AI Transformers and Attention

In the previous tutorials, we learned about:

We also learned about tokens, vectors, and embeddings.

Now we are going to understand one of the most important parts of modern LLMs:

Attention

Attention is one of the key ideas behind the Transformer architecture.

🤔 Why Do We Need Attention?

Let’s start with a simple sentence:

The boy went to the bank because he needed money.

As a human, you can understand that bank probably means a financial institution.

Now consider:

The boy sat near the river bank.

Here, bank means the side of a river.

The word is the same, but the meaning changes based on the surrounding words.

So the model needs to look at the context around a word.

This ability to consider relationships between different parts of a sentence is a major reason attention is so useful.

⚙️ How Do Humans Understand a Sentence?

Consider this sentence:

The developer fixed the bug because it was causing the application to crash.

When you read the word “it”, you naturally try to connect it with something mentioned earlier.

You understand that “it” most likely refers to the bug.

In your mind, there is a relationship:

A language model also needs a way to understand such relationships.

This is where attention comes in.

📱 The Problem With Older Approaches

Before Transformers became popular, neural networks such as RNNs and LSTMs were commonly used for language-related tasks.

A simplified view is:

The model processes the sequence step by step.

This works reasonably well for short sequences.

But consider a very long sentence or document.

When the model reaches “it”, it needs to connect it with information that appeared much earlier.

As the sequence gets longer, this becomes harder.

We therefore needed a better way to capture relationships between different parts of the input.

🔎 What is Attention?

The basic idea is simple:

Attention allows the model to focus more on the parts of the input that are useful for understanding a particular part.

Imagine you are reading:

The developer fixed the bug because it was causing the application to crash.

When understanding “it”, some words are more useful than others.

Conceptually:

The model calculates these relationships mathematically.

It doesn’t use a simple rule like:

“Always look at the nearest noun.”

Instead, the model learns patterns from training data.

💡 A Simple Attention Example

Consider:

The cat drank the milk because it was hungry.

When the model processes “it”, it needs to determine what “it” refers to.

It may assign different levels of attention to the words:

The exact attention values are produced by the model.

The important idea is:

Some parts of the input receive more attention than others.

🔎 What is Self-Attention?

Transformers use a very important mechanism called Self-Attention.

The word self means that the model looks at relationships between different parts of the same input sequence.

For example:

The cat drank milk.

The model can consider relationships such as:

The   ↔ cat
cat   ↔ drank
cat   ↔ milk
drank ↔ milk

So instead of treating each word independently, the model can consider how the words relate to one another.

🤔 Why Is Self-Attention Powerful?

Imagine this sentence:

The software engineer who joined the company last year developed the application.

The model doesn’t only need to understand neighbouring words.

It needs to understand relationships such as:

Self-attention gives the model a mechanism for considering these relationships.

Conceptually:

⚡ What is the Transformer?

A Transformer is a neural-network architecture that uses attention mechanisms to process relationships between different parts of an input.

The Transformer architecture was introduced in the 2017 research paper:

“Attention Is All You Need.”

The Transformer became extremely important for modern AI because it provided a powerful way to process sequences and model relationships between their elements.

A simplified view is:

A real Transformer is much more complex, but this gives us the right mental model.

🕸️ Transformer Is a Type of Neural Network

This is important.

A Transformer isn’t something completely separate from neural networks.

It is a neural-network architecture.

Think about it like this:

Different architectures are designed to handle different types of problems.

Transformers became particularly important for language and later became useful for many other types of AI tasks.

🧱 What Are Transformer Layers?

A Transformer usually contains multiple layers.

A simplified representation is:

Each layer processes the information and builds richer representations.

This is similar to what we learned about Deep Learning:

The actual Transformer architecture is much more sophisticated, but the idea of progressively processing information is useful for understanding it.

🔎 What Are Query, Key and Value?

Now we can introduce three terms that you will see whenever you study attention:

• Query
• Key
• Value

They are often written as:

Q = Query
K = Key
V = Value

At first, these names can be confusing.

Let’s use a simple example.

💡 Query, Key and Value — Simple Example

Imagine you are in a library and want to find information about Java.

Your question is:

“I want information about Java programming.”

This is your query.

The library has many books. Each book contains information that can be compared with your query.

Conceptually:

In attention, the model uses Query, Key and Value vectors to calculate which information should receive more attention.

This is a simplified explanation, but it gives you the right intuition.

We’ll study the actual mathematics later, when we need it.

⚙️ How Attention Works at a High Level

The simplified process is:

The model therefore doesn’t treat every piece of information equally.

It can give more importance to the information that is useful for the current calculation.

🤔 Why Did Transformers Become So Important?

Transformers solved several important problems very effectively.

They made it easier to:

• Model relationships between words
• Process long sequences
• Train large models efficiently
• Scale models to very large datasets

This helped enable modern Large Language Models.

The relationship is:

That is the path we’re following in this course.

⚡ Transformers Are Not Only for Language

Another important point is that Transformers are not limited to text.

Transformer-based architectures are also used in areas such as:

• Images
• Audio
• Video
• Multimodal AI

This is one reason Transformers have become such an important part of modern AI.

🤔 Why Attention Matters for ChatGPT

When you ask ChatGPT a question, the model has to process your input and understand relationships between different pieces of text.

For example:

“I am building a RAG application using PostgreSQL. Which vector database options should I consider?”

The model needs to understand relationships between:

RAG
 ↕
application
 ↕
PostgreSQL
 ↕
vector database

Attention helps the model process those relationships.

The model can then generate a response one token at a time.

📱 What Happens Inside an LLM?

At a very high level:

We don’t need to understand every mathematical operation yet.

The important thing is understanding where Transformers and attention fit into the overall system.

⚖️ Transformer vs LLM

These two terms are also easy to confuse.

Transformer

A neural-network architecture.

LLM

A large language model built using a suitable neural-network architecture, commonly a Transformer-based architecture.

So:

while:

For example, GPT-style language models are built using Transformer-based architectures.

💡 A Simple Analogy

Think about building a car.

The architecture is like the overall design of the car.

The trained model is the completed car with all the components working together.

In our case:

This is not a perfect analogy, but it helps distinguish the two concepts.

🧩 The Bigger Picture

We have now covered:

And within the LLM, we have started looking at:

We will continue adding more pieces to this picture.

🧩 Key Points

• What is a Transformer?

  A Transformer is a neural-network architecture that uses attention to process relationships between different parts of an input.

• What is Attention?

  Attention allows the model to give different levels of importance to different parts of the input.

• What is Self-Attention?

  Self-attention allows different parts of the same input sequence to interact and influence each other.

• What are Query, Key and Value?

  They are components used by the attention mechanism to calculate which information should receive more attention.

• Why are Transformers important?

  They became a powerful foundation for many modern AI models, especially Large Language Models.

🧩 Where We Are Now

Our LLM foundation now looks like:

We have now understood what the model processes and how the Transformer processes relationships between those pieces.

The next important question is:

What makes an LLM “large”?

We’ve already heard the word parameters several times.

We’ll understand what parameters are, why a model can have millions or billions of them, what a parameter actually does, and why more parameters does not automatically mean a better AI model.