LLM Tokens and Tokenization

In the previous tutorial, we learned about Large Language Models (LLMs) and how they generate text.

We saw that an LLM does not work with text in exactly the same way humans do. Before the model can process text, the text needs to be converted into a form the model can work with.

This brings us to an important concept:

Tokens

Understanding tokens is important because tokens are used throughout modern LLM applications. They also affect context windows, cost, performance, and RAG.

Let’s understand tokens from the beginning.

📌 What is a Token?

A token is a small piece of text that a language model processes.

A token can be:

• A complete word
• Part of a word
• A punctuation mark
• A number
• A special piece of text

For example, consider this sentence:

I love programming.

A tokenizer might break it into something similar to:

"I"
" love"
" programming"
"."

Each of these pieces is a token.

The exact tokens can be different depending on the model and tokenizer.

⚙️ What is Tokenization?

The process of breaking text into tokens is called tokenization.

The basic process looks like this:

For example:

The tokenizer converts a sentence into smaller pieces that the language model can process.

⚖️ Is One Word Equal to One Token?

No.

This is one of the most important things to understand about tokens.

You should not assume:

1 word = 1 token

That is not always true.

A common word may be represented by one token, while a longer or less common word may be split into multiple tokens.

For example, a word such as:

"hello"

might be represented as one token.

But another word could be split into smaller pieces:

This is only an example. The actual split depends on the tokenizer.

So remember:

A token is not necessarily a complete word.

🤔 Why Don’t LLMs Simply Use Words?

You might wonder:

Why doesn’t the model just process complete words?

The main problem is that language contains a huge number of possible words and word variations.

Consider:

program
programmer
programming
programmers
programmable

If every possible word and variation needed its own separate entry, the vocabulary would become very large.

Instead, tokenizers can break words into smaller reusable pieces.

Conceptually:

The same pieces can then be reused in other words.

This makes the vocabulary more practical for a language model.

✂️ Tokens Can Be Parts of Words

A token doesn’t have to represent an entire word.

For example, a tokenizer may split a word into smaller parts:

Or:

Again, these are examples to understand the idea. The actual tokenization depends on the model.

This approach is commonly called subword tokenization.

❗ Tokens Can Also Be Punctuation

Tokens are not limited to words.

Consider:

Hello! How are you?

The tokenizer can represent pieces such as:

Hello
!
How
are
you
?

So punctuation can also contribute to the token count.

This becomes important when working with LLM APIs because the amount of input and output is generally measured in tokens.

🔢 What About Numbers?

Numbers can also be represented using tokens.

For example:

2026

may be represented as one or more tokens depending on the tokenizer.

The same applies to:

• Dates
• Decimal numbers
• IDs
• Mathematical expressions
• Phone numbers

So tokenization is not limited to normal English words.

📌 What About Programming Code?

Tokens are also used when an LLM processes code.

Consider this Java code:

public static void main(String[] args) {
    System.out.println("Hello");
}

The model processes the code as pieces of text.

Conceptually, those pieces could look like:

public
static
void
main
(
String
[
]
args
)
{
...
}

The actual tokenization may be different.

This is one reason modern LLMs can work with programming languages as well as natural language.

🤔 Why Does Text Need to Become Numbers?

Now we come to an important question.

We know that neural networks work with numbers.

So how does the model go from:

"I love programming."

to something the neural network can process?

The simplified flow is:

Let’s understand token IDs.

🏷️ What is a Token ID?

Every token in a model’s vocabulary is associated with a numerical identifier called a token ID.

For example, purely for illustration:

These numbers are only examples.

A token ID is simply an identifier used by the model’s tokenizer.

For example:

does not mean that 7291 represents the meaning of Java.

It is simply the ID assigned to that token in that vocabulary.

⚠️ Token IDs Are Not Embeddings

This is an important distinction.

A beginner may see numbers and think:

“So those numbers represent the meaning of the word.”

Not exactly.

A token ID is just an identifier.

Later, the model converts these token IDs into numerical representations that are useful for the neural network.

This leads us to another important topic:

Vectors and embeddings

We’ll study those separately.

For now:

Don’t confuse the token ID with an embedding.

⚙️ How Does an LLM Use Tokens?

Let’s take a simple question:

What is RAG?

The simplified process is:

The model then processes those tokens and generates an answer.

But the answer is also generated as tokens.

So the overall process is:

⚙️ How Does the Model Generate the Answer?

Remember what we learned in the LLM tutorial:

An LLM generates text by predicting the next token.

Suppose we give the model:

The capital of India is

The model considers possible next tokens.

Conceptually:

The model selects a token.

Then it predicts the next token again.

This continues until the response is complete.

So the model is essentially generating a sequence of tokens.

📤 Input Tokens and Output Tokens

When you use an LLM, there are two important token counts.

Input Tokens

These are the tokens you send to the model.

For example:

Explain RAG in simple terms.

becomes input tokens.

Output Tokens

These are the tokens generated by the model.

For example:

RAG stands for Retrieval-Augmented Generation...

becomes output tokens.

So:

This distinction is important when building applications.

🤔 Why Do Developers Care About Tokens?

You may wonder:

Why should I care how a sentence is split into tokens?

Because tokens affect several important things.

1. Context Window

An LLM can only process a certain amount of information within a single context.

That amount is measured in tokens.

For example:

We’ll study context windows separately.

2. API Cost

Many LLM providers calculate usage based on the number of input and output tokens.

So:

The exact pricing depends on the model and provider.

3. Performance

More tokens usually mean more information for the model to process.

For applications handling large documents or long conversations, token usage can become important for performance and latency.

4. RAG

Tokens become especially important when we build RAG systems.

Imagine a company has a large PDF containing hundreds of pages.

You might have:

Sending the entire document with every question is not a good approach.

Later, RAG will allow us to retrieve only the relevant parts of the document.

That is one reason token knowledge is important before learning RAG.

⚖️ Tokens, Words and Characters

These three terms are easy to confuse.

Suppose we have:

programming

Characters

The individual characters are:

p r o g r a m m i n g

Word

programming

That is one word.

Tokens

It may be represented by multiple tokens.

For example:

program + ming

So:

Characters ≠ Words ≠ Tokens

They are different concepts.

⚖️ Tokenization for Different Languages

Tokenization is not limited to English.

LLMs can process languages such as:

• English
• Hindi
• Tamil
• Telugu
• Marathi
• French
• German
• Japanese
• Chinese

However, the same amount of text may use different numbers of tokens in different languages.

This is important when building multilingual applications because token usage can affect:

• Context limits
• Cost
• Performance

The tokenizer and model determine how the text is split.

✨ Special Tokens

Some tokenizers also use special tokens.

These tokens do not necessarily represent normal words.

They may be used for things such as:

• Marking the beginning or end of text
• Representing message boundaries
• Representing special instructions
• Handling padding

The exact special tokens depend on the model.

For now, just remember:

Not every token is an ordinary word.

⚠️ A Common Mistake

A beginner may think:

“The LLM reads the sentence just like I do.”

It doesn’t.

A simplified view is:

The LLM receives something more like:

This is one of the first major differences between how humans work with language and how language models process language.

🗺️ The Complete Picture

Let’s put everything together:

This is the basic token flow in an LLM.

🎯 Why Tokens Matter for Our AI Journey

We started with:

Now we are adding an important layer:

But there is still something missing.

We now know how text is broken into pieces and represented with token IDs.

The next question is:

How does the model represent these pieces as meaningful numerical information?

That’s where we introduce vectors.

🧩 Key Points

• What is a token?

  A small piece of text processed by a language model.

• What is tokenization?

  The process of breaking text into tokens.

• What is a token ID?

  A numerical identifier assigned to a token.

• Is one word always one token?

  No. A word can be one token or several tokens.

• Why do tokens matter?

  They affect:

  • Context windows
  • Cost
  • Performance
  • Input and output limits
  • RAG design

• The important flow

We now understand what tokens are and how tokenization works.

But token IDs are just identifiers. They don’t by themselves capture the meaning of the text.

So our next question is:

How can AI represent information using numbers?

We’ll start with the simplest possible explanation of a vector, and then connect it to embeddings, semantic similarity, semantic search, and eventually RAG.