AI Tokens and Tokenization

In the last lesson, you learned AI Prompt Improvement Loop. Now we need tokens, because an LLM does not read text exactly like we read words.

Here is how text becomes tokens:

🔤 What Is a Token?

A token is a small piece of text that a model reads or writes.

You can think of tokens as the model’s reading units.

• Humans read letters, words, and sentences: We naturally understand full words and sentence meaning.
• Models read tokens: Before text reaches the model, it is split into smaller pieces.
• A token may look like a word: A simple word like cat can be one token.
• A token may be part of a word: A longer word can be broken into multiple pieces.

A token can be:

• A full word: A simple word like cat may become one token.
• Part of a word: A longer word may be broken into smaller pieces.
• A punctuation mark: Characters like ., ?, or , can also count as tokens.
• A space joined with a word: Some tokenizers attach the space before a word to that word token.

For example:

"Generative AI is powerful"
        |
["Generative", " AI", " is", " powerful"]

Different models can split text differently, so this example is only to show the idea.

You do not need to manually split text into tokens in the beginning. The model provider or tokenizer usually does that.

🧪 More Token Examples

Token splitting can feel strange at first, so look at a few simple cases.

These examples are not exact for every model. They show the idea.

So do not count tokens by only counting words.

For beginners, remember this:

• Token is not always a full word.
• Token is not always one character.
• Token is just the piece the model receives after text is split.

Bigger sentence examples

Now look at bigger statements.

Again, these are approximate examples. The exact split depends on the tokenizer used by the model.

Statement:
"The user wants a simple explanation of AI."

Possible token idea:
["The", " user", " wants", " a", " simple", " explanation", " of", " AI", "."]

What to notice:

• The sentence has 8 words.
• The token idea has 9 pieces because punctuation can count too.
• Some tokens include the space before the word, like " user" and " wants".

Statement:
"Please summarize this support ticket in three bullet points."

Possible token idea:
["Please", " summarize", " this", " support", " ticket", " in", " three", " bullet", " points", "."]

What to notice:

• This looks like one normal instruction.
• For the model, it becomes multiple small pieces.
• If you send many instructions like this, token count grows.

Statement:
"Create a JavaScript function that checks whether an email address is valid."

Possible token idea:
["Create", " a", " JavaScript", " function", " that", " checks", " whether", " an", " email", " address", " is", " valid", "."]

What to notice:

• Programming words can also be tokens.
• A word like JavaScript may stay together or split differently depending on the model.
• The model does not see this as one big sentence. It sees pieces.

Statement:
"The customer says the payment failed, but the amount was still deducted."

Possible token idea:
["The", " customer", " says", " the", " payment", " failed", ",", " but", " the", " amount", " was", " still", " deducted", "."]

What to notice:

• The comma and period may count as separate pieces.
• Support messages can become many tokens quickly.
• If the app also sends old chat history, the request becomes even larger.

🌍 Tokens in Real Tools

🧱 Characters vs Words vs Tokens

Do not think of tokens as exactly equal to words.

• Short common word: A word like cat may become one token.
• Long word: A word like unbelievable may be split into smaller token pieces.
• Space before a word: Some tokenizers attach a space to the next word token.
• Code symbols and punctuation: Characters like {, }, ., and () can count as tokens.
• Rare or unusual words: Less common words can split into more token pieces than simple common words.

So when a provider shows token usage, it is not simply “number of words”.

🤔 Why Do Tokens Matter?

Tokens matter because models measure work in tokens.

This is like how mobile data is measured in MB or GB. AI text usage is often measured in tokens.

• Prompt uses input tokens: Everything you send to the model becomes part of the input token count.
• Answer uses output tokens: Every piece of text generated by the model also counts.
• Context window has a token limit: The model can only handle a limited amount of text in one request.
• Cost is often token-based: More tokens usually mean a higher API cost.
• Latency can increase: More tokens usually means the model has more work to process.

So when an AI app feels slow or expensive, tokens are often part of the reason.

📥 What Counts as Input Tokens?

Input tokens are not only the user’s latest message.

They can include:

This is why long chat history can make an app slower over time.

The app is not only sending your latest sentence. It may also send old messages and instructions.

Input tokens:
System message + user prompt + documents + conversation history

Output tokens:
The model answer

📏 Token Limits

Every model has a context limit.

That means only a certain amount of text can fit in one request.

Prompt
+
Conversation
+
Retrieved context
+
Output
        |
Context window

If you exceed the limit, the application must reduce the input.

Common strategies:

• Summarize older conversation: Keep the important details and remove repeated chat text.
• Retrieve only relevant chunks: Send only document parts that match the user’s question.
• Ask the user to narrow the question: If the request is too broad, ask what exact part they need.
• Split the task into smaller steps: Handle one document section or one subtask at a time.

🧮 Token Budget Thinking

Before sending a request, think like this:

• What is the user’s actual question? Do not send extra documents if the user only needs one small answer.
• Which context is truly needed? Include only the text that helps answer the question.
• How long should the answer be? A short answer should not allow unlimited output.
• Is this quick or detailed? A quick answer needs fewer tokens than a full tutorial-style answer.
• Can old conversation be summarized? A summary may keep the important details while reducing token count.

This simple habit helps a lot in RAG and agent apps.

💰 Tokens and Cost

Most hosted models charge based on tokens.

The exact price changes by provider and model, but the idea is stable:

• More input tokens usually cost more: Sending a huge document costs more than sending one useful paragraph.
• More output tokens usually cost more: A long model answer costs more than a short answer.
• Larger models usually cost more than smaller models: Stronger models may be better, but they are often more expensive.

🧩 What You’ve Learned

• ✅ Tokens are the small text pieces a model reads and writes.
• ✅ Token count affects cost, speed, and how much text fits in one request.

🚀 What’s Next?

When an LLM answers a question, it is using patterns it learned before our prompt. For example, it has seen many question-answer examples during training. Next, let us see how LLMs learn.