LLM Temperature

In the previous tutorial, we learned about inference.

We saw that an LLM generates an answer by predicting tokens one after another.

For example:

But there is an interesting question:

If the model has different possible next tokens, how does it decide which one to choose?

One of the settings that affects this process is called temperature.

🌡️ What is Temperature?

Temperature is a setting that controls how much variation there can be in the model’s token selection during generation.

In simple terms:

Lower temperature → more predictable output

Higher temperature → more varied output

It does not change what the model learned.

It changes how the model selects from the possible outputs during inference.

💡 Let’s Start With a Simple Example

Imagine the model sees:

The sun rises in the…

The model may consider several possible next tokens:

The exact probabilities are determined by the model.

Now imagine we generate the answer multiple times.

With a low temperature, the model is more likely to choose the strongest prediction.

The sun rises in the east.
The sun rises in the east.
The sun rises in the east.

The output is more predictable.

With a higher temperature, the model can give more weight to less likely choices.

This can produce more variation.

❄️ Low Temperature

A low temperature makes the selection process more focused on the higher-probability tokens.

Conceptually:

This can be useful when you want consistency.

For example:

• Factual-style responses
• Structured output
• Classification-like tasks
• Repetitive business workflows

However, temperature alone does not guarantee factual correctness.

🔥 High Temperature

A higher temperature makes the selection process less focused on only the highest-probability tokens.

Conceptually:

This can be useful when you want different ideas or creative variations.

For example:

• Brainstorming
• Creative writing
• Generating different ideas
• Exploring possibilities

Again, higher temperature does not automatically mean “better creativity.”

💡 A Simple Comparison

Imagine we ask the model to complete:

“The small house was surrounded by…”

At low temperature, you might get responses that are more similar:

trees
trees
trees

At higher temperature, you may see more variation:

trees
flowers
mist
tall grass

These are simplified examples to understand the concept.

The actual output depends on the model, prompt, and other generation settings.

🧠 Does Temperature Change the Model?

No.

This is important.

Temperature does not retrain the model.

It doesn’t change:

• Parameters
• Training data
• Model architecture
• What the model learned

It affects the generation process during inference.

So:

🎲 Temperature and Probability

To understand temperature properly, we need to connect it with what we learned in the previous tutorial.

During generation, the model produces scores for possible next tokens.

These scores are converted into probabilities.

For example:

These numbers are just an example.

Temperature changes the distribution of these probabilities before the token is selected.

❄️ Low Temperature

With a lower temperature, the probability distribution becomes more concentrated around the strongest choices.

Conceptually:

The model strongly favours the most likely option.

🔥 Higher Temperature

With a higher temperature, the distribution becomes more spread out.

Conceptually:

Less likely tokens have a greater opportunity to be selected.

Again, these diagrams are only for intuition.

🎲 Is Temperature Randomness?

People often describe temperature as a randomness setting.

That’s a useful beginner-friendly description, but it is slightly incomplete.

A better explanation is:

Temperature controls how strongly the model favours high-probability tokens during sampling.

When sampling is being used, changing temperature can change the likelihood of selecting different tokens.

So:

🎯 Temperature = 0

You may see an API or model configuration where temperature is set to 0.

People often describe this as:

“No randomness.”

That’s a useful practical shorthand, but the exact behaviour depends on the model and API.

The general intention is:

However, you should not assume that every system guarantees perfectly identical responses just because you specify zero temperature. Other factors and implementation details can matter.

🧠 Does Higher Temperature Make the Model Smarter?

No.

Temperature does not increase the model’s intelligence.

For example:

The underlying model hasn’t changed.

Only the generation behaviour has changed.

So don’t think:

“High temperature = smarter AI.”

Instead:

Temperature changes how the model chooses among possible outputs.

⚠️ Does Higher Temperature Cause Hallucinations?

It can affect the behaviour of generation, but temperature is not the root cause of hallucinations.

A model can produce incorrect information at low temperature too.

For example:

is possible.

Temperature controls token-selection behaviour; it does not give the model access to new facts or a verification mechanism.

We’ll study hallucinations separately.

🤖 Does Every LLM Use Temperature?

Not necessarily.

Different models and APIs expose different generation controls.

Some systems provide temperature.

Others may use different or additional decoding controls.

So don’t think:

Every AI model must have a temperature setting.

Instead:

Temperature is one common generation parameter used by many language-model systems.

📋 Temperature and Use Cases

A simple way to think about choosing temperature is:

These are general guidelines, not strict rules.

The best setting depends on the model and task.

💡 A Simple Mental Model

Imagine the model has these choices:

At lower temperature, the model strongly favours:

A

At higher temperature, the probability distribution becomes less concentrated, giving alternatives a greater chance of being selected.

That’s the core idea.

⚠️ What Temperature Does NOT Do

Temperature does not:

• Add knowledge to the model
• Change the model’s parameters
• Retrain the model
• Increase the context window
• Make the model inherently smarter
• Guarantee factual answers

It only affects the generation process.

📌 One Important Point

Temperature is only one part of how an LLM generates text.

There are other techniques and settings for controlling token selection.

For example:

• Top-p
• Top-k
• Maximum output tokens
• Stop sequences

We don’t need to learn all of these right now.

They belong to the broader topic of LLM generation and decoding.

For AI Fundamentals, understanding temperature is enough for now.

🧩 Key Takeaways

• What is temperature?

  A generation setting that controls how strongly an LLM favours high-probability tokens.

• Low temperature

  More focused
  More predictable
  Less variation

• High temperature

  More varied
  Less predictable
  More possibility for alternative outputs

• Does temperature change the model?

  No.

  It only affects how the model generates its response during inference.

• Does high temperature mean better AI?

  No.

  It simply gives the generation process more variation.

🧩 Where We Are

Our journey now looks like:

We now understand how an LLM generates an answer and one of the settings that controls its generation behaviour.

The next fundamental question is something almost everyone notices when using AI:

Why can an AI sometimes give an answer that sounds completely confident but is actually wrong?

That brings us to:

🤖 AI Hallucinations