LLM Inference and Generation Settings

In the previous tutorial, we learned about the context window.

We saw that an LLM receives information as context and uses that information to produce a response.

Now let’s understand what happens when we actually use a trained AI model.

That process is called inference.

⚙️ What is Inference?

Inference is the process of using a trained AI model to produce an output from an input.

In simple terms:

Training teaches the model. Inference uses the trained model.

For example:

If the input is:

“The sun rises in the”

the model may predict:

“east”

The process of making that prediction is inference.

🏋️ Training and Inference

Let’s make the difference very clear.

During training, the model learns.

After training is complete, we can use the model.

That second process is inference.

So:

Training = learning

Inference = using what was learned

⚙️ Inference in an LLM

For an LLM, the input is usually text.

For example:

“The sky is”

The LLM processes the input and predicts what token should come next.

Conceptually:

The model then has:

“The sky is blue”

It can continue predicting the next token.

This process continues until the response is complete.

⚙️ LLMs Generate Tokens

We learned about tokens earlier.

An LLM doesn’t directly generate an entire paragraph at once.

It generates a sequence of tokens.

For example, a simplified response:

“The sky is blue.”

could be generated as a sequence like:

The exact tokens depend on the tokenizer.

The important idea is:

An LLM generates its response step by step by predicting tokens.

📌 What Does the Model Predict?

The model looks at the information currently available to it and calculates possible next tokens.

For example:

The sky is

The model might assign different probabilities to possible next tokens:

These values are only an example.

The model is essentially asking:

“Based on everything I have seen so far, what should come next?”

It then selects a token according to the model’s generation settings.

📌 Then It Predicts Again

After generating one token, the new token becomes part of the sequence.

For example:

Now the model has:

The sky is blue

It can predict the next token again.

So the overall process looks like:

This continues until the model stops generating.

📌 What Information Does the Model Use?

The model uses the information available in its current context.

We just learned about the context window.

So the simplified process is:

The newly generated token becomes part of the sequence used for subsequent predictions.

💡 A Simple Example

Let’s imagine we ask:

“Complete this sentence: Water freezes at”

The model might predict:

Then:

Then:

The exact token sequence depends on the tokenizer and model.

But the basic idea is the same:

The model keeps predicting what should come next.

⚙️ Is Inference Only Used for Text?

No.

Inference is a general AI concept.

For example, an image-classification model could receive:

The prediction might be:

“Cat”

Similarly, a speech model could receive:

So:

Inference means using a trained model to produce a result.

For LLMs, that result is commonly generated text.

🤔 Why Is Inference Important?

Inference is the part of AI that users actually interact with.

When you:

• Ask an AI a question
• Generate text
• Summarise something
• Translate text
• Generate code
• Ask for an explanation

you are using a trained model through inference.

The model has already learned its parameters during training.

Now it is using those parameters to produce an output.

💡 A Simple Real-World Analogy

Imagine a student preparing for an examination.

Learning

The student studies books and learns concepts.

Using the knowledge

During the exam, the student receives a question and gives an answer.

This is only an analogy, but it helps us understand the difference:

⚙️ Does Inference Change the Model?

Normally, no.

When you ask an LLM a question, the model is not normally retraining itself from that question.

The trained parameters remain unchanged during ordinary inference.

So:

does not mean:

These are different processes.

⚙️ How Does an Answer Stop?

The model needs to know when to stop generating.

Generation can stop when:

• The model produces an appropriate end condition
• A configured stopping condition is reached
• The maximum output length is reached

You don’t need to remember the technical details yet.

Just remember:

Inference continues until the model or application decides that generation should stop.

⚙️ Inference Can Be Fast or Slow

Inference requires the model to perform calculations.

A small model may produce responses relatively quickly.

A larger model may require more computing resources.

The speed can also depend on:

• Model size
• Hardware
• Number of input tokens
• Number of output tokens
• Application configuration

So when people talk about AI inference performance, they are talking about how efficiently the model can produce its output.

🧩 The Complete Picture

Let’s put the concept together:

This entire process of using the trained model to generate the result is inference.

📌 Training vs Inference — Remember This

This is the most important thing from this tutorial:

If you remember only one thing:

Training is when the model learns. Inference is when we use the trained model.

🧩 Key Takeaways

• Inference means using a trained AI model to produce an output.
• In an LLM, inference involves generating tokens.
• The model predicts tokens one after another.
• The current context influences those predictions.
• Normal inference does not retrain the model.
• Inference is what happens when we actually use an AI model.

🧩 Where We Are

Our journey now looks like:

During inference, the model has different possible next tokens with different probabilities.

So our next question is:

How can we control how predictable or varied the model’s responses are?

That leads us to our next topic:

🤖 AI Temperature