LLM Parameters

In the previous tutorial, we learned about Transformers and Attention.

We saw that a modern LLM uses a Transformer-based architecture to process text and understand relationships between different parts of the input.

Now let’s answer an important question:

What makes a Large Language Model “large”?

One important answer is:

Parameters

You may have heard statements such as:

• “This model has billions of parameters.”
• “The model has 70 billion parameters.”
• “More parameters means a larger model.”

But what exactly is a parameter?

Let’s understand it step by step.

🎛️ What is a Parameter?

A parameter is a numerical value that a machine-learning model learns during training.

In simple words:

Parameters are the values a model adjusts while learning from data.

Remember our neural-network tutorial.

We had connections between neurons:

Those connections can have numerical values called weights.

Weights are one type of model parameter.

So you can think of parameters as the model’s learned values.

💡 A Simple Example

Imagine a very small model that predicts whether a student will pass an exam.

It receives:

Hours studied
Attendance
Previous marks

The model may learn that these inputs have different importance.

Conceptually:

The weights are learned during training.

For example, purely for illustration:

These numbers are parameters.

The actual neural network contains far more parameters and much more complicated calculations.

🧩 How Are Parameters Learned?

This connects directly to what we learned about Deep Learning.

During training:

The model keeps adjusting its parameters so that its predictions become better.

Over many training steps, the parameters become values that help the model represent useful patterns in its training data.

⚠️ Parameters Are Not Rules

This is an important distinction.

A traditional program might contain a rule like:

IF age >= 18
    allow voting

A neural network doesn’t normally store knowledge in this simple form.

Instead, many learned numerical parameters work together.

So:

while:

The model’s behaviour comes from the interaction of many parameters rather than one simple rule.

🤔 Why Do LLMs Have So Many Parameters?

Language is extremely complicated.

An LLM needs to deal with relationships involving:

• Words
• Sentences
• Grammar
• Context
• Meaning
• Programming languages
• Different subjects
• Different languages

A model needs enough capacity to represent many of these patterns.

This is one reason modern LLMs can contain very large numbers of parameters.

A simplified picture is:

🎛️ What Does “7 Billion Parameters” Mean?

Suppose someone says:

“This model has 7 billion parameters.”

It means the model contains approximately:

7,000,000,000

learned numerical values.

These values are used throughout the model’s neural-network layers.

The number of parameters is one way of describing the size of a model.

🎛️ Does More Parameters Mean a Better Model?

No.

This is a very important point.

It is tempting to think:

But this is not always true.

Model quality depends on many things, including:

• Model architecture
• Training data
• Data quality
• Training method
• Number of training tokens
• Post-training
• Alignment
• Inference setup

A smaller model can sometimes perform better than a larger model on a particular task.

So:

Parameter count is an important model characteristic, but it is not a direct measure of intelligence or quality.

⚖️ Parameters vs Model Size

You may also hear people talk about model size.

These concepts are related.

A model with more parameters generally requires more storage.

For example:

However, the actual size of a model on disk depends on how the parameters are stored.

This leads to another concept called precision.

🤔 Why Does Precision Matter?

A parameter is stored as a numerical value.

There are different ways of storing those numbers.

For example:

• 32-bit
• 16-bit
• 8-bit
• 4-bit

A simplified example:

Reducing the precision of model parameters is commonly called quantization.

We’ll cover quantization later when we discuss running LLMs locally.

For now, just understand why parameter count and memory usage are connected.

⚖️ Parameters vs Training Data

Another common misunderstanding is:

“Parameters are the training data.”

They are not.

Training data is the information used to train the model.

Parameters are values learned during that training.

Think of it like this:

So:

Training data → used to learn the model

Parameters → values learned by the model

⚖️ Parameters vs Tokens

We have now discussed two different things:

Tokens

Pieces of text processed by an LLM.

Parameters

Learned numerical values inside the model.

Parameter 1
Parameter 2
Parameter 3
...

So:

These are completely different concepts.

⚙️ Parameters During Inference

Once training is complete, the model has learned its parameters.

When you ask the model a question, those parameters are used to generate the response.

For example:

The parameters are not normally being retrained each time you ask a question.

This is important when we later discuss training vs inference.

⚙️ What Happens During Training?

Let’s take a simplified example.

Suppose the training text contains:

“The capital of India is Delhi.”

The model receives:

"The capital of India is"

It tries to predict the next token.

Maybe it predicts:

Mumbai

But the expected token is:

Delhi

The model calculates the error.

Then its parameters are adjusted.

This happens again and again across a huge amount of training data.

🤔 Why Are Parameters Important for LLMs?

Parameters are one of the main things that allow an LLM to represent complex patterns.

You can think of the model as:

The architecture determines how the model is structured.

The parameters contain the values learned during training.

Both are important.

🎛️ Parameters and Knowledge

You may hear people say:

“The model stores its knowledge in its parameters.”

This is a useful simplification, but we should be careful with it.

The model’s learned parameters contain patterns and information learned during training.

However, an LLM is not simply a database where you can look up:

There isn’t normally one parameter corresponding to one fact.

Knowledge and behaviour are distributed across many parameters.

So:

A model’s learned knowledge and behaviour emerge from the interaction of many parameters.

💡 A Simple Analogy

Think about learning mathematics.

You don’t have a separate brain cell containing:

“2 + 2 = 4”

and another one containing:

“5 + 5 = 10.”

Your knowledge comes from many connections working together.

Similarly, an AI model doesn’t normally store one simple fact in one parameter.

Many parameters work together to represent learned patterns.

This is only an analogy, but it gives us a useful mental model.

🎛️ Parameters and Model Families

Different models can have different numbers of parameters.

For example, a model family might offer:

Small Model
Medium Model
Large Model

The larger models may have more parameters and require more computing resources.

But choosing a model should depend on the actual problem.

For a simple task, a small model may be enough.

For a complex task, a larger model may provide better results.

🤔 Why Parameters Matter When Running Models Locally

This becomes especially important if you want to run an LLM on your own computer.

Suppose you have:

7B model

and another:

70B model

The second model has roughly ten times as many parameters.

It will generally require much more memory and computing resources.

This is one reason local LLM users pay attention to:

• Parameter count
• Quantization
• GPU memory
• CPU memory
• Model precision

We’ll explore this later in the Local LLMs section.

🎛️ Parameters and Fine-Tuning

Parameters also become important when we learn fine-tuning.

During normal pre-training:

During fine-tuning:

This is why fine-tuning can change a model’s behaviour.

We’ll study this properly in the Fine-Tuning tutorial.

⚖️ Parameters vs Hyperparameters

There’s another term you will hear frequently:

Hyperparameters

Don’t confuse the two.

Parameters

Learned by the model during training.

Examples:

Weights
Biases

Hyperparameters

Values chosen by the developer or training process.

Examples:

Learning rate
Batch size
Number of training epochs

A simple comparison:

We’ll explore hyperparameters when we go deeper into model training.

🧩 The Bigger Picture

Our LLM fundamentals are now growing:

We now understand what goes into the model and what the model has learned internally.

But there’s still a very important question:

How did the model learn all those parameters in the first place?

That takes us to the next topic.

🧩 Key Points

• What is a parameter?

  A numerical value learned by a model during training.

• Are weights parameters?

  Yes. Weights are one type of model parameter.

• Why do LLMs have billions of parameters?

  Because large language models need significant capacity to represent complex patterns in language and other data.

• Does more parameters always mean better?

  No.

  Model architecture, data, training and many other factors also matter.

• Parameters vs tokens

• Parameters vs hyperparameters

🧩 Where We Are Now

We have reached:

The next question is the natural one:

How does an LLM actually learn these billions of parameters?

We’ll take the entire learning process from the beginning and explain:

Then we’ll separate pre-training, fine-tuning, and later inference, so the whole LLM lifecycle becomes clear.