LLMs Explained: Tokens, Context Windows, Parameters & Embeddings
Large Language Models (LLM) refers to a type of AI model (typically a neural network). It has quickly become one of the most important technologies, utilised in tools such as ChatGPT, Gemini, Claude, etc. It is a key component for AI to process and generate human-readable language.
Despite how frequently LLMs are discussed, some of the fundamental concepts behind them can be surprisingly difficult to distinguish. What exactly is a token? How does a context window differ from a model's parameters? And where do embeddings fit into the picture?
Below are some of the key concepts behind LLMs, that will provide a useful foundation for learning how LLMs function:
Tokens: how text is represented as discrete units
Context windows: how much information a model can process at once
Parameters: the learned values that determine how a model behaves
Embeddings: numerical representations that capture semantic relationships
Tokens: How LLMs Read Text
While humans can naturally understand words, phrases, and sentences to parse meaning and information, machines can only process numbers. Hence, the text must be converted to numbers.
One way of doing this is using tokenisation. Where text is broken down into small tokens, discrete units that can then be associated to a numerical integer index.
The Tokenisation Algorithm
A common misconception is that one word corresponds to one token.
This may be true in some cases, but it is not necessary.
A single token may represent:
an entire word
part of a word
punctuation
whitespace
numbers
special characters
The exact tokenisation depends on the model and tokeniser being used.
A relatively uncommon or long word may be split into multiple small subword tokens for processing.
This approach allows models to represent a very large vocabulary without requiring a separate token for every possible word.
Practical Tips for Tokenisation
If an application has a context limit of a certain number of tokens, the limit applies to the tokenised representation of the input, not simply to the number of words.
This is particularly important when working with:
long documents
source code
chat histories
retrieved documents
structured data
multilingual text
A prompt that appears relatively short to a human can still consume a substantial number of tokens depending on its content.
Context Windows: The Model's Working Context
A model's context window refers to the amount of tokenised information it can consider within a single processing context. It is the max amount od information the AI model can process at one time.
This can include both the user's input and other information provided to the model, such as:
previous conversation messages
system instructions
retrieved documents
tool outputs
other contextual information
The larger the context window, the more information will be considered in processing a suitable response. But more memory and computation are required.
It is similar to RAM in concept, where only so much data can fit at a time, and anything outside is not immediately computed.
Context window ≠ model knowledge
It is a common misunderstanding that the context window limits the model's full scope of information.
An LLM may have been trained on a large amount of information, but that does not mean all of that information is simultaneously present in its context.
A useful distinction is:
Training data → learned into parameters
Prompt/context → information currently available to the model
The context window determines how much tokenised information can be supplied to the model during a particular interaction.
Why Context Windows Matter
Larger context windows can be useful for tasks involving large amounts of information, such as:
analysing lengthy documents
reviewing large codebases
maintaining long conversations
comparing multiple sources
processing large datasets
However, simply increasing the context window does not automatically guarantee better reasoning or better answers. The quality, relevance, and organisation of the information provided to the model still matter.
Parameters: What the Model Learns
If tokens describe the information being presented to a model, parameters are part of what the model has learned during training.
Parameters are numerical values that are adjusted during the training process.
A simplified view of training looks like this:
Through repeated training, the model adjusts its parameters so that it becomes better at predicting the next token.
Modern neural networks can contain billions or even hundreds of billions of parameters.
What Are Parameters?
It is tempting to think of parameters as a database containing facts.
That is not quite accurate.
The parameters are numerical values distributed throughout the neural network. Collectively, they encode patterns and relationships learned during training.
These patterns can include relationships involving:
language
syntax
semantics
concepts
code
factual associations
patterns of reasoning
The model does not generally retrieve a sentence from a database every time it generates an answer. Instead, its learned parameters influence the probability distribution over possible next tokens.
More parameters ≠ automatically better
Model size is often discussed using parameter count, but parameter count alone does not determine capability.
Performance also depends on factors such as:
architecture
training data
training methodology
optimisation
inference techniques
model specialisation
context handling
Consequently, a smaller model can outperform a larger model on particular tasks.
Embeddings: Representing Meaning Numerically
Embeddings provide another important piece of the LLM ecosystem.
An embedding is a numerical representation of some piece of information—such as a token, word, sentence, document, or other object—in a high-dimensional vector space (embedding space).
Encoding refers to the process of choosing an initial numerical representation of data to train the model on.
An embedding represents each item in n-dimensional space with n floating-point numbers (typically in the range –1 to 1 or 0 to 1).
For example, a simplified embedding might look like:
"cat" → [0.21, -0.43, 0.17, 0.82, ...]
The embedding is in the form of a numerical array called a vector, which maps the coordinates of the data point in a high-dimensional vector space.
The following example is from Google Machine Learning Crash Course, because I lack the creativity to invent a more suitable case:
A one-dimensional representation of food, using "sandwich-y" to determine the positioning of a food data point.
A two-dimensional representation of food, plotted by both "sandwichness" and "dessertness".
An AI model typically uses a much higher-dimensional space to plot out its data.
The important idea is not each numerical vector itself. The important part is the relationship between each of these vectors.
Semantic Similarity
Items with related meanings can be represented by vectors that are closer together in the embedding space.
Eg. A shawarma is more similar to a hot dog, and may be grouped together. Or a borscht and a salad can be grouped together due to similar characteristics, as determined by their similar vector positions.
This allows systems to perform operations based on semantic similarity rather than simply matching exact words.
For example, a search system could retrieve a document about "automobiles" when a user searches for "cars," even though the exact words may differ.
How These Concepts Fit Together
These four concepts describe different parts of the LLM pipeline.
| Concept | What it represents | Why it matters |
|---|---|---|
| Tokens | Discrete units of text data | Determine how text is processed |
| Context window | Amount of tokenised information available at once | Determines how much information can be considered |
| Parameters | Learned numerical values inside the model | Encode patterns learned during training |
| Embeddings | Numerical vector representations | Enable semantic comparison and retrieval |
It is important to consider each component when designing or debugging LLM applications.
For example:
If a document is too large to process, context limitations may be relevant.
If an application exceeds its token budget, tokenisation matters.
If a model lacks useful learned patterns, training and parameters become relevant.
If a system needs to find semantically relevant documents, embeddings and vector search become important.
Building a document-question-answering system may require decisions about:
how documents should be chunked
how many tokens can fit into the context
which embedding model should be used
how vectors should be stored and searched
how retrieved information should be passed to the LLM
how much context should be included in each request
A Simple Mental Model
One way to remember the differences is to think of an LLM as a system processing information through several stages.
Tokens are the units the model processes.
The context window determines how much tokenised information can be considered at once.
Parameters contain the learned numerical patterns that shape the model's behaviour.
Embeddings represent information as vectors that can be compared based on learned relationships.