Goals

• To learn what AI agents are, not just in theory, but in practice.

• Look at the ideas behind modern agents.

• Build your very first AI agent step by step using Google's ADK.

• Understand how AI agents reason, act, and adapt, and you'll have code that you can actually run yourself.

So What are AI Agents?

At the simplest level, an agent is software that doesn't just answer; it can decide and take action. Instead of generating a single response like a traditional chatbot, it looks at your request, figures out what steps need to be taken, maybe calls an API, runs code, looks at the results, and then decides what to do next.

One of the clearest explanations comes from the research paper ReAct: Synergizing Reasoning and Acting in Language Models.

// To read the research paper: https://arxiv.org/pdf/2210.03629

The idea in that paper was simple but powerful. Language models shouldn't just generate text in one go. They can actually reason step by step, take an action like calling a tool or API, observe the results, and then decide what to do next.

That cycle of reasoning, acting, observing, and adjusting is the foundation of how modern AI agents work, and it lines up with how it's defined: Systems with reasoning, planning, and memory, with enough autonomy to adapt and make decisions on behalf of the user.

Now, not all agents behave in the same way. A useful way to think about them is in three broad patterns:

1. First, we have Sequential Agents. These run step-by-step like an assembly line. Predictable and rigid. >> Good for simple, predictable flows.

2. Second, we have Reactive Agents. These decide in the moment; they look at the current state and ask, "What should I do next?" They're flexible, but they don't plan ahead. >> Good for dynamic problems.

3. Lastly, we have Deliberate or Planning Agents. These pause to sketch a plan, then execute. Think about booking travel. You don't just buy a flight randomly. You pick dates, hotels, and then that's the order of steps that you follow through. >> Good for multi-step goals with dependencies.

Each of them should be applied to different problems.

SDK vs ADK

• SDK (Software Development Kit): Comprehensive suite of tools used to build traditional platform applications.

• ADK (Agent Development Kit): Specialised framework designed to build autonomous AI agents capable of reasoning, planning, and tool execution.

Building a Blog-Writing Agent with Google ADK

You give it a topic, it will generate a structured outline, turn that outline into a full blog post, and then suggest alternative titles and hooks.

This is where ADK's categories come in.

As we walk through the code, notice how the planner and writer agents are LLM agents, which are powered by a model. They're wrapped in loop agents, which are workflow agents in ADK. These handle retries automatically if validation fails. And if we ever wanted something beyond that, ADK would also let us extend a custom agent by subclassing the base agent.

So let's set up the environment and then go line by line through the code.

First off, you need to install UV.

Environment Setup (Terminal)

// Environment Setup
pip install uv
pip install google-adk

Step 1: Imports and Config

# agent.py
import os
import sys
from pathlib import Path
import datetime

from dotenv import load_dotenv
from google.adk.agents import Agent, LoopAgent
from google.adk.tools import agent_tool

# --- env/config -----------------------
load_dotenv()

MODEL = os.getenv("MODEL", "gemini-flash-latest")

Meet the Planner Agent

• We give it the name ("BlogPlanner") so it's easy to identify.

• And we tell it which model to use (model=MODEL), which comes from our environment file.

• The (description) is just a human-readable note.

• The important part of the introduction is in the (instructions). Here we tell it what ot produce.

• Finally, we look at the (output_key), we set it to blog outline. This means whatever this agent writes will be stored in shared state under that name so that the next agent to pick it up can do so automatically.

# --- Sub-Agent: Planner ----------------
blog_planner = Agent(
    name="BlogPlanner",
    model=MODEL,
    description="Creates a practical, skimmable outline in Markdown.",
    instruction="""
        You are a technical content strategist. Produce a clear Markdown outline with:
        - Title
        - Short intro
        - 4-6 main sections (each with 2-3 bullets)
        - Conclusion

        If `codebase_context` exists in state, weave in specific sections/snippets.
        Return only the outline in Markdown.
    """,
    output_key="blog_outline"
)

The Outline Validation Checker

Next, we add a second agent. The Outline Validation Checker. This one doesn't write anything new. It just checks the outline. It's instruction is strict. That way we can automatically decide whether to accept the outline or try again.

A critical gate validating the planner's output with 2 simple results: Okay / Retry.

# Outline Validation Checker Class
class OutlineValidationChecker(Agent):
    def __init__(self):
        super().__init__(
            name="OutlineValidationChecker",
            model=MODEL,
            description="Validates that the outline is usable.",
            instruction="""
                Check the outline in state `blog_outline`. If it has a title, intro, 4-6 sections, and a conclusion, respond exactly "ok".
                Otherwise respond exactly "retry" and list missing pieces.
            """,
            output_key="validation result",
        )

Robust Blog Planner

To connect the planner and the checker, we've wrapped them inside a loop agent called Robust Blog Planner.

The loop first runs the planner, then the checker. If the checker says "ok", we're done. If the checker says "retry", the loop runs the planner again with the same instructions. It can do this up to three times. This gives us a safety net. So if the model forgets something, it has another chance to get it right.

The (blog_planner) agent is passed as a sub-agent.

The (OutlineValidationChecker) is another sub-agent responsible for validation.

# The robust_blog_planner
robust_blog_planner = LoopAgent(
    name="RobustBlogPlanner",
    description="Retries planning if validation fails.",
    sub_agents=[
        blog_planner,
        OutlineValidationChecker()],
    max_iterations=3,
)

The Blogwriter Agent

Transforming outlines into full-length technical blog posts.

Now we create another agent called blogwriter, this one takes the outline we just saved in blog outline and turns it into a full markdown blogpost.

The instruction tells it how to write.

The finish draft is saved under (blog_post) in shared state, ready for the next step.

# --- Sub-Agent: Writer ----------------
blog_writer = Agent(
    name="BlogWriter",
    model=MODEL,
    description="Writes a technical blog post from the outline.",
    instruction="""
        Write a complete Markdown article from the outline in `blog_outline`.
        
        Guidelines: 
        - Audience: Software engineers; skip basics and focus on practical insight. 
        - Explain both the 'how' and 'why'/
        - Include concise code snippets when helpful.
        - Follow the outline's structure (H2/H3).
        - Output only the final article in Markdown (no fence around the whole post).
    """,
    output_key="blog_post",
)

The Blog Post Validation Checker

class BlogPostValidationChecker(Agent):
    def __init__(self):
        super().__init__(
            name="BlogPostValidationChecker",
            model=MODEL,
            description="Validates the final post.",
            instruction="""
                Check `blog_post` for: intro, clear sections matching the outline, conclusion, and technical clarity.
                If passes, respond "ok". Else respond "retry" with the specific fixes.
            """,
            output_key="validation_result",
        )

Robust Blog Writer

Just like with planning, we combine the writer and its checker into a loop agent, called Robust Blog Writer. This makes the system automatically retry the writing step until the validation passes, up to three times.

# --- The Robust Blog Writer Loop ----------
robust_blog_writer = LoopAgent(
    name="RobustBlogWriter",
    description="Retries writing if validation fails.",
    sub_agents=[
        blog_writer,
        BlogPostValidationChecker()],
    max_iterations=3,
)

Using agent_tool.AgentTool

Now we wrap each loop, the (robust_blog_planner) and the (robust_blog_writer) as tools. This is what lets the root agent call them directly, without worrying about what's happening inside.

# --- Wrap Each Loop as Tools --------------
# Expose planner/writer as tools so the root agent can call them explicitly
planner_tool = agent_tool.AgentTool(agent=robust_blog_planner)
writer_tool  = agent_tool.AgentTool(agent=robust_blog_writer)

Root Agent: Blogger

Finally, we put it all together with the root agent called Blogger.

The instructions Blogger are simple. When a user gives it a topic, first it calles the planner tool to make an outline. Then it calls the writer tool to draft the full post. And finally it adds three alternative titles and two sweet tweet length hooks.

The root agent only has access to the two tools we exposed, the planner, and the writer. So it's workflow is clear and controlled.

# --- Root Agent: Trends → Plan → Write ----
root_agent = Agent(
    name="Blogger",
    model=MODEL,
    description="Minimal multi-agent blogger that plans and writes.",
    instruction=f"""
        If the user gives a topic:
        1) Call the planner tool to generate the outline.
        2) Call the writer tool to produce the full draft.
        3) End with 3 alternate titles and 2 tweet-length hook.
        
        If the trends tool fails or times out, continue with a sensible outline and draft anyway.

        Date: {datetime.datetime.now().strftime("%Y-%m-%d")}
    """,
    tools=[
        planner_tool,
        writer_tool,
    ],
)

You've successfully written your first agent script. The next step is to run it.

Running with ADK Web

ADK Web (Terminal)

adk web

Head into your terminal and launch it using this command with ADK web. So you can view everything inside a simple UI.

Local ADK Web Link (Browser)

Once that is running, open the local ADK web link in your browser and start interacting with the agent directly.

To kick things off, I'm going to ask it to generate a blog on the top 10 use cases for AI agents. Let's see how it plans, writes, and delivers the results. Your subagents will then work together to create a blog post.

Gemini API Key

https://ai.google.dev/gemini-api/docs/api-key

Create a new key.

You can pass your valid api key via 2 methods.

(1) Use an .env file (I did this)

ADK automatically reads environment variables from a file named .env in your project folder.

Create a file named .env (make sure it has no .txt extension) directly inside your project directory.

Add your Gemini API key to the file.

GOOGLE_API_KEY="YourActualAPIKeyGoesHere"

(2) Set it directly in PowerShell (I did not do this)

Save the file and restart your agent with (adk run .) or (adk web .)

Gemini Overloaded with Requests

Try again, or switch to a faster, lower-demand model.

Switched from latest model to the latest LITE model.

# --- env/config ---------------------------
load_dotenv()

# MODEL = os.getenv("MODEL", "gemini-flash-latest")
# gemini model currently experiencing high demand, cannot run the latest model...
MODEL = os.getenv("MODEL", "gemini-flash-lite-latest")

Boom it works perfectly now.

And that's it. You've built and run your first AI agent with Google's ADK. We started from the idea of what agents are, then walked through the code step by step, and finally saw it come to life in the terminal.

I remember when he came to Singapore for an award ceremony at MBS, and I got the chance to bro-fist him in 2013, which feels like a lifetime ago...

I'm definitely interested to see how this plays out.
Here I'll jot down some quick notes on what I know so far!

Situation Summary:

• PewDiePie has released his custom local AI model - Ajax, designed for users to run privately on their home PC for everyday web searching, email, and calendar management.

• While developing Ajax, he was banned from OpenAI twice for “distillation”: using OpenAI to generate training data for his own AI model development.

YouTube Video:

https://www.youtube.com/watch?v=ODDJXGY_1kQ

My Notes on the Video

(Mostly content from Pewdie’s video, personal comments are indicated by //)

Ajax Introduction

• Ajax is a fine-tuned AI model that can browse the web, do private searches, write your emails, manage your calendar, etc.

• Ajax is tuned for the Odysseus harness.

• Ajax is a private “first base” foundation, it does not collect any data whatsoever from its users.

• Most other open source projects also collect user data, but not this one.

• Instead of stealing data, Pewdiepie hoped to ask his follower base to donate their data.

• Creating data.pewdiepie.com for people to submit their data.

• He tried to reduce friction and made tutorials, but still no one submitted their data…

• // Wtf is the difference between Ajax and Odysseus systems? What exactly are they?

• // Also, it's nice that they're named after the rival Greek heroes, with convenient timing after most people are now familiar with some of the mythology post-release of the Odysseus movie.

• // Hmm but Ajax did go temporarily insane and killed because of his hallucinations though, so maybe not the best name for an AI model when everyone is still fighting to reduce AI hallucinations…

OpenAI and Decryption

• He went to OpenAI to distil a bit - to increase the performance of his smaller AI model, he can distil the knowledge from a larger model to train on

• // To Study: Knowledge Distillation Wiki Page

• // https://en.wikipedia.org/wiki/Knowledge_distillation 

• // Also TIL that "distil" is UK English. "distill" is American English.

• He wanted to distil OpenAI's GPT -5.6 Sol model because he liked it. 

• The problem was that OpenAI encrypts their reasoning tokens, all the information that gives it chain-of-thought thinking on how it solves the problem.

• Pewdiepie found a study that decrypts the data using their own API, and it worked.

• // To Study: Reasoning tokens, chain-of-thought thinking in AI

• // To Study: Data Encryption

• // To Study: arXiv - stealing reasoning traces from proprietary LLM APIs

• // https://arxiv.org/abs/2608.09867

• Pewdiepie then went on to talk about how these giant tech companies “worked hard” to steal everyone’s data to make their AI machines. How dare we, as consumers, take part of their hard work stolen from us?

• RIP he got banned from OpenAI, appealed and got accepted.

• Then he ran OpenAI GPT Sol again to create his model’s training seed data, and OpenAI banned him a second time.

• // This ofc brings up the very interesting ethical and legal debate surrounding AI developers collecting and scraping internet data, there is plenty of ongoing discussion on privacy, data ownership concerns, lack of opt-out options, consent, uncompensated use of personal content, etc.

• // I don’t have answers to questions about the fairness of data collection and use, but it is fascinating to hear different perspectives and to weigh the trade-off between the convenience of powerful AI tools and digital ethics.

• // Before watching the video, I also did not expect Pewdiepie to be so vocal about his takes on AI and data collection, but it makes sense given that his internet presence would also be subject to lots of scrutiny, surveillance, and data harvesting.

Decensoring AI

• Decensoring AI is like removing its ability to say No.

• Pewdiepie compared it to brain surgery: 

  • Imagine I’m asking you for your deepest, darkest secret.

  • You are thinking it in your head.

  • You don’t even have to answer it out loud.

  • I am actually running an MRI scan on your brain right now, and I can spot exactly what part of your brain was active when you refused to answer it.

  • I “zippity zap” it (remove it)

• By repeatedly feeding the AI model all these different dialogues that it does not want to reply to, it is possible to identify the parts responsible for the AI model’s censorship (data set of cursed questions).

• These parts can then be ablated, removed.

• This does cause “brain damage”, and the more extensive the removal, the more damaged the AI model becomes.

• // I really like the brain surgery comparison, it makes it much easier to understand the general concept of de-censoring AI.

• // To Study: Decensoring AI, how are the parts identified, how are the parts ablated/removed.

• // To Study: Program called Heretic, open-source project that makes the process super simple.

• This gives Pewdiepie the flexibility to decide and control the extent and areas of censorship that he wants to retain in his model. To think about where to draw the line, and what kind of AI use is responsible use.

• There’s a bazillion other models out there uncensored, so it does not matter too much.

• But Pewdiepie drew the line at causing harm to other people, or causing harm to yourself, as that goes against his personal philosophy.

• // I like that Pewdiepie has a strong stance on his personal philosophy and avoiding doing harm.  

• // Creating a small, independent model does give this kind of freedom to exercise one’s personal ethics instead of playing it as safe as possible, unlike the large firms.

• // But he isn’t being irresponsible about it.

Teaching the AI Model

• Pewdiepie ran GRPO to train his model Ajax, and he found it really fun because it's a more interactive way of teaching the model.

• The model will run the same task a set number of times (16). If it solves the task even one time out of the 16 times, that becomes valuable training data. 

• You reinforce the ability to generate the solution and hope for it to improve the model.

• // To Study: GRPO

• // Group Relative Policy Optimization (GRPO) is a memory-efficient reinforcement learning algorithm that eliminates the need for a separate critic model by evaluating multiple outputs within a group to calculate relative advantages

• After 4 days of training, he ran out of tasks to fit the criteria, so he had to quickly synthesise more.

• There was a back-and-forth process where it first improved, then flatlined, and then it started improving again.

• He had to run reinforcement training, then run the decensoring ablation again, and then have to quantise,The capabilities of open models and harnesses point the way forward for users to use uncensored, less restricted intelligence on their own hardware, an important goal in keeping the technology free. and benchmark it to ensure nothing was lost.

• Spent 8 weeks going through the dos and don'ts, by doing the don'ts over and over and over.

Small Model Advantages

• Pewdiepie originally planned to train a way bigger model, but then he realised no one would be able to use it. 

• The main idea behind Ajax was to give everyone access to run a local AI, and for that it had to be a small model.

• // I like that he is coming from a place of accessibility and thinking of the average use of AI for most people.

• Pewdiepie’s napkin maths: OpenAI/Anthropic models are rumoured to be one trillion parameter models, which would take roughly 27 copies of his computer to run one instance of a model, about 150 houses in terms of electricity use.

• It makes no sense to poke “that giant beast” to ask it for basic tasks like checking emails.

• He really believes strongly in creating small models trained for a specific harness.

• Most people use AI for searching, researching, browsing; just saving time going through lots of pieces of information. Getting information quickly to you.

• That is where his small model works best.

• Important for AI to not replace creativity, to not replace thinking.

• A lot of AI problems can be solved by running it on your own.

• That is a future that he thinks is exciting, and that’s what he wants to build with Odysseus and Ajax.

• He strongly believes that AI should not replace learning new things or thinking for yourself.

• Everyone still needs the basic knowledge of different subject matters, and when they get to the point of understanding, that is when they can utilise AI to start accelerating their work and process.

• // I agree with all that he said! I think it's well put too!

Other Notes

Tom's Hardware - PewDiePie unveils ‘uncensored’ Ajax AI model built to run on home PCs — creator says OpenAI banned him twice over model distillation used to build his product

• Ajax is built for Odysseus, his self-hosted AI workspace, as an autonomous, always-on agent and assistant.

• The setup is similar to OpenClaw and Nous Research’s Hermes Agent.

• Ajax, a fine-tuned Qwen3.5-9B model...

• // I wonder why he used Qwen as the basis for Ajax, maybe they're less strict idk how Alibaba compares as a Chinese company vs the western based AI companies that I'm more familiar with?

• The video announced the first version, using Alibaba’s model, as a follow-up to his workspace launch in May.

• Frontier AI model companies such as OpenAI and Anthropic make sure to hide their reasoning in numerous ways, including encryption and attempted distillation; such a campaign OpenAI recently attributed to actors linked to Moonshot AI is seen as a threat.

• To "uncensor" his model, he relied on automatic abliteration through the open-source Heretic, where you discover prompts the model refuses to trace and remove the refusal, preferably without doing “brain damage” by removing too much.

• // I just liked the use of the word abliteration here haha

• // The term combines "ablation" (surgically removing a part) and "obliteration" (destroying it).

• The capabilities of open models and harnesses point the way forward for users to use uncensored, less restricted intelligence on their own hardware, an important goal in keeping the technology free.

Tweets that I Saw

It was all over my X/Twitter feed, which was what initially drew me to watch PewDiePie's video to understand what was going on!

I am disappointed that for everyone's eagerness to comment on his video, there hasn't been much interesting input, praise, or critique on his work.

Too early to tell I suppose.

Everyone's just acting shocked that he's as knowledgeable about AI as he is.

Which isn't very meaningful, even for a tweet, but I'm glad to have been brought awareness about his interesting project!

TO BE ENTIRELY HONEST, my first impression of prompt engineering was that it would be reserved for the extremely technologically challenged...

It seems entirely intuitive to me that large language models (LLMs) as a prediction engine would produce more desirable outcomes when a user is able to provide more details on the task, context, and expectations.

However, I have learnt that prompt engineering is more than the obvious show of writing and optimising prompts. Many people get overly fixated on that part of the process. It is harder to set up a framework for evaluating the quality/accuracy of the LLM responses and output. It can be hard to define quality and measure how desirable a result is.

What is Prompt Engineering

See my previous post dedicated to this!

(https://tokenofmyattention.hashnode.dev/prompt-engineering-how-to-communicate-effectively-with-llms)

The simplest way to summarise this is "better prompt → better response".

There are multiple techniques and frameworks for people to consider when writing and improving their LLM prompts.

LLMs do respond to the information and instructions provided in their context. A prompt that clearly communicates the task, relevant context, constraints, and expected output can generally give the model a better basis for generating a useful response.

However, what defines usefulness or desirability in an LLM response may not always be obvious.

Prompt Engineering is not a literary competition. There is no basis for the cleverest prompt.

The more I learn about the topic, the more I think prompt engineering is better understood as a design and evaluation problem.

Prompt Engineering Starts with Defining the Task

Before optimising something, you need to know what you are optimising for. If the objective is vague, optimisation becomes difficult.

Imagine trying to improve a search engine without defining what makes a search result relevant. Or improving a recommendation system without deciding what "good recommendation" means. Or testing a software application without defining what constitutes correct behaviour.

Note: Personal preference is not the same thing as quality.

I might prefer an answer because it sounds polished, while someone else might prefer a more concise version. And both of us might be wrong because the answer contains technical inaccuracies that neither of us caught.

Furthermore, it is infeasible to manually read and rate every response from an LLM-powered application.

Optimisation Loop

Informally, a simple take on prompt engineering probably looks similar to:
Write prompt → Send to LLM → Get response → Looks good? → Repeat

The issue is that "Looks good" is not a rigorous evaluation methodology.

A more engineering-oriented workflow looks something like:

Define task →
Define quality criteria →
Build evaluation dataset →
Run prompt / model →
Evaluate outputs →
Analyse failures →
Modify prompt →
Re-run evaluation →
Compare against previous version →
etc.

The prompt is therefore only one part of the process, the evaluation loop is what allows us to determine whether our changes actually helped.

1. Start by Defining the Task

Before evaluating an LLM, we need to know what it is supposed to do. This sounds obvious, but LLM applications can involve surprisingly ambiguous objectives.

Consider a customer-support application.

We might ask an LLM to:

Classify incoming customer messages into categories.

That sounds sufficiently specific. But what does success mean?

Suppose the available categories are:

Billing 
Technical Issue 
Account Access 
Product Information 
Other

And there is an incoming customer message that reads:

"I was charged twice and I can't log into my account."

Should this message be categorised under Billing? Account Access? Both?

Suddenly, the evaluation problem exposes an ambiguity in the task definition itself. Before optimising the prompt, we may need to define the expected behaviour.

In this case, we may add the requirement:

Assign exactly one category based on the customer's primary issue. If multiple issues are present, select the issue most directly related to the customer's requested resolution.

The process of testing an LLM can reveal that the original task was not sufficiently specified.

2. Build an Evaluation Dataset

Once the task is defined, we need examples against which to evaluate the system. This can be thought of as an evaluation dataset or test set.

For our customer-support classifier, it might look like:

The expected outputs become our reference answers, sometimes called ground truth or golden answers, depending on the evaluation setup.

Now the criteria for a model's response has become "Does the model's output match the expected behaviour?"

3. Your Test Dataset Should Contain More Than Easy Examples

A common mistake would be to create a dataset consisting only of obvious examples.

For example:

"I forgot my password." → Account Access 
"My application crashes." → Technical Issue 
"I want to know your pricing." → Product Information

These examples are obvious, and most models will likely perform well on them. The more interesting test cases are the difficult ones.

Ambiguous Inputs

"I can't access my account after being charged."

Incomplete Inputs

"It doesn't work."

Unusual Phrasing

"Money got taken twice from me."

Typos

"I was chared twise."

Multiple Issues

"I can't log in and I was charged twice."

Out-of-distribution inputs

"Can I become a reseller?"

These cases tell us much more about how robust our prompt is.

This is also where evaluation datasets become valuable beyond simply measuring performance.

They can act as a catalogue of known failure modes.

4. Define Evaluation Criteria

Once we have our test cases, we need to determine how outputs will be judged.

For a classification task, this can be relatively straightforward. Comparing the LLM answer against our expected answer gives upfront matches or incorrect mismatches.

But LLMs have many applications outside of classification tasks. E.g. Consider a task to summarise a text or document. There may not be any singular correct answer. Two summaries could use completely different wording while both being excellent. This means traditional exact-match evaluation is insufficient to judge these tasks.

We may instead evaluate other dimensions of the answer, such as:

• Correctness: Does the output contain factual errors?

• Relevance: Does it address the intended task?

• Completeness: Does it include the important information?

• Groundedness: Is the answer supported by the supplied source material?

• Instruction Following: Did the model follow the specified requirement?

• Style: Does the output match the desired tone and format?

• Safety: Does the response satisfy the application's safety requirements?

The exact criteria will depend heavily on the application. Every LLM with a different task will have a different metric to evaluate the quality of its output.

5. Not Every Evaluation Needs The Same Metric

For LLM evaluation where multiple outputs may be considered acceptable or good, we have to diverge from traditional software testing. For conventional software testing, we can often determine deterministic assertions or outputs, as the expected result is exact and always the same.

E.g. assert calculate_tax(100, 0.1) == 10

For LLMs a few popular evaluation methods will be listed below:

Exact-match Evaluation

Same as traditional software testing. Actual output should match expected output.

This works well for structured tasks such as:

• classification

• yes/no decisions

• fixed labels

• structured identifiers

• exact values

Rule-based Evaluation

We can also evaluate specific properties of an output.

For example:

Does the response contain a valid email address? 
Does it contain exactly five bullet points? 
Is it below 200 words? 
Does it contain prohibited information? 
Is valid JSON returned?

These checks are deterministic and relatively easy to automate. So for structured LLM applications, output validation can be particularly valuable.

Semantic Evaluation

Sometimes two answers can be different strings but have essentially the same meaning.

For example:

Version 1: "The meeting was postponed until Monday."

Version 2: "The meeting has been moved to Monday."

An exact string comparison would mark these as different. So instead we can apply a semantic evaluation to assess whether the responses convey equivalent information. This is where techniques involving embeddings or model-based evaluation can become useful.

6. LLM-as-a-Judge

As always, you can count on AI to help automate repetitive tasks.

There is an interesting approach to use a secondary LLM to evaluate the output of your target LLM.

The evaluating Model B might receive instructions such as:

Evaluate the response for factual accuracy, relevance, completeness and instruction following. 

Give each dimension a score from 1–5 and explain any identified issues.

This is often referred to as LLM-as-a-judge.

This is appealing as LLMs can process information and text to evaluate qualities that are impossible to capture with traditional coding deterministic rules. No "if/else" assertion is able to encode the question of "Does this response provide a clear explanation appropriate for a beginner?"

However, we still have the problem of evaluating Model B.

Who judges the judge? Who evaluates the evaluator?

The judging model itself can be prone to mistakes. Being biased to certain writing styles, failing to notice factual errors, producing inconsistent evaluations, etc.

An LLM judge cannot be treated as an unquestionable source of truth. So it is likely that introducing an LLM judge will come with its own round of evaluation criteria, reliability, and other qualities testing in the first place.

It is a lot of work, and I can only imagine justifying it when there is a need to test a lot of LLM models.

7. Human Evaluation Still Matters

For subjective tasks, human evaluation can remain extremely valuable.

However, it can be difficult to process the variance in human responses and preferences in AI responses. Just simply asking "Do you like this response?" will result in plenty of vague and inconsistent judgement.

Suppose we are building an AI assistant that drafts customer-support responses.

We could ask reviewers to rate responses on a defined rubric:

This gives human evaluation more structure and makes the judgement more consistent and reproducible.

8. Evaluation Should Happen Across a Test Set

Imagine a scenario where we have 100 test cases for an LLM, and we calculate the accuracy of 2 prompts.

Accuracy = Correct Predictions / Total Predictions

Here we have clear evidence that Prompt B had higher success and accuracy than Prompt A, and will likely perform better across future use cases.

9. Look at the Failures, not just the Score

Suppose we achieve a prompt with a high 95% accuracy rate, that may seem like a successful test set result. However, it is important to investigate further into the failures.

Imagine that the 5 failures are all of the same category. Eg. "Ambiguous billing issue".

This tells us more about the prompt and the LLM mode. It may not have a general classfication problem, it may have a specific weakness around ambiguous billing requests in particular. This allows us to improve the prompt or system based on an identified failure mode.

Perhaps we need an explicit instruction in the prompt for ambiguous billing cases? Or we need additional examples. Or the category definitions themselves could be further refined. Or the model itself could be further refined with more data.

This is much more informative than the simple statistic of a 95% accuracy rate

10. Few-Shot Examples become Testable Interventions

This also changes how I think about techniques such as few-shot prompting. Instead of adding examples because they "seem helpful," we can test whether they actually improve performance.

Version A: Classify the customer request into one of the following categories...

Classify the customer request into one of the following categories. 

Example 1: 
Input: "I forgot my password." 
Output: Account Access 

Example 2: 
Input: "I was charged twice." 
Output: Billing 

Now classify: {input}

We can run both prompts across the same evaluation dataset. Perhaps Version B improves accuracy, perhaps it does not. Perhaps it improves ambiguous cases while making another category worse.

The point is that prompting techniques become hypotheses to test, rather than techniques that we assume must work.

11. Prompt Versions should be treated like Software Versions

This was unexpected to me. This is another area where LLM development starts to resemble traditional software engineering.

You can version out your prompts with incremental improvements along the way.

For example:

Each version represents a change, and if the same evaluatin dataset is maintained, it is possible to compare the performance of each version against each other. This creates a basic form of regression testing, where the goal is for each systematic change to improve the LLM behaviour without breaking or deteriorating any existing behaviour that was already working.

Having a clear version list with changes made makes it easier to keep track of progress and any mistakes or regression made.

12. Evaluation Datasets Should Evolve

A test dataset is not something we necessarily create once and forget. As we discover new failure modes, we can add them in. Over time, the dataset becomes a collection of cases that represent the behaviour we care about.

This creates an interesting feedback loop:

Real-world failures → evaluation cases → prompt improvements → better system

Thus, the evaluation dataset effectively becomes part of the engineering infrastructure.

13. LLM Evaluation is Inherently Application-Specific

Once again, this boils down to: There is no universal definition of a "good LLM response."

Consider two systems.

System A: Creative Writing Assistant

We might care about:

• creativity

• style

• coherence

• originality

• engagement

System B: Enterprise knowledge assistant

We might care much more about:

• roundedness

• citation correctness

• completeness

• instruction following

• refusal of unsupported claims

The same output could potentially be desirable in one application and unacceptable in another.

This means that evaluation cannot be separated from the purpose of the system.

14. The Evaluation Pipeline

Putting everything together, an LLM evaluation workflow might look something like:

The actual implementation can be considerably more sophisticated, but the fundamental loop remains similar.

My Takeaways

Rather than thinking about prompt engineering as: "How do I phrase my instruction?"

I now find it more useful to think of: "I have an LLM-based system with a desired behaviour. How can I define, measure and improve that behaviour?"

The prompt is merely one of the levers I can change and manipulate to get my target result. The evaluation framework tells me whether pulling that lever actually helped. And the test dataset gives me the evidence to make that decision.

This makes prompt engineering feel much closer to an empirical engineering discipline than a collection of clever prompting tricks.

Resources

• OpenAI - Evaluation Best Practices

• OpenAI - Getting Started with Datasets / Evals

• OpenAI - Model Optimization

  • Optimizing LLM Accuracy

  • Generative AI Evaluation

• A Survey on LLM-as-a-Judge - arXiv

Prompt 1: Write a cover letter for me.

Prompt 2: Write a professional but personable cover letter for a Software Engineer position at a technology company.

My background: I have a Computer Science degree and three years of experience working in software development and technical project coordination.

Emphasise my experience working across engineering teams, debugging software issues, and coordinating multiple projects. Keep it under 400 words and avoid generic phrases such as "I am excited to apply."

The second prompt gives the model considerably more information about the task, context, constraints, and desired output.

This is the basic idea behind prompt engineering.

Rather than treating an LLM like a search box, prompt engineering is about learning how to communicate with a model in a way that makes the desired outcome clearer and more reproducible.

What is Prompt Engineering?

Prompt engineering is the process of designing and refining instructions given to a language model so that it produces an output that better meets a particular objective.

A prompt can be as simple as:

"Explain quantum computing."

Or it can contain multiple components:

• A task

• Background context

• Constraints

• Examples

• Desired output format

• Tone or style

• Information the model should use

• Instructions about what to avoid

For example:

Task: 
Explain Retrieval-Augmented Generation (RAG). 

Audience: 
A beginner who understands basic LLM concepts. 

Requirements: 
- Explain the problem RAG solves 
- Describe the retrieval and generation stages 
- Give a simple real-world analogy 
- Include one practical example 

Output: Use headings and keep the explanation under 600 words.

The goal isn't simply to make a prompt longer.

The goal is to make the model's job clearer.

Why Does Prompt Engineering Matter?

LLMs don't inherently know what we mean when we give them an ambiguous instruction.

Consider:

"Tell me about Python."

Depending on the context, we could be asking about:

The Python programming language Python's history Python syntax Python libraries Python as a career Python the animal

A more precise prompt reduces this ambiguity:

"I'm a beginner learning programming. Explain what Python is, why it is commonly used for AI and data science, and show me three simple examples of Python syntax."

The model now has considerably more information about what constitutes a useful answer.

This is why effective prompting is less about discovering a magical combination of words and more about communicating requirements clearly.

OpenAI, Google, Microsoft and Anthropic all emphasise variations of the same fundamental principle: provide clear instructions, relevant context, and appropriate constraints, then iterate based on the results.

A Useful Mental Model: Task + Context + Constraints + Output

One way I find useful to think about prompts is to break them into four components. Task Context Constraints Output.

Task

What do you want the model to do?

Examples:

Summarise this article. Explain this Python function. Compare these two approaches. Generate five ideas for a blog post.

The task should be explicit.

Instead of:

"What do you think about this?"

Try:

"Identify the three strongest arguments in this proposal and explain one potential weakness for each."

Context

What information does the model need to complete the task?

For example:

I am a beginner learning about machine learning.

or:

This presentation is intended for non-technical business executives.

or:

The following text is an internal project update. Do not introduce information that isn't contained in the text.

Context gives the model information that might otherwise be missing.

This is especially important for domain-specific tasks.

Constraints

What boundaries should the model follow?

For example:

Keep the answer under 500 words. Use only the information provided below. Do not use technical jargon without explaining it. Return exactly five recommendations.

Constraints help define what a successful response should look like.

Output

What should the final response look like?

You can specify:

- Markdown
- Bullet points
- Tables
- JSON
- An email
- A step-by-step explanation
- A specific tone
- A specific length

For example:

Return the results as a Markdown table with these columns:

| Concept | Definition | Example | Common Mistake |

This is particularly useful when the output will subsequently be processed by another system or used as structured data.

Some Common Prompt Engineering Techniques

Once the basics are understood, there are several techniques worth experimenting with.

Zero-Shot Prompting

The simplest approach is to give the model a task without providing examples.

Classify the following review as positive, negative, or neutral: "The battery life is excellent, but the screen is disappointing."

The model has to infer the task from the instruction alone.

This is often sufficient for straightforward tasks.

Few-Shot Prompting

Instead of explaining the task abstractly, provide examples of the desired behaviour.

For example:

Classify each sentence as positive or negative.

Example: "I love this laptop." → Positive

Example: "The keyboard feels terrible." → Negative

Now classify: "The display is beautiful." →

The examples give the model a pattern to follow.

Few-shot prompting can be particularly useful when the desired output has a specific structure or classification scheme.

Role Prompting

Another commonly used technique is to specify a role or perspective.

For example:

You are a senior software engineer reviewing code written by a junior developer.

Review the following code for:
1. Bugs
2. Security issues
3. Readability
4. Performance

Explain your recommendations clearly and provide corrected code where appropriate.

This establishes a perspective and gives the model a clearer set of responsibilities.

However, I don't think of role prompting as simply saying:

"You are an expert."

and expecting better answers dramatically.

The more useful part is defining what the model should actually do in that role.

Anthropic's prompting guidance similarly treats roles as one technique among several, alongside clarity, examples, context and structured prompts.

Structured Prompts

For more complicated tasks, structuring the prompt can make both the prompt and the resulting output easier to understand.

For example:

## Role
You are a technical writer.

## Task
Explain Retrieval-Augmented Generation.

## Audience
Computer Science students who understand basic LLM concepts.

## Include
- What RAG is
- Why it is useful
- How retrieval works
- How generation works
- A simple example

## Constraints
- Avoid unnecessary jargon
- Use one analogy
- Keep the article below 1,000 words

## Output
Use Markdown headings and bullet points.

Markdown headings, lists and other delimiters can make logical boundaries clearer to the model. XML-style tags can also be useful, particularly when separating instructions from large blocks of reference material.

Prompt Engineering Is Iterative

One of the biggest takeaways from learning about prompt engineering is that writing a prompt isn't necessarily a one-shot process.

You might start with:

Explain RAG.

The answer might be technically correct but too advanced.

So you refine it:

Explain RAG to someone who understands what an LLM is but has never encountered RAG. 

Use a simple analogy and explain: 
1. Why RAG is needed 
2. How retrieval works 
3. How the retrieved information is used to generate an answer

Then perhaps you realise the explanation is still too long:

Keep the explanation below 500 words.

Then you might want to test whether the model understood the concept:

After explaining RAG, give me three questions to test my understanding. Do not provide the answers until I respond.

This creates a cycle:

Prompt → Output → Evaluate → Refine → Output

OpenAI explicitly recommends iterative refinement, while Anthropic recommends establishing success criteria and ways to evaluate the result before attempting to optimise a prompt.

Prompt Engineering Has Limits

It is also important not to treat prompt engineering as a solution to every LLM problem.

If a model doesn't have the information required to answer a question, simply making the prompt longer doesn't magically give it that information.

Similarly, a prompt cannot guarantee that an LLM will always produce a correct answer.

Depending on the problem, other approaches may be more appropriate, such as:

• Retrieval-Augmented Generation (RAG)

• Tool use

• Function calling

• Fine-tuning

• Model selection

• Evaluation systems

• Better source data

• Application-level validation

Anthropic's prompt engineering documentation makes a similar point: some problems that appear to be prompting problems may instead be better addressed through model selection, evaluation, or other system-level changes.

This is an important shift in perspective.

Prompt engineering is one component of building useful AI systems, not the entire solution.

(WIP) Add On: I should discuss LLMs doing their own prompt engineering.

I was going to dedicate another article to this, but I fear that the section is short enough to add here when I have the time.

Summary: You can prompt an LLM to help draft/improve/modify/write better. Especially useful for prompts that will be reused often.

Resources

• OpenAI - Prompt engineering best practices for ChatGPT

  • Can read up on other similar companies' documentation

  • OpenAI

  • Google Cloud

  • Anthropic

  • Microsoft

Large Language Models (LLMs) are AI models that are remarkably good at generating answers written in human-like text.

Retrieval-Augmented Generation (RAG) is a technique / AI architecture design that enables LLMs to retrieve and incorporate new information from external data sources, at the time when a question is asked.

Not all AI models can access information beyond the data they were trained on. Therefore, AI architects and developers must consider whether it is practical and effective to incorporate the required data directly into the model's training process, or whether the data should instead remain external and be retrieved by the model only when needed.

For example, imagine asking an LLM about a company's internal policies, a private knowledge base, or a document that was just freshly published. The LLM may simply not have that information, as such data was not included in its training process.

You could fine-tune the model on the company's employee handbook and private databases, but that makes the model inflexible to updates and policy changes, as updating and re-running the AI model training process can be a computationally exhausting task.

RAG takes a different approach: Instead of modifying the AI model itself, we can keep the company's documents and policies outside the model, and retrieve the relevant information as needed. So data and policy changes can be quickly modified and updated directly to the external database, without needing to retrain the model.

The model does not need to contain the entire knowledge base. It just needs be able to use the RAG system to obtain the right information in its context when it generates its LLM answer.

As shown in the flow chart above, the process of generating an answer to a user's question can be summarised as:

Retrieve relevant information → give it to LLM → generate an answer

This simple idea has become one of the fundamental patterns for building AI applications, especially when dealing with private, domain-specific, and frequently changing data.

3 Stages of RAG (Retrieval-Augmented Generation)

The name describes the 3 core stages

1. Retrieval: Finding information relevant to the user's question.

2. Augmentation: Add that retrieved information to the prompt sent to the LLM.

3. Generation: Ask the LLM to generate an answer using the additional context.

Microsoft describes this basic pattern as retrieving grounding data, providing it to the model, and generating a response grounded in that information.

(The concept of grounding to avoid hallucinations to be elaborated on further below... )

Why can't we just put the entire document into the prompt?

Sometimes we can!

But imagine a company has:

• 10,000 PDF documents

• thousands of web pages

• internal wiki pages

• product specifications

• technical documentation

• customer records

Often times we know the data we want is from a specific PILE of documents, but we have trouble identifying which one precisely. Sending everything to the LLM for every question would be extremely inefficient.

Instead, RAG creates a searchable representation of the knowledge base.

The system can then retrieve only the pieces that are likely to be relevant.

Overview of the Process

Step 0a: Split the Document into Chunks

Before we can retrieve information, documents usually need to be broken into smaller pieces called chunks.

For example:

Retrieval works better when the system can identify the specific section that answers the question.

There are different chunking strategies / methods that work to divide a document into smaller pieces.

It is often more complicated than simply choosing an arbitrary number of characters to divide the text by.

A chunk that is too small may lose important context. While a chunk that is too large may group lots of irrelevant information together.

This makes chunking strategy one of the important designs in a RAG System.

Step 0b: Turn Chunks into Embeddings

This is where the concept of embeddings becomes especially useful.

An embedding model converts text into a numerical representation (aka a vector coordinates in a high n-dimensional vector space).

E.g. "How many days of annual leave do employees receive?" → Embedding Model → [0.12, -0.43, 0.19, ..., 0.31]

Each chunk will receive its own embedding from the embedding model. The goal is not to encode the exact words, but the vector should represent aspects of the text's meaning. This will allow us to compare a user's question against document chunks based on semantic similarity.

E.g.

• User Question: "How much vacation can I take?"

• Document Chunk: "Employees are entitled to 15 days of annual leave..."

There is semantic similarity between the User Question and the Document Chunk. The words are different, but the concepts are closely related.

Ideally the embedding model will be able to convey the relationship between the 2 similar concepts, and make it easier to retrieve the information.

Step 0c: Store the Embeddings

The embeddings can be stored in a system designed for efficient retrieval. This might be a vector database or a search engine supporting vector search.

E.g.

Document Chunk → Embedding Model → Vector → Vector Index

The database may not only store the numerical vector, but also additional metadata. The metadata can later help with filtering, citations, and access control.

E.g.

{ 
    "document": "employee_handbook.pdf", 
    "section": "annual_leave", 
    "page": 42, 
    "text": "Employees are entitled to..." 
}

Step 1: Retrieve Relevant Chunks

When a user asks the LLM a question, the text of the question will also get converted into an embedding.

The Question Vector is then used to find similar Document Chunk Vectors.

Those vectors then become reference points to pull out the relevant document information.

The relevant information is then passed into the LLM to be processed.

E.g.

• User Question: "How much vacation can I take?"

• User Questions → Embedding Model → Query Vector → Search Database Vector Index to find similar Document Chunk Vectors → Return Most Relevant Chunks

• The retrieval system might return:

  • 1. Annual Leave Policy

  • 2. Leave Entitlement

  • 3. Leave Carry-Forward Rules

• These retrieved chunks then become the context that we give to the LLM.

Step 2: Augment the Prompt

Augment the prompt to be passed to the LLM.

Instead of simply sending: "How much vacation can I take?"

We can provide the LLM with the retrieved information:

• Data Context: 1. Annual Leave Policy | 2. Leave Entitlement | 3. Leave Carry-Forward Rules

• User Question: "How much vacation can I take?"

The LLM can now generate an answer using information retrieved from the company's knowledge base.

Step 3: Generate the Answer

Finally, the LLM receives: (User Question + Retrieved Context).

Then it processes this modified/augmented input.

And generates the final answer.

For example: "Employees are entitled to 15 days of annual leave per calendar year."

The important part is that the answer is grounded in retrieved information rather than relying entirely on the model's pre-trained knowledge.

The Complete RAG Pipeline

Putting everything together...

This separation between indexing and querying is an important part of understanding how real RAG systems work.

Does RAG Eliminate Hallucinations?

No. :(

This is one of the most critical and important misconceptions about RAG.

In fact I was also confused to learn otherwise as I have previously read articles that said RAG was one of the main techniques to counter AI hallucinations.

My improved understanding of the RAG method and the step-by-step process has cleared up my confusion and helped me to think more clearly about the relationship between LLM and the RAG method!

RAG can help ground an LLM's response in external information, but it doesn't guarantee that the final answer is correct.

There are several possible failure points.

Failure 1: Bad Retrieval

The correct information exists, but the system retrieves the wrong chunks.

User Question → Poor Retrieval → Wrong Context → LLM → Wrong Answer

Failure 2: Missing Information

The answer is simply not in the knowledge base.

Failure 3: Poor Chunking

The relevant information may have been split in a way that removes necessary context.

Failure 4: Poor Generation

The correct information was retrieved, but the LLM itself misunderstood, or incorrectly synthesized it, or still continued to hallucinate.

This means improving a RAG system isn't simply about choosing a better LLM.

You also need to improve the retrieval pipeline.

Beyond Basic RAG

A basic RAG implementation might look like:

Question → Vector Search → Top-K Chunks → LLM

But the real-world systems can become much more sophisticated by integrating other methods and technology.

They may include:

• keyword search

• semantic search

• hybrid search

• metadata filtering

• query rewriting

• query decomposition

• reranking

• multiple retrieval steps

• citations

• retrieval evaluation

• response evaluation

For example, hybrid search can combine traditional lexical matching with semantic/vector retrieval.

Microsoft's current RAG guidance discusses vector, full-text, hybrid and multiple-search approaches as part of designing retrieval systems. (To do further research in the future)

RAG vs Fine-Tuning an LLM

This is another area of confusion for many, but RAG and fine-tuning solve different problems.

RAG changes what information the model can access.

Fine-tuning changes how the model behaves.

RAG

Useful when you want the model to access:

• private information

• frequently changing information

• company documentation

• product documentation

• knowledge bases

• external sources

Fine-tuning

Useful when you want to change aspects of how a model behaves, such as:

• output style

• formatting

• specialised task behaviour

• domain-specific patterns

My Thoughts on RAG

I think RAG is an interesting example of how useful AI applications are often more complex than most of the public would assume, and how huge improvements in AI can come from updates in smaller components and tools that work together behind the scenes to produce better results as a whole.

An LLM by itself is essentially a powerful language model.

But a useful AI application often looks more like:

The intelligence of the application isn't necessarily contained entirely inside the model.

It's also in the surrounding system:

Data + Retrieval + Orchestration + Model + Evaluation

That's what makes RAG such an important pattern for modern AI applications.

Moving Forward

Understanding basic RAG is only the beginning.

The next questions become much more interesting:

• How should documents be chunked?

• How do we choose an embedding model?

• How many chunks should we retrieve?

• When should we use hybrid search?

• How do we rerank results?

• How do we evaluate retrieval quality?

• How do we handle conflicting documents?

• How do we cite sources?

• How does RAG work with tables and images?

• What is GraphRAG?

• What is agentic RAG?

These are the problems that turn a simple "chat with your PDF" demo into an actual AI engineering system.

And that's probably the most important thing I've learned about RAG so far: A RAG application isn't just an LLM with a vector database attached.

It's a retrieval system, a data pipeline, and a generation system working together.

Resources:

• Microsoft - Introduction to retrieval-augmented generation concepts

• Microsoft - Retrieval augmented generation (RAG) and indexes

• Microsoft - Generative AI for Beginners

• pinecone - Retrieval-Augmented Generation (RAG)

• Wikipedia - Retrieval-augmented generation

Despite how frequently LLMs are discussed, some of the fundamental concepts behind them can be surprisingly difficult to distinguish. What exactly is a token? How does a context window differ from a model's parameters? And where do embeddings fit into the picture?

Below are some of the key concepts behind LLMs, that will provide a useful foundation for learning how LLMs function:

• Tokens: how text is represented as discrete units

• Context windows: how much information a model can process at once

• Parameters: the learned values that determine how a model behaves

• Embeddings: numerical representations that capture semantic relationships

Tokens: How LLMs Read Text

While humans can naturally understand words, phrases, and sentences to parse meaning and information, machines can only process numbers. Hence, the text must be converted to numbers.

One way of doing this is using tokenisation. Where text is broken down into small tokens, discrete units that can then be associated to a numerical integer index.

The Tokenisation Algorithm

A common misconception is that one word corresponds to one token.

This may be true in some cases, but it is not necessary.

A single token may represent:

• an entire word

• part of a word

• punctuation

• whitespace

• numbers

• special characters

The exact tokenisation depends on the model and tokeniser being used.

A relatively uncommon or long word may be split into multiple small subword tokens for processing.

This approach allows models to represent a very large vocabulary without requiring a separate token for every possible word.

Practical Tips for Tokenisation

If an application has a context limit of a certain number of tokens, the limit applies to the tokenised representation of the input, not simply to the number of words.

This is particularly important when working with:

• long documents

• source code

• chat histories

• retrieved documents

• structured data

• multilingual text

A prompt that appears relatively short to a human can still consume a substantial number of tokens depending on its content.

Context Windows: The Model's Working Context

A model's context window refers to the amount of tokenised information it can consider within a single processing context. It is the max amount od information the AI model can process at one time.

This can include both the user's input and other information provided to the model, such as:

• previous conversation messages

• system instructions

• retrieved documents

• tool outputs

• other contextual information

The larger the context window, the more information will be considered in processing a suitable response. But more memory and computation are required.

It is similar to RAM in concept, where only so much data can fit at a time, and anything outside is not immediately computed.

Context window ≠ model knowledge

It is a common misunderstanding that the context window limits the model's full scope of information.

An LLM may have been trained on a large amount of information, but that does not mean all of that information is simultaneously present in its context.

A useful distinction is:

Training data → learned into parameters

Prompt/context → information currently available to the model

The context window determines how much tokenised information can be supplied to the model during a particular interaction.

Why Context Windows Matter

Larger context windows can be useful for tasks involving large amounts of information, such as:

• analysing lengthy documents

• reviewing large codebases

• maintaining long conversations

• comparing multiple sources

• processing large datasets

However, simply increasing the context window does not automatically guarantee better reasoning or better answers. The quality, relevance, and organisation of the information provided to the model still matter.

Parameters: What the Model Learns

If tokens describe the information being presented to a model, parameters are part of what the model has learned during training.

Parameters are numerical values that are adjusted during the training process.

A simplified view of training looks like this:

Through repeated training, the model adjusts its parameters so that it becomes better at predicting the next token.

Modern neural networks can contain billions or even hundreds of billions of parameters.

What Are Parameters?

It is tempting to think of parameters as a database containing facts.

That is not quite accurate.

The parameters are numerical values distributed throughout the neural network. Collectively, they encode patterns and relationships learned during training.

These patterns can include relationships involving:

• language

• syntax

• semantics

• concepts

• code

• factual associations

• patterns of reasoning

The model does not generally retrieve a sentence from a database every time it generates an answer. Instead, its learned parameters influence the probability distribution over possible next tokens.

More parameters ≠ automatically better

Model size is often discussed using parameter count, but parameter count alone does not determine capability.

Performance also depends on factors such as:

• architecture

• training data

• training methodology

• optimisation

• inference techniques

• model specialisation

• context handling

Consequently, a smaller model can outperform a larger model on particular tasks.

Embeddings: Representing Meaning Numerically

Embeddings provide another important piece of the LLM ecosystem.

An embedding is a numerical representation of some piece of information—such as a token, word, sentence, document, or other object—in a high-dimensional vector space (embedding space).

Encoding refers to the process of choosing an initial numerical representation of data to train the model on.

An embedding represents each item in n-dimensional space with n floating-point numbers (typically in the range –1 to 1 or 0 to 1).

For example, a simplified embedding might look like:

"cat" → [0.21, -0.43, 0.17, 0.82, ...]

The embedding is in the form of a numerical array called a vector, which maps the coordinates of the data point in a high-dimensional vector space.

The following example is from Google Machine Learning Crash Course, because I lack the creativity to invent a more suitable case:

A one-dimensional representation of food, using "sandwich-y" to determine the positioning of a food data point.

A two-dimensional representation of food, plotted by both "sandwichness" and "dessertness".

An AI model typically uses a much higher-dimensional space to plot out its data.

The important idea is not each numerical vector itself. The important part is the relationship between each of these vectors.

Semantic Similarity

Items with related meanings can be represented by vectors that are closer together in the embedding space.

Eg. A shawarma is more similar to a hot dog, and may be grouped together. Or a borscht and a salad can be grouped together due to similar characteristics, as determined by their similar vector positions.

This allows systems to perform operations based on semantic similarity rather than simply matching exact words.

For example, a search system could retrieve a document about "automobiles" when a user searches for "cars," even though the exact words may differ.

How These Concepts Fit Together

These four concepts describe different parts of the LLM pipeline.

It is important to consider each component when designing or debugging LLM applications.

For example:

• If a document is too large to process, context limitations may be relevant.

• If an application exceeds its token budget, tokenisation matters.

• If a model lacks useful learned patterns, training and parameters become relevant.

• If a system needs to find semantically relevant documents, embeddings and vector search become important.

Building a document-question-answering system may require decisions about:

• how documents should be chunked

• how many tokens can fit into the context

• which embedding model should be used

• how vectors should be stored and searched

• how retrieved information should be passed to the LLM

• how much context should be included in each request

A Simple Mental Model

One way to remember the differences is to think of an LLM as a system processing information through several stages.

Tokens are the units the model processes.

The context window determines how much tokenised information can be considered at once.

Parameters contain the learned numerical patterns that shape the model's behaviour.

Embeddings represent information as vectors that can be compared based on learned relationships.

Resources

• Google Machine Learning Crash Course: LLM

• Google ML Crash Course: Embedding

• OpenAI: Understanding and Counting Tokens

• OpenAI: Embeddings Documentation

• Attention Is All You (The Original Transformer Paper)

So I'm starting from the ground up, I want to understand questions like:

• How does a neural network actually learn?

• What exactly is an embedding?

• Why did Transformers change AI?

• How does attention work?

• What actually happens when I send a prompt to an LLM?

• How are LLMs trained?

• What's the difference between prompting, RAG and fine-tuning?

• Why do LLMs hallucinate?

• How do you evaluate whether an AI system is actually good?

And I plan to use this blog to document my learning experience.

Why write about it?

I've always believed that the best test of understanding should follow the Feynman Technique: to explain a concept simply in your own words, as if teaching it to a beginner or a child.

I want this writing process to force me to go beyond "I think I kind of get it" and to avoid muddling my way through half-understandings; I want to progress towards being able to articulate and explain the details of how AI has enabled computers and machines to mimic human thinking, to learn from data, and to perform complex tasks like problem-solving and decision-making.

I want this blog to be a public notebook of things I'm learning, building and trying to understand.

Some posts will probably be beginner-friendly explanations. Others will get more technical as I go deeper into the underlying concepts.

I'm also hoping that writing publicly will make me more disciplined about actually building things rather than endlessly consuming courses and tutorials.

What I'll be learning

I'm currently interested in the intersection of:

Artificial Intelligence · Machine Learning · LLMs · Software Engineering · AI Products

Currently, I want to start working my way through topics such as neural networks, embeddings, Transformers, LLM architectures, training, fine-tuning, RAG, evaluation and AI application engineering.

I will not be able to write from the perspective of an expert, but I hope that I can achieve meaningful learning here, and to encourage others who also wish to start their learning journey.

What to expect

I'll be turning my notes and experiments into short technical articles — partly as a way to reinforce my own understanding, and partly in the hope that they might help someone else who is learning the same things.

If I get something wrong, I'll fix it.

If I learn something interesting, I'll write about it.

And hopefully, over time, this blog becomes a record of how my understanding of AI develops from the fundamentals to actually building things with it.

Let's see where this goes.