The term context engineering has become the biggest buzzword in
AI in the past 10 days, and it's a word that if industry leaders
are to be trusted, you must now start to learn, with many people
calling it the next big evolution from prompt
engineering. In very simple terms, prompt
engineering is what you ask an AI model to do for you, whereas
context engineering is what a model knows when you ask it
something. This shift reflects very hard
one lessons when building AI agent systems.
You can have something that has all the bells and whistles, but
if it doesn't give the right context, it can never deliver
good outcomes for people. In today's episode, we're going
to discuss the evolution from prompt engineering to context
engineering, and I'll breakdown what context means when it comes
to AI. And finally, I'll actually give
a breakdown of some of the real things that you as a person are
going to start noticing as a result of improved context
engineering. This is in the Loop with Jack
Halton. I hope you enjoy the show.
So I'm going to assume most of you already know what prompt
engineering is, but just just for the sake of the episode, I'm
going to spend 20 seconds on it. Prompt engineering is a phrase
that gained mainstream adoption and traction generally after the
launch of ChatGPT in about late 2022.
And this is because millions of people are suddenly trying to
figure out how to talk to an AI system.
It was one of the very early job roles that were advertised for
by big companies. Prompt engineering became a
skill set that people went into courses about generally became a
really important skill. And engineering a prompt
basically meant using trial and error and intuition really to
find a set of I guess instructions and also the way
the instructions are worded to make the model do what you want
it to do. A really famous example of this
was the very famous prompt that literally just said, let's think
step by step. And this led to train of
thought, which is a form of AI reasoning, which typically
improves the performance of how an AI model performs when you
just ask it a simple question because it's making itself a
plan and following it step by step.
However, as the use cases for AI have become increasingly
advanced, so have the steps and the tasks that an agent must
follow to actually, I guess, deliver value today.
And as a result, there were very clear limitations with prompt
engineering because early prompt engineering is essentially the
A1 shot attempt. So meaning you write a single
prompt, which could maybe include a few examples or
instructions, and the model then give an answer as a result.
But many tasks require dialogue with users.
It requires the agent to memorize things.
It requires the agent to use tools, so integration into
separate systems to capture data from those systems.
And so, for example, asking a model to, say, answer this
question by searching the web and then write a summary can't
be done in a single one prompt attempt.
The model needs to take many different actions, like doing a
search and remembering what it found and then doing something
with that information. And many builders in the space,
including ourselves, do things called prompt chaining, which
essentially means chaining prompts together to give the
model more context to every single step.
So take a step and then follow another step and use the
information from that previous step.
And there was many different innovations that include RAG, so
retrieval, augmented generation, including what's called React,
So reasoning and action, which essentially are ways of getting
a model to prompt and capture subsequent information and use
that as context to provide a better answer.
The point being is that it's not actually just the instruction
sent to a language model that matters.
It's actually the context the language model has and how it
carries context throughout many different steps.
And this is where context engineering really starts to
take center stage. Because context engineering is
all about deciding what a language model should know
before it speaks and answers a human.
Not just what you ask, but what data or tools or memories,
instructions, and history should be fed into that system to guide
its answer. And the term itself became
largely popularized by Shopify CEO very recently on Twitter.
He said, I prefer the name context engineering to prompt
engineering. It better describes the core
skill set, which is giving the model the context it needs to
plausibly solve the task. And there are other big names
out there like Andre Kapathy who was one of the Co founders or
let's say founding team members of open AI.
And he's very famous and even coined the term vibe coding.
He described it more as an art and science of filling the
models context window just enough to make the next step in
its sequence work, but not too much and not too little.
It's just the right amount of information.
So you can almost consider context engineering like
orchestration. You're orchestrating an agent
over many subsequent steps and trying to control what
information and context it should have after each step and
how it should use that context in the subsequent one.
Let's go one level deeper and actually unpack what counts as
context when it comes to an AI model.
Because every piece of software built over a language model,
that's mindset, that's every provider is essentially this
fantastic piece of software that is dictating the context and the
instructions sent to a model and how that information is surfaced
to a user. And that is where all the hard
work comes from because it's flipping difficult to get it all
to work. And you'll see why in a second.
There's many layers to context. So you've got the first piece of
context, which is the immediate prompt.
So this is the direct question or instruction by a user, for
example, creating a marketing strategy.
And this is where prompt engineering has typically been
focused #2 you've got systems instructions, which essentially
a system message sent to the large language model to set a
behavior. So you are an expert travel
panel. The second layer is called
system instructions and essentially this is where you
send a system message to a large language model such as you are
an expert travel planner. So this is the context that
influences say the tone, the scope, and this alone can be
quite detailed. We've got an entire agent
builder around these which dictates how to behave, not to
behave, guard rails, policy, personality, how it should
structure its responses, but that's all system messages.
Again, the software built on top of these things a highly complex
in advance just to make sure the right thing is sent at the right
time with the right amount of context.
So third piece of context is the conversation history, which is
often called the short term memory.
So in a conversation which they ChatGPT, you want parts of that
chat to be included in every single prompt that goes to the
LLM. So when you ask a question, it
has the context of all previous conversations you've had in that
thread again, so you might say as I was speaking about earlier,
and therefore it knows what earlier means.
And that's because every time a message by a user is sent
ChatGPT is a piece of software also takes the message history,
condenses it down and sends it to the language model.
So it then has that context of the conversation.
The 4th type of context that's absolutely critical is long term
memory or persistent memory. This can be a summary of
previous interactions. This could be stored facts about
a user, such as my preferences or information about my profile
or even previous conversations between different threads.
So it's that context that persists across different
messages. So it knows that you work in a
certain job, you like response in a certain way, you hate
things that sound like this. We're actually as mindset
launching an entire long term memory layer next month, which
enables our customers to dictate what an agent should remember.
Then you've got retrieving external information.
So this might be both RAG or integrations into third party
systems. So sometimes you'll ask
questions that will require context of external documents or
code snippets or videos. And so the software that you'll
be interacting with will then call that information, bring it
into the prompt that is sent. So your message plus that
information from an external data source sent to the LLM, and
then the LLM will use that to answer your question.
Another example could be a customer support bot that pulls
in user's account data whenever it answers a question about
billing. You've then got another layer
down, which is tool definitions. So if the models can use tools
like say, making API calls or running calculations, to do
that, it needs to understand how to actually call those tools or
interact with external tools, and it must use the information
it gets back to answer or support what it's doing.
And so whenever it uses one of those tools, let's say
summarizes all my WhatsApp conversations, you then want it
to be able to retain the output of that query into WhatsApp for
the next step in the process. So yeah, that's different layers
of what context actually means in AI system.
Again, it just sounds like a human that requires context to
do good work, but it actually has very big implications on
what it means for a genetic systems.
And it's one of the big reasons we're launching a new
orchestration system that enables our customers to really
give models the right context at the right time.
And I think in summary, and I'll explain why in just a second,
simpler processes or sequential step by step processes are
better than multi agent systems right now.
We've talked about multi agent systems before in previous
episodes, but this is where you have teams of agents doing many
different things. And it's really relevant to this
conversation because it's all about the concept of context
engineering and really highlights why context is so
important. So let me delve a level deeper
by explaining the problem of having teams of agents doing
many different things at once. There are many different use
case ideas that people often think, oh, let's get a team of
agents to do this. Let's get a team of agents also
working on tasks in parallel to each other and then breaking
down those tasks into subtasks and getting other agents to help
with those subtasks. So the idea might be to have,
say, Agent A break a problem into small little parts,
delegate it to agent B&C, and then have a say combine all of
their information into one answer.
Because in theory it would just make the responses faster
because you have people working, or so I say, agents working in
parallel. And those agents could be
specialized, breaking down tasks, for example, or
summarizing information. So in theory it makes sense, but
often it will fail because of coordination.
And when we say coordination, what we actually mean is
context, contacts not being shared between different agents.
Now, there was a fantastic paper recently released by Cognition
Labs again, the one of the labs behind what's called the Devin
agent, one of the very famous coding agents before the new age
coding apps emerged. And that paper was called Don't
build multi agent systems. And to show why they actually
tasked a set of agents to build a Flappy Bird clone.
To accomplish this. The I created sub agents want to
create the background graphics and another one to create that
player bird, the thing that you're going to control.
Now without the full context, the first sub agent might
interpret the task a little bit different and generate a Mario
like background instead of a Flappy Bird pipes background.
Now they're very similar but slightly different.
But the second agent might then produce a bird that doesn't
therefore fit with the style of graphics.
So when these two agents merge the outputs to combine it into a
single game, it obviously doesn't work.
It feels like separate products. And the root cause of this is
that the sub agents lack the full context of the original
objective and the work of other agents going on at the same
time. Now the straightforward solution
you might say would just to be give all these sub agents all
the information they need, like the information on the task, the
description, any decisions of the agents have made so they can
go and do a good job. In practice, that means when
agent A, say, delegate something, it should pass along
not just the sub task, but also all of the information, the
conversation history, everything that's happened up until that
point, and all the decisions that led there.
However, even sharing the entire context isn't enough sometimes
because the sub agents then might work with full knowledge
of say the main objective, yet still produce something that's
incompatible because they may make assumptions throughout that
process that conflict with each other.
Like 1 deciding that the game should have a cartoon style and
the other a realistic style. Again, you'll notice that every
single decision or action by an agent will always have a knock
on effect. And so overall what Cognition
Labs have claimed and believed after much testing is that
single agent architectures, which means that one agent
follows sequentially a few steps is far easier to get a fantastic
output than multi agent systems. And it's exactly as I said, why
mindset is enabling this exact use case making enabling our
customers to dictate all the different steps and separate
integrations and the long term memory that an agent should
carry throughout a conversation and essentially what it should
do with each part of its data and output at each step.
And they're not the only ones that reported this because
Anthropic did a lot of research into multi agent systems for
their research assistance and hire the exact same problem.
We've personally found that same problem when trying to use multi
agent systems in our own business.
So for all the excitement of multi agent systems with teams
of agents doing things and and even a single agent doing
something really powerful, it all comes down to how well the
software product on top of those large language models can prompt
and give it context every single time it prompts.
So I think this is going to be a growing area of innovation over
the next sort of 12 months. And obviously I believe
everything will be solved and multi agent systems will
probably be possible in I'd say 12 months, but we can always
half it and everything happens faster than we thought.
So I guess the interesting question therefore, is as a
listener right now, how does this impacts you?
Like what should you know about this?
Why is this relevant to you? But really simply, I think good
context engineering, either as a user using just something simple
as ChatGPT, or using a product that's an AI product.
Good context engineering is where you're left with a sense
of that was actually helpful. It also massively influences the
types of experiences and features and options as a user
that you're going to be given by products.
So one of those are going to be personalization.
The system is going to increasingly be able to remember
everything about you, your preferences, and previous
conversations. You're going to start to see
more user interface and user experience patterns emerge.
For example, as AI becomes integrated and say, apps,
designers are going to have to come up with ways to give
context to agents without users actually realizing it.
For example, many chat interfaces have an upload file
button, which enables you to upload a document.
From a user's perspective, it's straightforward.
I want to talk to a document. But actually what you're doing
is giving the agent or the AI language model context.
You're also going to have different modes of working.
So even if you go on the ChatGPT app, you have the deep research
prompt. If you're a pro user, you have
the Google search, you now have study with me.
There's many different options. And what that does is
essentially give context to an agent that they built.
And it might just be a single agent architecture with a series
of steps. It gives it context on how it
should behave in that situation. Another big area is managing
context in multi tenant environments.
So many of our customers are technology providers, software
providers who have say thousands of enterprise customers and each
one of them have their own space in these platforms.
How therefore does an agent or an AI retain conversations and
contacts between all these people and stay secure?
So it's an area we've been working really harder and we're
excited to bring some out for. But this is the area that's
really important because let's take a SAS platform in sales.
It might integrate with every single salesperson's contacts.
It's CRM, it's e-mail. So when a salesperson asks to
say, let's say draft an e-mail, it'll just use all of that data
seamlessly, very cleverly, magically and answer you.
But all of that magic actually had to be done quite securely.
I think the other one is going to be privacy and trust for you
because I would be worried about what happens with all this data.
They're capturing a lot of data on us now.
I personally use it as a life coach as ever.
I use it for almost any question that I've got.
I will almost always go to ChatGPT or another AI system
first. So I guess let's just conclude,
early on we were obsessed with the model itself and how to talk
to it with prompt engineering. And over time we've realized
that an AI agent isn't just the model, it's the whole system
that feeds it contacts and interpret its output and uses
that output to inform its next set of contacts.
So in a way, we're moving one level above in terms of
abstraction. How do I get the model to output
X? Response is more about thinking
what information do I need to give it and guidance to get that
best response. Anyway, I hope you enjoyed the
episode, I certainly did, and I'll see you next week.