Next in Tech | Ep. 246: Context Engineering

View Full Transcript

Eric Hanselman: Welcome to Next in Tech, an S&P Global Market Intelligence podcast where the world of emerging tech lives. I'm your host, Eric Hanselman, Chief Analyst for Industry and Company Data at S&P Global Market Intelligence. And today, we're going to be talking about an aspect of AI that is just starting to be more widely discussed. It's context engineering and it's something that's being driven by the shift to agentic work, having agents actually start to do some of the AI heavy lifting that's taking place. And with me to discuss it are 2 of our analyst team, Alex Johnston and Jean Atelsek. Welcome back to the podcast for both of you.

Jean Atelsek: Thanks. Great.

Alex Johnston: Thanks, Eric.

Eric Hanselman: When we think about AI, I think most of the planet, it seems these days is familiar with the idea of crafting prompts for chat-based AI. But when we start thinking about what the shift to agentic action is and the agents that are necessary to really do that, we've talked a lot about how prompts have to be engineered well, how you have to build in enough different perspectives in the prompt, what's effective in terms of how you manage it. When we start to make that shift to agents, agents are going to be acting more independently. And in order to get effective results, they have to have context around what's actually being done. Hence, the idea around context engineering. And both of you have been focused on that. Can you give our listeners a little bit of background in terms of where -- what's really necessary in context engineering? And how it's different and the kinds of things we have to think about when we start moving to agents.

Jean Atelsek: Well, I'll start with that. Basically, I think what we found with prompt engineering, as you say, is that doing it well, it's more of an interaction-based activity. It has to do with you interacting in a back and forth during one chat, say, with an LLM. Context engineering is a way of -- it's sort of setting up the LLM to succeed in accomplishing the tasks that the agent wants it to do. And that's by giving it access to the tools it needs to the data sources it needs, giving it access to memory so that it can take into account past responses -- and basically, it's a superset of prompt engineering that gives the model and the access to the information it needs to deliver the results. So, it's sort of taking the probabilistic nature of large language models and adding deterministic factors so that to kind of narrow the work that it has to do to deliver a response.

Alex Johnston: Great. I'd say as well, it's partly a reflection of the fact that I think organizations looked at these large pretrained models and someone writing a nice prompt there's always talk about the prompt engineer as a role and a growing reflection actually, there's a need to have a broader conversation, I guess, about the kind of aggregate information environment these models are operating within, which is, I guess, is when we start about context on individual prompts. And there's some sort of facets of that, I think, are worth talking about. So, one is you mentioned agents already. Obviously, we think about agents have to operate from a sort of shared context layer, and that's something that has to be architected for. Organizations often move beyond just human beings, prompting is a reflection on the need for metadata for certain workloads, for example, -- there's things like memory of past interactions, there' a number of systems that are being reached out to as part of these operations. And that complexity is far more than having someone get better at curating a couple of lines and putting that in a large language model.

Eric Hanselman: Well, I guess, on a certain level, if we think about chat-based interfaces, this is really putting all the pieces together right now that humans have been doing that, we've been maintaining context. We've been understanding what really was around the prompt and then taking those results and working with them. when we start shifting to agents starting to pick up some of these capabilities, we now, Alex, to your point, we've got to ensure that the agents can work between themselves, that information can be passed, that the rest of the framing of what that request is, what that prompt is, what the action is that we're expecting to be taken somewhere along here is actually built into it. And this actually has to be built into the architecture of the environment in which they operate. And that's a level of sophistication, I think, that we haven't really seen in a lot of the uses that enterprises are doing in terms of how they leverage these capabilities today. It seems like this is a significant step up in what has to be done.

Alex Johnston: And I think there's almost 2 -- I almost refer to almost as waves, I guess, or reasons why this is taking place. So, one, we've discovered quite early on, the first wave of generative AI applications was very focused on retrievable workloads. Organizations effectively realized that it wasn't enough just to have these general-purpose models. We need to sort of tailor them with enterprise data. And sometimes that was things like better affecting company voice, but more commonly, it was a reflection of the fact that what we needed had actionable outcomes based on information that we had to hand. But the second one, which I think is perhaps a bit more interesting is actually, if we talk about competitive edge, where does that sit? Well, the answer is most organizations have access to exactly the same models. So, what do you have control over? Well, it's context engineering. So where are you going to put your focus? Well, that makes a lot of sense when it comes to how do we have differentiated outcomes, how do we perform better? That's the aspect we actually have control over rather than these models of which most organizations are using the same pool of vendors.

Eric Hanselman: Everybody has got access to the same LLMs. The question is or the differentiation is in how you actually put them to work. And that's something where -- I mean, if you look at some of the approaches towards prompt engineering, you've now got pretty voluminous prompts that were being put together to really manage some of the more sophisticated interactions with models. We built significant volumes of data to be able to actually give that context to the models themselves, but that's been the human side of this equation. Now we need to start thinking about how we actually automate that, what are the pieces that we're putting together around that. That's something that, as you're saying, in order to actually get something that's differentiated, we now have to take this up to that next level. We have to think about context. We've got to think about what that -- what has been the manual part of this and how to really work on making that better.

Alex Johnston: I think as well, the timing around the size of context windows is quite important. So obviously, Google really pushed the boat out, had a massive impact on the size of the context windows that models have available to them. You saw Grocfast has got 2 million. I think was it LamMa4 Scout at 10 million token context windows now. And when you have context windows of that size, I guess there has to be more thought as to how we take advantage of that.

Jean Atelsek: The large context windows are pretty impressive, but there are practical problems with actually filling them. The responses become -- the attack surface is much larger in terms of security. And even the large language model can't retain all of that information equally. In other words, they found that when you fill a large context, the model is more likely to access the first information that went into the context window and the latest, but that there's kind of this U-shaped drop in its ability to use data that came in the middle of that giant window. So not only is it expensive to have that much context, but the results can suffer and there's a higher risk.

Eric Hanselman: So, it means that we've really got to start thinking about what we're building from a set of contexts. Also, again, if there are these utilization problems within the models themselves, we've got to account for those as well. It sounds like this is something where there's got to be significant thought built into not only the context we want to frame, but also how the models are working with the context that's provided to them. This is starting to get into fairly complex interactions with both the model environments, the data we're providing and the outcome that we're trying to achieve. What are things we start thinking about this that we have to put into place, I mean, to perform context engineering. What is it that's going on behind the scenes? I mean, I guess we've got to build some background in terms of the necessary information for the query. But we seem to be getting into the same kinds of things about -- that we did around database engineering, around other data-sensitive context in terms of how we were actually putting some of these environments together, except now we've got dramatically greater flexibility in terms of both what we can deliver to the model and the kinds of information that we can feed in.

Jean Atelsek: Right. And in some cases, that requires restricting the -- I mean, that's what context engineering is. It's basically restricting the information that the model has the access to, but you don't want to be as prescriptive as you would in a set of microservices or an application. Otherwise, you're not taking advantage of the ability of the model to plan and execute an agent to plan and execute tasks. So, it is a balance between sort of the "reasoning ability of the model and what you're making available to it and that you can change the underlying context dynamically so that you're not -- you don't have a brittle rigid system.

Eric Hanselman: So, it sounds like we're getting back into the kinds of things that we had started to do with retrieval augmentation, things that we started to do from an architectural perspective to help enhance some of the data we have. We've got -- the model has some understanding with RAG kinds of environments. We were getting to a point at which we were supplementing the information that was available to the model to both do some anchoring in truth to be able to throw in additional context, maybe to bring in information that the model hadn't necessarily been trained in it sounds like with context engineering, we're really expanding that and looking at that broader set of facts, information, collections of data that we want to bring together. But we've got to balance that with the realm of the possible. I mean, so often, we've sort of come from this idea that, yes, just send all of the data we have to the model and life will be better. Of course, that leads to potential confusion, problems, all sorts of issues with efficiency and cost. How do we strike that delicate balance?

Alex Johnston: There's a variety of quite interesting things going on. So obviously, it's been quite challenging historically, particularly to have structured unstructured data, both included within a context window, and that's been a major area of focus. And there's context compression. You might have heard various innovations taking place on that side. The multimodal data also adds a whole dimension of complexity. And you've also mentioned agents and agents are quite interesting because not only do have this shared context layer, but they also almost work as a way of curating context for other agents. And at that point, you get into quite a challenging environment as to what data needs to hand off between various models. But you've touched upon already, I guess, some of the challenges that we faced around things like how do we ensure we have accessed the timeliest data, for example? Is that something you solve with the graph database? A lot of people think, yes. Do you have to rely on something like streaming technology as an alternative to that? And it's a very challenging space with a lot of different architectures trying to achieve relatively similar things.

Eric Hanselman: Well, if you think -- you mentioned this issue of what we're trying to accomplish with agents is that automation of both the information gathering and some of these process pieces. And a part of that is that you've got agents doing work that will then be fed to other agents. And that seems like we've now got that additional context bounding problem of you have a first agent that goes off, does a particular query, collects information, formats it, manages it, builds it in ways that are going to be useful for the next agent that's going to work on that. But you also have -- when we've got these cascades, you've got the potential for cascading errors that happen along with it that if you didn't query well, if you didn't build a good data foundation, you've now got a challenge as you move forward. That sounds like the fundamental aspects that we've got to work with in terms of working this out. How do we start tackling something like that? Again, these -- we're getting back to the principles that are necessary to make this happen. How do we look at addressing it?

Alex Johnston: It is interesting as well. So, one of the things that I find quite amusing almost the first wave of generative AI investment, the excitement that organizations had was effectively, we've got these large pretrained models, don't need to worry about our data strategy that historically prevented us from doing predictive AI. We've got a huge models trained on such a large corpus of data. And now exactly, as you say, like the focus on context engineering has basically brought organizations back to thinking, "Oh, no, the issue is our data. How do we structure that in a way? How do we make sure that people have access to what agents have access to, obviously, a major concern, not just with agents, but actually the fact that organizations are effectively just plugged in entire repository of data to these models in many instances. So, we're in this interesting environment where organizations having felt like we don't have to worry too much about data strategy, look at this exciting new technology and now exactly back to where we started a few years ago of the sort of core enabler being their own data.

Eric Hanselman: And I think that one of the things we get into and this is that, as you're pointing out, originally, the thought was let's just train the models on all the data that we have. And then, of course, the models were trained on all the data with all the varying levels of quality that existed. It sucked up all the old with the new. We didn't have a lot of understanding about what was the most important version of a particular document. If you wind up pulling together all the old and the new versions, it was much more complex to make sense of where that happened. it seems like context piece is really starting to bring that focus right back around to ensuring that if you have this large training database that you've got the ability to select what's best and understand how that gets fed into the sequence of models that are -- or sequence of agents that are going to be working with that data.

Jean Atelsek: Yes. I'll say that the -- what you described like just dumping all of the data undifferentiated into an LLM can lead to context confusion, right? Because as you say, there might be different ages of documents, contradictory, knowledge contained in different documents. And this is something that it is adding to the amount of investment and engineering that's required to actually make agents work, especially if you're talking about agents giving data to other agents. One thing that is important here is that the agent's ability to learn has to be accounted for. In other words, there have to be checkpoints in the process for the human to be able to come in and give feedback on the performance of the agent.

Eric Hanselman: We quite literally do not want agents picking up bad habits. That's correct. And yes, you found this piece of information. That's great. That's not the direction that we want this particular process to head in. Whether or not that happens to be an incorrect interpretation, poor perspectives and dated data, that now starts to be just as big a concern.

Alex Johnston: It's interesting as -- if we start to sort of map some of these challenges on to what it means to the vendor landscape. I mean, a few months ago, there's a lot of talk about super agents. If you look at a lot of partners, normal partners in the space, a lot of them are building in a similar direction. There's a perception, you might have a small number of vendors effectively responsible for a large number of processes because all these guys are building into more process steps using agents. But where context engineering becomes really interesting is often, we found that these agents being built natively into these sort of software stacks or vendors building out their own agents, those agents are best suited to work with that system, right? They've got perhaps some specialist domain knowledge. They've been built by specialists internally who know how to use these systems correctly. And therefore, this context engineering story almost leaves us with lots of specialist agents for each of those systems rather than perhaps a super-agent, even if there is an orchestrator that sits above those agents. So, it's fascinating that these challenges that organizations face might have quite a major impact in terms of how the market itself evolves.

Eric Hanselman: Well, it does seem like a lot of the expectation is moving from that one agent to rule them all approach, especially when you start thinking about the challenges that are necessary in just terms of bounding agent activity. If you can -- if you have an agent that has one specific task, it's a lot simpler to be able to look at that, what that agent is doing, what it's accomplishing, whether or not it's successful, whether or not it's wandering, whereas if you have one agent that can do many things, it's much more complicated to bound those actions to understand the quality of its output and to correct, retrain, manage the process that it potentially heads into.

Alex Johnston: And a lot of that goes back to what Jean just said about that context confusion piece effectively. The technological limitations almost led to us binding them in that way.

Eric Hanselman: So, when we think about what's actually taking place with that agent interaction, which seems to be the point at which context engineering is really the most important.

Jean Atelsek: Right. And what I found in speaking to enterprises who are actually have agent platforms and those building agents is that these multi-agent systems tend to be linear as opposed to this orchestrated very intelligent layer of agents working in collaboration. And one thing that is important for these agents to be effective is also to be able to learn from the user behavior and take into account human feedback in terms of how it's performing and kind of course correction?

Eric Hanselman: I mean the intimation is things like I have an agent that does travel booking for me and the agent learns that I like sitting in window seats when I book airline tickets, things of that nature. But the intimation is these are things that are going to be growing and learning over time.

Alex Johnston: I would say when we think about how that learning is architected as well, I think something to keep in mind is a lot of these insights are being stored in external memory. But I think there's a perception that people have this continuous learning during inference, which actually isn't how a lot of these solutions are architected. So, in most instances, we're not talking about agents seeing updated model weights continuously due to the sort of inference they're engaged in. But I think that's a perception that a lot of people have. And it might be that we start to shift towards those sorts of interactions. There are a few interesting white papers that have come out in that in recent weeks. But outside of niches, that sort of continuous inference training story isn't actually reflective of what's going on even if it potentially is part of the narrative of how agents are presented.

Eric Hanselman: It's not going to learn that I like Windows seats. It's just going to keep asking me each time it's like, wow, Eric, you still want a window seat. fascinating.

Alex Johnston: It will know you like your window seat. So, we'll tell it that in its instructions. But the model itself isn't going to have updated in real time if that makes sense.

Eric Hanselman: Well, and to your point, so the context engineering part of this is that Eric likes Windows seats. And we have to ensure that, that gets built in as opposed to expecting the agent to realize that, hey, the last 5 flights, I was selecting Windows seats. And that means that probably for the sixth time, I might want to start looking at Windows seats. So, this really sort of feeds into right back to that question of how are we integrating these capabilities and applications and what can enterprises expect? And as a result, what should enterprises really be planning for. This -- as we make this move from the traditional prompt integrations around LLMs in this wonderful new generative world we live in, this really means that we need to think about our environments in ways that in the same sort of depth that we've always thought about application architectures, but now in some very new and very different ways in terms of what we actually have to integrate. It's not just information flow within an application as we've done historically. This is now, once again, raising the level of abstraction in terms of how we're considering how this is put together. But that's a skill that most enterprises are really going to have to work to develop, right?

Alex Johnston: Yes. I think if I had to come out with some concrete practices based on interviews we've had with enterprise in recent years or recent months even, something I find really interesting is often organizations only think of their data strategy when they're scaling technology up. And at that point, you're in serious issues, right? Because in a nice pilot environment, you've got a nice, curated data set. And when you move away from that, you're in a really serious situation often. And organizations are often thinking about it so late. They haven't really got the data architecture set up, and you and I were both involved in a bunch of those interviews. You have really interesting things about data overflow problems and some of the unexpected expense because of the way that data pipelines are set up and the move of data over multiple locations and all those sorts of interactions. So unfortunately, having to think about data strategy as early as figuring out what you're going to pilot was a common learning that came out of that. Another one I thought was really interesting was this idea when we have to prioritize various projects with generative AI, actually applying AI to data management isn't a terrible place to start. you always end up with quiet, I suppose, virtuous cycle there in which you're applying AI to improve your data, which allows you to apply AI better. And actually, that ended up being quite a sensible starting place for organizations that started there. AI to improve AI and especially the foundational element that's critical to all of this, the data that actually makes us all work.

Eric Hanselman: Jean, do you want to close that with any final thoughts on that?

Jean Atelsek: I'll just say that what this all comes down to, you're right, is the data, the curating of the data, and that requires an investment of time, engineering, and money. And you have to start thinking about where an agent can do better than the tools that we already have, the grounding, the RAG databases, the vector databases and so on. So, it's only it's not in every case that an agent is going to be able to perform a task more efficiently, more cost efficiently than a human. even if it's a repetitive task, you've got to sort of weigh the capabilities of these models against the investment that you're going to have to make in creating and maintaining the boundary that they have to work in.

Eric Hanselman: Agents aren't going to do it all, gee, medical. Well, I think this is the piece that we keep coming back to about so much of AI. in order to be really effective and efficient and reasonable and rational from an ROI perspective, you've really got to put a lot of thought into how you're building out these capabilities. They are not magic. You cannot vibe code your way out of an insufficient amount of thought around the architecture that's actually building all of this. So yes, still, I guess, some work to do on these fronts. Well, it has been great having you both on. Thank you for being back.

Jean Atelsek: Thank you.

Alex Johnston: Thanks so much.

Eric Hanselman: And that is it for this episode of Next in Tech. Thanks to our audience for staying with us. And thanks to our production team, including Sophie Carr, Feranmi Adeoshun, and Kyra Smith on the marketing and events team. I hope you'll join us for our next episode because there is always something Next in Tech!