Next in Tech | Ep. 252: Agentic AI Use Cases

View Full Transcript

Eric Hanselman: Welcome to Next in Tech, an S&P Global Podcast where the world of emerging tech lives. I'm your host, Eric Hanselman, Chief Analyst for Industry Research at S&P Global. And today, we're going to be talking about some recent data on AI use cases. And joining me to discuss it is Alex Johnston. Alex, welcome back to the podcast. 
 

Alex Johnston: Thanks very much, Eric. 
 

Eric Hanselman: I wanted today to dig into the study that you run on AI use cases and give our listeners some of the highlights about what that data is showing. So, what is the study digging into? And what are you seeing? 
 

Alex Johnston: Sure. So, I guess I'll start with the context. Once a year, we do a very nice survey. We speak to organizations about AI use cases and AI strategy. So, what we're effectively looking at here is a broad spread of different industries. So, I think in terms of the largest cohort is financial services, that's around 17%, 10% retail, 9% manufacturing. It's very broad spread. biases towards the U.S., but not entirely. So, it's about 43% U.S. respondents. But the most important takeaway is effectively what we're looking at is organizations and within those organizations, individuals that have a senior role in which they have an AI purview, so oversight or control over the AI strategy. And we effectively have those organizations lay out their plans for the year, what they've invested in already, the sort of success of those projects or otherwise in many instances. And we go through some of the components of their strategies around things like where they're sourcing AI from, what sort of roles they have in place to do so. It's a very broad, wide-ranging study. I'd say in general beats in terms of main findings, I'm sure we'll dig in some of the components of that. But last year, when we presented the data, probably would have been a similar call to this one, I would have presented a maturity curve and I would have said, look, AI is rapid advancement. Since the rise of generative AI, organizations have moved from experimenting with this technology to having in production very quickly. It's a big maturity story. But actually, that stored this year. But that isn't to say that the technology isn't permeating. Effectively, what we've seen instead is organizations have seen the technology take off across almost every area of the business. We've seen a huge surge in the use of AI internally, but that use is very unstructured, very unplanned, very fragmented. So rather than a mature AI strategy, which is informing decisions, it's a very sporadic, often very decentralized approach to AI, but absolutely huge levels of permeation, massive growth in terms of uptake and usage compared to the last 12 months. 
 

Eric Hanselman: Well, it's something that certainly we've seen anecdotally, which is that there is this huge rush towards AI. their enterprises have got massive efforts to be able to go promote AI use. There is tremendous enthusiasm and yet a bit of a fragmented approach in terms of who's using it, where it's used and the levels of sophistication with which it's being put into use. It's great to see that in the data, always something that when we start to see some of the things that we sort of feel out of the anecdotal perspectives actually turn up in the raw data itself. Interesting to see, though that shift in maturity because we saw those early days, there really was a much more pronounced march up that ladder of maturity. It seems like maybe we've now hit a point at which enterprises have gotten good enough at AI. And that, as you're pointing out, now there's -- we're into a phase of maybe less organized proliferation within the organization. 
 

Alex Johnston: Yes. I'd say as well, something that really stands out is organizations, when we do look at their investments, those investments are often sort of siloed. So individual departments will be using their own AI capabilities. And in some instances, there hasn't been any overarching control or any overarching oversight. We've seen this massive issue with fraud effectively. So, the percentage of organizations that see AI is something that's integrated and embedded across the organization has remained static year-on-year. So, we're effectively in a scenario in which organizations are telling us AI -- in fact, I've got the data at hand here in terms of the case studies we looked at. Of the 38 use cases, 19 already had AI on average in terms of adoption, 14 would be planned for the next 12 months, only 5 of the areas we looked at had no plans to implement on average. So, technology is hugely permeated. It's hard to find case studies or examples now of areas of the business that haven't been impacted by AI in any meaningful way. But we're in a scenario in which that doesn't necessarily equate to maturity. So, organizations have all these capabilities, but they've got no ability to control or manage them and often very little oversight as to what's actually being used. 
 

Eric Hanselman: The study also looked at progress in terms of how organizations are moving forward. What were you seeing in terms of AI project progression and success? 
 

Alex Johnston: So obviously, there's a lot of data which looks at the percentage of projects that are seen as being delivered in production. And we can talk about this later. There's a big divide between the sort of optimist and the pessimist and people see these data sets and they see them as inherently contradictory. But we see that whilst only quite a small percentage of projects are being delivered in production in terms of projects that initiate within the last 12 months, -- it's around 37% being successfully deployed and was seen as currently operational, delivering business value. Only a very small percentage of projects are seen as failing. And what you see instead is that most projects are caught in some sort of limited deployment or caught while still scaling up to a certain degree or still in development effectively. So, you've got this quite clogged project pipeline, but that wasn't quite the same as project failure. So actually, a lot of organizations saw some benefit even from these projects only being partially deployed. And when you ask organizations, have those projects met your expectations across almost all the objectives you looked at, the answer was overwhelmingly yes. So, organizations may not necessarily be able to point to a huge surplus of fully laid out, fully deployed AI projects, which wasn't quite the same as saying these projects weren't delivering value because in most instances, they were. 
 

Eric Hanselman: Well, you can talk about the optimistic or versus the pessimistic view I'll get on my soapbox a bit. And one of the challenges I see in the market right now is the off quoted statistic of an MIT study that talks about 95% of AI projects fail, which is a wonderful headline if you're on the pessimistic side. But it's something that, for my take, misinterprets what the MIT study really is representing and is also something that we see borne out in our data, which is that, as you're pointing out, not all projects have made it all the way out to full production and deployment and yet organizations are seeing value in them and that in the MIT study, what it's key finding is that 5% of projects are getting to full success and deployment. And yet that doesn't mean that 95%, the remainder of that quotient, in fact, are failing. It just means exactly what the data that the study shows is that these are projects that are still in progress, that value is still being obtained. Are they actually at that end point where they need to be in terms of where they're headed? No, they haven't gotten there yet. That 37% number that you're identifying is that group that is working its way through. But it really does to catalyze some of this conversation about whether or not you're optimistic or pessimistic about AI futures, and it seems like there are ways to interpret the data. In both of those directions, I think I lead towards maybe the more nuanced interpretation, which is it's a continuum. There are various aspects to this. There are some that are succeeding and many that are in the middle. And now that I have droned on and on about this, what's your take? And what's your thought about what those numbers really mean and what that represents? 
 

Alex Johnston: So, I guess an additional aligned, but I guess, kind of auxiliary point would be to look at ROI. And similarly, I think as an isolated data point, what we might have developed here could look quite troubling. Only 46% of initiatives were seen as being on track to achieve ROI within 12 months of launch. As an isolated data point, you could point to that as saying, look, AI near-term value, it isn't where organizations thought it would be. We present as transformative technology, but actually near term, we're not seeing that. But actually, when you compare it and combine it with the other data points we've got, it's far more optimistic. So, organizations are largely satisfied about how the technology is delivering against efficiency objectives, around cost objectives, around revenue objectives, around process efficiency. And they might not see that as necessarily a 12-month return on investment, but they certainly see it in the near-term future, maybe 18 months, maybe 2 years. I think the concrete measurement of ROI, again, is quite a partial measurement within our data set. We find that organizations on the whole don't measure ROI for their entire AI initiatives. It tends to be isolated projects that perhaps as an ROI number attached to. But actually, when you dig into individual case studies, companies often have some very positive stories about how technology has had impact. There's often learnings from the ones that haven't. So, I do think the story is perhaps more optimistic than what that 46% figure would present on the surface. 
 

Eric Hanselman: Well, it's something I know that in previous studies, we've looked at key performance indicators and how they vary by industry and had seen that, in fact, there was a lot of variation. And some of that, I'd chalk up to a move up the maturity curve. One of the big challenges is really understanding how you're measuring returns and what you're trying to put together in this most recent data, those particular use cases, some of how those are being managed and even some of the decision-making and influence on how those projects are managing, it really is evolving. This is very much still a work in progress, it seems. 
 

Alex Johnston: I will say if you want to pull out a cohort of top performers and you could base this on who's seeing the best ROI or projects being successfully deployed, if you look at what those companies have in common within those cohorts, it's measurement and governance. So those organizations are far more likely to invest in MLOps. They're far more likely to invest in AI governance tools. They have more mature measurement approaches. And it feels like there is an advantage to organizations, as you say, are more mature in terms of the percentage of projects that, a, they're sort of able to measure, but b, they're able to then extract those insights in terms of informing other projects. And there is a big, I'd say, glaring gap between maybe 20% to 30% of our data set and the broader 70% when it comes to overall outcomes. 
 

Eric Hanselman: And which side of that is -- is it the 20% and 30% who are doing it right, who have the governance in place? 
 

Alex Johnston: Yes, I'd say they were the top performers. So that smaller cohort would be the organization performing better. I will say our data set is biased towards more mature organizations, but there was still some quite interesting patterns as well as the industry wasn't a great predictor of whether organizations sit in these various buckets. It did seem that financial services surprisingly, in some instances, further forward because a lot of those governance capabilities are something they've had to rely more upon because the sort of the importance of the areas they operate within some of the various data security concerns in that space. But on the whole, region wasn't a great predictor either. It did look like measurement and discipline maturity was the most important capability when it came to what dictated whether organizations performing well or performing comparatively poorly. 
 

Eric Hanselman: I wonder whether or not this ties into Melissa and Sarah's comment as we were talking about Agent on an earlier episode that those who are doing better were the ones who showed a healthy amount of skepticism about exactly what they were trying to achieve. And to your point, those that have put governance and measurement capabilities in place to really understand where they are, what they're achieving and what kind of progress they're actually making from an objective perspective. 
 

Alex Johnston: Yes. There was also -- I found it interesting when we look at organizations prioritize less mature organizations, those performing less well tended to look at quite a small set of components within the AI strategy. So very focused on cost. They're very focused on business impact. We didn't necessarily have a holistic vision of data availability, for example. There's a wide array of things that they were underprioritizing relative to those more mature organizations, and that really came through in the data set as well. 
 

Eric Hanselman: You bring up what is typically one of the most significant stumbling blocks, which is sophistication about data management and understanding that the AI bits are certainly important, getting models right, trying to figure out what that supporting infrastructure is, identifying data sources, building data pipelines and being able to get to a point at which you actually understand data quality is so critical. It seems like that's what that's pointing to. 
 

Alex Johnston: Yes. So that was certainly one thing that did come out. So, you look at the most serious issues that organizations face, data-related challenges were very well represented within that data set. So, if you look at technical challenges, data quality was not just the most frequently identified of those technical challenges, it was actually more severe when it was identified as well. Similarly, data privacy, a major concern, frequently cited -- so we do see a lot of those data concerns emerging within the data set. I'd say another thing that was unexpected perhaps in terms of top performers versus the general population relate to levels of autonomy. So, you might expect that organizations that are more mature or more advanced will be operating with higher levels of autonomy. And that wasn't the case. It appeared that organizations that are seeing more value had focused on human-the-loop applications as sort of the bulk of their projects. And perhaps that attempt to sort of leapfrog being like we'll give a high level of autonomy systems; we sort out a lot of our own data estate. That didn't seem to work out. I felt that organizations by having this more gradual human-e-loop approach could actually progress more quickly than those that are trying to sort of leapfrog straight into fully autonomous capabilities. 
 

Eric Hanselman: It reinforces the point that a little healthy skepticism, the AI pixie dust is not going to solve your problems and jumping directly to more significant levels of autonomy you're not going to fix any of the challenges you face behind this. As with so many things, but specifically reinforced with efforts around AI, these are things that you've got to master the fundamentals and work through what are fundamentally business process challenges in terms of how you understand this, how you actually adopt it and how you're moving forward as an organization. I'm curious in terms of the kinds of things that organizations are trying to tackle, what were you seeing across use cases? And what were those distributions? And where are people seeing success? 
 

Alex Johnston: Yes. So, in terms of the use cases, we are seeing most focus and unsurprisingly related to content and communication. So, there have been a lot of focus on summarization, on language translation, on meeting productivity, for example, these very general productivity terms. I mentioned upfront how far the technology has permeated in the sense of almost 50% of the use case we looked at on average were having AI applied to them, which is actually an outrageous figure. We're going to have to go away and try to find some more niche application areas next year to get a more holistic response. But what I found really interesting about looking through that data is not only how far the technology permeated, but actually where it had most rapidly progressed, as mentioned before, wasn't necessarily the most automated areas. So, we did see some interest in things like automated reporting, for example, we looked at some of the application and finance departments, they were often quite niche, but often quite automated. But actually, a lot of those more productivity-focused use cases were very human loop. And that's where a lot of organizations have initially started with their investments. 
 

Eric Hanselman: Well, are these migrations out of what maybe historically might have been robotic process automation and maybe some small in the process pieces there, add a little more sophistication, a little more nuance in terms of what's being done. 
 

Alex Johnston: Yes. I will say the concept of moving away from RPA is quite an interesting one, though, because we find that when we look at, I guess, the 4 modes of technology maybe or 4 patents, you can talk about generative, you can talk about rules-based, you talk about patent recognition. And I find it interesting that those technologies are progressing at a similar pace or perception increasing at a similar pace. And I think the membranes between those algorithms or algorithm types, I suppose, is sort of broken down to a certain degree. You have an agentic interface, it might be looking at rules-based models, patent recognition models and generative AI models. And maybe that's why organizations see these things are progressing at a certain rate. But I would also say it does look like there's still a role for simple rule-based automations. And in some cases, organizations have been trying to apply some very large models, but there isn't a need for very large models. And you can just go back to 5 macros for certain use cases. It feels a bit like people have been trying too almost overfit. Actually, I shouldn't use the word overfit. So obviously, there's a technical term associated with it. But people have been trying to apply generative AI to areas where there isn't necessarily a need for generative AI. And in some instances, you actually better with the older rules-based stuff or Pn recognition models still have their place effectively. And that's something that I think is very prevalent and obvious when you look at our data. 
 

Eric Hanselman: Really that this is the new thing and let's go put it to work. And why not use a 1 billion parameter model to go make some very elemental decisions. 
 

Alex Johnston: That's part of the reason why we see all these -- again, these data problems rising up in the mix as well. And I think over the last 3 years, probably, Eric, whenever we spoke about this topic, cost has become from something that people wouldn't care about at all with the initial eruption around ChatGPT, cost concerns have all increased again. And one of the reasons those cost concerns have progressed over the last 3 years quite markedly is because organizations are applying these very, very large general-purpose models to a wide array of use cases. Because often the user is in control of the models they want to use, they have a bias towards the most capable models. And that is a thing that's exposed to them in many instances. They have those controls. So, I do wonder if we might start to see more considerations about making sure that the right models are in the right places, maybe as organizations get that membrane of management and visibility, they can stretch over their investments. 
 

Eric Hanselman: Well, as we discussed in a previous episode, I wonder whether or not we're just on the leading edge of this shift in terms of concerns about cost in that if we take the cloud model, for example, initially, when people were provisioning things in cloud, they used very large cloud instances. And down the road, there was a realization that, holy cow, you had all these great massive cloud instances, taking away not actually terribly high levels of utilization. It seems like right now, we don't have some of those same sort of gauges on whether or not the models that are being put to work are being utilized to their fullest, but that there is a rightsizing that can take place here and maybe heading down this FinOps style model in terms of starting to look at how this happens. I guess, are there pieces that you're starting to see and expectations about some of the operational aspects that are taking place in terms of how people are actually running their environments? 
 

Alex Johnston: Yes, there's a few facets I think we should dive into. So, one, I guess, why is the problem so pronounced? We've already spoken already about just the sheer number of use cases that AI was in use and how much -- how markedly that increased from previous years, where organizations are only looking at 5 or 6 AI application areas to almost every area in the business now has some level of AI investment, but also where the capabilities are being sourced from. So, I found it interesting. I assume that organizations would either have an approach where maybe they lent on third-party vendors, or they were doing their stuff homegrown. And actually, what we found is organizations are doing quite a confusing combination of the 2. So, they do see a lot of internal development. They do see this embedded in the hardware. They do see it as something that they purchased part of a new application. They do see it as an add-on to existing applications. And actually, when you dive into where organizations are feeling the most pain often, it is the fact their existing software partners are integrating these capabilities, sometimes without even informing them beforehand that these are going to appear. -- but even custom solutions from consultancies. And that sourcing landscape is so broad that organizations really, how do you get control over that? I mean that's an absolute nightmarish scenario in terms of the numbers of models that might be in use by your organization at any given moment. And that's the officially sourced stuff. doesn't even take into account the fact there's a huge amount of shadow IT, people with their own ChatGPT or Gemini or whatever models they want to use who are also plugging these things in. So, you're in a scenario in which that landscape becomes so complicated that it's no wonder that it's only a very small minority of organizations that see themselves having any control or oversight at all over their AI strategy. 
 

Eric Hanselman: Well, it's something, of course, that we see on the security side, which is just the proliferation of all of these capabilities. As you're pointing out, the fact that you're only one new release away from your software vendors identifying now with AI, in ways that you may not have control over presents some pretty significant challenges. 
 

Alex Johnston: One of the other patterns within the data is a clear understanding within organizations that user proficiency is very variable. So, technology in some instances, being used very heavily, very effectively. In other instances, entire team perhaps, we're not seeing the technology in use, or it's being used very poorly, or it's been used unsafely, which is a common concern. And that very variable level of performance is further complicated by that lack of control because we want to make sure that people are using these effectively, but they're using such a spread of tools that actually understanding how to use those effectively is challenging. things like can we put prompt templates together is much easier when you've got an officially sanctioned single tool that everyone is running through. It's a lot harder if there's 56 interfaces in use within a single team, for example. 
 

Eric Hanselman: Well, what does this point out for enterprises as they're moving forward in this journey? I mean, as you're pointing out, clearly, being able to put together scaffolding to help users’ education certainly is paramount. What other aspects of the data identified as things that organizations should be looking towards? 
 

Alex Johnston: I'll push out a controversial one to start a debate maybe, Eric. I mean it's a personal issue. I think there's been a lot of concern initially about sourcing ideas. So, a very decentralized approach will empower staff to use the tools they want and apply to the use cases they want these workshops will encourage people to use these tools and we encourage them to use their initiative in terms of using generative AI. And that very decentralized distribute approach initially seems like a great idea, but there are some pain points. So, one, we still see a lot of pressure on technical staff. So, when organizations are thinking about, we can have this low-code, no-code, agent builders, actually, in many instances, it ends up being technical staff that have these things handed off to them in a position where they're already overwhelmed. They're not going to be able to integrate this within -- more broadly with the people's workloads. We've got that project pipeline we mentioned being very clogged. So, we've spoken already, only a small percentage of projects are actually reaching production. A lot of them are in the sort of pilot scale-up phase, however you want to phrase it, limited deployment. And that problem is being worsened by that decentralized approach. And again, we come to that governance and measurement story where organizations don't actually know how impactful a lot of these technologies are on the whole how safely these technologies are being used. And they don't know who's using what and when they're using it. And that lack of visibility created by that very decentralized approach, I think that's also a problem. And cost, again, because these capabilities are emerging, they'll build their own agents, often if there are consumption costs associated with that, that's going to balloon. So, in a circumstance in which organizations by trying to empower staff have made things very difficult for AI teams and people in charge of AI strategy, which I realize is an unpopular view to have, but that is sort of my concern. 
 

Eric Hanselman: Well, I know that in examples that I've seen and particularly in terms of some of the training, I wonder how many help desk tickets have been raised because my AI app that analyzes my e-mail inbox defines my superpower and then writes a poem about it is not working properly. 
 

Alex Johnston: Yes. And there is -- with those sort of training programs as well, you probably noticed a very general purpose to account for the sheer variety of tools in use. So, you end up basically having a page saying, please use this safely with very little further information. You get this very conceptual story about what AI is and how it's used without necessarily how should we be using the individual tools available in our teams. And that's partly a reflection of how fast stuff is evolving. It's very difficult to do that. But again, the spread of investments has made it very difficult for organizations to train up staff on each one of those individual investments. 
 

Eric Hanselman: Makes a lot of sense, especially being able to not only build in a deeper understanding about what this guidance really means but also building the resources that this isn't just prompt engineering. This is ensuring that there are data sources that are vetted and whose use can be defined and constrained and really putting some boundaries around a lot of what's taking place. Yes, democratizing AI is useful in the long term, but there are some pieces to this that we need to provide sufficient guidance and focus to help this work out well. Well, this has been great, Alex. As always, there's so much more we can dig into. I think we're going to have to follow up with another episode with more of the data, especially we're talking about some of the case studies that study looked into some of these aspects. But we were at time for this episode. But thanks for being back on. 
 

Alex Johnston: Great. Thanks, Eric. Have a great day. 
 

Eric Hanselman: That is, it for this episode of Next in Tech! Thanks to our audience for staying with us. And thanks to our production team, including Sophie Carr, Feranmi Adeoshun, and Kyra Smith. I hope you'll join us for our next episode where we're going to be looking at the National Retail Federation Conference and not only AI, but some also retail, some finance some other aspects of what was actually taking place there. I hope you'll join us then because there is always something Next in Tech.