Next in Tech | Ep. 273: FinOps and AI

View Full Transcript

Ep. 273 - FinOps AI

[00:00:02] Eric Hanselman: Welcome to Next in Tech, an S&P Global podcast where the world of emerging tech lives. I'm your host, Eric Hanselman, chief analyst for industry research at S&P Global, and today we're going to be talking about an aspect of AI that enterprises are just starting to wrestle with: cost management. If we think about what's happened, so many of the impacts of cost in AI are only just starting to be felt by organizations. But with me to discuss this are two of the analyst team who were just at the FinOps X conference, taking a look at a lot of what had been the financial operational management pieces that have come out of cloud and that are now starting to find a home in the AI world. with me are Jean Atelsek and Melanie Posey. Welcome back to the podcast, you both.

[00:00:51] Melanie Posey: Thanks a lot, Eric. Always great to be here with you

[00:00:54] Jean Atelsek: Yeah. Thank you

[00:00:57] Eric Hanselman: So, what's your take? I guess for a little, can you give us a little background on what the FinOpsX conference has been about? Really, what has it been wrestling with? 'Cause it's now, I don't know, how many years has it been going now? A, a few now. but being driven out of concerns about cloud cost, right?

[00:01:13] Melanie Posey: All right. The FinOps Foundation was started about six years ago when they had the first FinOps X conference to get a bunch of practitioners together to talk about issues related to cloud costs and techniques for cloud optimization and right sizing and all of those things. And over the years, it's become a combination of a practitioner conference and a vendor conference in the sense that a lot of companies have sprung up with different tools that basically help organizations wrestle all their cloud billing data to the ground, find ways to categorize it, run analytics on it to get a really deep, granular view into what's being spent on cloud overall, who is that spending with, what is that spending on, and who is responsible for driving that spending. And what's developed over the last couple of years is how do we deal with the whole issue of AI costs? And now that the whole topic of tokenomics is out there in the wild with everybody talking about it, like your grandmother knows what tokenomics is now. So, I think that's why the FinOps Foundation has decided to kind of get there first and be the one that shapes the conversation around tokenomics and, FinOps for AI essentially, so that they're renaming next year's conference Tokenomicon I think that's it. And, to take a deeper dive into this whole thing. And the interesting dilemma that we're seeing in this space right now is a lot of organizations haven't yet gotten a handle on their, in quotation marks, traditional cloud infrastructure costs yet. And over the years, the FinOps Foundation has added different, what they call scopes, different silos of cloud spending. So that brings SaaS into the picture. Like how do you account for your spending on Snowflake and Databricks and some of those other software platforms? And, how do you do on-premises infrastructure costs? How do you bring that into the mix? And now how do you bring AI into the mix? So, lots of related but somewhat different problems that involve different personas within the organization who are responsible for driving that spend, and different vendors who could jump in and help.

[00:04:29] Jean Atelsek: I'll just add that, part of the, that effort is that the FinOps Foundation, has launched a companion foundation called the, Tokenomics Foundation, and that's going to be working, with the IT Asset Management, ITAM Forum, which, is also part of that same group. And it brings in a lot of the AI native, open-source community with, open, secure software foundation, the PyTorch Foundation, the Cloud Native Computing Foundation. In other words, it's something that's really moved beyond just cost management. Because AI touches so many departments that they really need to figure out how to corral those costs, and it's not something that's going to be handled just by finance people and engineers

[00:05:23] Eric Hanselman: So, this is really, taking a step... one, operating in, in parallel on the AI side, but taking a big step from what had been some of those original goals from the financial ops, financial optimization kinds of capabilities, the FinOps, the, that core capability. and now really moving into entirely new spheres. I guess to some extent, much the same way they had done with the cloud because, the original challenges with FinOps were all that issue of you're moving from what had been the historical IT approach of spending on capital, to now shifting that, because you'd go out and you'd buy infrastructure and, you'd set it up, you'd run it, but you'd spent upfront and then you work hard to maximize utilization of it. But that upfront cost had been the financial model. That shifted to an operating expense model, and that, when we started to move to cloud in which you are now, renting and/or leasing the capabilities you're working with. And for many organizations, that shift from CapEx to OpEx was a real jolt. In some industries, of course, that are driven to optimize capital performance, in fact, that was really complicated. and y- we got into all sorts of nuances, Melanie, as you were saying, that figuring out what the differences are between SaaS apps, like the Snowflakes and the Databricks' and the Salesforces of the world, to managing your cloud capabilities of, how big an instance do you need, when you happen to be in Amazon or Azure or GCP or OCI, to make sure that the instances were sized appropriately so that you weren't getting this great big powerful machine to do tiny amounts of work. To now really shifting to a very different environment, which is trying to drift towards that idea of AI tokens as being this uniform measure of AI work that gets done. and at the same time, starting to do a lot of the things that we had to do with other cloud capabilities, which is manage policy, as you were saying, who gets access to this, how do you manage the allocation of the cost? It seems though this pivot to AI is a pretty significant one in a whole set of procedural ways that, that many organizations are still just now starting to really understand how that impacts and how they integrate it with their operations.

[00:07:54] Melanie Posey: That's definitely a good point, Eric, because one of the most obvious things that is different from cloud infrastructure versus AI usage is that unit of value, right? Like tokens are a unit of measure here, and it's a very imperfect unit of measure, but it's the best we've got so far. But it's a lot more variable than some of the things that can happen in a cloud infrastructure environment to throw cost completely out of whack. Because first of all, you've got a plethora of different models that folks can use for, queries or for agentic AI operations. So that causes huge amounts of variability in the cost, right? So how do you optimize the types of models that are used in AI and agentic operations? Is there a way to route queries or agentic workflows to the best, in quotation marks, model based on cost, based on performance, based on latency, based on throughput? So, what AI basically does is turn a simple mathematical equation into a quadratic equation in terms of the numbers of variables. Like you've got, X, Y, and Z are the variables, and they're not any constants in the equation. This is part of the difficulty in developing the tools that can help organizations get a handle on these AI costs. Because a lot of times when people talk about IT costs, the real answer is, "Well, it depends." And, you get this exponential it depends response in AI, along with, it's complicated as well

[00:10:12] Eric Hanselman: So, is this something that we should be using large language models to figure this out? And how much are we gonna pay for it?

[00:10:19] Melanie Posey: Exactly. That's exactly what we heard quite a lot about at the event. We started hearing it last year, and we heard it a whole lot this year, where folks are talking about FinOps for AI, so applying the principles of cost optimization to AI. And then there's AI for FinOps, where a lot of the FinOps providers are integrating AI in the way that they deliver value to their customers. And one thing that's going to be really important going forward in FinOps for AI is this ability to predict costs, right? Because if you had surprise bills from your cloud provider during the cloud era, the frequency and probably overall size of those surprise bills is exponentially bigger in the AI era.

[00:11:26] Eric Hanselman: And you raise actually a really important point there, which is, there was the whole cloud sticker shock, and we went through the various phases of cloud sticker shock. First, just the cost of individual instances in the cloud, then, egress fees for data. that was its whole, revelation when suddenly volumes of data got big enough to be expensive.This is something where we're stepping into an area where, we like to talk about the democratization of AI and, the employee coder and all of the work we're doing to extend AI capabilities to everyone in an organization. And guess what? When everyone in the organization has the ability to start kicking up bills of potentially significant size for AI. How do we manage that? Because yes, we want everybody to have access, but holy cow, do we want to pay for everybody to have access, or at least to have unlimited access? gets us into some complicated areas.

[00:12:22] Jean Atelsek: I was just going to say, AI was a big topic also at last year's conference. But what changed between last year and this year was the agentic workflows, which are not just launching an agent that does its job and then, retires back into its spot. There's reasoning, there are retry storms, there are loops and so on, and Companies were blowing their budgets, using agents, and as a result, the, model providers changed their pricing model. So, when it was a big land grab situation, you could buy a premium, subscription and have an extremely high cap on your usage, and that led to a great deal of abuse. So now, instead of an all-you-can-eat model, there is a subscription, seat license or in many cases, a seat license with a certain modest amount of usage, and then token-based, pricing on top of that. And that is something that, you know, these companies have got to get a handle on. And the FinOps personnel who, had managed to really, do a lot of automation and optimization in terms of getting the cheapest rates for their, for their resources and also using fewer resources, all of a sudden they were back at square one. So that's the problem with that.

[00:13:53] Eric Hanselman: It sounds like, the various AI providers out there are having an all-you-can-eat shrimp moment

[00:13:59] Jean Atelsek: Yeah.

[00:14:00] Eric Hanselman: Some interesting spin beyond this in that they, of course, need to ensure they protect themselves, but they also have an incentive to keep their customers happy and give their customers ways to limit those bills as well. Because of course, if suddenly, they get skyrocketing bills, customers are going to shut off use. and presumably it is in their best interests to help them manage, for the AI providers to help their customers manage that use. So, as you're identifying, per seat, token-based, starting to put limits in, we saw this kind of evolution in cloud services, although there, because the folks actually doing the consumption were a much smaller audience, you could do things like alarms and limits and things that were. That didn't have to go quite so far. Now that we've got everybody in our organization potentially using these capabilities, it seems like there has to be. There are just a lot more controls and the complexity of those controls have to be significantly greater to be able to accommodate all these different use patterns.

[00:15:02] Melanie Posey: That's exactly it, Eric. There weren't that many people within organizations who were spinning up cloud instances and then leaving them idle or getting a bigger instance than was actually required for the workload.

[00:15:22] Eric Hanselman: It was all those darn developers,

[00:15:24] Melanie Posey: you can always blame developers for things. And, continuing into the AI era, you've got, developers still being front and center because they are building agentic capabilities into external customer-facing websites, thinking of retail, the chatbots and, AI personal shoppers and that kind of thing. But at the same time, you've got the developers and the IT folks building internal AI capabilities that are used for back office and middle office types of workloads. So, there are a lot of places where costs can spin out of control. And I think where we are in the AI era right now, we're building the plane as we fly the plane. Whereas in the cloud era, FinOps came along quite a few years after the cloud came along. So, the thing itself was built. Now you're building a control and navigation system to keep everything under control, and we don't actually have that luxury of first this, then that in the AI era. It's all happening at the same time

[00:16:52] Eric Hanselman: a recipe for complexity. Certainly, maybe not one for success, although I don't know. What's your take on at least how people are trying to apply this? Because as you're pointing out, this is the ground shifting under everybody's feet around this. It’s not just the models are evolving. It’s the data pipelines. It’s how we're actually building consumption. Yeah, this whole environment's moving pretty quickly.

[00:17:16] Jean Atelsek: I'll say one interesting phrase that we heard quite a bit, at this year's FinOpsX was model routing. And, in most cases, the practitioners that were speaking and that we spoke to, they said really the easiest way to do that is by persona. So, they're going to give the developers that are working on the customer-facing agents the latest, greatest models. But if you're in marketing and you're writing copy for a catalog or something like that, you're automatically getting a cheaper, less capable model. So, that's one way that they're doing it. And there are other techniques for model routing, but that's the one that seems the easiest and most popular at this point .

[00:18:04] Eric Hanselman: I'm curious because we've been hearing about model routing from a bunch of different corners of IT. And it's everything from the big infrastructure providers. The Dells, the HPs, the Lenovos of the world are identifying that they've got capabilities to be able to do some of this. We've also got it on the software provider side. SUSE has got their AI proxy, Red Hat has got a routing capability with an OpenShift. Were you seeing any of that, and are they talking about third parties, or is this something where we ought to expect to be able to integrate model routing? Where does that fit and who are the players in that area?

[00:18:44] Jean Atelsek: I would say it is a policy discussion right now, the enterprises who are applying it, the persona-based ones are basically saying, "We are, cordoning off this particular model for these particular functions in the organization." And that's not a very sophisticated way of doing it, but that is as of now, seems like a popular FinOps way of doing it

[00:19:14] Eric Hanselman: So if you want to get to that trillion parameter model, you better have a really good application. But yet that still leaves open this question of, we've come through the waves of training consuming most of the work that's being done, but now we're getting to this next stage of actually getting the inferencing where we're delivering the work with the models. But it, it seems like we've still got some gaps of understanding that is the cost of what we built to do the inferencing actually worth what we're paying for it? And that, that starts to, that ROI on, wow, we're spending, umpteen zillion dollars on tokens for this customer service app, that in fact. It doesn't actually generate any revenue or sales. How do we calculate ROI? How much do we spend on this? And it's actually costing twice as much as our, customer support center actually used to cost. Was that a good thing? There’s some big questions out there.

[00:20:06] Melanie Posey: Yeah, that’s the missing piece to all of this is, how do you get to value? And how do you get to a point where your AI consumption metrics, it's not, just about overall number of tokens used. It’s something like cost per outcome. So that would actually be the fully loaded cost across the tokens, the infrastructure, the sort of back and forth agentic MCP server calls, getting all of that data in one place and then assigning it to a particular business outcome. That’s going to be the tricky piece of things. And once we get to that, people won't be talking about tokens as such. Tokens are just part of the AI workflow. They’re an input rather than some intrinsic unit of measure that relates to cost or value or something else.

[00:21:31] Eric Hanselman: Is this getting back to the thing we've always struggled with, which is understanding what that delivered value for any particular IT project is? Or are we getting to that quagmire?

[00:21:43] Melanie Posey: I, actually in a way it's something that it's nothing new, right? It's just more complicated with AI because you've heard for years, like back in the dark days of what we used to call strategic outsourcing, there were always questions about, if you spend billions of dollars with Deloitte or IBM, what does that get you at the end of the day? And there were lots of pricing models being bandied about that basically boiled down to some kind of revenue sharing or, some kind of sharing in the cost savings, that the cost savings, a percentage of that is basically what you pay for the solution in the first place. I think that's another part of this, that people have talked about this outcome based, I don't know, value metrics. But how you get to that's going to be a lot of work. And in some ways FinOps starts to look a lot like technology business management, which heretofore has been seen as a different thing, but we'll definitely see some convergence there.

[00:23:05] Eric Hanselman: That is a really interesting point because what that says is that much like the outsourcing or, services model, that what happened is we got to outcome-based capabilities or services that were outcome-based in terms of how they were measured. Maybe this is something where AI has taken us up the technology stack a little, and in fact, we have to treat this like a managed service. It seems like all we need now is for the AI providers to actually package it that way, but, details.

[00:23:41] Melanie Posey: Exactly. Yeah. Well, and another thing that kind of needs to happen as well is there's a certain amount of opaqueness in How many tokens you end up using for a particular query or for a particular workload. tokens are defined as being, as amounting to roughly three-quarters of a word. I think that's the convention that's out there right now. But, how do you decide that the reasoning that takes place inside of this or that model means, it takes this many tokens to spit out the response, where it might take a larger or smaller number of tokens to get more or less the same thing from another model? Do the models really show their work to an efficient extent that organizations can make informed decisions about is the juice worth the squeeze using this model versus that model? Those are really complex questions, and again, it's, calculus not algebra when it comes to trying to figure all this stuff out.

[00:25:10] Eric Hanselman: Jean, back to your point of that issue of model suitability, just simply constraining which models get used by which users is a fairly crude approach. And getting to something where we start to actually assess what is the request and route it to the right model because we know that this model works better with this type of request in this area or in this field, wow, that's, that, that does get really complicated really quickly.

[00:25:39] Jean Atelsek: Yeah, I'd say that the FinOps practitioners have developed a lot of muscle in terms of optimizing for OPEX, spend on compute resources. And now since the FinOps Foundation changed its messaging to say, they want their practitioners to manage technology value, not just cloud value, so they're expanding their remit, at the same time. And I think everyone realizes that some of the people who spoke at the conference were saying that as much as 80% of their FinOps effort is now going to AI spend management. So, they are putting in the effort to get a handle on this despite the opacity of what a token is even from model to model, and what the value is once you come out. One thing that they, that some of the vendors are doing, however, is they're using event correlation between token usage and observability to get a better idea. It’s almost back to square one. They’re working on allocating the cost to different teams, different projects and so on, and then, going from there. And that seems to be where a lot of the users and a lot of the vendors are at right now is putting that together.

[00:27:11] Eric Hanselman: And it seems like everybody can see the horse, dashing for the gate and are just desperately trying to get to the gate before the horse gets out. But, I guess we'll have to see if we can actually get there. These are great perspectives. Thank you both. I guess the answer for now is stay tuned, buckle up, and hopefully you can get your arms around this in terms of managing it.

[00:27:36] Melanie Posey: Yeah, I think that, lest anybody come away thinking that this is an unsolvable problem as far as FinOps for AI, it's not. Because FinOps itself for public cloud infrastructure spending has laid a great foundation as Jean was saying around basically doing the basics on who's using these resources, how much of these resources are being used, and what are they doing with the resources? Those fundamentals are already well-established on how to figure that out in your cloud FinOps world. What we'll start seeing with FinOps for AI, which I like better than the term tokenomics, is building upon that and accounting for some of the quirks and the different, let's say, runtimes of AI workloads as opposed to traditional workloads. But the fundamentals are there. I think we'll see more of an entire industry ecosystem effort around tackling the AI cost beast to the ground. So, look forward to seeing a lot of different types of companies being involved in this new kind of, I don't know, do we call the token, tokenomics foundation like a subreddit of, the FinOps Foundation? Maybe. You’ll have a lot of new faces and a lot of old faces involved in that effort as well.

[00:29:23] Eric Hanselman: so, it sounds like there are a lot of lariats heading for that horse, but we'll have to see. I think, Melanie, that's a good, more positive note to end on. That we have the tools, we have the basics. We understand what we're getting towards. More work to do, but, I guess this is clearly an area where we got a lot of people focusing on really how to make this, how to practically manage some of where this is going. But this has been great. Thank you both. And that is it for this episode of Next in Tech. thanks to our audience for sticking with us, and thanks to our production team, including Sophie Carr, Feranmi Adeoshun, Kyra Smith, and Dylan Scheible on the marketing and events teams. If you liked this episode, please subscribe or like us. I hope you'll join us for our next episode, because there is always something Next in Tech.