Listen: Next in Tech | Episode 88: Putting AI To Work at 451NEXUS

View Full Transcript

Transcript provided by Kensho.

Presentation

Eric Hanselman

Welcome to Next in Tech, an S&P Global Market Intelligence podcast with the world of emerging tech lives. I'm your host, Eric Hanselman, Chief Analyst for Technology, Media and Telecom and S&P Global Market Intelligence. And today, we're going to be talking about what it takes to scale artificial intelligence and machine learning. With returning guests Nick Patience, who's managing analysts for data, AI, Infosec and risk. Nick welcome back to the podcast.

Nick Patience

Thanks, Eric. It's good to be back.

Eric Hanselman

And it is great to have you because not only are we digging into a subject that we've touched lightly on, I think, before, but this is also something you're going to be talking about at the 451Nexus conference. You've got a session that's talking about scaling AI.

Nick Patience

Yes, that's right. That's right. So we're going to sort of examine some of the subjects, examine what it kind of means to be actually scaling AI and some of the tools and techniques that organizations are employing to do that.

 

Question and Answer

Eric Hanselman

Well, this is something that -- as I said, we've touched on in the past, but we think about what's really happened with the artificial intelligence and machine learning. Organizations are getting into this next phase. They've gone from that sort of dabbling in it, trying to figure out what's going on to now actually running business processes on top of it. And that means they're having to take what was those early-stage operations and bring them up into full production scale. And that's not as straightforward as one might seem. What are the kinds of things they're having to wrestle with?

Nick Patience

Yes, that's right. It isn't as straightforward as it might seem. I mean just to start with partly because AI and machine learning is a different way of writing and deploying software than we've gone before. So we've obviously gone for decades with essentially rules-based software. And so humans write the rules, the software follows the rules and executes functions within the applications. And then when you want to do something different, the humans come back and change those rules.

And obviously, the key difference between that and machine learning is that the models that are deployed learn from new data that comes through and they change and they adapt essentially by themselves. And that means they need to be monitored and managed to make sure that they're still actually making predictions that make sense in the context of whatever application it is.

So it presents a -- it's a fundamentally -- it's a bit of a shift. And so we see organizations wrestling initially with that concept. But then once you're deploying, then there's other issues that you have to think about. So I guess 1 of the things that will be interesting to talk about is what does it mean to be scaling AI. And I don't mean that in a kind of esoteric sense, but what are the actual the actual challenges.

So we found actually in our research, our latest Voice of the Enterprise: AI & Machine Learning, Infrastructure survey. Just one -- yes, a couple of little data points. I'll just throw at you. So the -- compared to the year ago survey, we found that the average number of models in production has grown by 73% in the past 12 months.

And so we've got an average figure at the moment of the people who respond to just under 3,500 models in production. Now I was talking to organizations 4, 5 years ago, and they would had 5, 10, 15 models. And so you can see those numbers are quite different from 3,500.

Eric Hanselman

Yes, just a little bit.

Nick Patience

A little bit. Yes. So what kind of -- I guess it's what is asking, what does that actually mean? And what problem does that throw up essentially? Yes, as I said earlier, the issue of that models change. And over time, based on data means you need to essentially check in on those models to make sure they're still working properly. So that's a kind of key challenge we find organizations facing.

Eric Hanselman

Well, managing what was that process. I guess when a lot of organizations sort of headed into sort of the AI and ML environments, there was that idea that you do the training, the inference upfront, and then you deploy the model and then it would give you results. But yet the whole point as you've identified, is that, that model needs to continue to learn and grow. But you've also got to ensure that it continues to be valid that it's continuing to produce valid results.

There's a lot of that sort of reproofing that's taking place, the validation that's happening. It seems like that creates -- when you start getting it up into the very large numbers of models that ramp, that winds up being a whole maintenance process into and of itself.

Nick Patience

Yes, that's right. It's a kind of new -- essentially a new era of software where you're looking -- if you go back to the something like the application performance management space of a decade ago or something like that, a whole sector of software cropped up to essentially manage something else. And I think we're seeing the same thing now.

And so here, it's -- the space is called ML Ops. And there's a related field called AI governance, but the ML Ops stuff is very much that managing the performance, monitoring the performance and effectively scheduling retraining of models as there are certain periods of time that the organizations need to do because you obviously -- you can't do it essentially by hand when you've got thousands of models or tens of thousands or hundreds of thousands.

And so we see that kind of a new sector of the software industry cropping up. And that's something we're going to -- we're definitely going to talk about at 451Nexus.

Eric Hanselman

Well, I mean, ML Ops really seems is a -- I was going to say a different kind of automation. But well, I don't know, maybe is it a different kind of automation? It seems like you're having to shift the way in which we think about automation from doing things like managing basic configuration, deployment kinds of issues to now being something that really is much more tightly bound into the actual process itself.

Nick Patience

Yes, that's right. It is essentially a set of processes. The data comes in 1 end and a prediction comes out at the other end. And you're obviously hoping that those predictions continue to make sense. This model is embedded in software, and they are a key part of what drives whatever the process is that, that software is trying to achieve. So yes, I totally agree with that.

Eric Hanselman

I guess I look at traditional software processes and you've got unit test and what you're really doing is you're proving out what are the rules that you've built into the software. It seems like with AI and ML environments, you've got something that's much more adaptive. Is this something where you're revalidating training sets? How does that get tackled?

Nick Patience

Yes, it is. It is. That is a key part of it. The revalidating of training sets. And I guess it depends on what you're trying to do in terms of whether this needs to be real time or near real time or batch, that's certainly a key element of it. And then obviously, if the data is -- if there's a bias or there's anything else that's crept into the data or crept into the model, and there always is some sort of bias going on, whether we like or not. Yes, that can cause organizations to do some retraining.

And when we found actually in the same server I just mentioned, Voice of the Enterprise: AI & machine Learning, Infrastructure survey, we found, I think, the largest cohort of organizations in terms of the frequency of their training was monthly or retraining was monthly. I think we found 18% said monthly, 17% said daily, 14%, so within minutes, 9% said within seconds. And then it's out there like only 2% said yearly. So it's kind of -- you can see which way things are going.

And then another aspect to think about is, as we mentioned, the number of models, models themselves, obviously, they're not massive databases. They're a very small bits of software. But there's other things that are driving all this opportunity was the data, and then the networks and the storage and everything that goes with it. And those things are -- do put burdens on organizations.

We found relating to the retraining, 56% of organizations said their infrastructure limitations are actually preventing them from retraining as often as they would like. And so you see a kind of a real issue there. And everybody needs real time, far from it, but there is a desire to certainly make sure the model is working off the latest data and making sense of that data and selling out predictions that actually -- what the original intention of the application was.

Eric Hanselman

So when we think about some of the infrastructure limitations, I mean, what are the kinds of things that people are running into? I mean I know we talked in the past about challenges with access to data and ensuring that you have the data that you need, where you need it and the data distribution becomes a significant limitation. Is that mostly a networking issue? Is it something where data distribution becomes more of the problem? Which aspects are things that people are struggling with?

Nick Patience

Yes, we find generally speaking, the data availability or data, the data is abundant access is the challenge, essentially. So we...

Eric Hanselman

Water, water everywhere and not a drop to drink.

Nick Patience

Exactly. Exactly. So we found in that survey, 90% of enterprises have just enough or more data than they need for the AI and ML projects. So that's kind of -- it's taken care of. They have the data. I mean, sometimes I'll dispute some of that because obviously, that doesn't get into the nuance of is this well labeled data? Has this got decent metadata around it? Or is...

Eric Hanselman

Is it actually usable? Can you put it to work?, Do you know enough about it? And we can get into the whole process of all of the data conditioning that's necessary, but yes.

Nick Patience

But then we also found in the same cohort of people we spoke to, 65% of organizations had difficulty accessing the data. So we knew it was out there. They just couldn't get their hands on it for the purposes of building and training and deploying their machine learning. And so that's a key problem and it may always be.

But we also asked them what would you do to your -- with your infrastructure resources to improve the performance? And I think as we said before, the #1 is high-performance networking. I think last year, it was #2. In the year before, it was #1. And this year is #1. So it's pretty clear that networking is a major issue.

And as we know, why is that? That's because you're moving data around. And that's what takes up a lot of the issues. And that was clearly the #1 pick for organizations we found in terms of how they would improve things.

And the second and third were more scalable, high-performance storage and then accelerators in the cloud to cloud-based accelerators specifically for training and for inference, but mainly for training. So there's definitely some things that people can do, but it does -- a lot of it does come back down to the data, how much you have to move it around and how fast you can do that.

Eric Hanselman

Well, that's 1 of those things that it's a limitation we keep running up against over and over again. People sort of think, "Hey, we've got all of this data, we've got connectivity but it takes time to move bits around. It costs money to move bits around." This is stuff where now that starts to be that limiting constraint.

It was interesting that accelerators were a limitation. I'm assuming that, that's getting into whatever the choose your particular accelerator availability within cloudy environments because the data is actually there. Is that sort of what that's hinting at?

Nick Patience

I think it also talks about -- without getting into a pricing discussion, I also talked about cost. It's certainly availability of certain circumstances, depending on your -- the scale of the workload you're trying to train. But not everybody can afford cloud-based accelerators. And so we also -- and we also asked them about on-premises accelerators. And that was about halfway down the list. So people are a lot.

These are kind of -- this is a desired list. Yes, what would you do? If you could, what would you do? What kind of resources would you pick to improve the performance? And so accelerators have been on top or second or third every time we've asked this kind of question.

Eric Hanselman

Okay. So this is something where, certainly, there's capacity, but the cost of instances that have got accelerators in the cloud winds up being a limiting factor.

Nick Patience

Yes, totally. I mean, when you look at something like in the area of natural language processing, these large language models, which have come around. So Google kind of came up with the idea a few years ago, and then you see lots of other organizations like Open AI and Microsoft and lots of other specialists doing these things, they take up massive amounts of compute resources.

And so they're 1 of these things that if organizations want to do that, they are going to need a lot of access to probably cloud-based accelerators and maybe on-prem as well, depending on what the kind of thing they're doing. So there's only so many organizations who can really afford to do those kinds of things. And that's probably an extreme example because that's not for everybody, not everybody needs those kinds of things yet. But for those who do, it can take up a huge amount of compute resources and storage and networking.

Eric Hanselman

So for things like natural language processing, is that something where most of that processing work is done upfront at the training stage? Or is this -- you're still going to need a lot of computational resources when you're actually doing NLP work with whatever the corpus is you're trying to recognize against?

Nick Patience

Yes, I think that is mainly upfront at the training stage. So we're seeing organizations -- some organizations on the research side and others training on massive corpus is from the web. So yes, you can imagine the amount of work that does both input it and to filter it and make sure getting what you need in the right language and all that kind of stuff. So yes, that is more of an upfront exercise.

And then we see on kind of the other way around, we do see this desire for more and more real-time inference, and that's the other way around. So that's where the strain as it were on infrastructure will happen at the inference point to the point where the predictions are made. And so we find a lot of organizations wanting to do it. But for instance, we find -- I think it was 67% of those in retail who we survey said data locality prevents them from doing that.

So you can imagine, obviously, if you've got 200 stores around the U.S. and they're spread right across the country. It was then unlikely to be doing real-time inference on the data that is actually stored in the store as it were. So you can see those limitations. So no, it's an interesting point about that is the work done upfront is work done at the other end. And it's -- the answer is, it really depends on the use case.

Eric Hanselman

Got it. Well, which brings me to ideas around the value of the model themselves because, let's say, you've got a big NLP model, this is something where there's been a significant investment in computational power to create the model, to train it to get it to this point. The model itself starts to become the really valuable piece of this entire puzzle.

And that then starts to get into some tricky issues about how do you manage, protect deal with that model? And how are you -- are you able to get it out into the wild on its own? Do you have to protect it? Do you have to wrap it? How do you have to deal with it? It's -- we've been looking at some of this stuff in confidential computing stuff. That's an aspect that I think is really fascinating. The idea that it's now -- we've come from a day in which protecting the source code was the key piece.

Now again, to your point, we've now got a very different model in with -- a different model about the model that the model themselves and the way in which it's been trained and all that inference information starts to become that really valuable element.

Nick Patience

Totally, it becomes a valuable piece of IP, essentially. And so -- and sometimes, that will depend on whether the model is based on a pretrained model, which you may have either licensed from a commercial vendor or got from an open source as it were or you come from scratch.

And we're actually doing some more survey work now, which will come out in the new year about asking organizations where they -- how much they depend on pretrained models versus their own custom work because you can think of if you just want -- let's -- I'm not going to say pictures of cats because that's too obvious.

But if you just want pictures of let say, laptops, there's plenty of models that have been trained around pictures of laptops. That doesn't really -- you don't need -- if you don't need any particular added value on that, then it doesn't make sense for you to start importing 100,000 images of various laptops and training your own model. You can get that from the cloud vendors or -- and from open source.

But if obviously, you're doing something peculiar and unusual or secret or something you really don't want other people to know that you're doing for commercial reasons, then they're going to do a lot of custom model building and say that can become incredibly, incredibly resource-intensive.

Eric Hanselman

So I'm just looking for ring-tailed lemurs, there may not be a model readily available, and you've got to train it up. And when you build it, for that particular community, hey, it's got a lot of value.

Nick Patience

Yes. That's a much more interesting example than my one of laptops, but it's exactly the same point, yes.

Eric Hanselman

I'm good at coming up with sort of random animals and here and there. It's a podcast post specialty, so. But the other thing that we've been talking about some of the security perspectives when we start getting into confidential computing is then once you've got that valuable model of being able to then ensure that you've got -- you can turn it loose because if it's open source, hey, it's easy to get to.

But if you've built it, now you've got to think about where it executes. And we've been talking with some clients about being their concerns about taking models out to edge environments. So doing something that's a very valuable detection with a particular model, but -- and they want to be able to get it out and license it and have people use it.

But their concern is that when it gets out more broadly distributed, it's now that much more available for pilfering, and people can copy it, people can make off with it. But now you've got that IP that is -- it's much easier for it to wander off into places where you're not able to monetize it anymore.

Nick Patience

That's a really interesting point, and it's certainly something we expect to see a lot more of. So we -- in our latest survey, 95% of organizations said they would prefer to do more training or more influence or both at the edge. So that's basically everybody. But currently, only 11% are using the edge as a primary venue for training models and only 12% using it as their primary inference. So it doesn't mean then that's the primary one. So not they're doing inference in various places.

And then why we also say, why is that? And physical security is a big part of it. Storage capacity is another part of it, compute performance is. But you're right, there's also this whole idea of AI governance and auditability and who's -- when was this model trained by whom, when was it last retrained? How many times were you retrained? All of those kinds of things.

And then as you say, literally, are these models safe or can they be altered in any way beyond your control? And that's as ever, security looms large in everything that we see, especially in whenever a new way of doing something comes along that has a lot of perceived value, such as AI in general, I'm talking about. A lot of people are going to be looking for let's just say, nefarious opportunities.

Eric Hanselman

Well, you talked about checking for bias, and I mean, we could probably do an entire episode on auditability. And the challenge is that when you get to the edge, the things that you're typically doing at the edge, are those things that are closer to the humans in the world. And there are a whole bunch of those aspects of ensuring that the models that are operating at the edge are operating well for operating as intended.

And that's also the place where you start to get into the kinetic world of having models controlling physical things. You've now got cars, robots that whole set of aspects that are interacting with the world in the kinetic sphere. And that means that you want to be that much more careful about ensuring that they're working well, they're doing what you intend, really operating much more specifically and yet, you've got a bunch of constraints that make it complicated to figure out how well they're performing.

Nick Patience

Yes. And I guess this -- I mean kind of that sums up what it doesn't sum up. It's -- not 1 other aspect of all these issues we've been talking about today about the challenges of getting AI to scale. So you could be -- it's one thing to be doing it in, let's say, entirely within a single cloud environment. But as most organizations, as we know, don't operate in a single cloud environment.

And then you have all the other aspects around the edge and the near edge and all these other things. These are all scaling issues. It's not just about who has the most models or who has the most data. It is a kind of -- you could have a lot of edge environment, as you say, if you work in oil and gas and things like that in some dangerous places around the world, you have lots and lots of models, making decisions close to critical infrastructure, and that's a scaling issue. It's just a different kind of 1 than if you operated within a large bank and you had a data center in the basement.

And so it's -- there's lots of these lots of new problems that have been thrown up. And one of the things we're going to look at on the 451Nexus now, and our panel discussion is obviously how organizations can deal with some of those problems and make sure they can get their AI to continue to scale.

Eric Hanselman

And we have indeed come full circle. Back to scaling and ML Ops right from where we started and, of course, 451Nexus. Hopefully, our listeners have heard 451Nexus is back in its virtual form kicking off October 31. There is a link in the show notes to point you to the conference itself. We hope you can be there live to hear it actually when it's running, but you can also come back in, it will be recorded, you come back and check out all the content.

A whole set of different sessions and conferences. If you go a couple of episodes back, Melanie Posey is talking about everything that we're going to be talking about in various forms, including scaling AI. So Nick, thank you very much. I look forward to hearing the deep dive that you'll be doing at 451Nexus.

Nick Patience

Great. Thank you, Eric. And just let everybody know, you have the live session will be on Tuesday, November 1, at 1:00 p.m. Eastern. So -- and I say, we can always listen to the recording afterwards.

Eric Hanselman

Well, and in fact, actually, there are going to be some overlapping sessions. So you'll have your pick of the things that are interesting to you, and you can come back and catch the ones that you happen to miss. So I hope that we will see you all there. Well, thanks, Nick. This has been great. Maybe I'll have to take a raincheck on auditability episode as we head into the new year.

Nick Patience

Yes. That sounds like a good subject to come back to.

Eric Hanselman

Well, thank you very much, and thank you to our audience. And that is it for this episode of Next in Tech. Thanks for staying with us. And thanks to our production team, including Carolyn Wright and Ethan Zimman on the marketing and events teams; and our studio lead, Kyle Cangialosi.

I hope you'll join us for our next episode where we're going to be digging into all set of topics about what's taking place on COP27 ESG issues, a range of different things that we'll be digging into. So I hope you'll join us then because there is always something Next in Tech.