Listen: Next in Tech | Episode 133: Digital infrastructure roundup

View Full Transcript

Presentation

Eric Hanselman

Welcome to Next in Tech, an S&P Global Market Intelligence podcast where the world of emerging tech lives. I'm your host Eric Hanselman, Chief Analyst for Technology, Media and Telecom at S&P Global Market Intelligence. And today, we're going to be discussing a lot of the things we've been hearing about in infrastructure in its various forms with sort of a roll up of a couple of different major vendor events that have happened in the recent past and really some perspectives about a lot of what's been going on recently.

And joining me to discuss these lofty topics are Jean Atelsek, Melanie Posey and Henry Baltazar. Welcome back to you all.

Melanie Posey

Hey, Eric.

Jean Atelsek

Great to be here.

Question and Answer

Eric Hanselman

Great to have you here in what has been a pretty tumultuous set of vendor announcements, all sorts of other stuff that's been going on, and of course, the theme that seems to be running through everything these days is generative AI.

And I'm referring back to VMware had their conference 2 weeks ago, Google had their conference last week. And of course, generative AI was a big push in all of what they were looking forward. And I wanted to get your thoughts. Are we ahead of the game here? Have we jumped the shark on this? What's our thought on this?

Melanie Posey

Yes. I think we're definitely ahead of the game as far as generative AI goes. But one thing people have to remember is that AI existed before generative AI did. So think of generative AI as being transformational. You transform voice into text, voice into images, text into images. That's what gen AI is.

But for quite a long time, I would say it's more appropriate to say machine learning has been out there in the wild for quite a while now as a way to process a lot of data really quickly. Then I would say gen AI is one of the things you do with that data you process fairly quickly through the medium of large language models. So that's a use case.

But in a larger sense, the impact that it has on the IT industry is pretty significant because a lot of the infrastructure out there, whether it's on-premises, whether it's in the cloud, whether it's some kind of hybrid mix of the 2, it's not really up to the task in terms of the compute processing capacity, also the networking you need to move data from place to place and also the performance of the storage where all of this data is going to live.

So I think we see ourselves on the cusp of a pretty large-scale IT modernization wave that's kind of been catalyzed by all the interest in gen AI.

Eric Hanselman

And you raised something we got into a few episodes ago of should we be talking about legacy AI in terms of what we've been doing with AI? But you also raised some really good points about, is the infrastructure we've got ready to do this? And are we heading in the directions that are going to take a lot of upgrades or you can imagine -- and certainly, from a vendor perspective, this is something that I think a lot of people are thinking is going to catalyze reinvestment.

So maybe we're headed in interesting directions or so it goes. I mean, Henry, you were talking about data. One of the things that I think was coming up in both of these conferences was a lot of that conversation about access to data. Melanie was mentioning performance and a lot of the concerns around that. I don't know, what's your take? Do we have enough data in the right places to make all this stuff really fly?

Henry Baltazar

Yes. I think the issue with storage has always been about data growth and how to deal with data growth and where to put it. And that problem is only getting worse. If we look at our recent study findings, the amount of data people have under management a year ago, that was growing at 24%. Now it's close to 30%. So we seem to be creating data at a larger rate.

The other elements that we have to worry about is we also have to make sure we have other copies of the data, right? Copies of data are not necessarily bad. I mean to me, it's the price of doing business. Sometimes you want to have copies just for data protection's sake. Sometimes you want to have copies so somebody in a different geo could work with that data or for a different use case like test dev, whatever else.

And a lot of the organizations I talk to are struggling with that. Like how do we deal with these copies? Where should those copies go? And more and more, I'm sure we'll get into this discussion later, but more and more, a lot of organizations are starting to leverage cloud for that as well in terms of providing another site to put that data from a data protection standpoint and also from being able to leverage that data, that critical data with other innovations in the cloud, whether it be serverless or other algorithms and things like that.

Eric Hanselman

Well, it gets into those questions then of getting it in the right places and the capacity and cloud. And Jean, you certainly saw a lot of the cloud perspectives on what people are looking at this as well.

Jean Atelsek

Yes, exactly. I think in terms of training large language models, very few companies have the compute capacity to do that. And what we're seeing among the big cloud providers are these sort of tie-ups with AI startups so that they're basically investing in the OpenAIs and the Coheres and so on. And then basically, those companies are using that investment to buy compute capacity back from the hyperscalers.

So they're really -- the hyperscalers are really being the kingmakers at this point in terms of the large language models. And what I have been struck by is how much what they call dogfooding is going on in terms of providers themselves, the AI start-ups themselves, the software startups, global system integrators.

They're all still applying the technology internally looking for use cases with credible return on investment. So yes, that's the interesting thing that -- it definitely feels like we're ahead of the curve, but the use cases still need to be developed and proved out financially.

Eric Hanselman

Well, it's interesting. The internal uses and whether or not it's dogfooding, champagne drinking, however palatable you want to make that. Yes, it was interesting to see actually all of the announcements.

You've got Microsoft with Copilot. You've got Google with Duet. You've got all of the capabilities that VMware is doing on the front-end pieces of the adviser kinds of aspects of this. Everybody is starting to leverage it in some form or fashion. And in some cases, it looks as though it's starting to get beyond just the useful query front end to actually now making some more intelligent analysis of the data and the query that's actually being put to them.

Melanie Posey

Yes, Eric, I think that's one thing to think about here is, I guess, with any new technology, you have this paradigm of does this new technology help me do all the old things I've always done in my business better, cheaper, faster, more efficiently, whatever Or does it enable me to do when new to the world kind of things that actually transform my business rather than just optimize it.

And I think for legacy AI and to some extent, for gen AI as well, we're still talking about doing old things in new ways that with gen AI, what we talk a lot about is chatbots and news, obviously, have applications in any kind of customer-facing, customer experience sort of situation.

So it's kind of like cloud and all the other technologies that have come before it. It does actually have a demonstrable ROI. But as long as you shut down all the old ways that people did things and move them over to the new ways and then maybe your ROI is 2, 3 years out once you've shut down all legacy. But I think in gen AI, when it gets down to the business processes it's being applied to, you're going to have that same old technical debt problem that you've always had.

Eric Hanselman

Well, I'll raise an interesting counterpoint to this, which was looking at the software applications for basically gen AI writing code, and there were some reasonably useful examples that we've seen recently. And certainly, some of what we saw at Google and some of those pieces we're interesting to me. Jean, you've seen a lot of what they were trying to accomplish there. There were some interesting examples they've put up.

Jean Atelsek

Right. I think this is one of the most interesting applications we're going to see, which is I've been looking at app modernization for a while. And one of the really intractable problems is the challenge of modernizing code bases from legacy languages that were tied to these monolithic architectural models into much more nimble cloud native languages and configurations from monolith to micro services and so on.

And there are a couple of cases, IBM has released in preview a service that takes COBOL code and converts it into Java specifically for their z platform. And at Google Cloud Next, we saw an example of them using generative AI to take a C++ app and translate it into Go.

But in both cases, those models were trained on the platforms of the vendors. And it's basically sort of taking away of modernizing code, yes, for cloud deployment, but not in an especially open way, like it was basically having a fast on-ramp onto platform X, Y or Z.

Eric Hanselman

So continuing the march to cloud and specifically targeting, hey, you want to do ingest into our platform, we got tools for you. Generalized things, maybe not so much, which actually maybe brings up, I think, one of the things that I found another aspect of what we were hearing was this contrast between where generative AI takes place.

What's appropriate? What's suitable? And I'll contrast VMware's announcement of private AI infrastructure, their focus on keeping data resources on-prem, keeping the computational process on-prem.

And of course, I think the hyperscalers' approach, which is, sure, bring it all into cloud, and the different bits and pieces of that the fundamental aspect of that is a lot of this is going to have to be about -- or it's going to be driven by that data gravity piece, which is wherever your data is and wherever -- how comfortable you are about moving it there.

And again, we get back to the question about storage and where all this fits and how that's going to work because we've seen this also from other vendors, HPE and Dell in terms of we're looking to leverage their storage assets. What's your take on that angle of things, Henry?

Henry Baltazar

Yes. I mean I don't want to be the compliance police, but I mean that's going to wind up being an issue, right, I mean, as a certain semiconductor company found out, right?

Eric Hanselman

I'll also bring up that VMware had Raghu be interrupted and surprised by their General Counsel bringing up all of those compliance issues. So not exactly inconsequential.

Henry Baltazar

Yes. And I kind of shudder to think about what are some of these COBOL application that Jean's talking about. How sensitive are those things? How -- are they running like something really sensitive or something crucial? I would probably want that in a more isolated environment than throwing it out there.

So yes, I think a lot of organizations are going to struggle with that. And I think that's going to be an issue in terms of where that data placement is going to be and why. Again, I don't want to be the compliance police person, but it is a big factor in terms of where we do things and where that data should be stored.

Eric Hanselman

Well, and that's, I think, gets back to the point that Melanie was making about ROI on some of this transition. And hey, the ROI on an app modernization is probably pretty straightforward. The ROI on broader gen AI investments may be a little harder to quantify.

Henry Baltazar

But what's the negative impact of data leakage and other things that go out there like trade secrets or you spread vulnerabilities out there or you reveal vulnerabilities by having the code out there, which is probably what happened last time.

So yes, I mean, I think there's going to be a lot of issues. It's not going to -- it's not an easy case for me in terms of justifying where the data placement should be and why.

Eric Hanselman

Yes. We do now have a handful of cautionary tales about in-cautious use that has led to exposure of proprietary code, proprietary data. I'm not going to dive into all of the security aspects of gen AI. I think I need to put together a team to do a whole another episode on that part, but these are significant concerns. And maybe keeping things on-prem is actually not a bad way to at least cut down on the total number of unknowns in this transition.

Melanie Posey

Eric, one other thing to take a look at here is who's doing a lot of the hand-waving about gen AI right now. But one thing that's a bit different maybe than it has been in years past is that this messaging is targeted at the C-suite, right? The promise there is better, faster, cheaper, more efficient business outcomes, more customer satisfaction, all of that really good stuff.

So it's the people who were thinking about how we're going to digitally transform. Our business is -- that's who it's targeted at right now. I'm not really sure we've gotten to the level yet of the IT people who actually do the deployment or if you're more advanced type of organization, your platform engineering or site reliability engineering teams. We haven't really gotten to that point yet necessarily.

So I think a lot of these problems will come out in the wash. And the main thing here, which is all the cloud providers in talking about gen AI, one thing they tend to focus is, yes, in the context of a very hybrid-oriented world, a lot of the gen AI processing access to large language models, that's going to come through the hyperscalers.

And as Jean quite aptly said, they are going to be the kingmakers of gen AI because they can do a lot of these things at scale that you would have to be big, rich, mega corporation to be able to spend the money to get that kind of compute firepower. And even if you could, is that how you want to spend your money?

Eric Hanselman

You raised a whole bunch of these good ROI questions because you've got the harbor vendors on the one side saying, you can build it and run it cheaper than it would cost you in cloud. And again, a lot of this is that trade-off of are you going to be using infrastructure that you're going to be running all the time versus are you just doing some model training and more inference, and so maybe you train in the cloud, maybe you do inference on-prem or wherever your edge happens to be.

We're in the middle of a lot of healthy debate about what things cost, where you actually locate them and how you should really be thinking about it because as Melanie, Jean were pointing out, you do have the hyperscalers, and I look at what Google is doing with their model guard. If you've now got a marketplace of models, you've now got all access to a whole set of these higher-level abstractions about how you're actually going to work with generative AI and the pieces that you're actually going to be assembling.

I think we'd all agree that there is no one generative AI model to rule them all realistically. What we're going to be working with is a whole set of different models working together. And in fact, models helping to integrate the output of other models. And then that opens that question of how do you actually start to assemble that? How do you merge all of this data together? Or how do you get the data to the places that you need it? A whole set of really nontrivial conversations about where this has to go.

Melanie Posey

Yes. I think Nick Patience, our AI Research Director, was saying that generally, artificial intelligence is some of the most diverse workloads in terms of where the infrastructure is located, whether it's on-prem, off-prem, at the edge and so on. And I think that given the sort of intensity of the compute needed for this, that it's causing a reckoning.

And I would say this, combined with sort of macroeconomic conditions, rising prices for higher cost of credit and so on, it's really bringing about a reckoning in terms of balancing CapEx and OpEx investments in IT infrastructure in a way that before there was just this headlong rush into OpEx for everything. And as deployments are growing and as demands are becoming more intense, people are rethinking that. And I think that especially plays out in the storage space.

Henry Baltazar

Yes. I mean, from my perspective, I think it's going to be the same issues and some advantages that cloud has now. And this really highlights the ability to have that elasticity. I guess also on the supply side guess what? All those harbor suppliers and components providers, they're going to give their resources to them first. So they already have an unfair economic, and you don't hear about them getting hit by supply chain shortages as much as a lot of the other vendors that have suffered for the last couple of years have had to.

To me, I think this is going to highlight a lot of those capabilities. Again, I think it's like any other workload though, even if we're talking AI or whatever. It's about knowing what the workload is doing and when. If you actually know what's happening and what the resource consumption is going to be and you can actually predict it on a fairly accurate basis, yes, there probably is opportunity to save money by running it on-premises if you have the staff to do it and if you know exactly when you're going to have to need the capacity additions and whatnot.

If you don't, then that's when that elasticity has such a much higher value, that ability to be able to do that Black Friday thing or whatever emergency thing you need to do. It's just not that easy to go grab a bunch of GPUs and resources and a bunch of high-performance storage at the same time and be able to run those things if you don't have those assets ready.

Eric Hanselman

And it seems like we get back to that old adage of buy the base, rent the peak, the mantra from way back when. Well, I guess, way back when from a cloud perspective. But do we get into these issues of we got to balance, but yet balancing is kind of hard because we've got to move a lot of data around. We've got to have computational resources available. And again, some tough conversations about how you actually get ready to do this.

Melanie Posey

Well, one thing I find interesting is -- and I've seen this from the hyperscalers -- is that they're really leaning into the whole notion of abstraction and taking it to a higher level, if you will, just like how do you do generative AI. It's not like you can go to a website and drop like 3 items of gen AI into your shopping cart, click Place Order and go. It doesn't really work that way.

So what we're seeing is, I don't necessarily want to call it productization, but the hyperscalers putting together all the stuff you need to do this or that use case. And to Henry's point about the hyperscalers being first in line at the back door of NVIDIA's factory to get GPUs, that's definitely true. So I think that has some effect on the extent to which the hyperscalers and NVIDIA working with the hyperscalers wants to at least for people to get started with gen AI offer something that kind of looks like a one-stop shop.

And even beyond that, we've noticed this trend over the last couple of years that the hyperscalers have also leaned into their partner ecosystems. And Jean, you mentioned that the kingmaker aspect of the hyperscalers is, in part, due to the tight partnerships and/or ownership stakes they have in the large language model people. So I think all of that comes together to create sometimes collaborative, sometimes competitive ecosystem around helping organizations get into gen AI. And it's not something one company can take you there. They take you there with all of their ecosystem friends.

Eric Hanselman

Well, now I will also point those listeners who didn't catch the previous episode. The fact there are some middle grounds, things like some of the AI data center providers like CoreWeave or Point North and that class of folks. So there is that.

There are those points at which if you're not really fully on-prem but you still need a big bag of GPUs, there are some people you can go to. But I think from a comprehensive offering -- and you, of course, used my favorite word abstraction -- that really winds up being, I think, maybe the benefit of those more packaged offerings that, in fact, you don't have to do the low-level assembly, you can take it up to something that's maybe a little more LEGO brick like in terms of how you put the stuff together.

So what should organizations be thinking about? How do we start these hard conversations? And what are those pieces that we ought to be thinking about as we look at making sure that we're there? Should we just bind ourselves more tightly to the hyperscalers? Build out more on-prem? Jean, what's your take?

Jean Atelsek

I would say, I echo what Henry was saying in terms of knowing your workloads, and capacity planning is super important. These are concerns that you thought were a thing of the past, right, with OpEx. But I really think it's important to, first of all, at the -- having a data fabric strategy and a sort of governance policy in place is fundamental before you can start building AI applications on top of it.

And I think companies are wise to sort of wait and see what these start-ups and the SaaS companies and the GSIs come up with in terms of they are applying this technology internally, which are the applications that are going to have legs, right?

Eric Hanselman

And see what's actually going to deliver returns on a lot of these efforts. Henry, is that the direction that you've been thinking as well?

Henry Baltazar

Yes. I don't think there's -- it's an all or nothing thing or everything's set in stone. I think knowing your workload is one thing, but I think the other thing is how do workloads evolve? I mean maybe we start doing these things with test dev and betas and blah, blah, blah. And when we start dealing with more critical data or other aspects or data or trying to get to other geos, maybe that's when they go on-prem or even a hybrid tech model.

I don't think that we should be thinking about all or nothing. I think there's a different -- there's a better place for -- I think that the -- where the workload is and what's the criticality, what's the sensitivity, those things change over time. And as we start looking at those things, I think we need to have a more flexible model that will allow us to really take advantage of all these other different areas.

But at least, at the start, tell me it's hard to argue with going cloud for some of these things, especially because of the access to instant resources. The elasticity capability, the ability -- most important part to me is not elasticity. Oddly enough it's not the ability to get more. To me, the most important ability is to be able to turn it off because you can't do that if I buy a giant massive hardware and have it running in a data center, I can't just turn it off.

Eric Hanselman

That's that fundamental cloud scale, right?

Henry Baltazar

Yes, exactly.

Eric Hanselman

That's not going to work well for you if you can't figure out how to turn things off.

Henry Baltazar

Yes. So to me, I mean that elasticity. And then also, we can't ignore the fact that there's access to innovation in that marketplace in that execution venue that you may not have on premises either. Yes, to me, I think as life cycles emerge and as people get a better understanding of the workloads, they should be able to optimize.

But I think the groundwork has to be there from the start in terms of, okay, we decided to start development in cloud, let's figure out how, what's our migration or replication strategy so we could have another locations if we decide to move that or if we decide to expand it to another venue that might not have that cloud or might not have those resources or might be at a price that's not palatable in that geo. I think orientation should have some flexibility in mind because it's not a black and white type of decision.

Eric Hanselman

So Melanie, zero-sum game? All in one way? All in the other?

Melanie Posey

No, I mean nothing is ever a zero-sum game, except maybe dead and alive. But beyond that, I think we've talked about this on the podcast several times before, Eric, that hybrid is a de facto operating model of IT going forward. So as much as the hyperscalers want you to move everything into their cloud right away, you don't have to.

And I think a lot of the hyperscalers have gotten more realistic about that, too, that in some cases, they will bring the cloud to you on-prem, just like a lot of on-premises hardware incumbents are taking on-premises to the cloud. So that's a trend that will definitely continue.

Eric Hanselman

And we got people talking about multi-cloud and cross cloud and low code.

Melanie Posey

Exactly.

Eric Hanselman

They're making -- they're swearing on stacks of Bibles. They mean it.

Melanie Posey

Exactly. And there's a lot of acknowledgment. Now you should have our networking analyst, Mike Fratto, on to talk to you about this some time, that the more hybrid and multi-cloud things get, the more important networking becomes. And networking and security are tightly linked. So there is a lot of evolution in that space as well.

But one thing I would just say to kind of close out my piece of things is we're talking about digital infrastructure here. We're talking about digital transformation. And I think at the end of the day, what enterprises need to think about is why are we doing this? Is our primary motivation to reduce cost? That's a perfectly valid way to do things as long as you start shedding technical debt along the way.

Or is the ultimate motivation how do we become a faster, more agile, more successful business overall where we have optimized processes, where we optimize the way we interact with our customers, we boost customer engagement? So what's the business strategy behind all of this IT transformation and modernization?

I think getting everybody in the organization on the same page, the IT people, the app dev people, the line-of-business people, the C-suite people, I think ultimately, you'll make better decisions when you know why you're making the decision in the first place.

Eric Hanselman

Having the business drive the technology decisions? Melanie, you're radical.

Melanie Posey

Shocked, shocked.

Eric Hanselman

Well, again, another one of those conversations that we seem to have over and over and over, but definitely one that I think most organizations could certainly profit from having in more detail and hoping to guide strategy.

Well, thanks to all of you for all these perspectives. It's been great. And clearly, we've got many more conversations to come on helping people figure out where this is all going. So thank you all for being back. And I guess, we'll say until next time.

Melanie Posey

Thanks a lot, Eric.

Henry Baltazar

Thanks, Eric.

Jean Atelsek

Thank you.

Eric Hanselman

And that is it for this episode of Next in Tech. Thanks to our audience for staying with us. And thanks to our production team, including Caroline Wright, Ethan Zimman, our marketing events teams and our agency partner, the 199.

I hope you'll join us for our next episode whereas Melanie alluded to, I'll actually have Mike Fratto on talking about observability, all of the things that you do when we start getting this wildly distributed on-prem cloud and all the rest kind of bits. I hope you'll join us then because there is always something Next in Tech.