Listen: Next in Tech | Episode 136: AI and datacenter challenges at HCTS

View Full Transcript

Eric Hanselman

Welcome to Next in Tech, an S&P Global Market Intelligence podcast where the world of emerging tech lives. I'm your host, Eric Hanselman, Chief Analyst for Technology, Media and Telecom of S&P Global Market Intelligence. And today, we're going to be talking about the Hosting and Cloud Transformation Summit coming up October 10 in New York.

And with me today are two of the team who are going to be presenting sessions at HCTS, Nick Patience and Dan Thompson, both repeat guests in the podcast. Welcome back to you both.

Dan Thompson

Thanks so much.

Nick Patience

Thanks, Eric.

Eric Hanselman

And great to have you on to talk about everything that we're doing at HCTS, and as well as a lot of the core infrastructure conversations. We're talking about the impact of AI. Because, of course, generative AI is changing everything, right?

Nick Patience

Does indeed.

Eric Hanselman

Dan and I were talking just a few episodes ago about, is there such a thing as an AI data center? But of course, fundamentally, we've got the issue of dealing with all of the aspects of what's happening with this next stage of AI.

Not that we haven't been dealing with what are a lot of these same challenges, would have been looking at a lot of machine learning, analytics, various earlier stages of AI. We want to talk about -- I guess we're talking about legacy AI now. But Nick, you're going to be digging into a whole set of aspects about AI in your session.

Nick Patience

Yes, that's right, Eric. I'm going to -- the title of my session is, What the AI Explosion Means for the Digital Infrastructure Industry. So I'm going to be trying to take a fairly high-level look across the whole space, but obviously, focusing on generative AI and talk about how this is different.

You mentioned there's sort of legacy AI. There's certainly a common place now to hear about traditional AI. So the kind of shift from more traditional AI and machine learning was doing around patent classification to the shift to content generation, to the shift from very specific and narrow technologies that machine learning was being applied to.

In theory, general purpose use cases, generative AI may also get quite narrow as we go along. And then look at the market in general. We have our own market forecast, which we've talked about on previous episodes. And also just trying to deal with some of the upfront, trying to deal with some of the perceived negativity that may be around.

Have we reached peak generative AI already? What are the kind of things like the slight drop off in the web traffic of ChatGPT's website tell us, if anything at all? And then contrast that with the way the industry itself doesn't think that's the case at all. And that's demonstrated by the level of investment in the industry, especially hyperscale cloud providers putting in.

And so they're very much betting -- for the long-term, betting for the next 5 to 10 years, not the next 18 months. And so yes, that's how we're going to sort of kick it off. And I'm also going to talk a little bit about skills and the chip market and then sort of look to the future.

Eric Hanselman

Well, is this interesting -- I don't know, I was going to say quandary, we find ourselves in, I guess. About once the heavy-duty buzz starts to roll off a little bit, there's that moment of thinking, "Oh, this was just a passing fad." And yet, there is an awful lot foundationally that's going on.

As you're pointing out, it's the level of investment that's taking place currently. It's the kind of things that are actually underway in terms of a lot of the foundational elements that are being brought together to help move towards this next stage. And this is not really a situation that I think is slacking off or really moderating at any time soon, and we certainly see that broadly.

Nick Patience

Totally. I mean if you just look at this, in the last few days, the announcement by AWS putting up to $4 billion into Anthropic, mirroring the kind of similar deal that Microsoft struck more than 4 years ago now with OpenAI. There's a huge amount of appetite to invest in this technology.

And as you say, it's building very deep foundations on which we will build other things going forward. So I think one of the things that I'm going to talk a little bit about, and obviously, doing in our research as we look forward to 2024, how the space will evolve in terms of the actual product market fit?

And talking about things such as large language models and foundation models aren't the product themselves and shouldn't be seen as the product, and the products will be built on top of these things. And so that does -- still that takes a lot of money, but it also takes ingenuity.

And this is how -- this is where the challenge is going to be for the industry in the next 18 months or so is, how can we turn these things quickly from fun experiments that people may get a bit JD with quite quickly into products that actually help people do their jobs more efficiently?

Eric Hanselman

And it's that shift into really starting to leverage the model infrastructure to actually do work, whereas I think, to your point, so much right now gets focused on those near term, how do I build the model? How do I train the model? And yet the real value piece is the actual work that the models are going to be doing down the road.

Nick Patience

Yes. And we're very -- it's very, very early days in the second part of that. What is it actually going to do to help us do our jobs more efficiently, whether you'd be looking at operational savings or whether you're looking at revenue generation. And so we're certainly seeing early use cases in all these areas, but it is very early.

And I guess the temptation as ever, with a lot of any kind of major shift, and I think this is a major platform shift. So -- I mean I think there's something pretty fundamental going on here, but there's -- obviously, it can scare some people into thinking obviously their jobs or their roles might be threatened.

It also gets regulators very excited. It also get lawyers very excited. The copyright laws, for instance, are not really going to be -- don't seem to be fit for purpose for this kind of technology. There's so many different ways this is going to affect, but not just the tech industry, but much more importantly than that, the whole society.

And so what I'm going to be talking about at HCTS is really how we are -- the industry is committed to building the foundation layers for this technology. And then to a certain extent, what comes next? And what should the attendees at our conference be looking out for as we head into 2024?

Eric Hanselman

Well, and talking about foundational layers, I mean that's something, Dan, you're going to be talking about. I think it's really an extension of some of the conversations we've had about -- what really matters and what makes sense in AI.

And one of the things that I think is catching regulators, various market participants, which is a lot of the energy perspectives. And what it really means to have AI capabilities in a data center, really what's necessary to support them?

Dan Thompson

Yes, that's right. I mean there's a lot of focus that had been put on. What is the infrastructure requirements for AI? And when we get down to my area of coverage is, what are the data centers need to look like? But it really begs the question is, like can AI data centers be sustainable? And that's what I'm hoping to present on and hopefully, come to a conclusion at HCTS.

And so just quickly, as a bit of a recap to our last podcast that we had, Eric, we talked about what is an AI data center? And where we landed there was an AI data center is essentially just a high-density data center or, to say this another way, it's basically just a data center operating at a different power scale than a traditional data center.

And so to ask the question, can an AI data center be sustainable? It's really just to ask a question of, can a data center be sustainable? And that's very much something that the data center industry is still wrestling with, I think. But, like as we've dug into this, there are some interesting places where -- what we would agree and believe is an environmental issue.

So talking about the carbon emissions and other greenhouse gas emissions associated with operating a data center, where that environmental issue becomes a social issue. And so just to kind of give you an example of this. There's something that I have coined a term for calling the Virginia Paradox.

And so what's happening there is Dominion Energy, which is a power company in the state, have recently announced that they are going to be extending the lives of one gas power plant and then several coal power plants to ensure the reliability of the grid and to meet the current demand.

In the same conversation, actually, Dominion, specifically cites data centers as one of the major contributors to the demand situation there in Virginia. And so on the one hand, it's true that the data center companies and the hyperscalers are bringing on a tremendous amount of new clean and renewable energy to the grids around the globe.

But on the other hand, here, we have this community where the power companies having to walk back on their decommissioning plans for our plant, and there's -- created this paradox that I was describing, where now what do we do? And how do we think about that?

You would -- if I'm Microsoft, it would be easy enough and it would be in agreement with the Paris Climate Change Agreement to say, "Well, I have -- just use round numbers here. I have 10 megawatts of demand. I have brought on 10 megawatts, plus some of clean and renewable energy. So I'm good."

Eric Hanselman

Yes. But it's in this specific situation -- it is the thing we talked about in the previous episode. It's in the specific situation in that local context. That's where the paradox lies.

Dan Thompson

Yes, that's right. That's right.

Eric Hanselman

And that, yes, hey, you've got purchase agreements to be able to get all of this green energy, but not right there in Northern Virginia.

Dan Thompson

Yes, exactly. So the dilemma also -- and really why this is as much a social issue than it is an environmental issue is -- so the dilemma here is that Dominion has a state mandated requirement of sourcing 50% of its energy from renewable sources by 2032.

They're now having to walk back on some of the plans, which is going to make it hard for them to hit that number. They're having to walk back their decommissioning plans. And they have said, as part of their strategy, it's going to actually bring on more gas power plants. And again, this is to help with reliability of the power grid and meet demand.

And so like we are, I'm sure, very familiar with the rising power cost -- or the rising cost rather of natural gas. And so again, we're sort of contributing to the scenario where now the citizens who are purchasing power from Dominion may have a higher price longer term because of this use of natural gas.

And so it's just not a good look, I guess. I'm struggling to come up with how to exactly articulate this even now. But it's just like -- I think we would agree that it's just not a good look. And it's not fair, I don't think, for the hyperscalers and others to take a step back and say, "Well, I did my part. Sorry, Dominion." It seems like that's not the right place to be.

Eric Hanselman

Right. This is a promise that then being able to have the ability to bring on sufficient green energy sources, and that presents a challenge. And now they're having to turn to short-term fixes because they can't find the capabilities to bring on green resources.

Even though they've already made commitments to actually purchase green energy from other sources, it's just those electrons can't make it into their current environment. And in fact, it's having that regional impact that goes far beyond the data center environment.

Dan Thompson

That's right. So another issue or another area rather where environmental issues can become social issues is around the idea of water consumption and data center. So data centers will oftentimes use water as a means to cool the data centers. So they use a process called evaporative cooling.

This can consume in order of millions of gallons of water per day. And this is not just like river water, this has to be clean water. And so in areas of water stress, you could have a situation where data centers are competing with the citizens for access to clean water.

And again, like this is not a good look. This is not what we -- a position that we want to be in. Obviously, citizens would take first precedent when it comes to drinking water and clean water. But man, it's sure it's hard to say no to Google when they show up with these really large checks and they're wanting to participate in that utility. So these are all definitely things that we should think about.

Eric Hanselman

All right. That puts us into some interesting situations. And to your point, a real paradox. Again, all of these environmental stresses that come into play. I mean water -- you've got evaporative cooling that gets used a lot.

Dan Thompson

That's right.

Eric Hanselman

I think it's one of the questions that I wind up hearing a lot is, "So what are they doing with all this water?" Well, evaporative cooling is a big piece of it. But of course, a big chunk of it also is just being able to do heat exchange across that environment.

But whatever that end use is, there's just an awful lot of water that's getting consumed as part of the data center operations. But the grid piece is just as big a part. And we've talked about power purchase agreements. And what the real result is...

And that yes, you can have a particular cloud operator who is buying lots of renewable energy, but that end constraint is going to be what their local grid is actually able to deliver to them directly. And if that local grid is already stressed and doesn't have a lot of green supply or doesn't have sufficient green supply, it's going to be consuming just regular, old, dirty electrons generated by fossil fuel.

Dan Thompson

Yes, that's right. Yes. Sort of this -- like -- I guess these are some of the unforeseen consequences. I think if we delve in with Nick about AI and just some of the -- he mentioned a second ago, folks are worried about losing their jobs or thinking, at a larger scale, my role altogether will go away.

I think it's true of many situations where there are unseen consequences that you just sort of don't know about until you're in the thick of it, right? And this would be one of those, is that Northern Virginia has seen such dramatic data center growth. There are situations that exist there that we haven't seen play out elsewhere.

But you could also put that on its head and say, "This could be coming for other places like Phoenix, which are seeing dramatic growth as well out in Arizona." And we're already seeing some constraints in terms of distribution and transmission of electricity there. So you could potentially see situations there and in other places around the globe where problems are created, just simply because the data centers are growing at such a fast pace.

Eric Hanselman

Well, I guess, Northern Virginia is interesting because there's such a strong concentration there. But we are already seeing -- now that you've got most of the major cloud providers actually giving you some level of energy scoring on various data centers, you see workloads start moving to regions that have more green power.

Moving out to the Nordics, out to Oregon, places where there is a greater abundance of greener power that's available. I guess that sort of gets back to these questions of, "Can we start breaking up some of the training process, for example, and doing the training in places where it's better?"

But I don't know, Nick, is that something where if you think about how approaches are looking at being able to manage that, is that something from an AI perspective, from a workload perspective, that's starting to get integrated into some of this?

It seems like there would be the potential to do some level of orchestration to make workload -- or to push workloads from at least some part of that process into environments where there was more abundant energy or more abundant green energy. It seems like that's still relatively early days for those sorts of things. Or are there parts I'm missing there?

Nick Patience

Yes, it is very early. I mean when you think about the two main areas of AI, the training of the models and the making predictions from those trained models, which, as most people probably know by now, is known as inference, they do operate in quite different spheres at the moment.

And so -- especially now we're back with another kind of early wave. So if you think back to kind of the last big wave of enterprise AI, 2015, '16 period, we saw a similar pattern to now. A lot of the early experimentation is not exclusively done in public cloud, but it's very, very much geared towards that.

And we're seeing this again now. Obviously, this is exacerbated by the need to have access to both the hardware accelerators and large foundation models, especially with large language models that we all know and love. But that will change.

And I think most people I speak to in the industry think that the large-scale inference is for the infrastructure providers. And now the people, for instance, at our conference at HCTS in New York, those kind of companies that provide that digital infrastructure, their largest opportunity is on the inference side, in the long...

Well, not even in the long run, in fairly short order, we could be talking the next couple of years. So -- whereas a lot of the demand on the infrastructure is due to training at the moment, and that will continue for some period of time.

There's only so many large language models the world needs. What they are going to need is lots and lots and lots and lots of much smaller models that are trained on company data, organizational data and actually solve more of a specific problem.

And then the inference from that is going to be out closer to the use cases. So a lot of people think of this -- I mean this is not a statistic I have any basis for, but if it's 80/20 and put in favor of training, a lot of people I speak to think it might flip completely the opposite way around within a couple of years, which is quite interesting.

And a major challenge for infrastructure underneath, where we're talking about network, storage, memory, chips, all those good things. Because if you do get to the point where there's a lot of inference going on, and the inference could be at the near edge, it could be in network edge, it could be in retail stores, it could be in all sorts of places, that puts a whole different complexion on the nature of digital infrastructure.

And even if it's not that dramatic, it's certainly where everybody thinks the shift is going to -- that's the direction or travel for fairly obvious reasons, I think, because inference is where you get the value from the train model.

Eric Hanselman

It's where the actual work gets done. And the results happened, the control gets generated, whatever the particular action is, inference is the place in which you're actually starting to do the work. But to your point, right now, everybody is spending a lot of time training models, figuring out why are the models really appropriate, those sorts of things.

But there's this shift coming. And I guess maybe that's the point at which there's less focus on Northern Virginia and core data centers, and now starting to move that out to the edge. And that then starts to say, "Regions, edge data centers, all the things have been talking about, pushing more of this workload closer to the end users starts to come into play."

But that generates a very different mix in terms of where that load goes and what the power demands are going to be in terms of -- power and infrastructure demands are going to be in terms of getting those workloads out to where they actually need to do real work.

Nick Patience

Yes. And I don't even think we're there yet in terms of thinking about sustainability at the edge. Now the reality of it is, this is my personal opinion anyway, is that we are still very much focused on the large problem in front of us, which is the corporate data centers, these massive power consumers.

I said my personal observation is that we're still very much focused on those and not as much this future edge data center. Or whether that means the factory floor, whether that means some corner of the office somewhere, whatever that happens to look like. I don't even think that we're to a point yet where we're thinking about the sustainability of those pieces. Yes, we're still wrestling with, do we use diesel or do -- we don't use diesel? Like what is it today?

Eric Hanselman

Well, we seem to, once again, be into this set of pendulum swings of, there's been this big push to get out of corporate data centers because they're inefficient. When you look at power utilization affect those PUE numbers, corporate data centers are hovering around what 3-ish, high 2s, somewhere in there.

And so the shift to cloud was a good thing because they were running at much lower PUEs, and so the power effectiveness was much better. And now we're going to swing right back out to -- but these edge environments, however you want to characterize them, that are going to be a challenge to be able to make efficient and manage well in just the same ways, but a much higher scale.

Because, Nick, to your point, it's at the edge where so much of this inference is going to be going. I mean that's that big challenge, right? So much inference is going to be pushed out to the edge. We're going to have to be managing models in edge environments. We're going to have the infrastructure to manage all of that, and that creates its own set of challenges. But again, I guess, we're just on the cusp of that sort of transition.

Nick Patience

Yes, we are. I mean our research -- our survey show there's a massive pent-up appetite to do more at the edge, whether it'd be training or inference. I mean, yes, our numbers are somewhere near -- somewhere in the 90% of organizations and say, "We would like to do more of both, right?"

And then it actually comes down to, "Okay, how do you do that?" It's not entirely clear how they do that, both from an infrastructure point of view, but also even from skills point of view. And by that, we mean, "Who maintains the edge device when there's no IT department in that retail store, in that very small factory, or anything like that?"

And so yes, there's lots of multifaceted issues around, this is not just pipes and machines, there's also people and their skill levels. So -- and that's -- there's a whole lot of other skill challenges as well, the kind of soft challenges you may think of them, these kind of unintended consequences of these kind of areas.

And we're hoping -- in some ways, generative AI may create some issues with that, but it also may solve some of them. So when you think about cogeneration and the ability for generative AI models to produce code and code documentation and those kind of things, there's both the ying and yang to this. This could both be the source of the problem and perhaps a solution to some of the problems as well. That's the theory anyway.

Eric Hanselman

Well, and it seems like that transition then is one that we've got some options to help manage it. We're, I guess, still in the early days of being able to get to this new standing to be able to actually build enough skills, to be able to have enough models distributed far enough, to be able to actually leverage them.

And maybe some of those models are the things we'll be putting to work for energy management within the data center itself. That's something -- when you think about AI, operational efficiency management just for data center operations, those are things that we see some of today in the data center, right, Dan?

Dan Thompson

Yes. I mean that is sort of an interesting use case is, "Can AI be brought to bear in the data center?" Meaning, "Can it be used to manage the data center?" So kind of what Nick was just talking about, "Where is the problem and the solution? Can AI better manage the data center? Can it make these facilities more efficient?"

Because it can respond to things quicker or it can gain deeper insights. This is really interesting. And I would say that the industry is focused on this through our data center information management systems and even just [ broader ability ] management systems.

The key challenge though is that as anything, I mean, Nick would tell you, when it comes to trading models, garbage in, garbage out. And so we have to get to a place where the data that is going in these data center information management systems is good data.

Where companies have spent the time to really tune the sensors and making sure that false positives are chased out of these states or at least to an ability where the system can learn that those are false positives. And then we need to get to a place where functions are happening on an automated basis rather than a manual basis.

And what I mean by that is, today, even companies that have really well-run DCM, data center information management, when they have really well-run DCM systems, they're just using it, for a large part, just as information or alerts rather. And so an alert happens, a human being [ staged at ], and then they go perform some action rather than letting the system do it themselves.

This is quite common. I mean we've seen this play out over the years as different technologies have rolled out. It's a trust thing. And so as we get more and more comfortable with the systems, we get more and more comfortable with them -- with the systems making decisions for us and then taking actions for us. And so really it is a time thing.

Once we get good data in there, and we've had time enough to get comfortable with it and realized it's not going to make mistakes, or you've got a person on call, correct those mistakes or override mistakes that may happen, that's where we need to get to. I mean I think if you look at the research the Uptime Institute have done over the years and some of the other companies, they would say that failures in the data center almost always related to human error.

Whether it'd be -- we chose not to have a budget for that or a human actually made a mistake, they threw the wrong switch or whatever. And so to the extent that we can take the humans out of that, I think, would be powerful. We have a little bit of ways to go to get there.

Eric Hanselman

I think you identified precisely that in terms of where we are for a lot of the environments in which we're seeing AI applied today.

Nick Patience

Yes, totally. I thought about this for a long time. This issue of humans and roles and jobs and tasks. One of the main conclusions I've come to, which I will be talking about briefly at HCTS as well, is we're incredibly bad at making predictions about the future of work.

Without picking on Geoff Hinton, the AI pioneer who invented bank propagation, a key part of machine learning and deep learning, he said in 2016, "People should stop training as radiologists now. It's just completely obvious, within 5 years, deep learning is going to do better than radiologists now in 2016."

So I guess this is a key example of technology being too important to be left to technology people alone who may not be able to see the bigger picture. And it's -- not that I'm some great prognosticator here, but at least kind of step back and think -- let's look back on how various waves of technology have been applied to society and what's happened to them.

And we've had this over and over again, where you end up with jobs that nobody ever thought would have possibly existed, including machine learning engineer, being one of them, that didn't exist even 10 years ago. Kind of post-production video editor, that didn't exist. All these kind of roles didn't exist.

And I'm not saying that there won't be operational efficiencies, and some people, their jobs reduced or eliminated, I think there will be. But overall, I think the overall impact will be a positive one, a net increase in the kind of opportunities that people have and the kinds of jobs that AI will enable them to do, but enable them to do more effectively.

Eric Hanselman

Prompt engineering, they're all output validation. All these kinds of things that we've started, to suddenly realize they're a key part of how we work with a lot of these new technologies. Well, it's a lot of interesting conversations that we can extend at the Hosting and Cloud Transformation Summit coming up in New York on October 10. I hope we'll see many of our listeners there, but many thanks to both of you. It's been a great conversation that, of course, we look to extend when we get to New York.

Dan Thompson

Yes, I look forward to seeing everyone there.

Nick Patience

Yes, likewise. Good to be back in New York.

Eric Hanselman

And that is it for this episode of Next in Tech. Thanks to our audience for staying with us. And thanks to our production team, including Caroline Wright and Ethan Zimman on the Marketing and Events teams, and our agency partner, The 199.

I hope you'll join us for our next exposed, where we're going to be talking about news about AI and society, some interesting study results that are looking at basically a decade of perceptions of AI and what it's meant and possibly where we're headed. I hope you'll join us then because there is always something Next in Tech.