Listen: Next in Tech | Episode 132: Are AI datacenters a thing?

View Full Transcript

Presentation

Eric Hanselman

Welcome to Next in Tech, an S&P Global Market Intelligence podcast, where the world of emerging tech lives. I'm your host, Eric Hanselman, Chief Analyst for Technology, Media & Telecom at S&P Global Market Intelligence. And today, we're going to be diving into the wild and wonderful world of AI data centers. And to discuss that with me today, I've got Dan Thompson returning to the podcast. Dan, welcome back.

Dan Thompson

Hey, thanks a lot. I wish we could have walkout music onto this. Because if we could, I would say let's do splish-splash, taking a bath, the AI washing.

Eric Hanselman

All right. Well, I guess, you've sort of thrown down the gauntlet for at least our first take on this. But let me start with a little background. I think one of the reasons why we wanted to cover this today in this episode was that it seems like over the course of the last month or so, there has been an awful lot of discussion about AI data centers and how a whole range of different participants in this market now have AI data centers.

And I guess, the thing to first define is, is there such a thing as an AI data center? If so, what's it made up of? And what really are we talking about when we start thinking about that? Because clearly, there is a move towards a very high capacity.

There is this mad dash to be able to get massive amounts of computational power that's necessary to power generative AI and large language models and all those other things. But is that all that different from what we've been doing in data centers and overall infrastructure builds to begin with? It sounds like you've got an angle on that.

Dan Thompson

Yes. No, these are all good questions. And I think the first thing that we should level-set on, when we think about the data center industry, the data center industry loves a good hype story. And if we think back over popular topics at data center conferences over the last several years, it's been blockchain is going to change the world. IoT is going to change the world. 5G, oh, yes, I said it, 5G is going to change the world. And then lo and behold, it has not happened yet, at least not from an infrastructure standpoint, right? And so that's not to say...

Eric Hanselman

But Dan, you left out metaverse.

Dan Thompson

Well, is it too early, is it too soon to...

Eric Hanselman

Well, Dan, and I'll throw -- while we're talking about topics, I'll throw in cryptocurrencies. Because there are all of these different things have washed through the field of what data centers can support, what they're going to do, what's going to drive consumption. And we're still not in a world where any one of them has really gained sufficient traction. I mean...

Dan Thompson

That's right, that's right.

Eric Hanselman

But fundamentally, we are seeing changes in what requirements are for data centers.

Dan Thompson

Yes, yes. And I think that's the thing to circle in on. So I mean, if we think about just sort of the hype that has been around all these topics, what I commonly point to, when we think about the whole context of this is it seems like the data center industry is an industry that's constantly in search of a new problem to solve. And so my personal belief is the reason why they jump on these hot trains so quickly is because this is them trying to show value of, "Hey, here's another problem that we can solve."

So well, actually, before I say what I was about to say, so it's at least reasonable to question -- because there's been so many of these things that have kind of come and gone over the years, it's reasonable to question, is this real or is this not real? So let me say that upfront. However, in this sort of hype chain has also been cloud. And I think if we think of all the topics, cloud has actually radically changed what is happening in data centers and has actually changed sort of the momentum of the data center industry.

So like it's good to remember that just because there are things that sort of are flashes in the pan, so to speak, there have been things that have stuck around. And cloud is one of them. So the question is, to your point in the lead-in there, is will AI be another one of these big drivers or will it disappear? And there's a couple of things that I would like to point to. So let's start with just the infrastructure and kind of what we're seeing from a data center demand right now.

So if we look at -- so I'm just going to use the Phoenix market as an example, it's just top of mind. There are several markets across the United States, Phoenix being one of them, where we have seen a tremendous amount of new demand. And it's all from the hyperscalers, so Google, Microsoft, Amazon. I would even throw in Facebook and Twitter in those in some cases. And so there has just been a tremendous amount of demand and a tremendous amount of just construction going on to support something.

It seems like this is net-new. But we don't know that for sure. And so just to set the record straight, when Google, Microsoft, Amazon or whoever come in and they say, "Hello, Digital Realty, hello, CyrusOne, hello, Aligned Data Centers," whoever the data center provider is, when they come in, they say, "Hey, I need 50 megawatts of capacity." They don't say for the purpose of AI. They say, "I need 50 megawatts of capacity, mind your own business." And so it's not...

Eric Hanselman

"We're going to do with it what we see fit," and there we go.

Dan Thompson

Exactly, yes. So it's not like all these companies sort of secretly know, "Oh, this is all being driven by AI and building this new AI data center." What they know is this is the capacity that's being asked for. This is the density requirements of that capacity, meaning how much power per rack is being necessitated by the ask. And they know, is there something weird or interesting or new about the tooling requirements for all those racks? And so...

Eric Hanselman

And you've hit on, I think, one of the things that I think is really important, which is -- and one of the things that I'll make the case, is changing whether or not that's AI-driven or not. We can certainly debate which is the amount of power per rack. Because when they get a spec for a data center, they know what the square footage is, they know what the power requirements are.

And what they're then building is a data center that is going to be able to deliver a certain power density per rack basically per unit of space within that data center. And that power -- the power dissipation for that individual rack is the thing that, I'd make the case, has already been changing. And we've seen that shift pretty dramatically. And the hyperscalers have driven part of that. The nature of the way in which we're consuming infrastructure has driven that.

I think you point out cloud is driving that because cloud helps to make more efficient use of the capacity that's in a rack. And the pre-cloud, what happened was we had even reasonably good virtualization. We were getting maybe 30%, 40%, 50% utilization of an individual server in that rack. And cloud bumped that up a bit because now we've got much better operational capability. We've got the ability to schedule more workloads to be able to make better use of the systems that are in play. And those numbers started creeping up significantly up into the 60%, 80% utilization.

Dan Thompson

Yes, yes. So let's dissect that a little bit, actually. So what we've seen on the ground in Phoenix -- and I think if we could just sort of use for the purpose of this discussion Phoenix as a bit of a bellwether because these are all brand-new builds and there's no limits on -- for the most part, in the Phoenix metro, getting power is not a problem if you're creative with where you want to buy land. And I'll explain that in a minute, if it's necessary.

But in any case, so what we were hearing from data center providers there is that around 2020, maybe even 2019, the ask of watts per square foot started going up. And so what we had previously seen in the Phoenix market was about 150 watts per square foot. What is now being asked for is north of 200 watts per square foot. And in many cases, it's 300 watts per square foot and above. Now that sounds like a doubling, and it is.

However, if we do the math, what that equals out to conservatively is about 10 kilowatts per rack. So we're still not -- I mean, like if you listen to the sort of the broader conversation of, "Oh, my gosh, it's going to be 1,000 watts -- or kilowatts per rack. It's going to be 2,000 kilowatts per rack." We're just not seeing that. We're not seeing like these crazy numbers.

We're seeing now what we want is an entire data center full of 10-kilowatt racks. In reality, what's happening is there's some number of maybe 50-kilowatt racks, and they're just putting fewer racks in the data center or they're mixing and matching, which is more likely what is happening. And so it is true that the ask of watts per square foot is going up. So that is an indicator that something new or something different is happening, but we're still not just seeing outlandish numbers.

And the reason why I want to make that point is if we're going to call something an AI data center, it strikes me as if you're going to fill the entire data hall with AI workloads, you're going to need more than 10 kilowatts per rack on average. So that's at least worth considering. It is true though that, like as I mentioned, in reality, what's happening is they're just putting fewer racks on the floor. Those racks are being more dense.

It could be that some of this is AI, that's probably true. Some of it is supporting workloads, also probably true. And that's sort of the question that we don't know is, for every AI instance, how many other instances are there in support of that AI instance? And so we have to remember that there's other infrastructure that's going around there. In addition to Microsoft, Amazon and Google, they're still selling just regular old instances, right? Like they still sell a ton of just infrastructure not associated with AI. So all that stuff is still...

Eric Hanselman

Strangely enough, not everything that's out there is all AI.

Dan Thompson

Right.

Eric Hanselman

Despite the level of conversation. But before we move on too far from that, I also want to focus on the other aspect of that growing increase in power and some of these dichotomies in terms of demand. The other piece, you touched on the cooling aspect and the cooling technologies that are there. And so take a look at how we cool servers in data centers, historically, that's all been with air. We're basically dumping air towards servers. We're pushing it in the right places.

We've gotten good at doing things like managing airflow front to back, making sure that one side of the server rack is where the cool air comes in, the other side is where the hot air goes out, cold aisle, hot aisle kind of designs, things that are helping to manage some of the airflow. But you get to a point at which when we start pushing more and more energy through those racks, that air alone is not going to do it for you.

Dan Thompson

Yes, yes. No, that's exactly right. And this is sort of another indicator that I would point to. So all the providers that I spoke to in Phoenix said that they felt like they could continue to cool the data halls in the same way they have been doing traditionally with these current generation of data centers that are going up right now. And so what that says to me is, yes, these numbers may be rising, but they're not getting so significant that we're having to re-architect things yet.

Now every provider also conceded that they could see now on the horizon where there would be a need to re-architect things. There were several providers that noted, "I'm going to be now having to deliver water or some sort of fluid to the data center floor." It mostly seemed like experimenting and instead of like entire data centers that were using water distribution or some sort of coolant distribution.

However, they are all now talking about, "This is on the horizon." So while this current generation of data centers, we can still use traditional methods to cool them, all the providers did see something now on the horizon where this seems more real, this seems like something that they will need to address.

So recently, just to kind of call some names out and points and some things out here, so recently, Digital Realty made an announcement that they are going to be offering high-density racks. Essentially, what they're doing is rear door heat exchangers. By their math, if you go a little bit more than 15 kilowatts per rack and probably less than 30 kilowatts per rack, you can do that easily by just using traditional air methods.

If we just, for just a second, assume that, that's true sort of globally and that nobody has any magic sauce -- I know there's providers out there that would say they have magic sauce, but just for the sake of discussion, if we're thinking about broadly the entire data hall, nothing freaky is happening just yet to...

Eric Hanselman

Well, one important point I want to make in this is that it's not as though the data centers are not already using water. Because they're already using water. And they're just -- they're chilling water and distributing it out to individual heat exchangers that then cool the air closer to different parts of the data center. But it's not as though they don't already have the water.

And when you talk about rear door heat exchangers, they're just simply extending what is already that cold water flow into a heat exchanger that rather than sitting up in the ceiling or wherever they're actually doing the air-cooling and the air-handling pieces, they're just simply extending that cool water flow all just to a heat exchanger that's targeted to one particular rack.

Dan Thompson

That's right, that's right.

Eric Hanselman

So they're making like that initial little journey in just extending the regular water flow. But the other piece that people are talking about is actually looking at specialized water or coolant capabilities that actually circulate inside the server itself. And that seems to be that next step-up in terms of transition in technology.

And that's the point at which now you're actually starting to have to do plumbing in the server that's moving whatever that cooling is through the server, which is that next stage up, which is the thing that seems to be the more specialized, the 50-kilowatt and beyond rack kind of thing. And that oftentimes is actually something that is in addition to the regular cooling loops that are already circulating within data center.

Dan Thompson

That's right, that's right. And when we talk to providers there in Phoenix, what they were telling us, totally unprompted by me, they all -- like literally, all the wholesale providers just kind of came out with this is that the hyperscalers are coming to them and saying, "Let's talk about a 5-year plan that looks at how we can accommodate 50-kilowatt racks in an entire data hall."

So the change that we're talking about seems to be -- like if we want to put a date on a calendar, it seems to be within the next 5 years is when this is going to happen or at least this is when the hyperscalers are anticipating this is going to happen. And so when...

Eric Hanselman

And there are set of technologies that are already -- if you look at the things that an Equinix or a Digital or a CyrusOne or any of these folks are doing now, they've already done sort of intermediate things to help that. Because they've had high-density technologies in which they'll actually, whether or not it happens to be back-of-rack or inter-rack coolers or -- there are all these kinds of technologies that they've already started to apply. But it's moving into some new and uncharted territory.

Dan Thompson

Yes, that's exactly right. I mean, they have -- all the major data center providers, wholesale data center providers have been experimenting with this for at least the last 12 to 18 months, if not longer. And so yes, they are all thinking in this direction. And now that the hyperscalers are sort of pushing them forward, I think we'll eventually start to see this being deployed in a major way on the data center floor, so...

Eric Hanselman

And the thing that has to happen in order for this to actually take place, in order to get your coolant into a server is that the server has actually got to be built with these capabilities in mind. So you have to have the server manufacturers actually build servers that can be cooled this way. So there's a bit of a chicken and egg. And actually, something that Perkins has been talking about on our coverage on that.

Dan Thompson

And not to mention that chip manufacturers also had to catch up with the demand. So there are multiple supply chains kind of playing out here, but -- so let me also comment that like -- so now that I've been like mildly a bit of a naysayer on there's no such thing as an AI data center yet. But let me like take half a step back here, and let's comment on what's happening sort of in the broader industry, right?

So first and foremost, we know that NVIDIA, they just came out with their earnings. Data center compute, revenue that nearly tripled year-on-year. This is primarily being driven by the cloud service providers buying up things. We know the cloud service providers, at least Microsoft and Amazon, anyway, in their earnings call said, "Hey, we're about to spend a ton of money on infrastructure. And the yield on your stocks may be a little bit lower over the next couple of quarters, but that's just sort of a necessary evil so that we can make the net expense."

It seems that there is an arms race happening. So it's not unprecedented that people are saying, "Oh, it's AI everywhere, and there's just so much AI investment." That's all actually true, right? We've seen it play out. So that is happening in the background. Interestingly though -- and I mean, this is something that I would call folks' attention to. So if you caught the CEO of NVIDIA's keynote speech about a week or so before their earnings call, he was talking about this new Grace Hopper chip, which I believe is the GH200.

And what he was saying is that this new Grace Hopper chip can accomplish with 0.26 of -- basically 0.25 megawatt with what it used to take 5 megawatts' worth of energy for x86 chips, there we go, to accomplish. So we're talking like a fraction of the amount of power required to do the same amount of work with this new level of GPU. So it is also worth noting that the future requires less energy.

Now it could also -- and people would say, oh, well, then they'll just take the same amount of spend. Rather than just doing 0.26 of a megawatt, they'll use the same amount of megawatts. He actually makes this point and says -- again, if we scale up to the same investment. So like his numbers were if you spend $100 million on CPUs -- x86 CPUs, that would require 5 megawatts of power.

If you spend that same amount of money for the Grace Hopper chips, that would require 3 megawatts. So no matter how you spend this, the amount of processing power to accomplish things is going to go down. I think it's sort of a foregone conclusion that it will be more dense though, to the point of the conversation you and I were having a second ago.

The other thing that was interesting from that CEO speech was it seemed to him as like a foregone conclusion that most AI was going to be happening in the cloud. Again, this is backed up by their earnings call that a ton of their chips -- a ton of their revenue rather is coming from the cloud service providers. So it seems like to say that AI is growing is to also say that the cloud is growing. And we're certainly seeing that, right? There's no refuting that we're seeing pockets of extreme and interesting growth from the cloud providers.

So I just want to have a moment of balance to say, yes, there are a lot of interesting things happening. Yes, I think the next 12 to 18 months are going to be really, really interesting, when we start seeing what's happening from an infrastructure perspective. Can we say we've seen that already? We're seeing hints of it, yes. Has something been radically changed yet? Maybe not. But we do see it on the horizon.

Eric Hanselman

Well, and to your point, I think Jensen Huang's point is that, okay, we maybe are starting to maybe blunt the rate at which power consumption is growing. And we've also got other silicon advances that are taking place. The new Hopper is more efficient because it's going to finer geometries in the semiconductors and the silicon that's made up. You've got new architectures.

Things like Intel's Habana Labs division is coming out with their Gaudi device, which, in theory, does deep learning more efficiently, more power-efficient, so -- but I think we can generally expect with this crazy level of growth with -- that's anticipated in the number of AI workloads, there's an awful lot that's going to be going on around that as well. But power is still going to be a concern.

Dan Thompson

Yes, yes. I mean, at the risk of sounding contradictory to what I just said, it is true -- I mean, to kind of tease out what you just said, it is true that every generation of processor chip requires less energy to do the same amount of work as the previous generation. It's one of the benefits of keeping up with the latest model.

It is also true that we have not seen a reduction in demand from the data center space. So what the cloud providers are coming in and asking for is not going down. And so it seems like a reasonable thought to think they're just going to keep throwing more and more stuff in the pile. Is there a point at which and when they say, "This is enough," or at least, "This is enough for a data center," we don't know that.

Folks would be guessing to say that. But anyway, when you're thinking about this, there are -- to kind of keep the balance level, you've got to kind of consider both sides of the coin in that it will require way less energy according to the CEO of NVIDIA. At the same time, companies may just cram more in the same footprint.

Eric Hanselman

Well, will we ever have enough capacity for all of that AI demand that's out there? I guess, that's something we'll have to see. But I mean, there are folks who are coming at this with an approach of GPUs as a service.

Like if you look at CoreWeave, who's raised just a crazy amount of money, and they're not necessarily building data centers insomuch as they're building data centers that are already pre-populated with really high density computational workloads, lots of GPUs and things, and other folks who are taking that approach, I guess, maybe sort of heading towards the purpose-built world of things. But demand is certainly going to keep growing, and we're going to be looking at other ways to be able to meet that demand.

Dan Thompson

Yes. I mean, they're sort of -- CoreWeave is an interesting case, so is Northern Data and GDA, among a number of other companies that have kind of been in the crypto world and are trying to change their focus. Now that the value of many of the crypto coins has dropped, they're trying to change their focus and chase other workload types. In the three cases...

Eric Hanselman

We've moved from one heavy-duty computational load to another.

Dan Thompson

Yes, exactly, exactly. And so there's a number of companies that are trying to make this jump from cryptocurrency now to AI-type processing. The jury is still out, just to be fair. This is very early days to see how successful this will be. But it is an interesting new case. Maybe they have AI data centers and I've just -- I haven't seen it. But in any case, I remain skeptical.

Eric Hanselman

Well, it sounds like we'll have to leave it there. I guess, maybe we've got some use cases. But I guess, it's a certain amount of caveat emptor in terms of really making sense of what exists out there. Well, this has been great. Yes, thanks very much.

Dan Thompson

Yes, yes, yes, thanks for having me.

Eric Hanselman

And that is it for this episode of Next in Tech. Thanks to our audience for staying with us. And thanks to our production team, including Caroline Wright and Ethan Zimman on the Marketing & Events teams and our agency partner, the One Nine Nine.

I hope you'll join us for our next episode, where we're going to be digging into digital infrastructure, which, surprise, surprise, also has some AI pieces in it, talking about a lot of the announcements that are coming out at the beginning of the conference season, a lot of perspectives on where enterprises are going with AI and what the infrastructure is required to run them. I hope you'll join us then because there is always something Next in Tech.