Next in Tech | Ep. 275: AI CEO Series: Dr. Varun Sivaram

View Full Transcript

[00:00:00] Eric Hanselman: Welcome to Next in Tech, an S&P Global podcast where the world of emerging tech lives. I'm your host, Eric Hanselman, Chief Analyst for Industry Research at S&P Global, and this is the next in our AI CEO series, where we're looking at places where AI is being put to work. With me today is Dr. Varun Sivaraman, founder and CEO of Emerald AI, formerly the Group Chief Strategy and Technology Officer of Orsted, a Fortune 500 energy company, a physicist, and Rhodes Scholar. Welcome to the podcast, Varun.

[00:00:33] Varun Sivaram: Eric, thanks so much for having me

[00:00:35] Eric Hanselman: It's great to have you on. We've talked about a number of different applications of AI and where it's being put to work, but this is a particularly interesting intersection that actually hits a number of the things that we've talked about on the podcast, both on data centers, energy consumption, grid operation, and a lot of those interconnected challenges that we face really in helping to manage getting sufficient energy where we need it, when we need it, to be able to manage AI demand, but also to more flexibly work with, data center build-outs and to be able to meet what is that, that ever-expanding need for capabilities. But, what you've been doing is actually leveraging AI capabilities to be able to help manage some of that. Could you give us some details on some of that?

[00:01:25] Varun Sivaram: Absolutely, Eric. Look, I think there's a collision course today between the AI revolution and this unprecedented historic growth of AI data centers and what the energy grid is really capable of delivering. And because of this collision course, because we can't get enough energy to the AI revolution, we risk the United States' competitiveness and innovation in the AI race, and we also risk raising energy bills for communities and making AI a historically unpopular topic. That's not good for anyone. It's not good for the AI companies. It's not good for America. It's not good for communities. Now today, United States wants to add fifty gigawatts or more of AI data centers just in the next three years, between now and twenty twenty-eight. And you might recognize that number, Eric. It's because I'm quoting S&P Global statistics. But between now and twenty twenty-eight, just half of that capacity in AI data centers can actually get connected to the grid, twenty-five gigawatts or less. So that's a very difficult position that we're in, and we need a serious course correction in order to achieve all of the goals: get enough AI data centers connected as fast as possible, keep bills low for communities and grids reliable, and advance our position in the AI race.

[00:02:43] Eric Hanselman: If you think about what the depth of current interconnect queues are for the grid, we're seeing such tremendous delays in order to both bring on new capacity as well as to interconnect new data centers to the grid. that presents some, I guess what seemingly are insurmountable challenges.

[00:03:01] Varun Sivaram: It would seem insurmountable, Eric? And the reason for this is, you can think of the power grid as a highway. This highway that you probably take your morning commute on has rush hour at 8:00 or 9:00 AM, right? In fact, the grid is a highway with horrible utilization. It has rush hour maybe once or twice a month. The rest of the time, that highway is relatively empty. There's plenty of room. That's how the US' electricity grids function. There's plenty of spare capacity 99% of the year. And so, because the grid isn't able to serve new data centers for 1% or less of that year, they make them wait, as you said, for 10 or 12 years in that interconnection queue, and that's what's holding up the AI revolution

[00:03:44] Eric Hanselman: Well, that really speaks to the challenge of design of the grid, right? Which is that issue of, the grid capacity has to be able to meet peak demand. That’s the design criteria. And that, that's really what's been guiding design principles, and as you're pointing out, is one of the things that's increasing the time to get connected and the way in which we manage grid capacity broadly.

[00:04:06] Varun Sivaram: Exactly. But I would flip the paradigm there, Eric, and I would say this is not just about grid design and about how you operate the grid. This is about how the users use the grid. Because if there is ninety-nine percent of the year plenty of spare capacity, wouldn't it be great if users could just use that spare capacity? Wouldn't it be great that during those one or two rush hours a month, instead of an eighteen-wheeler, you could shrink that eighteen-wheeler down a little bit? That is what flexible data centers represent, and that's what Emerald AI, our company, is doing. We have developed technology backed by NVIDIA, backed by seven Fortune five hundred companies. We've developed technology to help data centers to control their power demand during these rare rush-hour periods. And if data centers can flexibly modulate their demand in response to those rare moments of peak grid strain, you could fit one hundred gigawatts worth of data centers on today's power grids right now. So again, recapping, the US wants to connect fifty gigawatts in the next three years. We can only connect twenty-five, but really there's one hundred gigawatts of capacity that we could connect right this second if we had flexible data centers. That's the opportunity. It's four trillion dollars of AI data centers. And the good news, every time you connect a flexible data center, you avoid having to build out another lane of that highway, that's already a massive superhighway most of the time, you save customers money. Local communities will actually see their power bills go down and not up with more flexible data centers. And policymakers are taking notice. Around the country, states and even the federal government and the Trump administration is saying, "We should really find a fast lane for these flexible data centers." Because if you're flexible and you're going to bring rates down and you're going to make the grid more reliable, and you're going to use all that spare capacity without having to build out extra lanes of that highway, you should get a benefit. You should get a ten-year head start in getting connected to the grid, and that's what we're seeing. On June eighteenth, we saw a major new Federal Energy Regulatory Commission announcement that flexible large loads, flexible AI data centers, are now potentially going to have a benefit. And Texas has passed a similar rule to give flexible data centers a fast lane onto the grid.

[00:06:29] Eric Hanselman: But that's a complicated problem. the data center design philosophy has always been that we're going to be able to go meet demand, for our full capacity all the time, and there's been this disconnect between both the power demand side and the workload automation side of, what do you run and where do you run it? And it seems like that gap has been pretty vast. Th- there's some specialized applications, like crypto mining. You take a look at how crypto miners have set up camp in various places, not only electric grid, but also in natural gas environments, where they're looking at stranded capacity to be able to manage that. But, crypto mining is one of those workloads that you can start it, you can stop it and shut it down. Typical data center workloads, though, there's no coordination between when that workload runs and what power demands look like. and it seems like in most data center environments, th- there isn't, isn't even a mechanism for it. You look at most cloud providers, you got spot markets to be able to take advantage of spare capacity, but there's no view from the workload perspective into what the current availability of energy happens to be. How do you bridge that gap?

[00:07:43] Varun Sivaram: Eric, that's such a good point. You're absolutely right that making data centers flexible and controllable power users sounds like an alien concept. And you're right, Bitcoin miners can do this, but it's relatively trivial to ramp up and down how much Bitcoin you're mining. Traditionally, with data centers where you might be running PayPal and Venmo transactions, or a human may be controlling a surgical robot, you really don't want to mess with the availability and speed of any of those transactions. And so traditional cloud computing has been a terrible, unsuitable candidate for this power modulation. And if you just tried to slap a bunch of batteries onto a data center, it would be an expensive way in order to modulate the power draw from the grid. But let me tell you that although this sounds alien and hard and something the data centers may not want to do, compare it to the alternatives and you realize how attractive this solution is. The alternatives for data centers are you continue down the current path, you wait 10 years, and oh, in the meantime, the grid has to build out additional lanes to that highway and raise everybody's power bills. And then you've got pitchforks coming out from communities saying, "We don't want you here." Or another alternative is you just say, "I'm going to go off the grid." It's technically complicated and expensive to build your own natural gas plant and your own microgrid. You happen to raise prices on everybody else in the process because you're buying up all the supply of electrical equipment, and then you've got to maintain and run your own little electricity grid when right next door there is a massive power grid that is built literally to do that function for you. Or you go into space. And, space is great because there's a lot of literally space However, it comes with its own engineering challenges of getting the data centers to space and operating them and using photovoltaics and doing heat dissipation, et cetera. All of these are an order of magnitude more complicated than making the data center power flexible, making an AI data center controllable in the same way that a Bitcoin miner has been controllable. And I believe that the rise of the AI factory, these specialized data centers filled with GPUs or other accelerators are actually purpose-built to be exquisitely controllable because they do one thing really well. They convert watts into tokens of artificial intelligence. They may be running training or fine-tuning or inference. And Emerald AI has built a series of AI tools, AI agents that will, in a very intelligent way, ramp up and down the power by orchestrating these workloads, by keeping the critical ones running at 100% while slowing down and throttling some of the more flexible ones, or by moving certain workloads between locations at the speed of light so that in Virginia, as we demonstrated with Oracle, you can reduce power when the grid needs you in the middle of a winter peak period by moving them to Chicago during a period where there's excess grid capacity in Chicago and excess data center capacity. So, there is technology to do this. Yes, it's complicated, and no, it's not more complicated than the other alternatives.

[00:10:46] Eric Hanselman: Ah. So, it seems like part of this is really inverting the equation on this. You’re talking about moving workloads. This is something where you're overcoming grid inflexibility because the challenges we've talked a lot is that the grid has a really hard time distributing energy. It wasn't really built to move energy across long distances. It was built to be relatively universal or relatively unidirectional from generation out to consumers. but it sounds like where you're headed with this is that you're able to move workloads to places where they're needed, and you've also got an understanding of workload characteristics to be able to manage that. Is that where you've headed?

[00:11:26] Varun Sivaram: That's exactly right, Eric. And the reframe that I'd say is think of AI not just as a liability on the grid, a big energy suck. Think of AI as the next great energy technology. I was at S&P's CERAWeek earlier this year. I spoke on the main stage, and we made a big announcement with the six biggest American power companies and with NVIDIA. And we said, "Think of AI factories, these new specialized AI data centers, as a living, breathing component of the energy grid that supplements it." Because as you said, Eric, now suddenly you've got an asset that can move the use of electricity, AI workloads around the country and around the world at the speed of light, or in one single location, you can throttle the use of energy up and down on command. It's exquisitely controllable. It's virtually controllable, and that's the advantage of these AI factories. And if you think of them as an energy technology, as virtual generators or virtual power plants and batteries, we can use this network of AI not just as an energy suck, but as a real source of flexibility for the existing inadequate energy grid. So, we've done now five demonstrations at real commercial data centers all over the world, whether it's London or Phoenix, Arizona, and we've proven with NVIDIA and with our partners like Oracle and with the local utilities like the National Grid in the UK, we've proven that AI factories can flex on demand. Just like a Bitcoin miner, you can turn the dial up or down based on what the utility needs, and the data center will precisely follow the signal because we understand the workloads and we understand the constraints of the AI factories Now we're going to the commercial phase. We're less than two years old, but we're really excited that within those two years we'll have taken a brand new energy technology from invention all the way through commercialization. A nuclear reactor, which is another kind of energy technology, takes decades to go from invention to commercialization. But this one, AI factories is an energy technologies. Because it's software defined and controlled by AI, we can move at the speed of light. So, we're about to do our first two commercial-scale deployments. The first one's in Santa Clara with NVIDIA and the utility Silicon Valley Power. There, we're going to increase the capacity of the NVIDIA data center immediately because Silicon Valley Power will reward them for being a flexible power user by giving them additional capacity. And the second one is a hundred megawatts, a large scale, multi-billion-dollar facility, a data center Digital Realty and NVIDIA are building in Manassas, Virginia. And at that Digital Realty data center, we're going to be working very closely with NVIDIA to make it a power flexible AI factory that can respond to the local utility and the local grid operator called PJM. The goal is, Eric, going forward, the next thousand AI factories all built to the NVIDIA spec or reference design will have this feature, power flexibility. So, you can control their power use up and down, and a utility anywhere in the world can see the sticker that it's got Emerald inside and say, "Ah, I really want to connect that one ten years early because I want to have that data center on my grid. That's the one that lowers my customers’ bills. That's the one that helps keep my grid alive." The last thing I'll quickly say, Eric, is in London when we did a demo, we proved that at halftime of a soccer game, when all the British people turn on their tea kettles and there's a one gigawatt spike of energy on the grid, the data center can actually help to stabilize the grid by reducing its power use. That was a really cool and fun demonstration because these AI factories are now going to become the most community and grid-friendly assets. They're going to go from villains to heroes, and it's because we've reimagined what a data center is. It's not just an energy suck, it's an energy technology.

[00:15:23] Eric Hanselman: Now that the World Cup is upon us, there must be a lot of electric kettles kicking on and off with all the games that are going on. I guess the demand is all that much greater. But, I wanted to touch on one of the points you made about battery energy systems and energy storage, because this is one of those things that's been hung out there as, all we need to do is have, batteries as buffers. But we seem to see a whole set of cautionary tales about not only the limitations of battery energy storage systems, but also some of the durability. we were talking actually a few episodes ago about, the challenges that, the XAI data center environments seem to be seeing just simply because of the wear and tear on the battery environments, that, that seem to start poking some holes in, the, I guess that sort of panacea that has become the battery idea.

[00:16:14] Varun Sivaram: Yeah, absolutely. Well, look, first, the last thing you want to do is have a really polluting diesel generator running all the time in order to provide you flexibility. The community will absolutely hate you for that. Then if you start to use batteries, there are a bunch of technical constraints that come into play. First of all, it's expensive, right? If you have a battery set up and you want to make sure that this data center is very flexible so that the utility says to you, "Okay, I'm willing to connect you immediately and not make you wait ten years." then they're going to say, "When I run into a difficult condition, we might have to ask you for four or six or eight or twelve hours of flexibility," and a battery just can't do it. And so you're going to have to build a very expensive, massively overbuilt battery to handle all of the different contingencies for which the grid might ask you to be flexible. By far, the most graceful, lowest cost way to achieve flexibility is by modulating your workloads, right? The virtual AI jobs that you are able to orchestrate by slowing them down, by pausing them, by moving them somewhere else, that is the most elegant way to solve this problem, and you can actually do it in close collaboration with the battery. You can imagine you've got a two-hour battery sitting on your site, and you have some flexibility in your AI workloads, and you coordinate the two. Emerald has a product that does this. In fact, we collaborate with some of the world's biggest energy companies so that the local energy equipment at the data center can be coordinated with the AI virtual flexibility that our AI agents are managing. That, that's the best way to minimize the expense of your equipment, your capital equipment on site, while also precisely meeting what the grid needs so that the grid can give you an earlier and a larger power connection. Today, there's no more valuable currency, there's no coin of the realm, to use a Game of Thrones analogy, than time and speed to power. Emerald delivers that for our data center clients, and we hope that, alongside these amazing new commercial deployments with NVIDIA, with Digital Realty in California and Virginia, we get the chance to work with customers all over the world, and we're now expanding into the United Kingdom, into Japan, because every country needs this. They need data centers to be heroes and not villains, to be energy, controllable assets, energy technologies. And if we can continue to deploy that, we'd love to see far more data centers built on existing power grids than the other way around. Far more existing grids are needed just to power a small number of data centers

[00:18:43] Eric Hanselman: So, it's interesting to see some of what you're able to accomplish around that. You're really managing demand side. I'm curious about how you're putting AI to work, and is that scheduling? Is that management? I'm curious about what aspects of that you're taking advantage of and what you've seen to actually get real results.

[00:19:02] Varun Sivaram: It's a great question, Eric, and it can be a little confusing because we're almost a recursive AI company, right? We are AI managing AI, and so we think about AI foundationally. We are a frontier research laboratory for AI and energy. You can think of Anthropic as a frontier research laboratory for AI. We are the frontier research lab for AI and energy because we're doing foundational AI research. All AI workloads, or almost all of them, have some amount of flexibility inherent in them. If you're fine-tuning a particular model, that workload might have some temporal flexibility. You might be able to slow it down a little bit. If you're running AI inference, it can be batch inference, which has some temporal flexibility or serving inference, you need the answer right this moment, that might have some spatial flexibility. You might be able to move it at the speed of light to another data center. And there are hundreds of different AI workload types, and we work on understanding what are the different flexibility parameters for each of these AI workloads. We've used AI to train our own model in order to quickly and rapidly pattern recognize what sort of workload is running on an existing data center, and we have to use AI agents in order to orchestrate dozens or hundreds or thousands of AI workloads running at one data center or across a network of data centers. No human could make the decisions that we had to make when in London, a lightning strike hit, and we had to have the AI factory respond within seconds to reduce the power consumption by over thirty percent in under thirty seconds. And so, the AI that we use is orchestrating the AI that's running on the data center. And we've published our research now, peer-reviewed platforms around the world, like one of the world's top scientific journals, Nature Energy. So, we have real results, and we are at our heart a research laboratory, an AI frontier research lab for AI and energy

[00:21:02] Eric Hanselman: Oh, wow. So, you are taking what you understand about the workloads, about, the dynamic nature of the grid, what the capacity capabilities are, and being able to, in real time, do that scheduling so that you really have that impact, on grid demand at any point in time.

[00:21:19] Varun Sivaram: That's exactly right. And so, utilities look to us as their best friend. Utilities need to look at AI factories with Emerald inside as a real resource. If you're running into a pinch, the AI factories can help bail you out. Now, we don't want to do this too often because AI factories, I want to be clear, run the world's most valuable workloads. You might think that generating cat images from ChatGPT is not very valuable, but this is. Whether it's that or you're creating slides or you're doing coding work, knowledge work is going to become the most important part of the global economy, so we don't want to take these assets for granted. We want to judiciously flex them when it will not disrupt the core mission of those AI workloads. But because they are exquisitely controllable and our AI agents are capable of controlling them in such a way as to protect the customer's experience while also precisely meeting grid targets, we believe that this can become an indispensable resource to the electric power grid.

[00:22:23] Eric Hanselman: As you're pointing out, those that are looking to get their own version of astronauts riding horses on the moon, might actually be able to accept a bit more of a delay, especially when that gives us, better demand profiles and the capabilities to actually work cooperatively with the grid. Oh, well, it's been fascinating. Thank you, Varun.

[00:22:40] Varun Sivaram: Eric, thank you so much for having me. Really an honor to join S&P.

[00:22:44] Eric Hanselman: Thank you very much, and especially on such an important topic, but we are at time for this episode. Thanks to our audience for staying with us, and thanks to our production team, including Sophie Carr, Feranmi Adeoshun, Kyra Smith, Dylan Scheible on the marketing and events teams. If you liked this episode, please subscribe or like us. Please keep in mind that statements made by persons who are not S&P Global employees represent their own views and are not necessarily the views of S&P Global. I hope you'll join us for our next episode, because there is always something next in tech.