Next in Tech | Ep. 242: Open Compute Project Summit

View Full Transcript

Eric Hanselman: Welcome to Next in Tech, an S&P Global Market Intelligence Podcast where the world of emerging tech lives. I'm your host, Eric Hanselman, Chief Analyst for Industry and Company Data at S&P Global Market Intelligence. And today, we're going to be talking about the recent Open Compute Summit and all the things that are taking place there, especially with the greater focus on AI. There's a lot that's been happening. And the event itself has grown, and that's an indication of what's taking place in the market. And with me to discuss it is analyst Perkins Liu. Perkins. Welcome back to the podcast. Great to have you here. 
 

Perkins Liu: Thank you, Eric. 
 

Eric Hanselman: You were just out in San Jose at the summit. I guess to start, it's been, I think, maybe a couple of years since we covered OCP specifically, but I wanted to start on a little bit of background. And the open compute project is something that started originally as Meta's effort to really standardize what was or what they saw as a new set of specifications for how you build out data center infrastructure. And if we look at the summit from -- back from those early days in which there were just a handful of people. I was at one of the intermediate summits that was actually closer to here up in New Hampshire. It was a set of relatively small number of people that were just really looking at the really nitty gritty of this, and it was pretty specialized. But this year's OCP Summit was something that was a little bit different and I guess, a little larger. 
 

Perkins Liu: That's right. So I'll give you a little bit background information like if you look back to the history, OCP Foundation was actually initiated by Facebook that time. Facebook now Meta, with the mission to take advantage of the open source and open collaboration to speed up and foster the hardware innovation starting from the core of computing, including server, storage, networking and supporting racks and the entire data center infrastructure. It came from a data center Facebook designed itself with this concept. A lot of products at the data center infrastructure was none of those standard products of the market, pretty much designed by themselves. So the inspiration expanded across industry in the coming years. Later on, Microsoft joined in 2014. Apple joined in 2015 and Google joined in OCP in 2060. So OCP really becomes a technical innovation community for the industry to attack and advance the data center ecosystem, not just the hardware, but starting from networking, storage, all kinds of stuff. So before the pandemic, participants or attendees remained around 2,000 to 3,000 for years. Since the summit, it's always like -- I think the peak was 3,600 at 1 year. But since the summit returned to an in-person format in 2022, attendees growth accelerated sharply for consecutive years until this year, 3,500 in 2022, 4,400 in 2023. Last year was a big jump, 7,047. But this year, it break into like 5 digit, 10,835 attendees. The conference to move out of the building, the conference, they built a huge TAM to host the keynotes and in the lunch and breakfast sessions, you can imagine. of the OCB Summit. 
 

Eric Hanselman: Well, it really speaks to how the importance of data center infrastructure has changed. It's gone from a world in which it was very specialized to now having a much broader ecosystem of players and a set of people are really -- now this is much broader set of interested parties who are all now there. And you're bursting at the seams at the San Jose Convention Center, the fact that they actually had to go outside, that's a strong testament to how interest has grown. 
 

Perkins Liu: So my take is that it definitely is taking advantage of a technical community for the entire ecosystem, entire industry to collectively attack the challenges that we're facing especially driven by the boom of AI applications. Actually, yes, last year, I had a conversation with the CTO of OCB, said, well, is this really an OCP Summit or is a GTC NVIDIA, right? But he, wow, this is the biggest application in our industry these days. Why not? I think that tells what OCP foundation could do to the industry, definitely a fast environment and ecosystem for everybody to work together to advance and innovate the data center is just one layer, while all the other stuff associated with the AI application, which is the biggest one in the ecosystem these days. 
 

Eric Hanselman: Well, it does seem like AI is dominating so much of the conversation. But if we look at what's happening really in the conference itself, I'm curious what you're seeing about in terms of major themes in that integrated ecosystem is a piece of it, but we've moved from the big debate about whether or not you move from 19-inch racks to 21-inch racks to now really talking about so many different pieces of this. This is -- it's not only the basic form factors. It's, as you're pointing out, there's networking that's now involved. There are all sorts of things that are included, including a chiplet component. So we're now starting to get down even into the semiconductor specifications and the means for extending this ecosystem, not only much more broadly in terms of technology pillars that's working at, but much more deeply now down getting into the silicon level. 
 

Perkins Liu: Yes. So my focus area is the data center infrastructure layer. So I have more to say about this. But this year's OCP Global Summit had a major theme of leading the future of AI. As I mentioned, everything is driven by AI. What I saw at the summit was a perfect reflection of this theme. So under this theme, the summit emphasized on how the open hardware and open software community is collaborating to build a scalable, efficient, sustainable infrastructure for next-generation AI data centers, covering the areas like high-density rack design, power distribution, liquid cooling, memory and compute architecture advance and edge AI deployment. I'll give you 2 examples. At the center of the stage, adjacent booth on EXPoloor, Meta and AMD expanded Open Rack wide or ORW and Helios platform, respectively, echoing each other, the advancement in data center architecture are designed to address the increasing power density, thermal and the serviceability requirements associated with AI infrastructure. So the ORW is contributed by Meta to OCP, positioning as the next-generation double white rack standard for AI scale data center. ORW is their next-generation rack form factor, development in response to the limitation of traditional OCP OpenWack, which is 21-inch wide. This thing is 42 inches, double the size. okay? Because the original design was primarily optimized for CPU and storage workloads. However, the rise of AI workload has led to the GPU part demand between 80 to 150 kilowatts per rack, requiring enhanced liquid cooling, large power feeds and improved service access. So this ORW addresses these challenges by adopting a double white rack footprint, right, expanded double the size, allowing for expanded space, dedicated power shelves, liquid cooling, manifolds, cable routing and designated service zones at both front and the back of the rack. So in response to this, AMD published the Helios, which is a rack scale AI platform serves as the first reference implication of the ORW specifications. So it's co-developed with Meta. So that's the biggest thing I see is that power density rising up, pushing people to look at expand the size of the rack. 
 

Eric Hanselman: And you talk about 80 to 150 kilowatts per rack. Back in the days, I mean, when OCP was first starting out, I mean, we were -- if people were talking about 10 kilowatts per rack, that was big. And now it has really gone to an amazing acceleration. And hey, the 80 to 150 is probably not even the greatest part of this, although you were touching on the importance of liquid cooling. What's your take about where liquid cooling is and where it's headed? 
 

Perkins Liu: Well, at the OCP Summit this year, what we see is the liquid cooling continue to kick a center stage as one of the most transformative trends shaping the next generation of AI data centers. After years of development or debating, if you want to call it, the entire industry has explored the liquid cooling technology in great details. There are several noticeable trends I observed. One of the main force behind the liquid cooling evolution is the redless increase in heat and power density. Modern AI racks cannot exceed several hundred kilowatts and approach megawatt range. To handle this, designers are implementing different approaches like low temperature different loops, high-flow distribution manifolds and even exploring 2-phase cooling technology. The goal is to efficiently dissipate the heat from extreme compact and powerful compute nodes far beyond the capability of air cooling. So last year, we see a main share of CDE at 1 megawatt. This year, its capacity is doubled, increased to 2 megawatts, highlighted by the Google's project that shoes a handful of companies demonstrate the markup units for 2-megawatt CDUs, big. Another key shift is the focus on retrofitting cooling architectures because the industry recognized that most of the existing data centers cannot be rebuilt from scratch. Vendors and players are offering liquid cooling system that integrate into the existing aisle and the racks in the mix with air coolings,xamples, including like a side car liquid blue read heat exchange and the modular CDUs that can drop in brownfield environments. This approach allows operators to adopt liquid cooling incrementally and develop in steps to accelerate the deployment without disrupting their ongoing operations and business. Now the industry is also moving decisively from bespoke custom vendor-specific system towards a standardized and modular solutions, allowing interoperation across the multiple suppliers, companies showcase the next generation. Modular cold plate and liquid to liquid CDUs built on open hardware standards, highlighting the growing of maturity of OCP's liquid cooling systems. 
 

Eric Hanselman: Well, it's something where -- I think to your point, it's gone from a situation where this really was a special case to now really being an absolute requirement with the kinds of power dissipation that we see in the current generation of devices that are there. and where we see all this dissipation going, it's a challenge to be able to do this in anything other than liquid cooling. Now granted, there also were the other benefits, lower power in terms of managing cooling, so the power savings, you're taking out fans and such. Now you still have pumps. But in theory, significant reductions there. It is such a large shift in such a short period of time. It was interesting, one of the other headlines I saw coming out of it was the fact that the standards body that handles a lot of cooling specifications and does a lot of the rating, the ASRA organization has -- is now starting to move after a long period on redefining some of the liquid cooling standards. they, for a long time, have been the ones that set specifications for how air was delivered in a data center that if you had a cabinet in data center, you could count on the Aray standard air being delivered at specific temperatures and volumes. And they've had to scramble to be able to shift to liquid because there's been such a shift. And good to see that at least there's a standardization effort because to your point, originally, a lot of the liquid approaches were relatively customized, but we're now starting to move to something where there's at least commonality and standardization so that depending upon who you're getting your equipment from, you've got at least a form factor in which they can all plug. 
 

Perkins Liu: Yes. The message from the OCB Smit definitely is very clear. Liquid cooling is no longer an experimental. It's a real execution of industry that will go down to the levels in depth to make sure this application is well executed across the industry. Now the other part of this is also power -- not only power dissipation, but power distribution was once again, stun to see some of the additional changes that are taking place, the move to DC delivered power. And this is actually -- if we think back to the early days of the OCP, this was -- one of the things about design was that they were looking at 48-volt DC power distribution, being able to eliminate various step-down transformers in the data center itself, so bringing higher voltage AC directly to the rack and row. That's continued to advance to now what surprising levels. If you take a look at some of the voltage, they're kicking things up to 800 volts DC now in terms of distribution planning. Yes. That's -- if the transformation of data center cooling solution brought up by liquid cooling has become an unstoppable trend for years, right? So the rise of 800-volt DC power distribution architecture is a revolution in the power distribution side, also driven by the ever-increasing rack power density. Actually, talking about the DC, OCB actually has a history. So Meta actually developed a 48-volt DC architecture probably more than 10 years ago. they didn't bring this topic or talk it like loudly at OCP until last year. Last year, it was some small group talking about the 800-volt architecture, but it was very limited, okay? Even before that, we -- the industry actually had experimental about the 240-volt DC in China market. But in North America, European market, it pretty much is 48-volt DC, but it was limited to a certain application or certain like data centers, not in a wide range. But this year, because as we said, NVIDIA's AI-related application is taking the center stage, and they demonstrated NGX ecosystem at the floor. At the Summit, -- this NGX ecosystem is fully loaded with all the gears supporting the integration of 800-volt DC power architecture. If you go to any of the power distribution-related panels or discussions, 50% of the chance it's related to 800 volt. I'm not exerating. So the 800-volt architecture function as a high-voltage DC power distribution system and can replace or supplement the traditional AC to DC power stages in AI data centers. Its primary advantage include reduced conversion losses, which lead to improved overall power usage, PUE and higher power density, supporting hundreds of kilowatts per rack alignment with the requirement of 100 to 150 kilowatts per rack. So compared to 48-volt DC -- and because the voltage increase, it dramatically reduced the parent, which reduced the loss and reduce the usage of copper. Just because that you can kind of offset the increase of power density of the rack. So this architecture simplifies power distribution by minimizing the need for bus bars and the cables, facilitating easy integration with module racks. Additionally, it future-proofs the system by ensuring compatibility with next-generation high-voltage accelerators and GPU chips. I have seen there's like over 20 of NVIDIA's partners showcase solutions or components related to 800-volt DCs, companies like Eaton, Vertiv, Delta Electronics, Li and MacMeta, they all presented the 800-volt system during the event with markup unit. However, it is important to note that all the power module capacities, layout and dimension of each vendor are still -- they still varied significantly, indicating that we're still at the very early stage of this development, and there's still a long way to go before the standardization of product specifications and architectures to reach the level to the like Open Rack version 3, which is a very interchangeable, interoperationable, and you can pretty much plus the shelves into the rack without any problems from different vendors. but we're not there yet for 800-volt architecture. 
 

Eric Hanselman: Well, it's -- if you think about how rapidly this shift has taken place, I guess it's not surprising because you've got so many different concerns across this ecosystem, so many different things that you have to be able to tackle. Although certainly, it's absolutely needed. The biggest proof point to my mind, I was looking at the new Verorubin designs. And one of the call-outs was that it had liquid cooled bus bars. And for those that aren't familiar, the bus bars are the power distribution systems that are basically the main power rails. And they're just great, big, huge chunks of copper. And the whole point is that they've got relatively low resistance, and they're just simply moving a lot of this energy point-to-point for distribution. And the idea that they are now starting to move so much power, this particular case, they're looking at peak loads of like 5,000 amps, I think, was what they were talking about at a point at which you pass that through even a very large chunk of copper and it's still going to heat up. The move to 800 volts now starts to shift that equation because if the voltage is higher, you don't have to move quite -- the current levels don't have to be quite as higher. You're still moving a lot of electrons. You just happen to move them at greater potential rather than greater volume. It's the steps that are being taken across this ecosystem are so startling in the scale -- these are systems that are dissipating so much energy that it really presents some ecosystem challenges. But I guess not surprising right now that the power infrastructure part of the ecosystem is still working on standardizing this because you've also got to come up with new safety standards. You've got a lot of the challenges in power distribution, things like art flash mitigation, a lot of the -- making sure that all these systems are safe has to step up as well. So yes, it's so many moving parts, although being driven by what is this seemingly insatiable demand for energy. 
 

Perkins Liu: Yes, absolutely. The rising power demand, rising rack density is definitely driving the entire ecosystem to advance at the speed unprecedented seen we have to be able to deliver that much amount of power, and we have to be able to take the heat generate associated with it. So that's a challenge on both power side and the cooling side. What I have seen is that what I call a hockey stick effect. So before the emergence of ChatGPT like end of 2022, the entire data center industry, we have seen the rack density. We talk about 10 kW right? We see it advanced from probably 4 kW to 10, 12 kW for almost 10, 15 years. The slope is very stable and slow. But in the past 2 years, if you look at the curve, it ticked up quickly, jumping from 12 now we're talking about in practical, 140 k MVL72 rack and projection several hundred or 1 meg of rack in the coming years, dramatic. The ecosystem, the industry has a very little time to respond to this kind of change. So that's why we -- I used to call it like a 48-volt DC or these OCP like innovation was a nice to have like incremental improvement. Now it's a necessity because if we don't advance, we can't take care of this challenge. We can't deliver the power. We cannot take the heat away. It's a must-h. So that's why the industry is excited and also feel pressed with time to develop our ecosystem of infrastructure in response to this demand from the AI compute. 
 

Eric Hanselman: Well, and it's something that's also shifting the way the OCP is operating. And they're now expanding to not only moving beyond a lot of the core capabilities. It was interesting to see that this year, they've announced the OCP Academy. So they're actually going to be getting into the business of helping to facilitate education, building an OCP fluent workforce. They're taking on all sorts of different aspects in terms of what the organization itself is handling. 
 

Perkins Liu: Yes. I think that's also a mechanism to help the industry to advance in the speed that required. It's not just a natural growth, right? So OCB, it's becoming widely accepted community that all the participants collectively working together closely, never seen before to advance IT and data center infrastructure. Yes, it's academy. OCP Academy is a new online learning platform launched just at this summit to support the growing need for education and skills development across the global open hardware and data center community. It transforms OCP's extensive technical library spanning from open rack design, power system, cooling technology and AI infrastructure into structured interactive learning courses. The goal is to make OCP knowledge more accessible and actionable for engineers, facility operators, vendors and new community members. 
 

Eric Hanselman: Now to meet this growth, we've got to make sure that we've got a workforce that can support it. So -- absolutely -- both sides of that coin. So what are your thoughts on where this is headed and sort of your takeaways from what the indications were from the summit? 
 

Perkins Liu: Well, I think, like I said, it's an exciting time for the entire industry. A lot of people, a lot of friends, a lot of colleagues, networking people that I talk to every say, this is the time I have ever seen once a lifetime in my career. It's unprecedented. These are the key words I have seen. And also to me, myself as well, I use this hockey stick effect. So it's unprecedented innovation and opportunities for the industry, but also were challenged with all the pressure to get all these done in a timely manner, right? So last year's event, we see the dominance of GB200VL72 racks solution, created like illusion impression of a definitive peak level solution that seemed perfect in its execution. However, this is a totally different. Not many people demonstrated that because that obstacle is overcome, right? However, the relentless demand for power and cooling driven by AI computing continued without question as traditional single 21-inch or 19-inch rack struggle to meet these escalating requirements, the industry is adapting by doubling the rack width, expanded, right? The 40-volt power shells faced challenges related to efficiency losses and copper consumption. Now a huge amount of tension and concentrated shift towards higher voltage 800-volt DC solution to minimize the current copper usage. With the increased demand of liquid cooling, the capacity of CDUs rise from 1 megawatt to 2 megawatt -- so all these signs, we see unstoppable advancement and innovation. This summit to me underscored an industry dedication to push the boundaries and explore new heights, building upon the achievement of what we have done in the past. 
 

Eric Hanselman: Well, the achievements so far have been impressive and stunning. And clearly, there is so much more that is coming in the year ahead. Well, this has been great. Thank you, Perkins. It's been great to have you back. 
 

Perkins Liu: Thank you. It's my honor. 
 

Eric Hanselman: And we are at time for this episode. I want to thank our audience for staying with us. And thanks to our production team, including Sophie, Carr, Feranmi Adeoshun, and Kyra Smith on the marketing and events team. I hope you'll join us for our next episode where we're going to be talking about the Money2020 conference and a lot of what's going on in terms of not only the stablecoins stuffing back into crypto, some of the things we've been talking about, advances in FinTech. Hope you'll join us then because there is always something, Next in Tech.