Next in Tech | Ep. 271: AI Networking

View Full Transcript

Eric Hanselman   00:00:02

Welcome to Next In Tech, an S&P Global Podcast where the world of emerging tech lives. I'm your host, Eric Hanselman, Chief Analyst for Industry Research at S&P Global. And today, we're talking about a different aspect, networking in AI and the different levels at which we're starting to think about networking differently now that we're headed into the guts of AI infrastructure. And with me to discuss it is the analyst leading research around networking for us, Mike Fratto. Mike, welcome to the podcast.

Mike Fratto   00:00:33

Great, Eric. Thanks for having me.

Eric Hanselman   00:00:35

Thanks for being back on. And if we think about a lot of what's been happening in discussions around AI, everybody gets all tangled with GPUs and storage and all the other things that are happening. But AI is also forced assessment maybe, rethink possibly of really how we build networks and what the requirements are for interconnect. Really is -- we're having to shift a bit in terms of really how we're thinking about building networks.

Mike Fratto   00:01:08

Yes, we are, and it touches everything from the data center to the wide area to the cloud, all of it is changing and has to adapt for these AI applications and use cases.

Eric Hanselman   00:01:19

Well, we've sort of gone through this period of thinking about hybrid and multi-cloud and what that transition looked like. And I think historically, if you look back at a lot of the voice enterprise data, there was this movement for ages that enterprises wanted to get so they just had one cloud provider. And then really, I don't know, 2, 3 years ago, they realized that no, they actually were probably going to have more than one. And then AI. And now suddenly, they've realized that they need all of the cloud providers and they all have to be interconnected.

Mike Fratto   00:01:57

Yes. And that interconnection needs to be fast and reliable and all the requirements for any kind of distributed application. And it's not just the cloud providers, it's where are the AI applications being consumed in the office, in the remote office, on the road. What's going on? You haven't mentioned Agentic, but I'm sure it's going to come up at some point. How is Agentic going to impact the architecture and the use and the utilization of the wide area and talking to all of these different cloud services. So it's really kind of a crazy time, and it's changing things that are happening. from the IP layer, from Layer 3 from just interconnect and basic routing and switching all the way up into the application layer where load balancers live in application security and all of those networking features that are above the wire basically?

Eric Hanselman   00:02:47

Well, yes, bringing in security, suddenly, we've moved from something that was relatively simple just in terms of doing application protection to now having to have network-based security be something that actually has to be able to do semantic parsing of prompts in order to understand whether or not the prompt is malicious. That's a very different world. But I want to drill down on one of the things that actually is also starting to have some market activity, which is the need to manage this dynamic interconnection at multiple layers. And that's something where we've seen on the data center interconnect and what we used to call the wide area network, the need to be able to go get configurable connectivity for data center environments and the enterprise environments that house most of the data that's needed for training. This is one of the things that was behind some M&A activity that's going on. Lumen acquired Alkira, a start-up that is doing dynamic interconnect and automated stitching between cloud environments. That's been an interesting shift to actually have what has been a traditional interconnect provider now start to be dabbling in the dynamism that is the interconnect world.

Mike Fratto   00:04:10

Yes. It's a strategy that Lumen has been pursuing for several years now and trying to create a service that's dynamic and can change very quickly and is on demand and the other service providers are doing the same thing as well. And so that acquisition, what Alcya brings to the table is just a software layer. And so it provides not only that foundation for that sort of dynamism to take place, but the nature of what Alkya did, reaching deep into the cloud services, being able to configure the networking components in those cloud services literally in a hands-off kind of fashion, all of a sudden can be completely driven by a software. So you can go in there, spin up all the routing and security components in AWS or GCP or whatever other supported cloud. And you're just not really thinking about it anymore. And in addition, it gives Lumen this global reach. So it's not just in the U.S., it's anywhere, which is pretty fascinating.

Eric Hanselman   00:05:13

And I think that starts to get back to the legacy challenge that has been levels of automation. And the initial work with AI, okay, great. You want to be able to dynamically connect data to do training, agents. As you said, we're definitely going to dig into agents. Agents shift the scope and scale and speed of what has to be able to happen. And I think it gets to your point, in order to be able to capitalize on that kind of velocity, you have to get fully automated and the network is a fundamental part of that.

Mike Fratto   00:05:49

Yes. In supporting applications, the network needs to be fast and reliable and secure and all those features and capabilities that organizations want out of the network, but it also has to stay out of the way. And so one of the things that cloud and AI is bringing to the business is more and more people are able to type into a prompt and build an application that's talking to all these data sources. And so you never really know where things are going to be used or spun up or accessed. And so that's what's driving this philosophy. It's not just organizations now just need to, in a very controlled fixed manner, bring up a new pipe somewhere. It's -- you've got some business analysts who's putting together a dashboard in some kind of in the AI platform, and they're just doing it on their own. And all of a sudden, it's going to touch 10 different things. And so that's not IT driving that. And that's what I mean by the network just kind of has to stay out of the way. So if they can go in and go, I need all these things and then they push go. the business analyst isn't driving network configuration, but what they're doing is triggering these automated processes that bring up these connections dynamically.

Eric Hanselman   00:07:10

And those dynamic connections are going to chew up a lot of the capacity that you got to be able to respond to.

Mike Fratto   00:07:16

Yes.

Mike Fratto   00:07:17

Yes, absolutely.

Eric Hanselman   00:07:18

This is also something that if we look at the internal networking, this is something where AI is having a significant impact there, although those are things that we've kind of seen ramping up for a while, thinking around both within data centers, the kind of capacity that's necessary to manage training. But that's another area where scale is starting to become an issue. But this is something where just the levels of data movement within clusters is now starting to consume dramatically higher levels of Internet requirements in terms of -- or networking requirements in terms of interconnection. And we think about where this is headed, it's 400 gig used to seem like a lot of capacity and now Geez, it's table stakes.

Mike Fratto   00:08:06

Yes. 400 gig is -- yes, it's table stakes, 800 is here in some cases and even beyond. And it's just the sheer capacity and the throughput is really the same, but the requirements for what the impact of loss is not just a retransmit time, it's in a training session or even in inferencing, it's time when expensive GPUs could be sitting idle, I think costing money and delaying that whole sort of run. So it's not like you're losing one GPU. -- potentially with latency, the entire chunk of GPUs that are being used for a job end up just sitting idle. So there's the impacts and the stakes are a whole lot greater.

Eric Hanselman   00:08:50

And the -- when we've gotten to a point at which computational power has always been expensive. But as you're pointing out, -- now we've got what is a dramatically more expensive resource, GPU capacity. As we start to go step up to ever and ever denser clusters to do the computational work that's around this, keeping them fed, the cost for not keeping them fed winds up going up dramatically. If you can't get data into a cluster at sufficient speed to keep it fully occupied, that's costing you money right then and there.

Mike Fratto   00:09:27

Yes, it is. And part of the architectural challenge is how do you build that network from the cluster front end all the way out to the wide area and beyond in such a way that you can get all the data to all of the endpoints reliably and not run into congestion and loss and that latency that we talked about and those other sort of network degradations. So there's a number of architectural and protocol and engineering challenges that the industry is facing, not only on the vendor side and the provider side, but also on the operations side of how do we build and operate and run and what technology do we use? And what are the standards and sort of what's that approach?

Eric Hanselman   00:10:19

Well, we came through the age of merchant silicon, where we were able to build networking gear with some fairly standardized set of silicon providers who really were building out the core. And the big difference for a lot of these systems really was the overlying operating system and the switch or the router that was really handling us. that really has picked up significantly as well in that now you've got silicon providers working very closely with the GPU providers to be able to go build platforms that can keep up with the capabilities they need. You've had, I guess, historically, we had that compute cluster interconnect of InfiniBand, which was very specialized at the time, the lowest latency interconnect that you could get to now shifting gears to actually adapting Ethernet to be able to carry on at these substantially higher speeds and to be able to give a lot of the latency and performance guarantees that clearly you got with some of these specialized networking technologies like InfiniBand and at ever higher rates of interconnect capacity. once again, we've got ever tighter coupling between the different players in this environment, the Broadcoms, the Marvells, now Cisco with -- now that Cisco is a silicon supplier as well. Now you've got such tight coupling of their capabilities really being one more tight integrated piece of this, where Cisco Live is going on. There's been a lot of other recent conferences that we've seen. The fact that Cisco is out there pitching the innovations of their silicon capabilities as a stand-alone device just seems to me like we're in a very different territory right now.

Mike Fratto   00:12:13

Yes, it is very different. And part of that difference is the demand, especially from the hyperscalers and the cloud scalers, which is where a lot of the revenue opportunity is right now, is so strong that the traditional standards processes just simply can't keep up. How long the data center bridging take for the whole sort of constellation of protocols take to come from inception to standards. And those couple of years have been several years, I think more, I'm getting old and forgetting. But I do recall that it was surprising that it was so fast for standards processes, which tend to be glacial, which because they're being thoughtful and their.

Eric Hanselman   00:13:01

It's a consensus process, and it takes a while to sort out. But yes, to your point, it took a while.

Mike Fratto   00:13:07

It took a while, but now it's like we have all of these different technologies that are coming to market, which are essentially proprietary, and I know that's considered a bad word, but they're proprietary, but they're effective within their own sort of ecosystem of technologies. And so it's going to be there's going to be competing and competition between different chip makers, different network vendors and so forth, not so much because they want to lock in the technology, but because they've got different viewpoints and opinions for what's best and what's the most sort of effective path forward.

Eric Hanselman   00:13:45

Well, in some ways, you've now got a different community that's driving those requirements. You've got NVIDIA as specifying standards for interconnecting with Spectrum-X and things like that because that's what they need to interconnect in their environments and because you've got such a strong focus on being able to scale the cluster itself, a strong focus on a single vendor, all these capabilities, you've now got networking vendors who are all ensuring they can bolt up to Spectrum-X and the capabilities that are in there.

Mike Fratto   00:14:19

Yes. And Cisco with their Silicon One and their fully scheduled fabric is a closed -- tightly closed technology. Broadcom has their -- what they call it cognitive eCMP and there are others. But they're definitely serving a very -- they're serving a very large but sort of limited in scope sort of problem. So it's not like this is -- they're competing over the general enterprise or service provider networks. It's really very much in these clusters, right?

Eric Hanselman   00:14:55

Very dedicated clusters that are a huge concentration of computational power, networking density, power density, the entire thing, all in a much more tightly integrated block, which in order to operate at these dramatically elevated rates, you got to be. Although I did notice that in some of the Cisco Live keynotes, the network troubleshooting problem that they happen to be working through was a spanning tree protocol problem, which I keep thinking we put a stake in the heart of things like spanning tree, but I guess not if it's on stage, it's Cisco Live. But I also -- while there's this greater focus on what that cluster capability looks like, we've also got a shift in the providers that are playing in this -- in the networking arena. One of the things I thought was interesting coming out of Dell Tech World is Dell is selling a lot of networking gear. And we -- maybe, I don't know, 7, 8 years ago, as we were moving through that merchant silicon piece, you had folks like Cumulus Networks, who were going to give us an overlay operating system and you could run it on any hardware and all the hardware was all going to be roughly the same because the silicon was all roughly the same. We're now at a point at which the silicon is really high performance, and there's a shift towards looking at open source efforts around the software. So converting that. You've got efforts like SONIC, which grew out of Microsoft's data center requirements for high-performance switching. -- that you've got some part of this market right now is saying, oh, the important part and the high value of networking is now moving up the stack. So the operating system for the switch of the router is no longer a big deal. I guess, differing viewpoints on how we're moving through open source and some of those generic networking options.

Mike Fratto   00:16:55

Yes. The options like SONIC are interesting. I think they're more interesting for those kinds of organizations that want to have more of an engineered solution or -- I hate to call it white glove, but like a managed solution, right? This is what Dell does really well. They have a whole professional services engagement built around their networking and their sort of full stack engagements. And that's where it fits in really well. We're using SONIC because it takes away a lot of the uncertainty of using this open source platform, not that SONIC is unreliable, but there's all of the soft things that come along like having to learn SONIC and implement it into your tooling and how to manage it and how to troubleshoot it and all of those things. Sonic works out really well for those sorts of organizations that are -- that have a technical expertise and talent, but they need some help getting them along to a full deployment. I still don't see...

Eric Hanselman   00:17:56

It's sort of more along the managed distribution model, something like the Red Hat for networking kind of thing.

Mike Fratto   00:18:02

Yes, with a little more of a managed or a little more of a professional engagement that goes along with it. Yes, that seems to be the path.

Eric Hanselman   00:18:10

Curated and where it seems to go. It's interesting to see that the -- Cisco is also offering SONIC as an option. So Well.

Mike Fratto   00:18:23

Yes. I mean they've been offering SONic for a long time. All the big network vendors have offered SONIC because they want to cater to that hyperscaler or cloud scale customer that wants to have SONIC or some other open source software on their silicon.

Eric Hanselman   00:18:39

As always, it's a matter of looking for options. And I guess one of the things -- I was at the Open Network Users Group a few weeks ago. And there was a discussion about when do we start to shift data center interconnect fully to optical because it has always been the promise for a long time, which is hey, fiber connections, you got power advantages, you've got all sorts of goodness that starts to come along with that. And yet a lot of the participants in ONUG, some major financial services organizations, a set of major logistics companies, all sorts of folks who are working on this stuff. They're still in the process of getting themselves fully on to the optical options.

Mike Fratto   00:19:22

Did they say why that is? I suspect it's partly the availability of co-packaged optics and other optical components.

Eric Hanselman   00:19:29

Well, especially when you look at top of rack and R, those sorts of things, clearly, there's a lot of good optical options there. But I think I was actually doing a panel at both Broadcom and Marvell folks on there, they were talking about a couple of different things, and most of it was the readiness of the ecosystem. because you're mentioning co-package optics, the idea that you're now actually delivering optical interconnect directly to the device, to the GPU. It's one of the things that NVIDIA has been talking about and they -- when they get up to Verubin, they're talking about actually starting to standardize some of the co-packaged optics that are around there. But you've got to be able to go prove out the problem right now. Of course, if you look at optical interconnects throughout the data center, one of the challenges is reliability. So if you've got your various lasers that are in the plug-ins for the optical stuff, if one fails, you can swap it out. If that happens to be bundled into the GPU device that's within your system, that winds up being a little harder to swap out. it seems like a lot of people working on ensuring that the reliability of co-packaged optics is there that you actually might be able to do swappable lasers. To me, the fact that there's still a lot of discussion around it says that maybe it's still not quite so mature as it needs to be able to get there, but I guess we'll have to see where that shakes out.

Mike Fratto   00:20:56

Yes. I think the benefits of co-packaged optics is going to drive the development of the technologies just in power savings and all of the downstream power savings that derives from. But yes, there's still some things that's interesting to think about some of the challenges if there's a failure in the hardware all of a sudden, you're limited -- you may be limited in what you can do, right?

Eric Hanselman   00:21:23

Yes. It's how far back do you have to go to actually resolve the problem and let alone diagnosing the problem, sorting this out, and it gets right back to that whole question that we started with, which is how do you keep all of this really expensive GPU infrastructure actually running at full capacity and that winds up being some of the biggest challenges. So what should we be looking for going forward? I'm curious what your take is on next stages and where we -- directionally, what we should be looking at?

Mike Fratto   00:21:55

So I think with interconnect across the wide area, I think there's going to be -- the emphasis is going to be on the operations and the automation and management plane of being able to connect all of the endpoints when they need to be connected, providing the right capacity, monitoring the traffic and being dynamic in how that ends up being driven. And then, of course, putting controls around that for things like cost, regionality and so forth, data sovereignty all of a sudden becomes a big component but has been, but shows up. So I think sort of in the wide area, it's more about services and software running those services and how they're being configured within the data center, all the action is really it's on the front end, the front end, so the part that connects to the enterprise network, the backside, that's just a stack. And so I would expect to start seeing more hardware and software that supports some of the standards that are coming out of the Ultra Ethernet consortium that might drive some unity around the networking protocols and the traffic management protocols that end up being used. And for probably the foreseeable future, we're still looking at a lot of vendor-specific technologies until those standards start to come into play.

Eric Hanselman   00:23:19

Well, it's coming back around to the point you made at the outset, which is if what you're trying to do is grind as much performance out of that capability as possible, we're going to be looking at specialized approaches, and that comes down to the various sets of enhancements that are going to be more de facto, I guess, than standardized. What are those trade-offs that we typically make? Cool. Well, this has been great, Mike. Thanks for all the insights.

Mike Fratto   00:23:49

Thanks for talking with me, and thanks for your insights. I always enjoy coming on and sharing what I know and what I don't.

Eric Hanselman   00:23:55

Well, we got links to some of the research in the show notes to show the stuff that we do know and we'll go from there. But we are at time for this episode. Thanks to our audience for staying with us, and thanks to our production team, including Sophie Carr, Feranmi Adeoshun, Kyra Smith and Dylan Scheible and events teams. If you like this episode, please subscribe or like us. I hope you join us for our next episode because there is always something next in tech!