Data & Dimensions | Ep. 8 - How Pokémon Go Guides Delivery Robots

View Full Transcript

Neil Barbour   00:00:00

Welcome to Data & Dimensions, a podcast at the intersection of entertainment, industry and interactivity. I'm Neil Barbour, an analyst with S&P Global Market Intelligence, Kagan. And with me today is Brian McClendon, CTO at Niantic Spatial. Brian was part of the team at Silicon Graphics in the '90s, was an early investor in geospatial engineering company, Keyhole, and was a director of engineering for Google Geo, which built the basis for Google Maps. He would later join Pokémon Go developer Niantic as vice president of augmented reality research and mapping. Brian, thanks for joining us.

Brian McClendon   00:00:39

Happy to be here.

Neil Barbour   00:00:40

I really want to get into the story of how Pokémon Go is helping power practical robotics, but you seem to have been attached to so many seminal projects over the years. Can you tell me what it was like working at Silicon Graphics, which played such a pivotal role in the look and feel of special effects and movies?

Brian McClendon   00:00:54

Silicon Graphics is one of the best places to work that I think any engineer could hope for. We were building the most modern 3D graphics. We were building GPUs before GPUs existed. Our GPUs were 3 very large VME boards that cost $100,000. But at the time, they were the fastest graphics in the world. And many of the people from those teams went on to work with AMD or NVIDIA and build the consumer graphics that we all use today. But as a pioneer in 3D graphics, high-performance 3D graphics, we ended up seeing all the amazing applications that 3D can be applied to and we're able to be at the beginnings of flight simulation and computational fluid dynamics and of course, movie editing and special effects. So Maya and 3D Studio Max both ran on SGI machines and many of the early CGI movies, Jurassic Park, Terminator 2 were all done entirely on SGI machines.

Neil Barbour   00:02:02

Yes. And maybe talk a little bit about what it was like to work on Google Maps. I think a lot of us regard that as the first killer app of the smartphone era.

Brian McClendon   00:02:10

Well, the transition from 3D graphics of the '90s to the more powerful or higher volume use case of mobile phones was bumpy. $1,000 is not a reasonable price for 3D graphics. And so NVIDIA worked hard on that. And by 2002 or '03, we had a relatively low-cost consumer graphics card that could run this early thing that would become Google Earth. And in 2004, Google acquired us and turned this app Keyhole into Google Earth. And initially, we focused on satellite imagery, but immediately joined with the Google Maps engineers that were there and built both Google Maps, Google Earth and then Street View and so on. And the change in audience from having a few low thousands to tens of thousands, hundreds of thousands of users to almost immediately jumping into 100 million users at Google was breathtaking. When we launched Google Earth in June of 2005, we basically nearly took down Google because we used more than half their bandwidth, and it was extremely popular. But I think the real takeoff on user base didn't occur until we went mobile. And so Google Maps for mobile launched in 2008, 2009. And by 2010 or '11, everybody was using Google Maps mobile on iPhone, on Android and getting turn-by-turn directions and eventually replacing all of those dedicated personal navigation devices that had existed before.

Neil Barbour   00:03:52

And certainly replacing MapQuest or printing out directions. So let's fast forward a little bit more to another killer app of the mobile era. Describe your work with the original Niantic and Pokémon Go.

Brian McClendon   00:04:04

So Niantic Labs was actually formed inside Google by John Henke, and he wanted to build more geospatial consumer apps that sort of helped explore the world. And the first 2 apps were Field Trip and Ingress but they were successful enough that the Pokémon company wanted to build a version of Pokémon Go based on mapping and based on this technology. So John spun the company out of Google, which was a challenge into itself and formed Niantic Labs and launched Pokémon Go in 2016. And one of the things that I don't think people fully understand is that when it launched in July, for the next 2 months, 500 million people installed that app. And there is no other app in the world that can claim that. even to this day. And many of us do remember sort of those videos of floods of people crossing intersections to chase after the Snorlax or whatever. It was incredibly popular. But from John's perspective, it was also terrifying. He had to somehow keep the service running with 500 million users. And there was all sorts of challenges with that and scaling the company up. But Pokémon Go survived, continued to retain users and made $1 billion in 2016 and is making more than $1 billion today.

Neil Barbour   00:05:28

Pokémon Go is now owned by mobile games publisher, Scopely after a $3.5 billion deal. And now you're with a spin-off that's doing something a little different, but still building on that success. Talk to me about the data you were able to collect from Pokémon, how that has turned into precision geospatial modeling.

Brian McClendon   00:05:46

So somewhere around 2018, John got the idea that we could have Pokémon Go players scan PokéStops to help model them and collect that data to build a very highly detailed 3D model of some sort of narrow portions of the world. And so Pokémon Go players had motivation to put -- take out their video camera and record for 30 seconds to a minute. They would upload those videos and they would get in-game rewards for doing so. And those videos became the basis of our sort of augmented reality localization capabilities, which we call the visual positioning system. And this surfaced in Pokémon Go, where players could point their camera at a Pokeop leave one of their Pokémon in a specific location with a specific pose and then all other players could then visit and see that -- see the Pokémon in the same place and also join it themselves. So up to 6 or 8 of the Pokémon could be put there. And so Pokémon Go benefited from it, but we also were able to build this location map of all these PokéStops with high-resolution 3D data and then offer that service to developers through an SDK and also to enterprises to collect their own equivalent of PokéStops to build models of the world.

Neil Barbour   00:07:09

So the way I first thought about Niantic's visual positioning system is, let's say, you're on the same block as your friend and you both have Find my Friend turned on, right? And if you look at Apple Maps now, it gives you an approximation of where your friend is within around like 100 feet or so. This tech potentially cuts that down to maybe just a few feet, so you could walk right to your friend in the crowd. Am I thinking about that correctly, generally speaking?

Brian McClendon   00:07:34

There's a lot of gradations to what a visual positioning system can do. At one level, yes, that's exactly what we do. We can improve upon the accuracy of GPS, especially in urban areas. And this is a technology related to, but not fully dependent on the product we talked about before. The visual positioning system at the precise level is more about. I can tell you, I know exactly where you're standing within a few centimeters, and I know exactly which direction you're looking. And so if there are 2 people that are both looking at a sort of virtual game object or a virtual project, if they're an enterprise, they both see exactly the same thing in the same place and can talk about it. And one of them can be at the location or another can be remote and seeing a 3D representation in VR goggles or whatever. And so this idea of shared understanding and precise referenced location is enabled by VPS.

Neil Barbour   00:08:35

Right. Yes, we talk a lot about the importance of the digital twin and sharing data like up and down the stack from the physical location to the remote location and how that can unlock certain workflows or commercial opportunities. So I think the first time I saw Niantic Spatial in action was like late last year, you were partnering with defense contractor, Aechelon, and you were showing how first responders could efficiently access a crisis one. That's still part of the stack. That's the same technology enabling?

Brian McClendon   00:09:03

Yes. So that's an enterprise application of very similar technology where the long-term plan is for a emergency responder who wants a Coast Guard helicopter to come in and land and take off a person who needs to be taken the hospital, but is in a dangerous location. This would allow the emergency responder to scan the location with their phone, create a 3D model and give it to the pilot ahead of time so the pilot understands exactly what the lay of the land is because helicopters are not easy to land, and it really matters where trees are, where -- which way the wind is blowing and being able to preview that accurately and even simulate it in the simulator before you go out there will increase the safety for Coast Guard and give them more opportunity to go out and apply these things. So that was definitely one of the first applications where sort of a very bespoke collection of data for what is probably a one-time use case but could save lives.

Neil Barbour   00:10:13

The question I have is that these are areas you probably don't have data from the Pokémon Go experiments. So is it -- it's inferring certain topography based on data you already have in the model?

Brian McClendon   00:10:25

Well, there's some amount of data that exists that's publicly available and whether it's satellite imagery or terrain data, you can have a base map, but the Coast Guard already has that. What they need is the precise detail of the current location. And so we have an app called Scaniverse that effectively does the same thing that Pokémon Go did, but for enterprise users. So you can take Scaniverse out and scan that location and upload the data and then build a 3D model that can then be simulated. And so think of that as sort of an enterprise version of the prior Pokémon Go scans.

Neil Barbour   00:11:03

The most recent partnership I've seen is the tie-up with Coco Robotics, which offers last-mile delivery robots in urban areas… cities. What are the challenges that companies like Coco face? And how is Niantic Spatial addressing those issues?

Brian McClendon   00:11:18

In urban locations, it turns out that GPS is not terribly accurate. And you may have experienced this yourself if you come out of a subway station and you see your blue dot actually showing up half a block away. The reason is these tall buildings reflect GPS signals, and they give a sort of illusion of being in a different place. And so the GPS does a lot of work to try and figure that out, but the errors can be up to a city block wrong. And if you're a delivery robot, you can't be that far off. You literally will be on the wrong block and get very confused about where to go. And so CoCo is looking to improve the accuracy of their location -- and our service allows them throughout a city to improve that quality to the point where they can be pretty confident about which side of the street they're on, where in the sidewalk they're on and then make the decisions about how to navigate from there.

Neil Barbour   00:12:14

I recently saw a video of a delivery robot crashing into a bus stop and shattering the glass. I can only assume it was not outfitted with Niantic VPS, but it does underline some of the challenges here. There are real people and property on these streets and robots may or may not be equipped to navigate real-world conditions. Is this tech in good enough shape to be working with live rounds like that?

Brian McClendon   00:12:34

I think the advantage of delivery robots in general, notwithstanding that particular example is that they're usually not very big and very heavy. And therefore, the risk -- because they move very slowly, then the risk to humans is much lower. Self-driving cars face a much different problem because they are fully capable of killing someone. And so their need for location tech is much higher still. And that's actually one of the limiting factors in the rollout is that Waymo maps a city very, very carefully before they start driving it. And until they have that map, it isn't safe. And I think with robots, the bar is lower, but it is not 0 and the robots really still need to know where they are.

Neil Barbour   00:13:26

So if you would say like a Waymo is self-driving Level 4, do you put the CoCo robots on a similar level just at lower stakes?

Brian McClendon   00:13:34

I think they should be measured with the same kind of levels. And because the stakes are reduced, it allows them to roll out, I think, more quickly. But I think that the measurement of quality is still the same, how much of it is assisted, how much of it is automated and how many different scenarios can the robot handle? Like Waymo handles most scenarios, but they intentionally navigate around those scenarios that they don't handle and delivery robots know what areas they can and can't go based on capabilities or risk factors, and they will avoid those where possible.

Neil Barbour   00:14:14

So I guess that takes me to my next question is once this is -- this tech is refined in maybe robotics or other low-stake scenarios, does it pull across to self-driving cars eventually? Is that kind of the end goal with this sort of technology?

Brian McClendon   00:14:29

I think at the end of the day, we kind of know how many cars there are going to be in the world, and they will eventually all be self-driving. Certainly, there will be self-driving supervision of human drivers because self-driving will be safer. But I think the real change will be when there are many more robots in the world. And delivery robots are a really good prototype for this because they are out there today. They're usually wheeled. They have a task. It is point-to-point. It's relatively simple. But long term, you'll see humanoid robots that need to get from arbitrary point A to arbitrary point B, and that will require navigation, not just along the streets, but in the stairwells and through doorways and all sorts of other complex situations. And all of these robots are going to need to have that positioning information so that they know where they are relative to the world around them.

Neil Barbour   00:15:28

Let me ask this. If an Amazon or other warehouse operator were to come to you and say, let's put this VPS on robots -- could this software slot right into there.

Brian McClendon   00:15:40

That would be a good application of it. I think though that the -- many of the factory floor robotics have, in many ways, already solved their mapping problem because they have a very limited floor plan. And so the robots are trained on that specific floor plan and never really leave the area. If you look at how Amazon operates its robots, they actually have a sort of segregated system where half of the delivery floor is robot and humans are not allowed. The second half is mostly human, but some robots are allowed, but they play by much different rules. The safety rules for that side of the fence are much higher. And if a human goes into the pure robot side, all the robots stop because they are designed to move with aggressive nature with low-risk, where the only risk is really a robot colliding with the robot.

Neil Barbour   00:16:37

So in a factory, there are a lot more known knowns and you could hand a robot, a specific digital twin and it could train itself and be ready to go. The difference with your tech is that out in the real world, there are a lot more unknown unknowns. And so this VPS is helping the robots deal with those scenarios.

Brian McClendon   00:16:53

That's right. The world is much bigger than a single factory floor and the variance of -- to minimize the complexity, you really need to know the map well because the world varies so much, the humans, the bicycles, the baby strollers and the cars are all very dynamic objects. And your perception system really needs to focus on those and not on the navigation aspect.

Neil Barbour   00:17:19

So it looks like there are a few projects coming to market over the next few weeks as well. Why don't you talk about advancements in Scaniverse and VPS 2? How is this moving the Niantic story forward?

Brian McClendon   00:17:29

Yes. So we're releasing a new version of Scaniverse, which has been our consumer scanning app. And so if you want to try out this new technology called Gaussian splats, we have discovered that there is a new way to 3D model the world. So this isn't about VPS and visual positioning. This is actually about visualization. And with Scaniverse, you can, with a simple mobile app, scan an object and create a 3D model of it that looks extremely real, and you can do this all on your device. And we've had this out for several years, actually 2 years with Gaussian splats, probably 4 years if you count the triangular meshes and texture meshes and so forth. But in the last 2 years, Gaussian splats have just radically changed the industry. They look better, they render faster and pretty much everybody who's working on visualization right now is working on moving to Gaussian splats. And they sound a bit silly, but visually, once you see them, you'll understand why they're differentiated. And our release is to enable this kind of capability, but for enterprise customers. So instead of just a single scan of a single object, you can scan relatively large areas and create 3D models that could be used for visualization, for construction validation, could be used for robotic simulation. So giving an environment that a robot could learn how to train itself in. But the most common one is actually for visual positioning where you scan this very large area and now you can point your phone or any camera in this area and we'll know exactly where you're standing and therefore, be able to place that object that you're looking at and yourself precisely in 3D.

Neil Barbour   00:19:23

To go back on Gaussian splats -- it's a way to realistically position a digital asset in the 3D space?

Brian McClendon   00:19:32

What it is, is it's a replacement for the previous way we looked at 3D graphics. So even back in 1991 at Silicon Graphics, we use textured meshes to render our 3D. And in every model you've seen, whether it was Google Earth or first-person shooter or any visual, it's almost always been in this textured mesh. But Gaussian splats, which were invented a couple of years ago by many definitions, look better, render faster and are able to visualize aspects of things that were not possible before. And we believe that many of the existing visualization capabilities are going to be replaced by Gaussian splats, both now and in the near future. And I think we're one of the leaders in that category.

Neil Barbour   00:20:21

It seems like there's a lot of crossover with smart glasses for this kind of tech as well. We recently talked to Sean Man of RP1 about servicing digital data and real-world settings through AR. I know you've partnered with Snap on their Snap spectacles before on some projects. What's your outlook for AR/VR hardware? Are you hoping to serve a lot of smart glasses or headsets and those kinds of use cases over the next decade?

Brian McClendon   00:20:44

So AR glasses have been talked about for many years. And with the release of the Microsoft HoloLens almost a decade ago, there was a lot of hope it was coming soon. And I think that we at Niantic, both at Labs and at Spatial, have wanted AR glasses to become more ubiquitous. And the challenge has been that visualizing on the glasses, the -- an accurate model of the world that tracks as you move your head around is very hard. And you've seen this with the Meta Ray-Ban specs. They didn't try to do it. They basically are an input-only set of glasses. The only output is the audio that you hear. And so the display side, even when they do display, like, I guess, a new version of them, they don't try to track the world around them.

Neil Barbour   00:21:37

It's just a flat heads-up display.

Brian McClendon   00:21:39

Exactly. And so the Snapchat spectacles are going to be the first consumer glasses that try the hard problem that really want to track the world around you, have 6 degrees of freedom of motion, but then render the world on top of it. And to do that accurately, you do need to know exactly where you are in the world. So if you have a map and you're standing somewhere, you need to know not just where you're standing, but where you're looking, if you want to overlay information on top of the world around you. And so Snap Spectacles, I think, are going to be one of the first use cases where VPS really matters for adding content and annotation to the physical world.

Neil Barbour   00:22:16

One last question I sort of kept coming back to this, it seems like maybe there's some concern or confusion about how Pokémon Go players data has been leveraged in Niantic Spatial. Do you want to take a minute to walk through how the data has been collected and how it's used?

Brian McClendon   00:22:29

Yes. I mean I think the biggest worry would be that somehow there is background collection happening while you're playing the game. And that absolutely was never the case. Each time a player chose to enable this feature of scanning a PokéStops for a particular reason, they finished scanning the video and they were asked, do you want to upload the video in exchange for this in-game award. It was a very conscious act. And it happened a lot, but in the grand scheme of the game was a relatively narrow use case. So there was on the order of 300 million of these, and they are very useful for creating this accurate model of these PokéStops. And that taught us, both Niantic Labs and Spatial a lot about reconstruction and about visual positioning. And we will continue to use that data to improve our models of the world. But there are scans that were consciously updated with multiple steps along the way that the players participated in. So it's no background activity.

Neil Barbour   00:23:32

Well, Brian, thank you for your time.

Brian McClendon   00:23:33

Thanks, Neil.