Driverless cars: mapping the trouble ahead

When a self-driving car looks at the world, there are many things it sees. It has radars that measure distance to the next car, it has cameras that take in colour images of the street and its Lidar sensors send out laser pulses that gauge the surroundings. For any robot-driven car, one of the most important components of the journey is not just what it sees but what it knows beforehand about the area it is travelling through. 

The robot needs a map, but not just any map — these cars need a three-dimensional representation of the environment around them, one that is updated continuously and is accurate down to the centimetre. As it cruises through the streets, a self-driving car collects more than a terabyte of data a day, enough to fill 1,400 CDs. With that much detailed information coming from the car’s many sensors, however, it is uneconomic to send it through a network like the internet.

Instead, companies have to physically move the data from one hard drive to another, a process sometimes called the “sneakernet” because engineers joke that the hard drives move at the pace of their own footwear. 

The data collection is part of a great race to amass knowledge about the physical world that can be used to train the new generation of cars. Researchers hope that eventually the base layer of information will have applications not just for transport and logistics, but also for the development of augmented reality technologies — becoming like a simulation of the real world that can be used by any robot, drone or car. 

The first step in realising this potential, however, is the development of effective digital mapping technologies for  self-driving cars. The cumbersome storage of data is just one of the technical issues that are occupying many of the brightest engineering minds in Silicon Valley. Without better 3D maps, the much-hyped self-driving car revolution will be much slower to materialise.

“This is a very hard problem,” says Brian McClendon, a mapping expert who previously ran Google Maps and co-founded the company that became Google Earth. Mr McClendon, who led Uber’s mapping efforts after he left Google, departed from Silicon Valley last year to go into politics in Kansas. He is an adviser to DeepMap, a mapping start-up founded by former Google colleagues.

The reason these maps are so important for self-driving cars, he says, is not only for location but also “because it reduces the amount of work that the autonomous software has to do to recognise the world around it”. 

By comparing their actual surroundings to what was predicted in the map, he says, they can focus their attention only on things that are different, like identifying a pedestrian or a bicycle. 

Investment in autonomous vehicle research has reached record levels in the past year and along with it has come a surge of funds to improve mapping. Start-ups such as Civil Maps, DeepMap and Lvl5 have attracted mapping engineers from Google, Apple and Tesla, and raised more than $40m in funding.

Meanwhile, the biggest autonomous car companies all have their own mapping systems. Alphabet’s mapping prowess is seen as a key advantage for its self-driving car unit Waymo, which has already completed more than 4m miles of autonomous driving. (Alphabet owns Google Maps, Google Earth, Google Street View and the navigation app Waze, which tracks real-time traffic).

Within the industry there is a vibrant debate about whether to call these visual representations “maps” at all, such is their complexity. The information collected can be roughly divided into layers: the physical location of the pavements, buildings and trees; road signs and traffic lights; and how the self-driving car should behave, such as observing the speed limit. Accuracy is so important that even small changes, such as the shifting of tectonic plates that move a few inches a year, can have an impact. 

“The word ‘map’ is an inaccurate way of describing it,” says Wei Luo, chief operating officer at DeepMap. She prefers to think of it as a piece of software that feeds the car the information about its surroundings. Her colleague James Wu, DeepMap’s founder, describes these maps as the “part of the brain” of the autonomous robot that allows it to understand its location.

“I tend to see the map as the ‘collective memory’ of all self-driving cars,” says Ralf Herrtwich, head of automotive maps at Here, which is mostly owned by German carmakers after a consortium bought the map-making unit of Nokia for $2.8bn in 2015. “It’s almost like a driving school for autonomous vehicles,” he jokes. 

Regardless of what they are called, making these maps is extremely difficult. The huge volume of data used in the maps is one dilemma. Another challenge is keeping them updated continuously, so that they provide the latest information to the cars.

“A lot of companies have not figured out how to actually store their data,” says Sravan Puttagunta, chief executive of Civil Maps, a mapping start-up. “That is why autonomous vehicles are geofenced. They physically cannot fit the data in the trunk of the car, so they are restricted to certain areas,” he says. 

Civil Maps is trying to deal with this problem by simplifying the map data so that it is easier to manage, but there is no single industry standard that has won out yet. Moreover, the artificial intelligence needed to generate these maps is still far from perfect. Humans are often needed to check the labelling on maps, assess the need for any updates and analyse why the cars make mistakes during test drives. 

“One thing that is really not talked about with AI is how much human work is really required behind the scenes to get this technology to really work,” says Alexandr Wang, an engineer whose company, Scale API, provides human trainers for AI. 

“As these companies try to produce autonomous vehicles they need a bunch of humans to go through and meticulously label these maps, just like Google Maps back in the day.” 

Another challenge is the deep fragmentation of the sector, with no obvious common standard for these high-definition, 3D maps, nor any sharing of data because companies consider this to be important proprietary information.

“Everyone is trying to develop their own in-house HD map solution to meet their self-driving needs, and that doesn’t scale,” says Mr Wu of DeepMap. “It’s all reinventing the wheel, and that’s wasting a lot of resources. That will probably be one of the reasons to block self-driving cars from becoming a commodity.” 

Because companies do not share mapping data and use different standards, they must create new maps for each new city that they plan to enter. “It will delay the deployment in certain geographies,” Mr Wang says. 

Different driving rules in each city also mean that further software tweaks must be made. “You will have to redo the software in every geography that you enter,” he says.

The emerging demand for specialised 3D maps has created a race between incumbents such as Here and TomTom, the satellite-based navigation device maker, and upstarts such as DeepMap. While the start-ups are focused only on maps for fully self-driving cars, Here and TomTom believe that high-definition maps will be useful even before the mass adoption of autonomous vehicles, because they will help with advanced driver assistance technologies. 

Willem Strijbosch, head of autonomous driving at TomTom, says the maps needed for driverless cars are different from the current map applications because they will need to “serve a safety critical function”, rather than just being used for navigation. 

“Another change is that you can no longer use GPS as your only means of localisation in the map,” he adds, because the global positioning system is not precise enough for self-driving cars.

The traditional automotive mapmakers believe high-definition maps will become a big source of revenue in the next few years. 

“It is like the evolution in the TV, going from traditional TV to HD and 4K, once that wave breaks there is no going back,” Mr Herrtwich says. “We see that much of our future maps we will be selling will be HD.”

Industry participants agree that a wave of consolidation is inevitable. “If you look at the number of companies in this space, it doesn’t fit the economics,” says Mr Strijbosch of TomTom. “There are very high fixed costs, so only a couple of players will make it to the end.” 

The possibility of a regulatory backlash adds to the technical issues for the sector. Just as the launch of Google Earth prompted some homeowners to insist that the shapes of their properties be scrubbed out of the images on privacy grounds and Google Street View blurred licence plates and human faces, so too will HD map makers have to deal with privacy concerns. Although most of these maps are not directly available to consumers — they are made to communicate with the robot brain, not the passenger — the high level of detail they contain could raise alarm among privacy advocates. 

Anthony Foxx, a former transport secretary in the Obama administration, says mapping and sharing information across vehicles are two areas where the government should get more involved. “There is going to need to be serious industry-government engagement on these issues,” he says. “It is not just mapping, it is things like cyber security and other arenas, what happens in one vehicle can and should be shared.”

Even with these reservations, the broader tech sector is excited about the potential spin-offs from 3D mapping, hoping that it could enable other technologies. The race is on to collect the most complete data set and once that happens it will be useful for technologies such as augmented reality, which requires a perfect map of the world.

Some in the industry believe that, once cars can roam autonomously, other types of robotic devices could be fitted with mapping software.

“I think all of robotics is going to follow suit and go in the direction of self-driving cars,” says Mr Wang. “Eventually every robot that we build in the future is going to involve the same suite of sensors.” 

Researchers expect that self-driving cars could eventually become less dependent on maps, as their Lidar (light detection and ranging) sensors improve enough to allow them to navigate their environment. The maps will be crucial to ensuring safety during the early years of self-driving cars but eventually, optimists hope, the technology could move past that.

“Long, long term, Google Maps in its current form is all you need, because cars will be as smart as humans and solve all the daily existential problems,” says Mr McClendon. “But that is many, many years from now, when your car is equivalent to a human.”

Graphics: Ian Bott

Letter in response to this article:

The threat to progress on preventing road deaths / From Antonio Avenoso, Executive Director, European Transport Safety Council