NVIDIA Enables New Experiences in the Automotive Industry with End-to-End Large Models

The automotive industry is experiencing fundamental changes, where AI-based algorithms, computing, and data are continually powering the evolution of the intelligent automotive industry.

Autonomous driving (AD) has evolved from the first-generation AD technology to the gradual introduction of AI and end-to-end large models. Integrating end-to-end large AI models allows for management and optimization of the entire automotive technology stack. The next-generation AD systems will not only improve the accuracy of decision-making, but will enhance vehicles’ capabilities to learn and adapt through generative AI and robust computing platforms.

The emergence of generative AI takes AI to new heights and brings far-reaching impact to the automotive industry. As a global AI enabler and leader, NVIDIA is committed to bringing AI to every aspect of vehicles through its leading NVIDIA DRIVE technology.

In addition to delivering generative AI and end-to-end large AI models, NVIDIA and its partners are designing and building digital twins of AI factories for vehicle production. The factory of the future will leverage autonomous mobile robots (AMRs) to significantly improve digital factory operations.

“We’re building a new type of data center. We call it an AI factory. The way data centers are built today, you have a lot of people sharing one cluster of computers and putting their files in this one large data center. An AI factory is much more like a power generator. It’s quite unique. We’ve been building it over the last several years, but now we have to turn this into a product. It will be everywhere. Cloud service providers will build them, and we’ll build them. Every biotech company will have it. Every retail company, every logistics company. Every car company in the future will have a factory that builds the cars—the actual goods, the atoms—and a factory that builds the AI for the cars, the electrons. " said Jensen Huang, Founder, and CEO of NVIDIA.

Empowering automotive manufacturing with AV 2.0 technological innovation

AV 2.0 was first proposed by Wayve in 2017. In contrast to AV 1.0’s focus on refining a vehicle’s perception capabilities using multiple deep neural networks, AV 2.0 calls for comprehensive in-vehicle intelligence to drive decision-making in dynamic, real-world environments. AV 1.0 enables Advanced Driver Assistance Systems (ADAS) or NOA for automated driving in motorways and urban environments. For fully autonomous driving and the highest levels of safety, a transition to AV 2.0 is required, which is a unified model of the larger converged world. Built on either GPT or Transformer, it is a multimodal large language model (LLM) that includes visual language models, generative AI, and other technologies.

Xinzhou Wu, vice president of automotive at NVIDIA, believes that generative AI will bring significant improvements to total productivity, including many aspects of human life, and that AV 2.0 is inseparable from generative AI.

Generative AI is propelling AV 2.0, a new era in autonomous vehicle technology characterized by large, unified, end-to-end AI models capable of managing various aspects of the vehicle stack, including perception, planning and control.

NVIDIA is actively steering the transition of AD technology to the AV 2.0 era, creating safer, smarter and more efficient AD solutions to partners by using end-to-end AD and other technologies.

End-to-end autonomous driving is critical to achieving safe and reliable general autonomy. NVIDIA researchers built Hydra-MDP, an end-to-end driving system for accurate perception and robust decision-making under a unified transformer model. Recently, NVIDIA won an Autonomous Grand Challenge at the CVPR conference with the Hydra-MDP model. In the contest, Hydra-MDP ingests one second of vehicle trajectory history and camera and LIDAR data at a reduced frame rate of only two frames per second, and generates the next four seconds of optimal vehicle path as an output. Hydra-MDP's simplified end-to-end architecture optimizes pipelines with less code and better performance. The Hydra-MDP model can learn from real-world and simulated driving data. This enables easier handling of rare corner cases and dangerous scenarios, as well as the ability to more easily mimic human driving, providing a more comfortable and predictable experience. And with NVIDIA Omniverse Cloud Sensor RTX APIs, AV developers can test and validate the results of AV models in physically-based virtual environments.

Photo credit: NVIDIA

Deep integration of generative AI and AD

The integration of generative AI with AD is a key element to AV 2.0 technology innovation. Generative AI’s contextual understanding, creative output and adaptive learning capacities mark a new era. Across the entire auto industry, companies are exploring generative AI to improve vehicle design, engineering, and manufacturing, as well as marketing and sales. Beyond the automotive product lifecycle, generative AI is also enabling new breakthroughs in autonomous vehicle (AV) development. Such research areas include the use of neural radiance field (NeRF) technology to turn recorded sensor data into fully interactive 3D simulations. These digital twin environments, as well as synthetic data generation, can be used to develop, test and validate AVs at incredible scale.

“Accelerated compute has led to transformative breakthroughs, including generative AI, which is redefining autonomy and the global transportation industry at large,” said Xinzhou Wu, vice president of automotive at NVIDIA. What began with the transformer model discovery has since unleashed incredible results, supported by massive models whose training has been made possible with leaps in performance from NVIDIA accelerated computing.

For example, Li Auto, one of China's new electric vehicle manufacturers, is deeply engaged in the field of intelligent driving. By utilizing NVIDIA's-acceleration solutions from the cloud to the vehicle, and by adopting the fusion strategy of System 1 (intuitive quick reaction) and System 2 (deep thinking) thinking, Li Auto accelerates the imitation of the human driving decision-making process, and enhances its adaptability to complex traffic environments. At the same time, NVIDIA uses cloud-based training and reasoning to help Li Auto's AI assistant "Lixiang Tongxue " provide a three-dimensional spatial interaction experience in the intelligent cabin, creating a more natural and personalized in-vehicle space. In addition, Li Auto has selected the NVIDIA DRIVE Thor centralized in-vehicle computing platform to empower its next-generation models.

NVIDIA DRIVE Thor is an in-vehicle computing platform architected for generative AI applications, which are becoming paramount within the automotive industry. DRIVE Thor, the successor to DRIVE Orin, can deliver feature-rich cockpit capabilities, plus safe and secure highly automated and autonomous driving, all on a centralized platform. This next-generation AV platform will integrate the new NVIDIA Blackwell architecture, designed for transformer, LLM and generative AI workloads, which was announced during NVIDIA founder and CEO Jensen Huang’s keynote at GTC.. NVIDIA DRIVE Thor is poised to revolutionize the automotive landscape, ushering in an era where generative AI defines the driving experience. 

BYD, the world's largest electric vehicle manufacturer, is working with NVIDIA on multiple parts, extending not only from the vehicle to the cloud, but also including AI factories, digital twins, vehicle design, and automotive smart manufacturing, etc. In addition to building its next-generation EV fleets on DRIVE Thor, BYD plans to use NVIDIA’s AI infrastructure for cloud-based AI development and training technologies, along with the NVIDIA Isaac™ and NVIDIA Omniverse™ platforms to develop tools and applications for virtual factory planning and retail configurators. 

XPENG has also announced it will use the NVIDIA DRIVE Thor platform as the AI brain of its next-generation EV fleets. The next-gen car computer will power the EV maker’s proprietary XNGP AI-assisted driving system, enabling autonomous driving and parking capabilities, driver and passenger monitoring and other functionalities.

The future of intelligent vehicles has arrived     

NVIDIA is committed to fully integrating AI technology into the automotive industry to drive      continued progress in the AI era. NVIDIA has put a lot of thought into the DRIVE Thor platform not only on the vehicle side, but also on the cloud side, to support the development of the future of AI vehicles, and to increase the computational power of the simulation models to a whole new level. The NVIDIA DRIVE Thor ecosystem is rapidly expanding to build on a powerful computing platform and to meet higher security standards and support generative AI and large-scale language models that will drive the AI-defined automotive industry forward.

The rapid development of AI technology and application of end-to-end technology are reshaping the automotive industry. NVIDIA is transforming all stages of the automotive product lifecycle with cutting-edge technology, embracing the future of intelligent vehicles that are smarter, greener, and more user-friendly.