Search SciPol

Brought to you by
What it does 

Each type of sensor used by automated vehicles has its own strengths and weaknesses. Incoming data from individual sensors is combined to form the most optimal “view” of the driving environment. The driving system uses sophisticated machine learning algorithms to draw meaningful conclusions from such data. This process, known as sensor fusion, allows the vehicle to overcome the shortcomings of individual sensors caused by malfunction or condition-related impairments. For example, if the driving system must navigate through heavy rain or fog, its camera sensors may not detect objects clearly, while its radar sensors would maintain full ability to accurately measure the distance between itself and other objects. Therefore, sensor fused systems can improve accuracy, provide redundancy, and better equip the driving system to address suboptimal circumstances.

Relevant Science 

Because of the high volume and complexity of the incoming sensor data, sophisticated software algorithms and computer hardware is required to convert the data into useful information that can be used to make decisions about the operational domain. This section includes several examples.

  • Machine learning allows the computer to analyze input data and make an informed prediction or determination about that data based on rules programmed into an algorithm. The computer does not need control inputs to direct it to perform a specific task, but rather can correlate data inputs to its algorithm to perform tasks. While this process allows the machine to “learn” from past experiences and self-improve, machine learning algorithms are still limited because they are only capable of processing data that falls within the domain of inputs they are programed to handle.   
  • Deep learning (DL) is an advanced form of machine learning that allows computers to process larger quantities of data and recognize complex patterns in data to make decisions. Deep learning algorithms are more powerful and accurate than other forms of machine learning, but only when fed lots of high quality data. Autonomous vehicles need deep learning algorithms to fuse and interpret sensor data inputs; these algorithms allow the vehicle to detect patterns in sensor data, correlate patterns to its algorithm, and ultimately perceive and understand its environment. There are three main functions that deep learning sensor fusion will serve for autonomous vehicles, which are (1) localizing the vehicle relative to a ground map, (2) detecting and identifying objects in the surrounding environment, and (3) tracking objects in the surrounding environment.  
  • Deep reinforcement learning is a form of deep learning that allows machines to learn through experiencing positive and negative rewards. The machine interacts with its environment through trial and error, determining that a behavior is ideal when it produces a reward and determining that a behavior is non-ideal when it produces a negative reward. This trial and error process reinforces ideal behavior, allowing the machine to learn how to process unfamiliar input data. Common algorithms include Q-learningDeep Q Network (DQN), State-Action-Reward-State-Action (SARSA), Deep Deterministic Policy Gradient (DDPG), and others. 
  • Deep Neural Networks (DNN) are the algorithms that enable deep learning. The algorithm is constructed to mimic neural activity with the human brain; just as a brain is comprised of an interconnected web of neurons, the DNN is comprised of many hierarchal layers of interconnected neuron nodes. Data is fed through the hierarchal layers of neural nodes starting at an input layer, through hidden intermediate layers, to an output layer. Each layer receives data as input, classifies the data, and passes the data on as an output to the next layer in the hierarchy. Each subsequent layer of neural nodes is increasingly complex because it contains classification criteria based on the output of the previous layer; as data is processed through each layer, the algorithm is able to recognize its more complex features. On a microlevel, individual neural nodes within layers receive input data and assign a weight to the data that either strengthens or diminishes the input based on classification criteria. Once raw input data has fully passed through the network, the machine classifies the data based on its weights at neural nodes in each layer of the algorithm. This multistep process that allows for complex pattern recognition is illustrated in the figure below. Deep neural networks in autonomous driving systems control the system’s perception and therefore need to be trained on sufficiently large datasets that include all possible driving conditions.  

Deep Neural Network

(Nielsen, Bengio, Goodfellow & Courville, 2016)
neuralnetworksanddeeplearning.com

  • Convolutional Neural Network (CNN) is a type of DNN that contains a convolution layer in the algorithm network. The convolutional layer is only connected to some neural nodes and has multiple neural nodes that are weighted equally.  
  • Recurrent Neural Network (RNN) is a type of DNN that contains loops between layers of the algorithm network. The loops allow the outputs of each layer to feed back into previous layers of the hierarchy. This network may be advantageous for driving tasks that are temporally dependent such as correcting a steering angle across multiple time frames.  
  • Chipsets are the onboard hardware that manage communication between the processor, memory, and other devices of the vehicle. The chipset is generally a set of integrated electronic circuits designed to work with specific processors, but there is variation in chipset architecture between different autonomous driving system manufacturers. For example, classic chipsets in lower level autonomous driving systems integrate preprocessors that filter and crunch sensor data before sending it to a centralized processor, but higher level autonomous driving systems that cannot afford processing delays may combine multiple processing functions in a single centralized processing unit. In general, ADS chipsets must be very powerful, as they must have the computing power to carry out intensive deep learning algorithms. 
  • Processing units are hardware that fuse sensor data, perform calculations, and carry out commands of the deep learning algorithms. The fusion of data from sensors by the processing unit is illustrated in the figure below. There are multiple existing hardware implementations that vary in terms of impact on deep learning performance, computational capacity, and cost.  

A multi-sensor data fusion system

A multi-sensor data fusion system (Nutaq)

  • Central processing units (CPU) are general-purpose processing units. CPUs have high data management capacity and can perform a diverse range of tasks but cannot perform certain tasks with high accuracy. For example, CPUs cannot perform graphic and video processing efficiently. Consequently, CPUs are primarily limited to autonomous driving systems of lower levels of automation or are used in combination with other processing units. 
  • Graphics processing units (GPU) are processing units specifically designed to accelerate image and graphics processing and display. GPUs can perform parallel processing of images, allowing them to handle multiple independent tasks simultaneously based on sensor inputs. This feature makes GPUs helpful for 3D object mapping, image translation, and other computer vision functions. GPUs have outstanding computational capacity but consume high amounts of power. GPUs are the core technology currently leading the market
Primary Author 
Nicole Kastelic
Editor(s) 
Michael Clamann, PhD, CHFP
License 
Creative Commons License This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. Please distribute widely but give credit to Duke SciPol, linking back to this page if possible.