ROS2 End-to-End Lane Following Model with LGSVL Simulator

This tutorial works with Simulator Release 2019.05

This documentation describes applying a deep learning neural network for lane following in LGSVL Simulator. In this project, we use LGSVL Simulator for customizing sensors (one main camera and two side cameras) for a car, collect data for training, and deploying and testing a trained model.

This project was inspired by NVIDIA's End-to-End Deep Learning Model for Self-Driving Cars


Table of Contents#

Getting Started#

First, clone this repository:

git clone --recurse-submodules

Next, pull the latest Docker image:

docker pull lgsvl/lanefollowing:latest

To build ROS2 packages:

docker-compose up build

Now, launch the lane following model:

docker-compose up drive

(Optional) If you want visualizations, run drive_visual instead of drive:

docker-compose up drive_visual

That's it! Now, the lane following ROS2 node and the rosbridge should be up and running, waiting for LGSVL Simulator to connect.


  • Docker CE
  • NVIDIA Docker
  • NVIDIA graphics card (required for training/inference with GPU)


Installing Docker CE#

To install Docker CE please refer to the official documentation. We also suggest following through with the post installation steps.

Installing NVIDIA Docker#

Before installing nvidia-docker make sure that you have an appropriate NVIDIA driver installed. To test if NVIDIA drivers are properly installed enter nvidia-smi in a terminal. If the drivers are installed properly an output similar to the following should appear.

    | NVIDIA-SMI 390.87                 Driver Version: 390.87                    |
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |   0  GeForce GTX 108...  Off  | 00000000:65:00.0  On |                  N/A |
    |  0%   59C    P5    22W / 250W |   1490MiB / 11175MiB |      4%      Default |

    | Processes:                                                       GPU Memory |
    |  GPU       PID   Type   Process name                             Usage      |
    |    0      1187      G   /usr/lib/xorg/Xorg                           863MiB |
    |    0      3816      G   /usr/bin/gnome-shell                         305MiB |
    |    0      4161      G   ...-token=7171B24E50C2F2C595566F55F1E4D257    68MiB |
    |    0      4480      G   147MiB |
    |    0     17936      G   ...-token=5299D28BAAD9F3087B25687A764851BB   103MiB |

The installation steps for nvidia-docker are available at the official repo.

Pulling Docker Image#

docker pull lgsvl/lanefollowing:latest

What's inside Docker Image#

  • Ubuntu 18.04
  • CUDA 9.2
  • cuDNN
  • Python 3.6
  • TensorFlow 1.8
  • Keras 2.2.4
  • ROS2 Crystal + rosbridge
  • Jupyter Notebook


  • Training mode: Manually drive the vehicle and collect data
  • Autonomous Mode: The vehicle drives itself based on Lane Following model trained from the collected data
  • ROS2-based
    • Time synchronous data collection node
    • deploying a trained model in a node
  • Data preprocessing for training
    • Data normalization
    • Data augmentation
    • Splitting data into training set and test set
    • Writing/Reading data in HDF5 format
  • Deep Learning model training: Train a model using Keras with TensorFlow backend

Training Details#

Network Architecture#

The network has 559,419 parameters and consists of 9 layers, including 5 convolutional layers, 3 fully connected layers, and an output layer.

Layer (type) Output Shape Param #
lambda_1 (Lambda) (None, 70, 320, 3) 0
conv2d_1 (Conv2D) (None, 33, 158, 24) 1824
conv2d_2 (Conv2D) (None, 15, 77, 36) 21636
conv2d_3 (Conv2D) (None, 6, 37, 48) 43248
conv2d_4 (Conv2D) (None, 4, 35, 64) 27712
conv2d_5 (Conv2D) (None, 2, 33, 64) 36928
dropout_1 (Dropout) (None, 2, 33, 64) 0
flatten_1 (Flatten) (None, 4224) 0
dense_1 (Dense) (None, 100) 422500
dense_2 (Dense) (None, 50) 5050
dense_3 (Dense) (None, 10) 510
dense_4 (Dense) (None, 1) 11


  • Learning rate: 1e-04
  • Learning rate decay: None
  • Dropout rate: 0.5
  • Mini-batch size: 128
  • Epochs: 30
  • Optimization algorithm: Adam
  • Loss function: Mean squared error
  • Training/Test set ratio: 8:2


  • Number of training data: 48,624 labeled images
  • Number of validation data: 12,156 labeled images

Center Image#

Left Image#

Right Image#

Original Image#

Cropped Image#

Data Distribution#

How to Collect Data and Train Your Own Model with LGSVL Simulator#

Collect data from LGSVL Simulator#

To collect camera images as well as corresponding steering commands for training, we provide a ROS2 collect node which subscribes to three camera image topics and a control command topic, approximately synchronizes time stamps of those messages, and then saves them as csv and jpg files. The topic names and types are as below: - Center camera: /simulator/sensor/camera/center/compressed (sensor_msgs/CompressedImage) - Left camera: /simulator/sensor/camera/left/compressed (sensor_msgs/CompressedImage) - Right camera: /simulator/sensor/camera/right/compressed (sensor_msgs/CompressedImage) - Control command: /simulator/control/command (geometry_msgs/TwistStamped)

To launch rosbridge and collect ROS2 node in a terminal:

docker-compose up collect

To drive a car and publish messages over rosbridge in training mode: - Launch LGSVL Simulator - Click Free Roaming mode - Select San Francisco map and XE_Rigged-lgsvl vehicle - Make sure the simulator establishes connection with rosbridge - Click Run to begin - Enable Main Camera, Left Camera, Right Camera, and check Publish Control Command

The node will start collecting data as you drive the car around. You should be able to check log messages in the terminal where the collect node is running. The final data is saved in lanefollowing/ros2_ws/src/lane_following/train/data/ as csv and jpg files.

Data preprocessing#

Before start training your model with the data you collected, data preprocessing is required. This task includes: - Resize image resolutions from 1920 x 1080 to 200 x 112 - Crop top portion of images as we are mostly interested in road part of an image - Data augmentation by adding artificial bias to side camera images or flipping images - Data normalization to help the model converge faster - Split data into training and testing dataset

To run data preprocessing and obtain datasets for training:

docker-compose up preprocess

This will preprocess your data and write outputs in lanefollowing/ros2_ws/src/lane_following/train/data/hdf5/ into HDF5 format for better I/O performance for training.

Train a model#

We use Keras with TensorFlow backend for training our model as an example. The hyperparameters such as learning rate, batch size, or number of epochs were chosen empirically. You can train a model as is but you are also welcome to modify the model architecture or any hyperparameters as you like in the code.

To start training:

docker-compose up train

After training is done, your final trained model will be in lanefollowing/ros2_ws/src/lane_following/train/model/{current-date-and-time-in-utc}.h5 and your model is ready to drive autonomously.

Drive with your trained model in LGSVL Simulator#

Now, it's time to deploy your trained model and test drive with it using LGSVL Simulator. You can replace your trained model with an existing one in lanefollowing/ros2_ws/src/lane_following/model/model.h5 as this is the path for deployment.

To launch rosbridge and drive ROS2 node in a terminal:

docker-compose up drive

Or, if you want visualizations as well, run drive_visual instead:

docker-compose up drive_visual

To drive a car in autonomous mode: - Launch LGSVL Simulator - Click Free Roaming mode - Select San Francisco map and XE_Rigged-lgsvl vehicle - Make sure the simulator establishes connection with rosbridge - Click Run to begin - Enable Main Camera (we don't need side cameras for inference)

Your car will start driving autonomously and try to mimic your driving behavior when training the model.

Future Works and Contributing#

Though the network can successfully drive and follow lanes on the bridge, there's still a lot of room for future improvements (i.e., biased to drive straight, afraid of shadows, few training data, and etc). - To improve model robustness collect more training data by driving in a wide variety of environments - Changing weather and lighting effects (rain, fog, road wetness, time of day) - Adding more road layouts and road textures - Adding more shadows on roads - Adding NPC cars around the ego vehicle - Predict the car throttle along with the steering angle - Take into accounts time series analysis using RNN (Recurrent Neural Network)