Scenario Runner

Introduction#

Scenario Runner is a tool for the SVL Simulator which executes driving scenarios written in a high level Scenario Description Language called Scenic.

Scenarios can be defined where nearly everything is specified including locations, speeds, distances, actors, and environmental conditions (referred to as "concrete"), or they can be more "abstract" where some or all parameters are chosen at runtime, either randomly or across a distribution to ensure coverage over a range of possible values or options.

Among the use-cases of Scenario Runner are:

  • Generating and testing random variations of abstract driving scenarios
  • Performing automated (e.g. acceptance) tests based on a set of repeatable scenarios
  • Performing automated regressions tests

This preview release of the Scenario Runner provides the ability to perform several example scenarios in the SVL Simulator using Apollo for Autonomous Driving. This document describes how to run these scenarios.

For example, one provided example is an abstract "cut-in" scenario which can be performed on a test map (shown in the first two images below) or on other maps such as San Francisco (also shown in the image below).

Another provided example is a "random placement" scenario where the ego is randomly placed on a map such as Shalun and given a random destination to drive to.

Additional examples demonstrate pedestrian-crossing scenarios which can be performed on the GoMentum map and also the Borregas Ave map.

Package overview top#

Scenario Runner is delivered as a zip archive with the following directory layout:

lgsvlsimulator-scenarios-2020.07P
├── docker
├── scenarios
│   └── Scenic
│       ├── borregas-intersection
│       │   └── scenic-borregas.sc
│       ├── cut-in
│       │   ├── scenic-cut-in.sc
│       │   └── scenic-cut-in-sf.sc
│       ├── maps
│       │   ├── BorregasAve
│       │   ├── GoMentum
│       │   ├── SanFrancisco
│       │   ├── Shalun
│       │   └── Straight2LaneSame
│       ├── random-placement
│       │   └── scenic-example.sc
│       └── scenario_2_1
│           └── scenario_2_1_Apollo.sc
└── scenario_runner.sh
  • The docker directory contains a docker image saved as a tarball.
  • The scenarios/Scenic directory contains four sample Scenic scenarios:
  • borregas-intersection is a pedestrian-crossing scenario on the Borregas map
  • cut-in is a vehicle cut-in scenario on a two lane road test map
  • scenario_2_1 is a pedestrian-crossing scenario on the GoMentum map
  • random-placement is a scenario with random ego placement and destination
  • scenario_runner.sh is the scenario runner script that executes scenarios and plays them in the simulator by controlling the simulator's Python API and also controlling the Autonomous Driving stack (e.g. Apollo).

Scenario Runner workflow top#

  1. setup simulator in web UI
  2. launch simulator in API mode
  3. run the scenario runner script

Requirements#

Scenario Runner has these dependencies:

  1. SVL Simulator -- the release tag of the simulator should match the release tag of Scenario Runner to ensure compatibility.
  2. Apollo 5.0 lgsvl fork
  3. Docker and Docker-compose

Downloading and launching the SVL Simulator top#

The Simulator release can be downloaded as a prebuilt binary from the premium release download page.

Download and extract the zip archive at the desired location. In the unzipped directory, run the executable file named simulator to launch the SVL Simulator.

The main window of the simulator will open. Click on the "Link to Cloud" or "Open Browser" button to launch the simulator Web UI in a web browser. The Web UI controls all aspects of the simulator.

You will need to create an account to login to the Web UI. Once logged in you will be able to link your simulator into a cluster, select maps, configure vehicles, and create simulations to run in the simulator.

For more information on initial setup of the simulator, refer to the Installing Simuator and Link to Cloud sections of the Running SVL Simulator documentation for an illustrated step-by-step walk through of this process.

Configure API Only mode in Web UI (if needed) top#

From the simulator Web UI, you will need to locate in the Store any needed maps and vehicles, and add each of them to your library.

Refer to the Store for information on how to add the "BorregasAve" and "SanFrancisco" maps and "Lincoln2017MKZ (Apollo 5.0)" vehicle.

Refer to API Only Simulation for an illustrated step-by-step walk through on creating an API Only simulation.

Start SVL Simulator in API Only mode top#

  1. In the Simulations tab in the Web UI locate the API Only simulation.
  2. Then click the red "Run Simulation" button under the API Only simulation to start the simulation.

Launching Apollo alongside the Simulator#

Installing (and Building) Apollo 5.0 top#

Please follow the instructions on the SVL Simulator documentation website to for Running Apollo 5.0 with SVL Simulator if you have not already done so. This will involve installing Docker CE, Nvidia Docker and the LGSVL Docker image, then cloning the lgsvl fork of the Apollo 5.0 sources, and finally building Apollo and the bridge.

Start Apollo 5.0 top#

  • Open a terminal, change directory to the apollo-5.0 project (lgsvl fork: https://github.com/lgsvl/apollo-5.0), and type:
    • ./docker/scripts/dev_start.sh to start apollo container
    • ./docker/scripts/dev_into.sh to enter the container
    • Make sure you have built Apollo with GPU enabled options; for more details, refer to Running Apollo 5.0 with SVL Simulator
    • bootstrap.sh && bridge.sh & to start the Apollo Dreamview Web UI and cyber_bridge
  • Open browser, enter address localhost:8888, and select Lincoln2017MKZ from the vehicle menu
  • Note: Apollo modules will be started and stopped by the Scenario Runner script; make sure all modules are off before running simulation.

Using the Scenario Runner#

Scenario Runner top#

Navigate to the unzipped directory of the Scenario Runner package in a terminal window.

The Scenario Runner script can be run from the lgsvlsimulator-scenarios-2020.07P directory as ./scenario_runner.sh

Scenario Runner Commands top#

The Scenario Runner supports the following commands:

  • scenario_runner.sh help for help using the Scenario Runner
  • scenario_runner.sh env to print useful environment variables
  • scenario_runner.sh run [PARAMETERS] SCENIC_FILE_PATH to run a Scenario with optional parameters
  • scenario_runner.sh run --help for help running a Scenario
  • scenario_runner.sh version to print version information

Scenario Runner Parameters top#

The Scenario Runner supports the following command line parameters when running a scenario:

-i, --num-iterations NUM_ITERATIONS Number of scenario iterations. (default: 42)

-d, --duration DURATION Maximum scenario duration in seconds. (default: 20.0)

-m, --lgsvl-map MAP_NAME Map (default: GoMentum)

-O, --output-dir OUTPUT_DIR Default results output folder

-s, --sampler Save sampler data after simulation is finished.

-t, --check Parse scenic file and exit

Scenario Runner Environment Variables top#

The Scenario Runner supports the following environment variables (and default values) when running a scenario:

SIMULATOR_HOST=localhost

SIMULATOR_PORT=8181

BRIDGE_HOST=localhost

BRIDGE_PORT=9090

If simulator is being run on a separate machine than the Scenario Runner make sure to set the SIMULATOR_HOST environment variable with the IP address of the machine running the simulator; for example export SIMULATOR_HOST=192.168.0.2.

The BRIDGE_HOST machine is the machine which is running Apollo (and the CyberRT bridge) and is most likely also the same machine running Scenario Runner. If simulator and Apollo are running on the same machine, use the default value of localhost. Otherwise, set BRIDGE_HOST to the IP address of the machine running Apollo; for example export BRIDGE_HOST=192.168.0.1.

Example Scenarios#

There are several example scenarios provided with the Scenario Runner in this preview release. Each scenario has an associated (and required) map, as summarized in the following table:

Test Case Map
random placement Shalun
cut-in San Francisco or Straight2LaneSame
pedestrian crossing GoMentum or Borregas Ave

You can add the required maps to your library from the Store.

Note that each map (environment) must be available in your library, and the corresponding Apollo HD map must be selected in Dreamview.

Also note that changing maps in Dreamview will disable the modules so you will need to re-enable them as described above.

There are several example scenarios provided with the Scenario Runner in this preview release.

As noted above, the Scenario Runner script can be run from the lgsvlsimulator-scenarios-2020.07P directory as ./scenario_runner.sh

Random Placement Scenario top#

The Random Placement scenario will randomly place the ego vehicle on the map and then drive to a random destination.

To run the Random Placement scenario three times on the Shalun map, navigate to the lgsvlsimulator-scenarios-2020.07P directory and type:

$ ./scenario_runner.sh run -i 3 -m Shalun scenarios/Scenic/random-placement/scenic-example.sc

Cut-in Scenario top#

The cut-in scenario is a vehicle cut-in scenario for a two-lane road. This is an abstract scenario that can be run on a test map. The NPC vehicle will start out next to the ego vehicle, start driving and then cut in front of the ego vehicle and come to a stop. There is a similar scenario that can be run on the San Francisco map.

To run the Cut-in scenario twice on a test map, navigate to the lgsvlsimulator-scenarios-2020.07P directory and type:

$./scenario_runner.sh run -i 2 -m Straight2LaneSame scenarios/Scenic/cut-in/scenic-cut-in.sc

To run the Cut-in scenario three times on the San Francisco map, navigate to the lgsvlsimulator-scenarios-2020.07P directory and type:

$./scenario_runner.sh run -i 3 -m SanFrancisco scenarios/Scenic/cut-in/scenic-cut-in-sf.sc

Note: The scenic-cut-in-sf.sc script is based on the scenic-cut-in.sc script but was modified to load the "SanFrancisco" OpenDRIVE HD map instead of "Straight2LaneSame", and the apolloHDMap param was changed to select the correct map from Apollo's Dreamview. The San Francisco HD map is very large and takes a while to parse before the first iteration can begin.

Pedestrian Crossing Scenarios top#

The Pedestrian Crossing scenario demonstrates a vehicle approaching a crosswalk or turning at an intersection with a crosswalk. A pedestrian will enter the crosswalk forcing the ego vehicle to stop and wait for the pedestrian. There is a scenario that can be run on the GoMentum map and a similar scenario for the Borregas map.

To run the Pedestrian Crossing scenario three times on the GoMentum map, navigate to the lgsvlsimulator-scenarios-2020.07P directory and type:

$./scenario_runner.sh run -i 3 -m GoMentum scenarios/Scenic/scenario_2_1/scenario_2_1_Apollo.sc

To run the Pedestrian Crossing scenario ten times on the Borregas map, navigate to the lgsvlsimulator-scenarios-2020.07P directory and type:

$./scenario_runner.sh run -i 10 -d 40 -m BorregasAve scenarios/Scenic/borregas-intersection/scenic-borregas.sc

Known issues top#

  • If the ego vehicle is not placed properly within a driveable lane, Apollo may not correctly drive the vehicle to the requested destination. This is a known issue and will be addressed in a future version of the Scenario Runner.
  • After Initializing Apollo... appears in the console, there may be one or more Module <name> is not ready... messages followed by a waiting to stabilize message and a small delay, followed by Initialized Apollo. If waiting to stabilize does not appear, then the scenario runner script should be restarted; if it happens again then Apollo may need to be restarted.
  • The SanFrancisco HD map is very large and takes a while to parse before the first iteration can begin.