TestCaseRunner

Video#


Introduction#

TestCaseRunner is a tool for the SVL Simulator which executes test cases created with the Visual Scenario Editor (as well as other Python tests). These tests are executed by the TestCaseRunner using the LGSVL Python API.

Note: This tool is only supported on Linux.

Package overview top#

Python Runner is delivered as part of the svlsimulator-linux64-{release-version}.zip archive with the following directory layout:

svlsimulator-linux64-{release-version}
├── TestCaseRunner 
│   └── scenarioRunner 
│       └── docker
│       └── scenario_runner.sh
  • The docker directory contains a docker image saved as a tarball.
  • scenario_runner.sh along with the above docker image is used for Python API / Visual scenario editor a.k.a VSE" runtime templates in web UI to run the scenarios automatically. Visual Scenario runtime template is a way to upload and run VSE test cases directly from web UI.
  • scenario_runner.sh is the Python Runner script that executes VSE test cases in the simulator through the simulator's Python API. Also, if you want to run the VSE scenario manually using "API only" runtime template, then you can use this file.

Python Runner workflow top#

  1. Setup simulator in web UI
  2. Launch simulator in API mode
  3. Run the Python runner script
  4. Start the ego vehicle

Requirements#

Python Runner has these dependencies:

  1. SVL Simulator -- the release tag of the simulator should match the release tag of Python Runner to ensure compatibility.
  2. Apollo 5.0 lgsvl fork
  3. Docker and Docker-compose

Downloading and launching the SVL Simulator top#

The Simulator release can be downloaded as a prebuilt binary from the github releases page.

Download and extract the zip archive at the desired location. In the unzipped directory, run the executable file named simulator to launch the SVL Simulator.

The main window of the simulator will open. Click on the "Link to Cloud" or "Open Browser" button to launch the simulator Web UI in a web browser. The Web UI controls all aspects of the simulator.

You will need to create an account to login to the Web UI. Once logged in you will be able to link your simulator into a cluster, select maps, configure vehicles, and create simulations to run in the simulator.

For more information on initial setup of the simulator, refer to the Installing Simuator and Link to Cloud sections of the Running SVL Simulator documentation for an illustrated step-by-step walk through of this process.

Configure API Only mode in Web UI (if needed) top#

From the simulator Web UI, you can find any needed maps and vehicles in the Store and add each of them to your library. Or, if you want to use a private map to create VSE tests, you will need to upload your map's asset bundle (refer to Adding a map for how to upload your own maps). Similarly, you can also use your own private vehicle (refer to Adding a vehicle for how to upload your own vehicles). Note for any Apollo compatible vehicle you use, you need to have ApolloControl sensor included in order to correctly start simulation. Refer to Apollo Control sensor configuration for more information.

Refer to the Store for information on how to add the "BorregasAve" and "SanFrancisco" maps and "Lincoln2017MKZ (Apollo 5.0)" vehicle.

Refer to API Only Simulation for an illustrated step-by-step walk through on creating an API Only simulation.

Start SVL Simulator in API Only mode top#

  1. In the Simulations tab in the Web UI locate the API Only simulation, you need to create a simulation with API Only if you don't have it (for steps refer to Runtime Template: API Only).
  2. Then click the red "Run Simulation" button under the API Only simulation to start the simulation.

Launching Apollo alongside the Simulator#

Installing (and Building) Apollo 5.0 top#

Please follow the instructions on the SVL Simulator documentation website to for Running Apollo 5.0 with SVL Simulator if you have not already done so. This will involve installing Docker CE, Nvidia Docker and the LGSVL Docker image, then cloning the lgsvl fork of the Apollo 5.0 sources, and finally building Apollo and the bridge.

Start Apollo 5.0 top#

  • Open a terminal, change directory to the apollo-5.0 project (lgsvl fork: https://github.com/lgsvl/apollo-5.0), and type:
    • ./docker/dev_start.sh to start apollo container
    • ./docker/dev_into.sh to enter the container
    • Make sure you have built Apollo with GPU enabled options; for more details, refer to Running Apollo 5.0 with SVL Simulator.
    • Make sure you have HD map files and vehicle calibration files for your VSE test case in apollo repository.
    • bootstrap.sh && bridge.sh & to start the Apollo Dreamview Web UI and cyber_bridge.
  • Open browser, enter address localhost:8888 to view Dreamview.

Using the Python Runner for VSE tests#

Python Runner top#

Navigate to the unzipped directory of the SVL Simulator Linux package in a terminal window.

The Python Runner script can be run from the svlsimulator/TestCaseRunner/scenarioRunner directory as ./scenario_runner.sh

Python Runner Commands top#

The Python Runner supports the following commands:

  • scenario_runner.sh help for help using the Python Runner
  • scenario_runner.sh env to print useful environment variables
  • scenario_runner.sh run to run a test case (with optional parameters)
  • scenario_runner.sh run --help for help running a test case
  • scenario_runner.sh version to print version information

Python Runner Parameters top#

The Python Runner supports the following command line parameters when running a VSE test case:

-d, --duration DURATION Maximum scenario duration in seconds. (default: 20.0)

-f, --force-duration Force simulation to end after given duration. If not set, simulation will end by given duration or at the time when all NPCs' waypoints have been reached.

Python Runner Environment Variables top#

The Python Runner supports the following environment variables (and default values) when running a test case:

  • SIMULATOR_HOST=localhost
  • SIMULATOR_PORT=8181
  • BRIDGE_HOST=localhost
  • BRIDGE_PORT=9090
  • SCENARIOS_DIR

If simulator is being run on a separate machine than the Python Runner make sure to set the SIMULATOR_HOST environment variable with the IP address of the machine running the simulator; for example export SIMULATOR_HOST=192.168.0.2.

The BRIDGE_HOST machine is the machine which is running Apollo (and the CyberRT bridge) and is most likely also the same machine running Python Runner. If simulator and Apollo are running on the same machine, use the default value of localhost. Otherwise, set BRIDGE_HOST to the IP address of the machine running Apollo; for example export BRIDGE_HOST=192.168.0.1.

You may set SCENARIOS_DIR in order to mount a local directory which contains a VSE script that is not available inside the Python Runner container; for example export SCENARIOS_DIR="/home/<user>/my-vse-scripts/". Or you can specify the path to the script on the command line.

Usage Example#

Create a test scenario using the Visual Scenario Editor and make note of the directory to which it is saved.

After placing an EgoAgent in the Visual Scenario Editor scene, click on the ego agent, then click the "Edit" button and select the desired ego vehicle from the scrolling list that appears. If the vehicle will be controlled by Apollo then select a vehicle that is configured for Apollo, e.g. "Lincoln2017MKZ (Apollo 5.0)", which is configured with Apollo sensors and also configured for test case analytics. You also need to select a destination point for the ego agent so that the VSE runner will set up apollo automatically, i.e., connect with apollo, start required modules and set destination.

Also note that changing maps in Dreamview will disable the modules so you will need to re-enable them as described above.

Running the VSE Scenario top#

To run a locally-generated VSE test case called my-vse-test.json which is located in the ~/my-vse-scripts directory, with a test duration of 30 seconds, navigate to the svlsimulator/TestCaseRunner/scenarioRunner directory and type:

$ export SCENARIOS_DIR=/home/<user>/my-vse-scripts/
$ ./scenario_runner.sh run my-vse-test.json -d 30

Setting SCENARIOS_DIR is not necessary if you specify the path to the script.

In addition, when finished testing in API-Only mode, click on the red "Stop" (square/stop) button (at the bottom of the Web UI) to end the API-only simulation.

Running VSE runtime template top#

The same VSE scenario file can be uploaded to web UI and run automatically. See instructions here.

Note: Visual Scenario runtime template based simulations can only be run on Linux simulator build. This is not supported on Windows.

Known issues top#

  • The VSE runner won't turn off apollo modules when the scenario finishes. You need to disable the modules once you finish the testing.
  • If your runner is not working correctly, please make sure apollo has corresponding HD maps and vehicle calibration files, and bridge is running.