Test Results

The Test Results contains the full collection of test results generated from running test cases. You can view individual test reports of executed test cases. Test reports contain detailed information about the test case, including the success/failure result, important statistics for each ego vehicle, interesting events that happened, and a video of the ego vehicle's drive during the test case.

Test Results top#

On the left-hand side of the web user interface, clicking "Test Results" brings up the Test Results page. This page contains a table of all executed test cases which enabled test report generation.

For each test case, you can see when the report was created, the name of the test case for the report, an evaluated test case result, and a button to view the detailed report.

In order to create and view a test case report, make sure to enable the "Create Test Case report" radio button when creating a simulation.

Test Cases must have test report generation enabled when they are executed in order to be shown in Test Results. Additionally, the Analysis Sensor must be included in the ego vehicle used in the simulation. For video recording, the Video Recording sensor must also be included. Please see the sensor configurations page for the JSON.

Each test result will show the evaluated result, with either "Success", "Failed", or "In Progress". You can view each test report for additional details or messages about the reason for the evaluation.

Viewing a Test Report top#

To view a detailed test report, click the "View" button on the right for a given test result.

The top bar indicates the Test Report name, the associated simulation, and buttons to expand the starting simulation configuration details, and delete the report.

The "Iterations" section displays all iterations for this test report. For test cases run with a Python API script, one test case can contain many iterations, each with its own result, statistics, and events. The highlighted box indicates the selected iteration.

Simulation evaluation top#

The test report indicates the evaluation status, or result, of the executed simulation. The below table specifies the conditions and criteria for a simulation result. In the future, additional criteria as well as user-customizable criteria will be supported.

Result Cause/Criteria
SUCCESS The test case did not result in FAILED or ERROR
FAILED The test case involved at least one failure event callback
ERROR The test case resulted in an error and failed to execute properly

The "Start Time" indicates the cluster's system date and time when the test case was started/stopped. For test cases with a distributed cluster, the master node's system time is used.

The "Simulation time duration" indicates the total duration of time of the test case in simulation, excluding paused time. Note that this can be different than the "Real time duration", which measures how much total time passed in the real world during execution.

In the vehicle section, for each ego vehicle in the test case, you can view the vehicle bridge type, bridge IP and port location.

If you included the Video Recording sensor in at least one ego vehicle, you can see the file location where the recorded video of the simulation has been stored.

The following section, Sensors, exists for every ego vehicle involved in the test case. In test cases involving multiple ego vehicles, you will be able to see sensor statistics and callback event information for each ego vehicle.

Sensors: Statistics#

The Callbacks section gives notable information about the ego vehicle during the test case run.

Metric Description Units
Distance travelled Total distance traveled, as measured by wheel odometry km
Average speed The average speed of the ego vehicle km/hr
Max speed The maximum speed of the ego vehicle km/hr
Min speed The minimum speed of the ego vehicle km/hr
Max longitudinal acceleration Maximum instantaneous acceleration along the vehicle's longitudinal axis m/s2
Max lateral acceleration Maximum instantaneous acceleration along the vehicle's lateral (perpendicular to longitudinal) axis m/s2
Max longitudinal jerk Maximum instantaneous jerk along the vehicle's longitudinal axis m/s3
Max lateral jerk Maximum instantaneous jerk along the vehicle's lateral (perpendicular to longitudinal) axis m/s3

Callbacks top#

Callback events are categorized as one-time events during a test case that either affect the ego vehicle, or that the ego vehicle causes in the environment. The following types of events are supported:

  • Collision involving ego (EgoCollision)
  • Speed limit violation (SpeedViolation)
  • Sudden braking (SuddenBrake)
  • Sudden steer (SuddenSteer)
  • Low simulation performance (LowFPS)

The occurrence of at least one of these events will cause a simulation to be evaluated as "Failed".

For each callback event, the returned information is reported, including the simulation time at which the event occurred.

Selecting the eye icon will cause the visualization to seek to the time of the event in order to view the state of each sensor at that time.

EgoCollision top#

A collision involving an ego vehicle.

Field Description Units
OtherType Type of agent with which ego collided NPC|Pedestrian|Obstacle
EgoVelocity Velocity vector of ego at time at collision time x, y, z in m/s
OtherVelocity Velocity vector of agent/object at collision time km/hr
EgoCollisionTotal Number of ego collisions in the event

SpeedViolation top#

A speed limit violation by an ego vehicle, based on the lane speed limit in the map annotations.

Metric Description Units
Duration Duration of speed limit violation event time
Max speed Maximum speed of ego vehicle during event. m/s
SpeedLimit The ego lane's annotated speed limit of the HD map m/s

EgoStuck top#

An ego stuck event by the ego vehicle, in which the ego travels a threshold distance and then stops moving for a threshold time

SuddenBrake top#

A sudden braking event by the ego vehicle, in which it exceeds a threshold deceleration value.

SuddenSteer top#

A sudden steering event by the ego vehicle, in which absolute magnitude of steering angle exceeds a threshold value.

LowFPS top#

A low simulation performance to indicate that results may not be accurate due to sufficiently low framerate during simulation on the cluster.