Distributed simulation enables a cluster to run with multiple computing instances (master, clients). This enables you to run simulations that may require a large number of sensors, or with multiple ego vehicles, which may normally require a large amount of computing resources. Distributed simulation solves the inherent limitations of trying to run real-time simulations on one single-GPU instance by synchronizing across multiple machines (each with a GPU).
Cluster is a single unit of simulation. It includes one or more machines that will perform a simulation. There is always one master machine in every cluster; the master machine performs the simulation updates like physics and synchronizes the simulation state with every client.
See Clusters for details on editing clusters in the SVL Simulator web user interface.
For the proper functioning of the sensors distributed to the clients, it is required to synchronize the whole simulation environment. When the simulation changes on master, the master sends update messages to every client. Simulator by default synchronizes every
Rigidbody in ego vehicles, NPCs, pedestrians and controllable objects instantiated by Simulator Manager.
Master receives and distributes vehicle actions like setting lights state or wipers state, and distributes controllables control actions so every client can clone the simulation state.
Every component that has to be synchronized between cluster machines requires messages sends on the master and messages receiver on the clients. Taking a vehicle with
Rigidbody as an example, Simulator adds
DistributedObjectcomponent to the vehicle GameObject to synchronize enable and disable calls and
DistributedRigidbody components which send required data to mock the state of
Rigidbody on clients.
By default, it is required that each
DistributedObject has a unique path in the hierarchy - objects on the same hierarchy level require unique GameObject names.
See Distributed Objects for details on distributing simulation objects.
See Distributed Components for details on distributing components data.
Master delegates sensors to clients, every sensor is simulated only on one machine. Some sensors like controls are simulated only on the master machine, and ultra high load sensors will always be delegated to the clients.
See Sensors Distribution for details on distributing sensors.
Distributed simulation can be extended with custom implementations.
See Distributed Python API for details on distributing simulation objects.
See Custom Distributed Class for details on implementing a custom class that will distribute messages.
See Distributed Messages for details on messages sent and received in the components.