Example

Code is available in configs/examples/growth_sim.

TL;DR

To run the example, execute the following commands in the base folder of the RoboVAST repository:

# initialize project
vast init configs/examples/growth_sim/growth_sim.vast

# show the configurations that will be executed
vast config list

# setup pods in cluster (kubernetes required)
vast exec cluster setup minikube

# execute the tests in the cluster (fire-and-forget: returns immediately)
vast exec cluster run
vast exec cluster monitor       # shows progress per run
vast exec cluster run-cleanup   # run this after jobs complete

# Multiple campaigns can run in parallel by default.

# optionally: remove result buckets from S3
vast exec cluster download-cleanup

# cleanup pods in cluster
vast exec cluster cleanup

# evaluate the results
vast eval gui

Introduction

The overall workflow in RoboVAST consists of three main steps:

Variation → Execution → Analysis

For each step, RoboVAST provides dedicated tools to facilitate the process. For details on specific tools, please refer to How to run.

Before running any tests, you must initialize the RoboVAST project configuration:

vast init <config>

This command sets up the required configuration files and prepares your project for further steps.

Run Description

In this example, we test a simple logistic growth simulator defined in configs/examples/growth_sim/files/growth_sim.py. We will do parameter sweeps for initial_population and growth_rate. The simulator writes its output to a csv file.

This test uses a simple scenario: a single action invokes the growth simulator. Three scenario parameters are defined, and two of them will be varied later using parameter overriding during scenario execution. RoboVAST allows you to vary any scenario parameter as needed.

Scenario
import osc.helpers

scenario test_scenario:
    timeout(10s)
    
    initial_population: string
    growth_rate: string
    carrying_capacity: string = "1000"
    
    do serial:
        run_process(common.get_scenario_file_directory() + '/files/growth_sim.py' +
            ' --initial-population ' + initial_population +
            ' --growth-rate ' + growth_rate +
            ' --carrying-capacity ' + carrying_capacity +
            ' --output ' + common.get_output_directory() + "/out.csv")

RoboVAST Configuration

The central part of RoboVAST is the configuration file, which defines all aspects of a workflow. It has the ending .vast and is written in YAML format.

In this example we use configuration file configs/examples/growth_sim/growth_sim.vast.

The settings are split into three main sections: configuration, execution, and analysis.

Configuration

The configuration section defines the run scenarios to be executed. Each scenario specifies:

  • name: A unique identifier for the scenario

  • parameters: (Optional) Fixed parameters that apply to all configurations of this scenario

  • variations: (Optional) Advanced variation types for complex test generation

In this example, we define two scenarios:

  1. test: Uses variations to create multiple configurations by varying initial_population and growth_rate using the ParameterVariationList plugin. This creates 4 × 3 = 12 configurations.

  2. test-fixed-values: Uses parameters to define a single configuration with fixed values (growth_rate: 0.07 and initial_population: 123).

Configuration section of RoboVAST Configuration File
metadata:
  name: growth_sim
configuration:
- name: test
  variations:
  - ParameterVariationList:
      name: growth_rate
      values:
      - 0.05
      - 0.1
      - 0.3
      - 0.5
  - ParameterVariationList:
      name: initial_population
      values:
      - 50
      - 100
      - 200
- name: test-fixed-values
  parameters:

Execution

Note

For the execution, it is expected that the connection to the Kubernetes cluster is set up properly.

The execution section of the .vast configuration specifies all necessary parameters for running the tests, including the scenario file to execute. For multi-container setups and CPU/memory allocation, see resources and secondary_containers in Configuration.

Execution section of RoboVAST Configuration File
  - growth_rate: 0.07
  - initial_population: 123
execution:
  image: ghcr.io/cps-test-lab/robovast:latest
  resources:
    cpu: 1
  runs: 2
  env:

The scenario_file parameter specifies which OpenSCENARIO 2 file to execute (scenario.osc). In this example, we configure 20 runs for each config to ensure statistically meaningful results. In this basic example we hand in the system-under-test growth_sim.py directly by specifying the pattern **/files/*.py in the run_files. In larger setups, it might be required to use a custom container image.

Check Generated Configurations

Before starting the execution in the cluster, it is recommended to first check the configurations.

vast config list

Check Result of a Single Execution

To check that the container image and test are correctly set up, it is recommended to test the execution locally.

The command runs the container using the docker command and the same parameters and test-files as the kubernetes execution. Afterwards the output can be analyzed manually.

vast exec local run --config config1 output_config1

Cluster Execution

To execute all tests in the cluster, run:

vast exec cluster run

This command is fire-and-forget: it launches an in-cluster controller pod that drives the whole campaign and returns immediately, printing the campaign id and controller pod name.

Monitoring a run

The campaign runs entirely in the cluster. Track its progress with:

vast exec cluster monitor

To clean up a run’s scenario jobs/pods (and its controller pod):

vast exec cluster run-cleanup

This removes the scenario execution jobs and their associated pods from the cluster.

Download Results

When the campaign finishes, the results are uploaded to the configured share service (Nextcloud, GCS, …) automatically. Retrieve them with:

vast results download

See Sharing Results for configuration details and Results Processing for a complete description of the result files.

Analysis

As result analysis is tailored to each test, users are expected to implement their own analysis routines.

There are two steps invoked to analyze results. First, the results can optionally be postprocessed to simplify later evaluation. The user might specify postprocessing commands in the results_processing.postprocessing section of the .vast configuration. Common scripts including converting ROS bags to CSV files or extracting poses from tf-data are available to improve usability.

vast results postprocess

Postprocessing is cached based on the results directory hash. If the results directory is unchanged since the last postprocessing, the postprocessing is skipped automatically. To force postprocessing even if the results are unchanged (e.g., after updating postprocessing scripts), use the --force or -f flag:

vast results postprocess --force

After postprocessing, the actual evaluation can be performed. To simplify this process, RoboVAST provides a GUI tool, which enables users to execute Jupyter notebooks directly from a graphical interface.

vast eval gui

The visualization can be customized by adapting the evaluation.visualization section of the .vast configuration file.

Evaluation section of RoboVAST Configuration File
  run_files:
  - "files/*.py"
  - "files/*.sh"
  pre_command: /config/files/pre_command.sh
  post_command: /config/files/post_command.sh

Although this example includes only one entry in the analysis list, you can add more. Each additional entry will appear as a separate tab in the GUI.

There are three reserved keys for analysis: run, config, and campaign. These allow you to specify Jupyter notebooks for different scopes:

  • run: analyzes an individual run.

  • config: analyzes all runs for a specific configuration/parameter set.

  • campaign: analyzes all tests within a campaign, covering all configurations and parameter sets.

You are free to implement the notebooks as needed. The only requirement is that each notebook includes the following line:

DATA_DIR = ''

During execution within the GUI the content of DATA_DIR is replaced by the currently selected test-directory.

To improve usability the output of the jupyter-notebook-execution is cached and once it was generated it will be displayed instantly.