Example

Code is available in configs/examples/growth_sim.

TL;DR

To run the example, execute the following commands in the base folder of the RoboVAST repository:

# initialize project
vast init configs/examples/growth_sim/growth_sim.vast

# show the configurations that will be executed
vast config list

# setup pods in cluster (kubernetes required)
vast exec cluster setup minikube

# execute the tests in the cluster
vast exec cluster run

# OR: execute in detached mode (exit immediately, cleanup manually)
# vast exec cluster run --detach
# vast exec cluster monitor  # shows progress per run
# vast exec cluster run-cleanup  # run this after jobs complete

# Multiple runs can run in parallel by default. Use --cleanup to remove
# previous runs before starting.

# upload results from the cluster to a share service
vast exec cluster upload-to-share

# optionally: remove result buckets from S3
vast exec cluster download-cleanup

# cleanup pods in cluster
vast exec cluster cleanup

# evaluate the results
vast eval gui

Introduction

The overall workflow in RoboVAST consists of three main steps:

Variation → Execution → Analysis

For each step, RoboVAST provides dedicated tools to facilitate the process. For details on specific tools, please refer to How to run.

Before running any tests, you must initialize the RoboVAST project configuration:

vast init <config>

This command sets up the required configuration files and prepares your project for further steps.

Run Description

In this example, we test a simple logistic growth simulator defined in configs/examples/growth_sim/files/growth_sim.py. We will do parameter sweeps for initial_population and growth_rate. The simulator writes its output to a csv file.

This test uses a simple scenario: a single action invokes the growth simulator. Three scenario parameters are defined, and two of them will be varied later using parameter overriding during scenario execution. RoboVAST allows you to vary any scenario parameter as needed.

Scenario

import osc.helpers

scenario test_scenario:
    timeout(10s)
    
    initial_population: string
    growth_rate: string
    carrying_capacity: string = "1000"
    
    do serial:
        run_process(common.get_scenario_file_directory() + '/files/growth_sim.py' +
            ' --initial-population ' + initial_population +
            ' --growth-rate ' + growth_rate +
            ' --carrying-capacity ' + carrying_capacity +
            ' --output ' + common.get_output_directory() + "/out.csv")

RoboVAST Configuration

The central part of RoboVAST is the configuration file, which defines all aspects of a workflow. It has the ending .vast and is written in YAML format.

In this example we use configuration file configs/examples/growth_sim/growth_sim.vast.

The settings are split into three main sections: configuration, execution, and analysis.

Configuration

The configuration section defines the run scenarios to be executed. Each scenario specifies:

name: A unique identifier for the scenario
parameters: (Optional) Fixed parameters that apply to all configurations of this scenario
variations: (Optional) Advanced variation types for complex test generation

In this example, we define two scenarios:

test: Uses variations to create multiple configurations by varying initial_population and growth_rate using the ParameterVariationList plugin. This creates 4 × 3 = 12 configurations.
test-fixed-values: Uses parameters to define a single configuration with fixed values (growth_rate: 0.07 and initial_population: 123).

Configuration section of RoboVAST Configuration File

metadata:
  name: growth_sim
configuration:
- name: test
  variations:
  - ParameterVariationList:
      name: growth_rate
      values:
      - 0.05
      - 0.1
      - 0.3
      - 0.5
  - ParameterVariationList:
      name: initial_population
      values:
      - 50
      - 100
      - 200
- name: test-fixed-values
  parameters:

Execution

Note

For the execution, it is expected that the connection to the Kubernetes cluster is set up properly.

The execution section of the .vast configuration specifies all necessary parameters for running the tests, including the scenario file to execute. For multi-container setups and CPU/memory allocation, see resources and secondary_containers in Configuration.

Execution section of RoboVAST Configuration File

  - growth_rate: 0.07
  - initial_population: 123
execution:
  image: ghcr.io/cps-test-lab/robovast:latest
  resources:
    cpu: 1
  runs: 20
  env:

The scenario_file parameter specifies which OpenSCENARIO 2 file to execute (scenario.osc). In this example, we configure 20 runs for each config to ensure statistically meaningful results. In this basic example we hand in the system-under-test growth_sim.py directly by specifying the pattern **/files/*.py in the run_files. In larger setups, it might be required to use a custom container image.

Check Generated Configurations

Before starting the execution in the cluster, it is recommended to first check the configurations.

vast config list

Check Result of a Single Execution

To check that the container image and test are correctly set up, it is recommended to test the execution locally.

The command runs the container using the docker command and the same parameters and test-files as the kubernetes execution. Afterwards the output can be analyzed manually.

vast exec local run --config config1 output_config1

Cluster Execution

To execute all tests in the cluster, run:

vast exec cluster run

By default, this command waits for all jobs to complete and displays statistics.

Detached Execution

For long-running tests, you can use the --detach (or -d) flag to exit immediately after creating the jobs:

vast exec cluster run --detach

When running in detached mode:

The command exits right after creating all Kubernetes jobs
Jobs continue running in the background in the cluster
You can monitor job status using kubectl get jobs
You need to manually clean up jobs after they complete

To clean up after a detached run:

vast exec cluster run-cleanup

This removes all scenario execution jobs and their associated pods from the cluster.

Upload Results

The output of an execution is stored on the cluster-internal S3 server and can be uploaded to a share service (Nextcloud, GCS, …) with:

vast exec cluster upload-to-share

This transfers archives entirely inside the archiver sidecar — nothing is downloaded to the local machine. After a successful upload, the S3 bucket is removed automatically. To retrieve the results on another machine, use:

vast results download

See Sharing Results via cluster upload-to-share for configuration details and Results Processing for a complete description of the result files.

Analysis

As result analysis is tailored to each test, users are expected to implement their own analysis routines.

There are two steps invoked to analyze results. First, the results can optionally be postprocessed to simplify later evaluation. The user might specify postprocessing commands in the results_processing.postprocessing section of the .vast configuration. Common scripts including converting ROS bags to CSV files or extracting poses from tf-data are available to improve usability.

vast results postprocess

Postprocessing is cached based on the results directory hash. If the results directory is unchanged since the last postprocessing, the postprocessing is skipped automatically. To force postprocessing even if the results are unchanged (e.g., after updating postprocessing scripts), use the --force or -f flag:

vast results postprocess --force

After postprocessing, the actual evaluation can be performed. To simplify this process, RoboVAST provides a GUI tool, which enables users to execute Jupyter notebooks directly from a graphical interface.

vast eval gui

The visualization can be customized by adapting the evaluation.visualization section of the .vast configuration file.

Evaluation section of RoboVAST Configuration File

  run_files:
  - "files/*.py"
  - "files/*.sh"
  pre_command: /config/files/pre_command.sh
  post_command: /config/files/post_command.sh

Although this example includes only one entry in the analysis list, you can add more. Each additional entry will appear as a separate tab in the GUI.

There are three reserved keys for analysis: run, config, and campaign. These allow you to specify Jupyter notebooks for different scopes:

run: analyzes an individual run.
config: analyzes all runs for a specific configuration/parameter set.
campaign: analyzes all tests within a campaign, covering all configurations and parameter sets.

You are free to implement the notebooks as needed. The only requirement is that each notebook includes the following line:

DATA_DIR = ''

During execution within the GUI the content of DATA_DIR is replaced by the currently selected test-directory.

To improve usability the output of the jupyter-notebook-execution is cached and once it was generated it will be displayed instantly.