Example

Code is available in configs/examples/growth_sim.

TL;DR

To run the example, execute the following commands in the base folder of the RoboVAST repository:

# show the configurations that will be executed
vast config list configs/examples/growth_sim/growth_sim.vast

# setup pods in cluster (kubernetes required)
vast cluster setup minikube

# push the project, then execute it (fire-and-forget: returns immediately)
vast workspace init configs/examples/growth_sim --name growth_sim
vast workspace run growth_sim growth_sim.vast --description "growth_sim: first pass"
vast cluster monitor       # shows progress per run
vast cluster jobs-cleanup   # run this after jobs complete

# Multiple campaigns can run in parallel by default.

# cleanup pods in cluster
vast cluster cleanup

# read the results in the browser
vast ui

Introduction

The overall workflow in RoboVAST consists of three main steps:

Variation → Execution → Analysis

For each step, RoboVAST provides dedicated tools to facilitate the process. For details on specific tools, please refer to How to run.

There is nothing to initialize. Every command names its own input: the local verbs take the .vast as an argument, and a campaign runs a workspace’s project, named by the pair (workspace, path).

vast workspace run <workspace> <config> --description "what this run is for"

Run Description

In this example, we test a simple logistic growth simulator defined in configs/examples/growth_sim/files/growth_sim.py. We will do parameter sweeps for initial_population and growth_rate. The simulator writes its output to a csv file.

This test uses a simple scenario: a single action invokes the growth simulator. Three scenario parameters are defined, and two of them will be varied later using parameter overriding during scenario execution. RoboVAST allows you to vary any scenario parameter as needed.

Scenario
import osc.helpers

scenario test_scenario:
    timeout(10s)
    
    initial_population: string
    growth_rate: string
    carrying_capacity: string = "1000"
    
    do serial:
        run_process(common.get_scenario_file_directory() + '/files/growth_sim.py' +
            ' --initial-population ' + initial_population +
            ' --growth-rate ' + growth_rate +
            ' --carrying-capacity ' + carrying_capacity +
            ' --output ' + common.get_output_directory() + "/out.csv")

RoboVAST Configuration

The central part of RoboVAST is the configuration file, which defines all aspects of a workflow. It has the ending .vast and is written in YAML format.

In this example we use configuration file configs/examples/growth_sim/growth_sim.vast.

The settings are split into three main sections: configuration, execution, and analysis.

Configuration

The configuration section defines the run scenarios to be executed. Each scenario specifies:

  • name: A unique identifier for the scenario

  • parameters: (Optional) Fixed parameters that apply to all configurations of this scenario

  • variations: (Optional) Advanced variation types for complex test generation

In this example, we define two scenarios:

  1. test: Uses variations to create multiple configurations by varying initial_population and growth_rate using the ParameterVariationList plugin. This creates 4 × 3 = 12 configurations.

  2. test-fixed-values: Uses parameters to define a single configuration with fixed values (growth_rate: 0.07 and initial_population: 123).

Configuration section of RoboVAST Configuration File
metadata:
  name: growth_sim
configuration:
- name: test
  variations:
  - ParameterVariationList:
      scenario: growth_rate
      values:
      - 0.05
      - 0.1
      - 0.3
      - 0.5
  - ParameterVariationList:
      scenario: initial_population
      values:
      - 50
      - 100
      - 200
- name: test-fixed-values
  parameters:

Execution

Note

For the execution, it is expected that the connection to the Kubernetes cluster is set up properly.

The execution section of the .vast configuration specifies all necessary parameters for running the tests, including the scenario file to execute. For multi-container setups and CPU/memory allocation, see resources and secondary_containers in Configuration.

Execution section of RoboVAST Configuration File
    scenario:
      growth_rate: 0.07
      initial_population: 123
execution:
  containers:
    scenario:
      resources:
        cpu: 1

The scenario_file parameter specifies which OpenSCENARIO 2 file to execute (scenario.osc). In this example, we configure 20 runs for each config to ensure statistically meaningful results. In this basic example we hand in the system-under-test growth_sim.py directly by specifying the pattern **/files/*.py in the run_files. In larger setups, it might be required to use a custom container image.

Check Generated Configurations

Before starting the execution in the cluster, it is recommended to first check the configurations.

vast config list configs/examples/growth_sim/growth_sim.vast

Check Result of a Single Execution

To check that the container image and test are correctly set up, run one configuration’s scenario in the image before a campaign: vast container run (or the MCP exec_in_container) starts it with the same parameters and test files a campaign would stage, and its log can be read afterwards.

vast workspace run growth_sim growth_sim.vast --filter config1 --runs 1

Cluster Execution

To execute all tests in the cluster, run:

vast workspace run growth_sim growth_sim.vast

This command is fire-and-forget: it starts the campaign on the robovast-service (which drives it in-process) and returns immediately, printing the campaign id.

Monitoring a run

The campaign runs entirely in the cluster. Track its progress with:

vast cluster monitor

To clean up a run’s scenario jobs/pods (and any auxiliary-container pod):

vast cluster jobs-cleanup

This removes the scenario execution jobs and their associated pods from the cluster.

Download Results

Ask the service for the campaign’s archive:

vast campaign download <campaign-id>

That writes <campaign-id>.tar.gz into the current directory and stops there — the archive is yours to unpack, keep, or hand to a colleague.

If the campaign was launched with --upload-to-share it also has a raw, pre-postprocessing copy on the configured share (Nextcloud, GCS, …), which is a separate system with its own commands:

vast share list                     # what is on the share
vast share download <campaign-id>   # the raw archive, to this machine
vast share import <campaign-id>     # have a service take it in, and postprocess it

See Sharing Results for configuration details and Results Processing for a complete description of the result files.

Analysis

As result analysis is tailored to each test, users are expected to implement their own analysis routines.

There are two steps invoked to analyze results. First, the results can optionally be postprocessed to simplify later evaluation. The user might specify postprocessing commands in the results_processing.postprocessing section of the .vast configuration. Common scripts including converting ROS bags to CSV files or extracting poses from tf-data are available to improve usability.

vast campaign postprocess <campaign-id>

It runs automatically when the campaign’s runs finish, so this is for running it again. Postprocessing is cached based on the results directory hash. If the results directory is unchanged since the last postprocessing, the postprocessing is skipped automatically. To force postprocessing even if the results are unchanged (e.g., after updating postprocessing scripts), use the --force or -f flag:

vast campaign postprocess <campaign-id> --force

After postprocessing, the actual evaluation can be performed. Open the web UI and use the Results Explorer: it executes the campaign’s analysis notebooks server-side, one per selected node of the campaign/config/run tree, and renders them as HTML.

vast ui

The visualization can be customized by adapting the visualization.results.explorer.notebooks section of the .vast configuration file.

Evaluation section of RoboVAST Configuration File
  - ENABLE_X11: "false"
  scenario_file: scenario.osc
  mode: base
  run_files:
  - "files/*.py"

Although this example includes only one entry in the analysis list, you can add more. Each additional entry will appear as a separate tab in the GUI.

There are three reserved keys for analysis: run, config, and campaign. These allow you to specify Jupyter notebooks for different scopes:

  • run: analyzes an individual run.

  • config: analyzes all runs for a specific configuration/parameter set.

  • campaign: analyzes all tests within a campaign, covering all configurations and parameter sets.

You are free to implement the notebooks as needed. The only requirement is that each notebook includes the following line:

DATA_DIR = ''

During execution within the GUI the content of DATA_DIR is replaced by the currently selected test-directory.

To improve usability the output of the jupyter-notebook-execution is cached and once it was generated it will be displayed instantly.