Developer Guide

Test your Robotic Software with RoboVAST

RoboVAST is designed to facilitate testing and validation of robotic software systems by generating diverse scenarios and executing them in simulation environments. This guide provides an overview of how to utilize RoboVAST for testing your robotic applications.

1. Containerize your Software

As RoboVAST relies on containerization to ensure consistent and reproducible environments, the first step is to create a Docker container for your robotic software.

There are some requirements your container image must fulfill to be compatible with RoboVAST: - the image must contain scenario-execution package installed in /ws/install (which currently is available for ROS2 jazzy) - the image must be accessible by Kubernetes, e.g. by pushing it to a container registry.

2. Define a Test Scenario

Use the examples and the documentation of scenario-execution to create a scenario that tests your robotic software.

Keep in mind, that variations are currently supported for all overwritable scenario parameters as described here.

To test your scenario locally, you can run:

ros2 run scenario_execution_ros scenario_execution_ros <scenario-file> -t -d

3. Create Initial RoboVAST Configuration

Create a RoboVAST configuration file, based on the existing examples in the configs/ directory. Do not set any configuration, as this will be done in the next step.

vast exec local prepare-run --config config1 ./test_run

Afterwards you can verify the scenario, the RoboVAST-configuration and the docker image.

# execute a basic run
./test_run/run.sh

# use different container image
./test_run/run.sh --image <your-container-image>

# analyze issues by using an interactive shell
./test_run/run.sh --shell

# analyze network traffic, by using host network mode
./test_run/run.sh --network-host

# check that a standalone non-GUI environment (like in Kubernetes) works
./test_run/run.sh --no-gui

# enable extra scenario-execution output: live py-tree (-t) and debug log (-d)
./test_run/run.sh -t -d

To enable GUI visualization (e.g. RViz) for local runs while keeping cluster runs headless, add execution.local.parameter_overrides in your .vast file (see Configuration).

Next, it is important to verify that the output (e.g. ROS bag) is stored correctly.

vast exec local run --config config1 ./test_out

# check that output is created in ./test_out/<campaign-name>-<timestamp>/<config-name>/<run_number>
ls -l ./test_out/*-*/config1/0/

Once you are satisfied that the scenario and configuration work as expected, you can proceed to the next step.

4. Define Configurations

Define configurations in your .vast file. A good procedure is to add configurations one-by-one and analyze the result.

# 1. add configuration in config file

# 2. list created configurations
vast config list

# 3. try local execution with one of the created configurations
vast exec local run --config <config-name> --runs 1 ./test_out

5. Execute in Cluster

Once you have defined your configurations and verified local execution, you can run the tests in a Kubernetes cluster.

A good practice is, to first run a single configuration to verify that everything works as expected.

# 1. run single configuration in cluster, once
vast exec cluster run --config config1 --runs 1

# 2. upload results to share service (or use download-cleanup to just remove S3 buckets)
vast exec cluster upload-to-share
# Results can then be retrieved with: vast results download
# Files are organized as: <results-dir>/<campaign-name>-<timestamp>/<config-name>/<run_number>/

For long-running tests, you can use detached mode to run jobs in the background:

# Run in detached mode (command exits after creating jobs)
vast exec cluster run --detach

# Monitor job status (shows progress per run when multiple runs are active)
vast exec cluster monitor

# Clean up after jobs complete (all campaigns, or use --campaign for a specific campaign)
vast exec cluster run-cleanup

By default, a new run does not clean up previous runs, so you can run multiple runs in parallel. Use --cleanup to remove previous runs before starting (e.g. vast exec cluster run --cleanup).

Running local container images in minikube

To test local container images in a minikube cluster, you can load the image into minikube’s Docker environment.

# first terminal
docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000"

# second terminal
./container/build.sh --push

# specify the image in your RoboVAST configuration file

6. Analysis

RoboVAST provides a GUI for analyzing run results, which is based on user-provided Jupyter notebooks.

To develop the notebooks, it is recommended to use e.g. VSCode. For the RoboVAST GUI to work, it is expected to contain a DATA_DIR definition. The RoboVAST GUI will replace this line with the actual path to the results directory. During development you can set this variable manually to point to your results directory.

# for single-run (specific run of a configuration)
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>/<config-name>/<run_number>'
# for configuration (all configurations)
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>/<config-name>'
# for complete run
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>'

In case you are using ROS bags as output format, it is recommended to postprocess the results before analysis. This can be done with the postprocessing commands defined in the configuration file. RoboVAST provides several conversion scripts for common use-cases.

Postprocessing is cached based on the results directory hash. To bypass the cache and force postprocessing (e.g., after updating postprocessing scripts), use the --force or -f flag:

Afterwards you can start the GUI:

vast results postprocess
# or, to force postprocessing even if results are unchanged:
vast results postprocess --force
vast evaluation gui

Container Image Compatibility Version

RoboVAST enforces a compatibility version between the host Python code and the Docker container image. This prevents cryptic runtime failures when the two sides are out of sync (e.g. after updating one without the other).

How it works

A single integer COMPAT_VERSION is defined in src/robovast/common/execution.py. The same value is baked into the container image as the file /etc/robovast_compat_version.

Before any container starts, the version is checked by reading /etc/robovast_compat_version from inside the container:

Local execution: the generated run.sh script checks the file before docker-compose up.
Cluster execution: a Kubernetes init container reads the file and compares it to the expected value.
Postprocessing: docker_exec.sh checks the file before docker run.

If the versions do not match (or the file is missing), execution fails immediately with a clear error message.

When to bump the version

Bump COMPAT_VERSION when the contract between host scripts and the container changes:

A new Python or system package is required inside the container
The ROS distribution changes
The interface of mounted scripts changes (e.g. ros2_exec.sh, entrypoint.sh)
A postprocessing script requires a new ROS package

How to bump the version

Increment COMPAT_VERSION in src/robovast/common/execution.py
Update the LABEL and RUN echo lines in container/robovast/Dockerfile to match
Rebuild and push the container image

The CI workflow (image.yml) validates that all three values are in sync before building the image.

Extending RoboVAST

Add Variation Plugin

Provide your custom variation type by creating a class that inherits from robovast.common.variation.Variation.

To your pyproject.toml, add an entry under [tool.poetry.plugins.”robovast.variation_types”] to register your variation type. The key is the name used in the RoboVAST configuration file, and the value is the import path to your variation class.

[tool.poetry.plugins."robovast.variation_types"]
"YourVariation" = "robovast_<yourplugin>.your_variation:YourVariation"

Add Command-line Plugin

To create a plugin for the vast CLI:

Create a Click group or command in your package
Register it in your pyproject.toml under [tool.poetry.plugins.”robovast.cli_plugins”]
The plugin will be automatically discovered and added to the vast command

Example plugin registration:

[tool.poetry.plugins."vast.plugins"]
variation = "variation_utils.cli:variation"

Add Metadata Processing Plugin

Metadata processing plugins run after the generic and variation-plugin metadata phases and can modify the metadata.yaml produced for each campaign. They are configured in the .vast file under results_processing.metadata_processing:

results_processing:
  metadata_processing:
    - my_metadata_plugin
    - my_metadata_plugin:
        param1: value1
        param2: value2

Each plugin must subclass robovast.common.metadata.MetadataProcessor and implement the process_metadata method:

from pathlib import Path
from robovast.common.metadata import MetadataProcessor

class MyMetadataPlugin(MetadataProcessor):

    def process_metadata(self, metadata: dict, campaign_dir: Path) -> dict:
        # Modify metadata as needed
        metadata["custom_field"] = "custom_value"
        return metadata

Register the plugin in your package’s pyproject.toml:

[tool.poetry.plugins."robovast.metadata_processing"]
my_metadata_plugin = "my_package.metadata:MyMetadataPlugin"

Add Variation Plugin Metadata Hook

The Variation base class defines an overridable classmethod that returns an empty dict by default. Subclasses implement it to attach domain-specific metadata to each configuration entry in metadata.yaml:

from pathlib import Path
import yaml
from robovast.common.variation import Variation

class MyVariation(Variation):

    @classmethod
    def collect_config_metadata(cls, config_entry, config_dir: Path,
                                 campaign_dir: Path) -> dict:
        """Load extra metadata from a YAML sidecar in _config/."""
        data_file = config_dir / "_config" / "my_data.yaml"
        if data_file.exists():
            with open(data_file) as f:
                return {"my_data": yaml.safe_load(f)}
        return {}

collect_config_metadata is called once per configuration that used the variation and returns a dictionary that is merged into the configuration’s metadata entry.

Add PROV-O Provenance Hook to a Variation Plugin

Variation plugins can contribute domain-specific nodes to the campaign’s PROV-O provenance graph by overriding collect_prov_metadata on the Variation base class. The default implementation returns None (no contribution).

This hook is the right place for provenance that is tightly coupled to a specific variation — for example, a floorplan generation variation knows which map and mesh files it produced and can declare their lineage in the graph.

Return type: ProvContribution (or None to contribute nothing):

from robovast.common.variation import Variation, ProvContribution

class MyVariation(Variation):

    @classmethod
    def collect_prov_metadata(
        cls,
        config_entry: dict,
        campaign_namespace,   # rdflib.Namespace for the campaign
        config_namespace,     # rdflib.Namespace for this config
        gen_activity_id: str, # IRI of the config-generation activity
        vast_id: str, # IRI of the vast file that contains it
    ):
        """Contribute domain-specific PROV-O nodes."""
        from rdflib import PROV, Namespace

        _ID, _TYPE = "@id", "@type"
        MY_NS = Namespace("https://example.org/metamodels/")

        config_cfg = config_entry.get("config", {})
        my_file = config_cfg.get("my_output_file", "")
        if not my_file:
            return None

        file_iri = config_namespace[my_file]

        return ProvContribution(
            # Extra graph nodes (entities, activities) appended to @graph
            graph_nodes=[{
                _ID: file_iri,
                _TYPE: PROV["Entity"],
                "wasGeneratedBy": gen_activity_id,
                MY_NS["someProperty"]: "value",
            }],
            # Properties merged onto the concrete scenario node
            scenario_properties={MY_NS["outputCount"]: 1},
            # IRIs that each run activity should declare as "used"
            run_used_iris=[file_iri],
        )

ProvContribution fields:

graph_nodes: List of JSON-LD node dictionaries appended to the PROV @graph. Use rdflib.PROV, rdflib.DCTERMS, or your own Namespace objects as keys/values.
scenario_properties: Dict merged onto the concrete scenario entity node for this configuration. Useful for adding counts or classification properties (e.g. number of goals, number of obstacles).
run_used_iris: List of IRIs that every run activity in this configuration will declare as prov:used. Typically the IRIs of entities generated by this variation that are consumed at runtime (e.g. a map file, a mesh file).

Note

collect_prov_metadata receives rdflib.Namespace objects (campaign_namespace, config_namespace) so you can construct campaign-relative IRIs with campaign_namespace["some/path"]. rdflib is a required dependency of the core robovast package.

Add Postprocessing Command Plugin

Postprocessing plugins are Python functions that process run result directories (e.g., convert rosbag data to CSV). They are registered as entry points and executed before analysis.

Return value: A plugin must return (success: bool, message: str). It may optionally return a third value, a list of provenance entries, so that each produced file is recorded (e.g. which CSV was created from which rosbag). Each entry is a dict with keys: output (path relative to results_dir), sources (list of paths), plugin (plugin name), params (optional dict). If returned, these entries are merged and written into postprocessing.yaml in each run folder (<campaign-name>-<timestamp>/<config>/<run-number>/).

Provenance for container scripts: Plugins that run scripts inside Docker (e.g. via docker_exec.sh) cannot return data directly. The orchestrator passes a provenance file path to each plugin (optional kwarg provenance_file). Container-invoking plugins must pass this to docker_exec.sh as --provenance-file HOST_PATH; docker_exec.sh mounts the directory at /provenance in the container and the script receives --provenance-file /provenance/<basename>. The script should write a JSON file at that path with format {"entries": [{"output": "...", "sources": [...], "plugin": "...", "params": {}}]} (paths relative to the results/input directory). Use the helper write_provenance_entry from rosbags_common (same directory as the scripts, so it works in the container) to append entries; the script gets the path from --provenance-file and uses its own plugin name when calling the helper.

Creating a Postprocessing Plugin:

from typing import Tuple, Optional, List

def my_postprocessing_command(
    results_dir: str,
    config_dir: str,
    custom_param: Optional[str] = None,
    provenance_file: Optional[str] = None,
) -> Tuple[bool, str]:
    """Convert custom data to CSV.

    Args:
        results_dir: Path to the <campaign-name>-<timestamp> run directory to process
        config_dir: Config file directory (for resolving relative paths)
        custom_param: Optional custom parameter
        provenance_file: Optional path for provenance JSON (for container scripts)

    Returns:
        Tuple of (success, message) or (success, message, provenance_entries)
    """
    import subprocess
    import os

    script = os.path.join(config_dir, "tools/script.sh")
    cmd = [script, results_dir]
    if custom_param:
        cmd.extend(["--param", custom_param])

    result = subprocess.run(cmd, capture_output=True, text=True)

    if result.returncode != 0:
        return False, f"Failed: {result.stderr}"
    return True, "Success"

Register in pyproject.toml:

[tool.poetry.plugins."robovast.postprocessing_commands"]
my_postprocessing_command = "your_package.postprocessing_plugins:my_postprocessing_command"

Usage in .vast config:

analysis:
  postprocessing:
    - my_postprocessing_command:
        custom_param: value

Add Publication Plugin

Publication plugins package or distribute the results directory after postprocessing. They are plain callables (functions or class instances) that operate on the full results directory.

Return value: A plugin must return (success: bool, message: str).

Creating a Publication Plugin:

from typing import Optional, Tuple

def my_publication_plugin(
    results_dir: str,
    config_dir: str,
    destination: Optional[str] = None,
) -> Tuple[bool, str]:
    """Upload results to a remote storage location.

    Args:
        results_dir: Path to the results directory (parent of campaign directories).
        config_dir: Directory containing the .vast config file; relative
            paths should be resolved from here.
        destination: Remote destination URL or path.

    Returns:
        Tuple of (success, message).
    """
    import subprocess
    dest = destination or "s3://my-bucket/results/"

    result = subprocess.run(
        ["aws", "s3", "sync", results_dir, dest],
        capture_output=True, text=True,
    )
    if result.returncode != 0:
        return False, f"Upload failed: {result.stderr}"
    return True, f"Uploaded results to {dest}"

Register in pyproject.toml:

[tool.poetry.plugins."robovast.publication_plugins"]
my_publication_plugin = "your_package.publication_plugins:my_publication_plugin"

Usage in .vast config:

results_processing:
  publication:
    - my_publication_plugin:
        destination: s3://my-bucket/results/

Add Cluster Config Plugin

To add a new cluster configuration option for RoboVAST, create a class that inherits from robovast.execution.cluster_config.base.BaseConfig. Register your cluster config in your pyproject.toml under [tool.poetry.plugins.”robovast.cluster_configs”]. The key is the name used to select the configuration, and the value is the import path to your configuration class.

[tool.poetry.plugins."robovast.cluster_configs"]
"YourClusterConfig" = "robovast_<yourplugin>.your_cluster_config:YourClusterConfig"

To test your cluster configuration, you can use:

vast exec cluster prepare-setup --cluster-config YourClusterConfig ./setup_output

The output directory will contain all necessary files and instructions to manually execute the setup steps for your cluster configuration and execution.

Add a MCP Plugin

Create a class with a name property and a register(mcp) method:

# my_package/mcp_plugin.py
from fastmcp import FastMCP

class MyMCPPlugin:
    @property
    def name(self) -> str:
        return "my_plugin"

    def register(self, mcp: FastMCP) -> None:
        @mcp.tool()
        def my_tool() -> str:
            """A custom tool."""
            return "hello"

Then register the class as an entry point in pyproject.toml:

[tool.poetry.plugins."robovast.mcp_plugins"]
my_plugin = "my_package.mcp_plugin:MyMCPPlugin"

The plugin is picked up automatically the next time the server starts.

Querying RoboVAST campaigns

Using [rdflib](https://rdflib.readthedocs.io/), you can query the generated metadata graph using [SPARQL](https://www.w3.org/TR/sparql11-query/).

Load the metadata graph

from rdflib import Graph

g = Graph()
g.parse("metadata.prov.json)

Loading SPARQL queries

To do so, you can load any of the queries below as text, and use the query method for any graph g:

with open("query-file.rq", "r") as f:
  query_string = f.read()

qres = g.query(query_string)
for row in qres:
  # Process your results
  print(row)

Below are a few example queries demonstrating the PROV relationships in the metadata graph.

Scenario inputs

FloorPlan models:

 SELECT ?floorplan ?creator ?date
 WHERE {
      ?floorplan rdf:type env:FloorPlanModel .
      OPTIONAL {?floorplan prov:wasAttributedTo ?creator .}
      OPTIONAL {?floorplan dcterms:modified ?date .}
}

Vast file:

SELECT ?vast_file ?creator ?date ?abstract_scenario
WHERE {
    ?vast_file rdf:type robovast:VastConfiguration .
    ?vast_file dcterms:references ?abstract_scenario .
    ?abstract_scenario rdf:type scenarios:AbstractScenario .
    OPTIONAL {?vast_file prov:wasAttributedTo ?creator .}
    OPTIONAL {?vast_file dcterms:modified ?date .}
}

Generation

FloorPlan Model-to-Model Transformation:

SELECT ?floorplan ?activity ?jsonld_file ?agent
WHERE {
    ?floorplan rdf:type env:FloorPlanModel .
    ?activity rdf:type robovast:FloorPlanTransformation .
    ?activity prov:used ?floorplan .
    ?jsonld_file prov:wasGeneratedBy ?activity .
    OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
}

FloorPlan Artefact Generation:

SELECT ?source_files ?activity ?gen_file ?agent
WHERE {
    ?activity rdf:type robovast:FloorPlanGeneration .
    ?activity prov:used ?source_files .
    ?gen_file prov:wasGeneratedBy ?activity .
    OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
}

Generation of Concrete Scenario

SELECT ?vast_file ?ref_file ?activity ?agent ?gen_file
WHERE {
    ?vast_file rdf:type robovast:VastConfiguration .
    ?activity prov:used ?vast_file .
    ?vast_file dcterms:references ?ref_file .
    ?gen_file prov:wasGeneratedBy ?activity .
    OPTIONAL {?activity prov:wasAssociatedWith ?agent .}

Test Execution

Test results generated from a test run:

SELECT ?scenario ?config_files ?activity ?agent ?gen_file ?start_time ?end_time
WHERE {
    ?scenario rdf:type smm:ConcreteScenario .
    ?activity prov:used ?scenario .
    ?activity prov:used ?config_files .
    OPTIONAL{?activity prov:startedAtTime ?start_time .}
    OPTIONAL{ ?activity prov:endedAtTime ?end_time . }
    ?gen_file prov:wasGeneratedBy ?activity .
    OPTIONAL {?activity prov:wasAssociatedWith ?agent .}

Postprocessing

Postprocessing of a bagfile:

SELECT ?bag_file ?activity ?agent ?gen_file ?start_time ?end_time
WHERE {
    ?bag_file rdf:type robovast:ROSBag .
    ?activity prov:used ?bag_file .
    ?gen_file prov:wasGeneratedBy ?activity .
    OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
    OPTIONAL{?activity prov:startedAtTime ?start_time .}
    OPTIONAL{ ?activity prov:endedAtTime ?end_time . }
}

Metadata and Graph postprocessing:

SELECT ?metadata_file ?graph_file ?md_activity ?graph_activity ?agent ?start_time ?end_time
WHERE {
    ?md_activity rdf:type robovast:PostprocessingMetadata .
    ?graph_activity rdf:type robovast:PostprocessingGraph .
    ?metadata_file prov:wasGeneratedBy ?md_activity .
    ?graph_file prov:wasGeneratedBy ?graph_activity .
    ?graph_activity prov:used ?metadata_file
    OPTIONAL {?md_activity prov:wasAssociatedWith ?agent .
    ?graph_activity prov:wasAssociatedWith ?agent .}
    OPTIONAL{?md_activity prov:startedAtTime ?start_time .}
    OPTIONAL{ ?md_activity prov:endedAtTime ?end_time . }
}

Analysis

Identifying which variation types were used on each config:

SELECT ?config ?variation_type
WHERE {
    ?config rdf:type smm:ConcreteScenario .
    ?config robovast:variations/rdf:rest*/rdf:first ?variation .
    ?variation rdf:type ?variation_type .
    FILTER (?variation_type != prov:Activity)
    FILTER (?variation_type != robovast:Variation)
}

Getting the failure rate by environment:

SELECT  ?env_model (SUM(?failures)/COUNT (?activity) * 100 AS ?result) (COUNT (?activity) AS ?total)
WHERE {
    ?conf rdf:type smm:ConcreteScenario .
    ?activity prov:used ?conf .
    ?activity rdf:type robovast:TestExecution .
    ?activity robovast:success ?success .
    BIND(IF(?success=true, 0, 1) AS ?failures) .
    ?conf dcterms:references ?env_model .
    ?env_model rdf:type env:FloorPlanModel .
} GROUP BY ?env_model