Developer Guide
Test your Robotic Software with RoboVAST
RoboVAST is designed to facilitate testing and validation of robotic software systems by generating diverse scenarios and executing them in simulation environments. This guide provides an overview of how to utilize RoboVAST for testing your robotic applications.
1. Containerize your Software
As RoboVAST relies on containerization to ensure consistent and reproducible environments, the first step is to create a Docker container for your robotic software.
There are some requirements your container image must fulfill to be compatible with RoboVAST: - the image must contain scenario-execution package installed in /ws/install (which currently is available for ROS2 jazzy) - the image must be accessible by Kubernetes, e.g. by pushing it to a container registry.
2. Define a Test Scenario
Use the examples and the documentation of scenario-execution to create a scenario that tests your robotic software.
Keep in mind, that variations are currently supported for all overwritable scenario parameters as described here.
To test your scenario locally, you can run:
ros2 run scenario_execution_ros scenario_execution_ros <scenario-file> -t -d
3. Create Initial RoboVAST Configuration
Create a RoboVAST configuration file, based on the existing examples in the configs/ directory. Do not set any configuration, as this will be done in the next step.
vast exec local prepare-run --config config1 ./test_run
Afterwards you can verify the scenario, the RoboVAST-configuration and the docker image.
# execute a basic run
./test_run/run.sh
# use different container image
./test_run/run.sh --image <your-container-image>
# analyze issues by using an interactive shell
./test_run/run.sh --shell
# analyze network traffic, by using host network mode
./test_run/run.sh --network-host
# check that a standalone non-GUI environment (like in Kubernetes) works
./test_run/run.sh --no-gui
# enable extra scenario-execution output: live py-tree (-t) and debug log (-d)
./test_run/run.sh -t -d
To enable GUI visualization (e.g. RViz) for local runs while keeping cluster runs headless, add execution.local.parameter_overrides in your .vast file (see Configuration).
Next, it is important to verify that the output (e.g. ROS bag) is stored correctly.
vast exec local run --config config1 ./test_out
# check that output is created in ./test_out/<campaign-name>-<timestamp>/<config-name>/<run_number>
ls -l ./test_out/*-*/config1/0/
Once you are satisfied that the scenario and configuration work as expected, you can proceed to the next step.
4. Define Configurations
Define configurations in your .vast file.
A good procedure is to add configurations one-by-one and analyze the result.
# 1. add configuration in config file
# 2. list created configurations
vast config list
# 3. try local execution with one of the created configurations
vast exec local run --config <config-name> --runs 1 ./test_out
5. Execute in Cluster
Once you have defined your configurations and verified local execution, you can run the tests in a Kubernetes cluster.
A good practice is, to first run a single configuration to verify that everything works as expected.
# 1. run single configuration in cluster, once
vast exec cluster run --config config1 --runs 1
# 2. upload results to share service (or use download-cleanup to just remove S3 buckets)
vast exec cluster upload-to-share
# Results can then be retrieved with: vast results download
# Files are organized as: <results-dir>/<campaign-name>-<timestamp>/<config-name>/<run_number>/
vast exec cluster run is fire-and-forget: it launches an in-cluster
controller pod that drives the campaign and returns immediately. The campaign
runs in the background in the cluster:
# Monitor job status (shows progress per run when multiple runs are active)
vast exec cluster monitor
# Clean up after jobs complete (all campaigns, or use --campaign for a specific campaign)
vast exec cluster run-cleanup
By default, a new run does not clean up previous runs, so you can run multiple
runs in parallel. Use --cleanup to remove previous runs before starting
(e.g. vast exec cluster run --cleanup).
Running local container images in minikube
To test local container images in a minikube cluster, you can load the image into minikube’s Docker environment.
# first terminal
docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000"
# second terminal
./container/build.sh --push
# specify the image in your RoboVAST configuration file
6. Analysis
RoboVAST provides a GUI for analyzing run results, which is based on user-provided Jupyter notebooks.
To develop the notebooks, it is recommended to use e.g. VSCode. For the RoboVAST GUI to work, it is expected to contain a DATA_DIR definition. The RoboVAST GUI will replace this line with the actual path to the results directory. During development you can set this variable manually to point to your results directory.
# for single-run (specific run of a configuration)
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>/<config-name>/<run_number>'
# for configuration (all configurations)
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>/<config-name>'
# for complete run
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>'
In case you are using ROS bags as output format, it is recommended to postprocess the results before analysis. This can be done with the postprocessing commands defined in the configuration file. RoboVAST provides several conversion scripts for common use-cases.
Postprocessing is cached based on the results directory hash. To bypass the cache and force postprocessing (e.g., after updating postprocessing scripts), use the --force or -f flag:
Afterwards you can start the GUI:
vast results postprocess
# or, to force postprocessing even if results are unchanged:
vast results postprocess --force
vast evaluation gui
Note
The GUI discovers campaigns exclusively from a per-campaign
``campaign.db`` store — it does not walk the results filesystem. Search
campaigns write this store live; batch campaigns are indexed post-hoc from
their results tree. vast evaluation gui indexes any missing batch stores
automatically before launching, but you can also (re)build them explicitly:
vast evaluation index # build/refresh campaign stores
vast evaluation index --force # rebuild even if up to date
The store also carries the campaign mode (batch/search), so the
GUI renders the search batch level and resolves the
evaluation.visualization notebooks from the recorded config_dir. See
Campaign Store and Results Indexing for the schema and internals.
Container Image Compatibility Version
RoboVAST enforces a compatibility version between the host Python code and the Docker container image. This prevents cryptic runtime failures when the two sides are out of sync (e.g. after updating one without the other).
How it works
A single integer COMPAT_VERSION is defined in
src/robovast/common/execution.py. The same value is baked into the
container image as the file /etc/robovast_compat_version.
Before any container starts, the version is checked by reading
/etc/robovast_compat_version from inside the container:
Local execution: the generated
run.shscript checks the file beforedocker-compose up.Cluster execution: a Kubernetes init container reads the file and compares it to the expected value.
Postprocessing:
docker_exec.shchecks the file beforedocker run.
If the versions do not match (or the file is missing), execution fails immediately with a clear error message.
When to bump the version
Bump COMPAT_VERSION when the contract between host scripts and the
container changes:
A new Python or system package is required inside the container
The ROS distribution changes
The interface of mounted scripts changes (e.g.
ros2_exec.sh,entrypoint.sh)A postprocessing script requires a new ROS package
How to bump the version
Increment
COMPAT_VERSIONinsrc/robovast/common/execution.pyUpdate the
LABELandRUN echolines incontainer/robovast/Dockerfileto matchRebuild and push the container image
The CI workflow (image.yml) validates that all three values are in sync
before building the image.
Extending RoboVAST
Add Variation Plugin
Provide your custom variation type by creating a class that inherits from robovast.common.variation.Variation.
To your pyproject.toml, add an entry under [tool.poetry.plugins.”robovast.variation_types”] to register your variation type. The key is the name used in the RoboVAST configuration file, and the value is the import path to your variation class.
[tool.poetry.plugins."robovast.variation_types"]
"YourVariation" = "robovast_<yourplugin>.your_variation:YourVariation"
A variation can also be loaded from a local file relative to the .vast
without packaging it — reference it as <path>.py:<Class> wherever a variation
name is expected (in a configuration[].variations list or a search.variations
template). This is the same ./path.py:Class convention used by search
strategies, extractors and postprocessing plugins:
variations:
- variations/wind.py:WindFieldVariation:
wind_speed: 5.0
See configs/examples/quadrotor_landing/variations/wind.py for a runnable
example (a wind model that derives the simulator’s wind_strength), wired into
the quadrotor search vasts.
Note
Packaging a variation plugin as its own distribution. If your variation
types live in a separate installable package (as robovast-nav does for
FloorplanVariation, PathVariation*, ObstacleVariation*), it must
be installed everywhere scenario variations get composed — not just
where scenarios run. For local and host-driven cluster runs that’s the
host venv; for vast execution cluster run (search/batch) composition
happens inside the in-cluster controller pod, so the controller image
(container/controller/Dockerfile) must install the plugin package too,
or composing a config that references your variation type will fail there
with Unknown variation class.
Two pitfalls when exposing the package as a poetry extra (e.g.
nav = ["robovast-nav"]):
The extra’s package name must also be declared as an optional dependency in
[tool.poetry.dependencies](e.g.robovast-nav = {path = "src/robovast_nav", optional = true}for an in-repo sibling package) —poetry checkcatches the mismatch if not.The controller’s dev-iteration fast path (
controller_launcher.build_dev_wheels) builds a wheel of the currentrobovastsource for quick redeploys; if your plugin lives in a separate poetry project undersrc/, it needs its own wheel built and shipped alongside (asrobovast_navdoes) or dev changes to it won’t reach the controller pod.
If your plugin’s package pulls in a dependency that itself needs system
shared libraries (e.g. robovast-nav hard-depends on
pyside6-essentials, whose bundled Qt6 libs need libGL.so.1 and
friends to even import, regardless of whether any GUI is ever shown), the
controller image needs those apt packages too — see the
container/controller/Dockerfile apt-get block for the list verified
against robovast-nav. A missing system lib shows up the same way as a
missing extra: the plugin’s entry point fails to load and the variation
type is reported as unknown.
Add Command-line Plugin
To create a plugin for the vast CLI:
Create a Click group or command in your package
Register it in your pyproject.toml under [tool.poetry.plugins.”robovast.cli_plugins”]
The plugin will be automatically discovered and added to the vast command
Example plugin registration:
[tool.poetry.plugins."vast.plugins"]
variation = "variation_utils.cli:variation"
Add Metadata Processing Plugin
Metadata processing plugins run after the generic and variation-plugin metadata
phases and can modify the metadata.yaml produced for each campaign. They are
configured in the .vast file under results_processing.metadata_processing:
results_processing:
metadata_processing:
- my_metadata_plugin
- my_metadata_plugin:
param1: value1
param2: value2
Each plugin must subclass robovast.common.metadata.MetadataProcessor and
implement the process_metadata method:
from pathlib import Path
from robovast.common.metadata import MetadataProcessor
class MyMetadataPlugin(MetadataProcessor):
def process_metadata(self, metadata: dict, campaign_dir: Path) -> dict:
# Modify metadata as needed
metadata["custom_field"] = "custom_value"
return metadata
Register the plugin in your package’s pyproject.toml:
[tool.poetry.plugins."robovast.metadata_processing"]
my_metadata_plugin = "my_package.metadata:MyMetadataPlugin"
Add Variation Plugin Metadata Hook
The Variation base class defines an overridable classmethod that returns an
empty dict by default. Subclasses implement it to attach domain-specific metadata
to each configuration entry in metadata.yaml:
from pathlib import Path
import yaml
from robovast.common.variation import Variation
class MyVariation(Variation):
@classmethod
def collect_config_metadata(cls, config_entry, config_dir: Path,
campaign_dir: Path) -> dict:
"""Load extra metadata from a YAML sidecar in _config/."""
data_file = config_dir / "_config" / "my_data.yaml"
if data_file.exists():
with open(data_file) as f:
return {"my_data": yaml.safe_load(f)}
return {}
collect_config_metadata is called once per configuration that used the
variation and returns a dictionary that is merged into the configuration’s
metadata entry.
Add PROV-O Provenance Hook to a Variation Plugin
Variation plugins can contribute domain-specific nodes to the campaign’s
PROV-O provenance graph by overriding collect_prov_metadata on the
Variation base class. The default implementation returns None
(no contribution).
This hook is the right place for provenance that is tightly coupled to a specific variation — for example, a floorplan generation variation knows which map and mesh files it produced and can declare their lineage in the graph.
Return type: ProvContribution (or None to contribute nothing):
from robovast.common.variation import Variation, ProvContribution
class MyVariation(Variation):
@classmethod
def collect_prov_metadata(
cls,
config_entry: dict,
campaign_namespace, # rdflib.Namespace for the campaign
config_namespace, # rdflib.Namespace for this config
gen_activity_id: str, # IRI of the config-generation activity
vast_id: str, # IRI of the vast file that contains it
):
"""Contribute domain-specific PROV-O nodes."""
from rdflib import PROV, Namespace
_ID, _TYPE = "@id", "@type"
MY_NS = Namespace("https://example.org/metamodels/")
config_cfg = config_entry.get("config", {})
my_file = config_cfg.get("my_output_file", "")
if not my_file:
return None
file_iri = config_namespace[my_file]
return ProvContribution(
# Extra graph nodes (entities, activities) appended to @graph
graph_nodes=[{
_ID: file_iri,
_TYPE: PROV["Entity"],
"wasGeneratedBy": gen_activity_id,
MY_NS["someProperty"]: "value",
}],
# Properties merged onto the concrete scenario node
scenario_properties={MY_NS["outputCount"]: 1},
# IRIs that each run activity should declare as "used"
run_used_iris=[file_iri],
)
ProvContribution fields:
graph_nodesList of JSON-LD node dictionaries appended to the PROV
@graph. Userdflib.PROV,rdflib.DCTERMS, or your ownNamespaceobjects as keys/values.scenario_propertiesDict merged onto the concrete scenario entity node for this configuration. Useful for adding counts or classification properties (e.g. number of goals, number of obstacles).
run_used_irisList of IRIs that every run activity in this configuration will declare as
prov:used. Typically the IRIs of entities generated by this variation that are consumed at runtime (e.g. a map file, a mesh file).
Note
collect_prov_metadata receives rdflib.Namespace objects
(campaign_namespace, config_namespace) so you can construct
campaign-relative IRIs with campaign_namespace["some/path"].
rdflib is a required dependency of the core robovast package.
Add Postprocessing Command Plugin
Postprocessing plugins are Python functions that process run result directories (e.g., convert rosbag data to CSV). They are registered as entry points and executed before analysis.
Return value: A plugin must return (success: bool, message: str). It may optionally return a third value, a list of provenance entries, so that each produced file is recorded (e.g. which CSV was created from which rosbag). Each entry is a dict with keys: output (path relative to results_dir), sources (list of paths), plugin (plugin name), params (optional dict). If returned, these entries are merged and written into postprocessing.yaml in each run folder (<campaign-name>-<timestamp>/<config>/<run-number>/).
Provenance for container scripts: Plugins that run scripts inside Docker (e.g. via docker_exec.sh) cannot return data directly. The orchestrator passes a provenance file path to each plugin (optional kwarg provenance_file). Container-invoking plugins must pass this to docker_exec.sh as --provenance-file HOST_PATH; docker_exec.sh mounts the directory at /provenance in the container and the script receives --provenance-file /provenance/<basename>. The script should write a JSON file at that path with format {"entries": [{"output": "...", "sources": [...], "plugin": "...", "params": {}}]} (paths relative to the results/input directory). Use the helper write_provenance_entry from rosbags_common (same directory as the scripts, so it works in the container) to append entries; the script gets the path from --provenance-file and uses its own plugin name when calling the helper.
Creating a Postprocessing Plugin:
from typing import Tuple, Optional, List
def my_postprocessing_command(
results_dir: str,
config_dir: str,
custom_param: Optional[str] = None,
provenance_file: Optional[str] = None,
) -> Tuple[bool, str]:
"""Convert custom data to CSV.
Args:
results_dir: Path to the <campaign-name>-<timestamp> run directory to process
config_dir: Config file directory (for resolving relative paths)
custom_param: Optional custom parameter
provenance_file: Optional path for provenance JSON (for container scripts)
Returns:
Tuple of (success, message) or (success, message, provenance_entries)
"""
import subprocess
import os
script = os.path.join(config_dir, "tools/script.sh")
cmd = [script, results_dir]
if custom_param:
cmd.extend(["--param", custom_param])
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
return False, f"Failed: {result.stderr}"
return True, "Success"
Register in pyproject.toml:
[tool.poetry.plugins."robovast.postprocessing_commands"]
my_postprocessing_command = "your_package.postprocessing_plugins:my_postprocessing_command"
Usage in .vast config:
analysis:
postprocessing:
- my_postprocessing_command:
custom_param: value
Add Publication Plugin
Publication plugins package or distribute the results directory after postprocessing. They are plain callables (functions or class instances) that operate on the full results directory.
Return value: A plugin must return (success: bool, message: str).
Creating a Publication Plugin:
from typing import Optional, Tuple
def my_publication_plugin(
results_dir: str,
config_dir: str,
destination: Optional[str] = None,
) -> Tuple[bool, str]:
"""Upload results to a remote storage location.
Args:
results_dir: Path to the results directory (parent of campaign directories).
config_dir: Directory containing the .vast config file; relative
paths should be resolved from here.
destination: Remote destination URL or path.
Returns:
Tuple of (success, message).
"""
import subprocess
dest = destination or "s3://my-bucket/results/"
result = subprocess.run(
["aws", "s3", "sync", results_dir, dest],
capture_output=True, text=True,
)
if result.returncode != 0:
return False, f"Upload failed: {result.stderr}"
return True, f"Uploaded results to {dest}"
Register in pyproject.toml:
[tool.poetry.plugins."robovast.publication_plugins"]
my_publication_plugin = "your_package.publication_plugins:my_publication_plugin"
Usage in .vast config:
results_processing:
publication:
- my_publication_plugin:
destination: s3://my-bucket/results/
Add Cluster Config Plugin
To add a new cluster configuration option for RoboVAST, create a class that inherits from robovast.execution.cluster_config.base.BaseConfig. Register your cluster config in your pyproject.toml under [tool.poetry.plugins.”robovast.cluster_configs”]. The key is the name used to select the configuration, and the value is the import path to your configuration class.
[tool.poetry.plugins."robovast.cluster_configs"]
"YourClusterConfig" = "robovast_<yourplugin>.your_cluster_config:YourClusterConfig"
To test your cluster configuration, you can use:
vast exec cluster prepare-setup --cluster-config YourClusterConfig ./setup_output
The output directory will contain all necessary files and instructions to manually execute the setup steps for your cluster configuration and execution.
Add a MCP Plugin
Create a class with a name property and a register(mcp) method:
# my_package/mcp_plugin.py
from fastmcp import FastMCP
class MyMCPPlugin:
@property
def name(self) -> str:
return "my_plugin"
def register(self, mcp: FastMCP) -> None:
@mcp.tool()
def my_tool() -> str:
"""A custom tool."""
return "hello"
Then register the class as an entry point in pyproject.toml:
[tool.poetry.plugins."robovast.mcp_plugins"]
my_plugin = "my_package.mcp_plugin:MyMCPPlugin"
The plugin is picked up automatically the next time the server starts.
Add Search Strategy Plugin
A search strategy drives the closed-loop search (see Iterative Search): it proposes
parameter sets, is told their evaluations, and produces a final report. Strategies
are algorithm-agnostic and share one config schema; only the per-strategy
strategy_parameters block differs.
Subclass robovast.search.strategy.SearchStrategy and implement the four
abstract methods. Optionally set PARAMS_MODEL to a Pydantic model — the
framework validates search.strategy_parameters against it and passes the
parsed object as params:
from pydantic import BaseModel
from robovast.search.strategy import SearchStrategy
from robovast.search.types import ParamSet, SearchReport
class MyParams(BaseModel):
step: float = 0.1
class MyStrategy(SearchStrategy):
PARAMS_MODEL = MyParams # optional; omit (None) for no parameters
def ask(self, n: int) -> list[ParamSet]:
"""Propose n parameter sets (keys match search_space dims)."""
...
def tell(self, evaluations) -> None:
"""Ingest the evaluations of the batch just run."""
...
def is_done(self) -> bool:
"""True when the budget is exhausted / converged."""
...
def report(self) -> SearchReport:
"""Return the deliverable (ranked best, archive, Pareto front)."""
...
self.search_space, self.objectives and the validated self.params are
available on the instance. For single-objective strategies, self.objective_value(ev)
returns the sole objective sign-oriented so that higher is always better.
Register the class under robovast.search_strategies; the key is the
search.strategy name:
[tool.poetry.plugins."robovast.search_strategies"]
my_strategy = "your_package.strategies:MyStrategy"
A strategy can also be loaded from a local file relative to the .vast without
packaging, using strategy: ./search/my_strategy.py:MyStrategy (the same
load_ref mechanism used for extractors and search postprocessing).
Add Extractor Plugin
The extractor is the single, SUT-specific scoring step: it reads a parameter set’s per-config result directory and returns named objectives (optimized) and measures (quality-diversity behavior axes; ignored by non-QD strategies). This is the one place system-under-test logic lives.
Subclass robovast.search.extractor.Extractor. It is constructed with the
extract.params from the .vast (so thresholds / column names can be swept
without editing code), and aggregation over the config’s runs is its
responsibility:
from pathlib import Path
from robovast.search.extractor import (Extractor, ExtractResult,
completed_run_dirs)
class MyExtract(Extractor):
# __init__(self, **params) is inherited; params land on self.params
def extract(self, config_dir: Path) -> ExtractResult:
runs = completed_run_dirs(config_dir) # helper: finished runs
failures = sum(1 for r in runs if _failed(r))
return ExtractResult(
objectives={"failure_rate": failures / max(len(runs), 1)},
measures={}, # {} when unused
)
objectives and measures are named dicts, so single- and multi-objective
use the same shape. The framework records how many runs backed each result.
Register under robovast.extractors (referenced by search.extract.plugin),
or load from a local file with extract.plugin: ./search/extract.py:MyExtract:
[tool.poetry.plugins."robovast.extractors"]
my_extract = "your_package.extractors:MyExtract"
The extractor reads what a postprocessing plugin produced (e.g. per-run
metrics.csv) — it no longer computes raw metrics itself. Pair it with a
postprocessing plugin (below) that writes metrics.csv from raw artifacts:
list that plugin in search.postprocessing (run before extract) and/or in
results_processing.postprocessing (analysis), so search and the analysis
notebooks read the same metrics. Postprocessing plugins load identically in both
lists — by entry-point name or a local ./path.py:Class file reference.
Campaign Store and Results Indexing
Every campaign — batch or search — is described by a single sqlite store,
campaign.db (robovast.common.store.STORE_FILENAME), written at the
root of the campaign directory. It is the single source of truth the results
GUI reads, and the seam an in-cluster controller or web UI can later read/stream.
A campaign runs one or more batches: a batch-mode campaign (no search:
block) has exactly one batch of the enumerated configs; a search campaign has one
batch per ask/tell round.
Schema
robovast.common.store.CampaignStore is a thin wrapper over three tables:
campaign (1) --< batch (1) --< unit (one per param set / config)
campaign —
mode(batch/search),config_dir(base directory against whichevaluation.visualizationnotebooks resolve),config_json(the full config), and an opaquestrategy_stateblob for resumable strategies.batch — one ask/tell round (search), or the single batch (
idx=0) of a batch-mode campaign.unit — one evaluated parameter set (search) or one configuration (batch): the sampled
params,objectives/measures(JSON;{}for batch),n_samples, an aggregatestatusand theresult_dir.
Who writes it
The controller (
robovast.execution.controller.CampaignController) writes the store live for both modes as each batch is evaluated, so progress is queryable while a campaign runs. It owns the campaign id, the flat results layout (<campaign>/<config>/<run>/) and the batch loop; anExecutionBackend(DockerBackendlocally) only dispatches one batch’s jobs.The post-hoc indexer
robovast.common.campaign_index.build_campaign_store(campaign_dir)reconstructs the same store by scanning a finished results tree (reusing thecampaign_datareaders). It is used for campaign dirs not produced by the controller — e.g. cluster results downloaded from S3 — and is idempotent (mtime-guarded;force=Trueto rebuild), invoked byvast evaluation indexand automatically onvast evaluation guilaunch. Controller-written stores are left untouched.
Store-driven GUI
The results GUI (RunResultsAnalyzer) discovers campaigns by scanning
<results_dir>/*/campaign.db — there is no filesystem-walk or depth-based
heuristic. It reads the campaign/batch/unit rows to build the tree
(campaign → batch, search only → config), resolves notebook workloads from
config_json against config_dir, and enumerates only the run-level leaves
from each unit’s result_dir.
Controller Control Interface
Every cluster campaign is driven by a fire-and-forget controller pod. While it runs (and after it finishes) the host talks to it through a small in-pod HTTP/JSON control channel — the live seam for monitoring and for issuing commands such as a graceful stop or an upload retry.
Server
robovast.execution.control_server runs a FastAPI + uvicorn server on a
daemon thread beside the synchronous controller loop (default port 8099,
ROBOVAST_CONTROL_PORT). Startup is best-effort: if it fails (e.g.
uvicorn missing) the campaign continues and the monitor falls back to its
Kubernetes-only view. FastAPI also serves the live OpenAPI schema at /docs,
so the same contract serves the CLI now and a web UI later.
Endpoints
GET /status— the controller’s liveStatus(loop phase, current batch, budget/run progress, history). The CLImonitorpolls this;phaseis the authoritative “done” signal.POST /command— an extensible RPC: a JSON body{name, args}is dispatched through the handler registry and returns aCommandResult({ok, result, error}). An unknown command returns HTTP 400.GET /healthz— liveness ({"ok": true}).
Status: phase and stage
phase is an open string advanced through a documented vocabulary, with
stage carrying finer markers so new states slot in without a schema change:
|
Meaning |
|---|---|
|
Campaign created; batch loop executing. |
|
Search stop condition met (or a |
|
Campaign published to storage; compressing/uploading to the share. The
|
|
Done ( |
|
Aborted. |
Commands
Handlers are registered in the HANDLERS
registry via the @register("name") decorator and receive
(state, **args). Built-in commands:
stop— cooperative graceful stop. During the batch loop it ends the search after the current batch; while the controller is parked waiting to retry a failed upload it abandons that wait and terminates.upload-to-share— (re)run the post-campaign upload.argsmay carry credential overrides (e.g. a corrected password); with no args the launch-time credentials are reused. The handler only signals — the actual upload runs on the controller’s main thread (see below) — so callers pollGET /statusfor thestagetransition.
State ownership and the retrigger handshake
ControllerState is the thread-safe
holder the controller writes and the server reads. The controller calls
update / set_phase at each batch boundary; the server thread reads a
consistent snapshot. Long-running work never executes on a request thread:
upload-to-share calls request_upload(overrides) (and stop calls
request_stop()), which wake the controller’s main thread blocked in
wait_for_retrigger() — it returns ("retrigger", overrides) to retry or
("abandon", {}) when a stop was requested.
Host-side client
robovast.execution.cluster_execution.control_client reaches the server over
kubectl port-forward (the same transport the launcher uses, so it needs only
the user’s kubeconfig — no extra RBAC): find_controller_pod locates the pod
by label, port_forward opens the tunnel, and get_status / send_command
call the endpoints. The CLI monitor, stop and upload-to-share are
thin wrappers over these.
The upload-to-share command may carry credential overrides, and because they
populate os.environ before the provider is loaded they can also switch the
share type (e.g. retry a failed gcs upload to sftp). The active share type is
reported in Status.share_provider and shown by monitor while uploading.
Warning
Trust model. The control server is unauthenticated — any principal
that can kubectl port-forward to the controller pod (RBAC verb
pods/portforward in the namespace) can issue commands. Because a retrigger
can switch the upload destination, such a principal can redirect a
campaign’s results to an arbitrary server (data exfiltration). A principal
with pods/exec could already read the data directly, so this only widens
exposure for port-forward-only principals. On shared/multi-tenant clusters,
restrict pods/portforward (and pods/exec) on the robovast namespace
accordingly. Adding a bearer-token check on POST /command is a possible
future hardening but is out of scope today.
API reference
In-controller HTTP/JSON control channel (state + RPC).
Every cluster campaign is driven by a fire-and-forget controller pod (see
robovast.execution.cluster_execution.controller_launcher). The controller
loop deletes each batch’s Kubernetes Jobs once their results are downloaded, so a
client that reconstructs progress purely from live Jobs cannot tell the gap
between search generations from the real end of the campaign, nor see the
loop-level state (current batch, search budget, run-level progress).
This module gives the controller a tiny FastAPI + uvicorn server, run on a daemon thread beside the synchronous controller loop:
GET /status— the controller’s liveStatus(loop phase, current batch, budget progress, per-batch run progress, history). The CLImonitorpolls this; thephasefield is the authoritative “done” signal.POST /command— an extensible RPC: dispatch{name, args}through theHANDLERSregistry. Ships one handler (stop); register more later.GET /healthz— liveness.
FastAPI auto-emits an OpenAPI schema (/docs), so the same contract serves the
CLI now and a web UI later (reached via kubectl port-forward now, a Service /
Ingress later — no code change).
fastapi / uvicorn are imported lazily (only build_app() /
serve_in_thread() need them) so the models and ControllerState
import cleanly anywhere; pydantic is a core dependency.
- class robovast.execution.control_server.Command(*, name: str, args: dict = <factory>)
An RPC request: a registered handler name plus its keyword args.
- args: dict
- model_config: ClassVar[ConfigDict] = {}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- name: str
- class robovast.execution.control_server.CommandResult(*, ok: bool, result: Any = None, error: str | None = None)
- error: str | None
- model_config: ClassVar[ConfigDict] = {}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- ok: bool
- result: Any
- class robovast.execution.control_server.ControllerState(**initial)
Thread-safe holder the controller writes and the server reads.
The controller calls
update()/set_phase()at each batch boundary (andupdate()for run-level progress within a batch); the server thread reads a consistentsnapshot().request_stop()/stop_requestedback the cooperativestopcommand.- request_stop() None
- request_upload(overrides: dict | None = None) None
Ask the controller’s main thread to (re)run upload-to-share.
overrides are optional
{ENV_VAR: value}credential corrections applied before the next attempt (the manual retrigger usually needs no args — the launch-time credentials are still in the pod).
- set_phase(phase: str, stage: str | None = None) None
- property stop_requested: bool
- update(**fields) None
- wait_for_retrigger() tuple[str, dict]
Block until an upload retrigger or a stop is requested.
Returns
("retrigger", overrides)to retry the upload, or("abandon", {})when astopwas requested (give up, terminate).
- class robovast.execution.control_server.Status(*, phase: str = 'starting', stage: str | None = None, mode: str | None = None, campaign_id: str | None = None, batch: int = 0, batches_done: int = 0, budget: list[~robovast.execution.control_server.BudgetItem] = <factory>, runs: ~robovast.execution.control_server.RunProgress = <factory>, best_objective: float | None = None, batch_history: list[dict] = <factory>, stop: dict | None = None, share_provider: str | None = None, extra: dict = <factory>, updated_at: float = <factory>)
The controller’s live state, served by
GET /status.phaseis an open string the controller advances through a documented vocabulary (starting→running→finishing→finished/failed);stageandextraexist so future markers (e.g."upload-to-share-done") slot in without a schema change.share_providernames the share type of the current upload attempt; it can change across retriggers (a failed upload may be retried to a different provider).- batch: int
- batch_history: list[dict]
- batches_done: int
- best_objective: float | None
- budget: list[BudgetItem]
- campaign_id: str | None
- extra: dict
- mode: str | None
- model_config: ClassVar[ConfigDict] = {'validate_assignment': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- phase: str
- runs: RunProgress
- stage: str | None
- stop: dict | None
- updated_at: float
- robovast.execution.control_server.dispatch(state: ControllerState, command: Command) CommandResult
- robovast.execution.control_server.register(name: str) Callable[[Callable], Callable]
Host-side client for the in-controller control channel.
The CLI monitor (and future tools) reach the controller pod’s HTTP server
(see robovast.execution.control_server) through kubectl port-forward —
the same kubectl transport the launcher and archiver use, so it relies only on
the user’s kubeconfig (no extra RBAC). Helpers here locate the controller pod,
open a port-forward, and call GET /status / POST /command.
- robovast.execution.cluster_execution.control_client.find_controller_pod(namespace='default', kube_context=None, campaign=None)
Return
(pod_name, phase)of the controller pod, or(None, None).Prefers a Running pod (the live controller); otherwise returns the most recent terminal pod so the monitor can report a campaign that has already finished. With campaign given, restricts to that campaign’s controller (
campaign-id=<label-safe>).
- robovast.execution.cluster_execution.control_client.get_status(base_url, timeout=5.0) dict
GET /status-> parsed JSON dict.
- robovast.execution.cluster_execution.control_client.port_forward(pod, namespace='default', kube_context=None, remote_port=8099, timeout=15.0)
Open a
kubectl port-forwardto pod and yield the localhost base URL.kubectl picks a free local port (
:<remote>); we parse it from kubectl’s output. The forward is torn down on exit.
- robovast.execution.cluster_execution.control_client.send_command(base_url, name, timeout=10.0, **args) dict
POST /commandwith{name, args}-> parsed JSONCommandResult.
Per-Cluster Resource Resolution
The resolution logic for per-cluster resources (the {context-name: value}
mappings documented under Per-Cluster Resource Limits in
Cluster Execution) lives in robovast.common.cluster_context:
Kubernetes context awareness and per-cluster resource resolution.
Resource values in the .vast config file may be given as a per-cluster
list keyed by the real Kubernetes context name instead of a scalar:
resources:
cpu:
- gke_my-project_us-central1_my-cluster: 4
- minikube: 8
memory:
- gke_my-project_us-central1_my-cluster: 10Gi
- minikube: 20Gi
Scalars always work and are the recommended default when a single cluster is
used. Pass the matching context name via --context/-x when running
commands against a specific cluster.
- robovast.common.cluster_context.get_active_kube_context() str | None
Return the name of the currently active Kubernetes context.
Reads the active context from the local kubeconfig (
~/.kube/configorKUBECONFIG). ReturnsNonewhen the context cannot be determined (e.g. kubeconfig is absent).
- robovast.common.cluster_context.get_config_context_names(config_path: str) set[str]
Extract all context names used in per-cluster resource lists.
Scans a
.vastYAML config for any field that uses the per-cluster list syntax ([{context-name: value}, …]) and returns the union of all keys.- Parameters:
config_path – Absolute path to a
.vastYAML config file.- Returns:
Set of context name strings. Empty when no per-cluster lists are found.
- robovast.common.cluster_context.list_all_contexts() list[tuple[str, str]]
List all available
(label, kube_context_name)pairs from the kubeconfig.- Returns:
List of
(label, kube_context_name)tuples sorted by name. Returns an empty list when no kubeconfig is available.
- robovast.common.cluster_context.require_context_for_multi_cluster(kube_context: str | None) None
Raise
ValueErrorwhen a multi-cluster config is used without--context.Discovers the project
.vastconfig file, scans it for per-cluster resource lists, and raises an informative error when more than one context name is present and no kube_context was specified.This is a no-op when:
kube_context is already set (the user supplied
--context).No project config can be found.
The config uses only a single context name (or only plain scalars).
- Parameters:
kube_context – The Kubernetes context name (
Nonewhen the user did not pass--context).- Raises:
ValueError – When multiple context names are found and kube_context is
None.
- robovast.common.cluster_context.resolve_resource_value(value: Any, context: str | None) Any
Resolve a resource value for the active Kubernetes context.
Handles two forms:
Scalar (
int,float, orstr): returned as-is.Per-cluster list (
[{context-name: value}, …]): the entry whose key matches context is returned.
- Raises:
ValueError – When the value is a per-cluster list but context is
None, or when the context has no entry in the list.- Parameters:
value – Raw resource value (scalar or per-cluster list).
context – Active Kubernetes context name, or
None.
- Returns:
Resolved scalar value, or
Nonewhen value isNone.
- robovast.common.cluster_context.resolve_resources(resources: dict, context: str | None) dict
Resolve all resource fields in a resources dict for the active cluster.
Calls
resolve_resource_value()for every key in resources and returns a new dict with all per-cluster lists replaced by their resolved scalar values.- Raises:
ValueError – Propagated from
resolve_resource_value()when a per-cluster list has no entry for context.- Parameters:
resources – Raw resources dict (e.g.
{'cpu': 15}or{'cpu': [{'gke_my-project_…_cluster': 4}, {'minikube': 8}]}).context – Active Kubernetes context name, or
None.
- Returns:
New dict with resolved scalar values (
Noneentries removed).
Querying RoboVAST campaigns
Using [rdflib](https://rdflib.readthedocs.io/), you can query the generated metadata graph using [SPARQL](https://www.w3.org/TR/sparql11-query/).
Load the metadata graph
from rdflib import Graph
g = Graph()
g.parse("metadata.prov.json)
Loading SPARQL queries
To do so, you can load any of the queries below as text, and use the query method for any graph g:
with open("query-file.rq", "r") as f:
query_string = f.read()
qres = g.query(query_string)
for row in qres:
# Process your results
print(row)
Below are a few example queries demonstrating the PROV relationships in the metadata graph.
Scenario inputs
FloorPlan models:
SELECT ?floorplan ?creator ?date
WHERE {
?floorplan rdf:type env:FloorPlanModel .
OPTIONAL {?floorplan prov:wasAttributedTo ?creator .}
OPTIONAL {?floorplan dcterms:modified ?date .}
}
Vast file:
SELECT ?vast_file ?creator ?date ?abstract_scenario
WHERE {
?vast_file rdf:type robovast:VastConfiguration .
?vast_file dcterms:references ?abstract_scenario .
?abstract_scenario rdf:type scenarios:AbstractScenario .
OPTIONAL {?vast_file prov:wasAttributedTo ?creator .}
OPTIONAL {?vast_file dcterms:modified ?date .}
}
Generation
FloorPlan Model-to-Model Transformation:
SELECT ?floorplan ?activity ?jsonld_file ?agent
WHERE {
?floorplan rdf:type env:FloorPlanModel .
?activity rdf:type robovast:FloorPlanTransformation .
?activity prov:used ?floorplan .
?jsonld_file prov:wasGeneratedBy ?activity .
OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
}
FloorPlan Artefact Generation:
SELECT ?source_files ?activity ?gen_file ?agent
WHERE {
?activity rdf:type robovast:FloorPlanGeneration .
?activity prov:used ?source_files .
?gen_file prov:wasGeneratedBy ?activity .
OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
}
Generation of Concrete Scenario
SELECT ?vast_file ?ref_file ?activity ?agent ?gen_file
WHERE {
?vast_file rdf:type robovast:VastConfiguration .
?activity prov:used ?vast_file .
?vast_file dcterms:references ?ref_file .
?gen_file prov:wasGeneratedBy ?activity .
OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
Test Execution
Test results generated from a test run:
SELECT ?scenario ?config_files ?activity ?agent ?gen_file ?start_time ?end_time
WHERE {
?scenario rdf:type smm:ConcreteScenario .
?activity prov:used ?scenario .
?activity prov:used ?config_files .
OPTIONAL{?activity prov:startedAtTime ?start_time .}
OPTIONAL{ ?activity prov:endedAtTime ?end_time . }
?gen_file prov:wasGeneratedBy ?activity .
OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
Postprocessing
Postprocessing of a bagfile:
SELECT ?bag_file ?activity ?agent ?gen_file ?start_time ?end_time
WHERE {
?bag_file rdf:type robovast:ROSBag .
?activity prov:used ?bag_file .
?gen_file prov:wasGeneratedBy ?activity .
OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
OPTIONAL{?activity prov:startedAtTime ?start_time .}
OPTIONAL{ ?activity prov:endedAtTime ?end_time . }
}
Metadata and Graph postprocessing:
SELECT ?metadata_file ?graph_file ?md_activity ?graph_activity ?agent ?start_time ?end_time
WHERE {
?md_activity rdf:type robovast:PostprocessingMetadata .
?graph_activity rdf:type robovast:PostprocessingGraph .
?metadata_file prov:wasGeneratedBy ?md_activity .
?graph_file prov:wasGeneratedBy ?graph_activity .
?graph_activity prov:used ?metadata_file
OPTIONAL {?md_activity prov:wasAssociatedWith ?agent .
?graph_activity prov:wasAssociatedWith ?agent .}
OPTIONAL{?md_activity prov:startedAtTime ?start_time .}
OPTIONAL{ ?md_activity prov:endedAtTime ?end_time . }
}
Analysis
Identifying which variation types were used on each config:
SELECT ?config ?variation_type
WHERE {
?config rdf:type smm:ConcreteScenario .
?config robovast:variations/rdf:rest*/rdf:first ?variation .
?variation rdf:type ?variation_type .
FILTER (?variation_type != prov:Activity)
FILTER (?variation_type != robovast:Variation)
}
Getting the failure rate by environment:
SELECT ?env_model (SUM(?failures)/COUNT (?activity) * 100 AS ?result) (COUNT (?activity) AS ?total)
WHERE {
?conf rdf:type smm:ConcreteScenario .
?activity prov:used ?conf .
?activity rdf:type robovast:TestExecution .
?activity robovast:success ?success .
BIND(IF(?success=true, 0, 1) AS ?failures) .
?conf dcterms:references ?env_model .
?env_model rdf:type env:FloorPlanModel .
} GROUP BY ?env_model