Developer Guide

Test your Robotic Software with RoboVAST

RoboVAST is designed to facilitate testing and validation of robotic software systems by generating diverse scenarios and executing them in simulation environments. This guide provides an overview of how to utilize RoboVAST for testing your robotic applications.

1. Containerize your Software

As RoboVAST relies on containerization to ensure consistent and reproducible environments, the first step is to create a Docker container for your robotic software.

There are some requirements your container image must fulfill to be compatible with RoboVAST: - the image must contain scenario-execution package installed in /ws/install (which currently is available for ROS2 jazzy) - the image must be accessible by Kubernetes, e.g. by pushing it to a container registry.

2. Define a Test Scenario

Use the examples and the documentation of scenario-execution to create a scenario that tests your robotic software.

Keep in mind, that variations are currently supported for all overwritable scenario parameters as described here.

To test your scenario locally, you can run:

ros2 run scenario_execution_ros scenario_execution_ros <scenario-file> -t -d

3. Create Initial RoboVAST Configuration

Create a RoboVAST configuration file, based on the existing examples in the configs/ directory. Do not set any configuration, as this will be done in the next step.

Push the directory to a service — the one-node deployment on your own machine (On this machine) is the development loop — and run one configuration through it. This is the same path every campaign takes, differing only in which service answers.

vast workspace init .                                   # upload the project
vast workspace validate <workspace> my.vast             # check it
vast workspace run <workspace> my.vast --filter config1 --runs 1

vast workspace list names the workspace; vast workspace update . <workspace> pushes an edit. Before a campaign, vast container run (or the MCP exec_in_container) runs a command or one configuration’s scenario in the image, which is where an import error or a missing package shows up in seconds rather than after a pull.

Next, it is important to verify that the output (e.g. ROS bag) is stored correctly.

vast workspace run <workspace> my.vast --filter config1
vast campaign wait <campaign-id>
vast campaign download <campaign-id>          # <campaign-id>/<config-name>/<run_number>/

Once you are satisfied that the scenario and configuration work as expected, you can proceed to the next step.

4. Define Configurations

Define configurations in your .vast file. A good procedure is to add configurations one-by-one and analyze the result.

# 1. add configuration in config file

# 2. list created configurations
vast config list my.vast

# 3. try local execution with one of the created configurations
vast workspace run <pinned-workspace> my.vast --filter <config-name> --runs 1

5. Execute in Cluster

Once you have defined your configurations and verified local execution, you can run the tests in a Kubernetes cluster.

A good practice is, to first run a single configuration to verify that everything works as expected.

# 1. run single configuration in cluster, once
vast workspace run <workspace> my.vast --filter config1 --runs 1

# 2. fetch the campaign's archive (or publish it to the share for someone else)
vast campaign download <campaign-id>       # -> ./<campaign-id>.tar.gz
vast share export -i <campaign-id>        # -> the configured share
# Inside the archive: <campaign-name>-<timestamp>/<config-name>/<run_number>/

vast workspace run is fire-and-forget: it starts the campaign on the robovast-service, which drives it in-process, and returns immediately. The campaign runs in the background in the cluster:

# Monitor job status (shows progress per run when multiple runs are active)
vast cluster monitor

# Clean up after jobs complete (all campaigns, or use --campaign for a specific campaign)
vast cluster jobs-cleanup

Campaigns run in parallel and a new one never removes an older one’s Jobs. Clear them when you are done with vast cluster jobs-cleanup (add --campaign for one campaign). It touches Jobs and pods only; the campaign’s results live with the service, and vast campaign delete is what removes a campaign.

Running local container images in minikube

To test local container images in a minikube cluster, you can load the image into minikube’s Docker environment.

# first terminal
docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000"

# second terminal -- publish the whole family into the local registry
make release-images PROJECT=localhost:5000 PUSH=1

# third: point runs at it (a .vast names no RoboVAST image; see docs/images.rst)
export ROBOVAST_PROJECT=localhost:5000

6. Analysis

RoboVAST provides a GUI for analyzing run results, which is based on user-provided Jupyter notebooks.

To develop the notebooks, it is recommended to use e.g. VSCode. For the RoboVAST GUI to work, it is expected to contain a DATA_DIR definition. The RoboVAST GUI will replace this line with the actual path to the results directory. During development you can set this variable manually to point to your results directory.

# for single-run (specific run of a configuration)
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>/<config-name>/<run_number>'
# for configuration (all configurations)
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>/<config-name>'
# for complete run
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>'

In case you are using ROS bags as output format, it is recommended to postprocess the results before analysis. This can be done with the postprocessing commands defined in the configuration file. RoboVAST provides several conversion scripts for common use-cases.

Postprocessing is cached based on the results directory hash. To bypass the cache and force postprocessing (e.g., after updating postprocessing scripts), use the --force or -f flag:

Afterwards, read the results in the browser (vast ui, against a running service):

vast campaign postprocess <campaign-id>
# or, to force postprocessing even if results are unchanged:
vast campaign postprocess <campaign-id> --force

Note

The Results views discover campaigns exclusively from a per-campaign ``campaign.db`` store — they do not walk the results filesystem. Search campaigns write this store live; a batch campaign’s is written by the controller that ran it.

A results tree with no store is therefore invisible to the UI, and nothing rebuilds one. That only affects a tree produced outside a controller — a hand-copied or pre-store campaign.

The store also carries the campaign mode (batch/search), so the Explorer renders the search batch level and resolves the visualization.results.explorer.notebooks notebooks from the recorded config_dir. See Campaign Store and Results Indexing for the schema and internals.

Working across the distributions

RoboVAST is four packages in one checkout (see Four distributions, layered by audience), which changes two things about the development loop. Both have bitten; both are silent.

Install the client last. robovast-client is a non-optional path dependency of robovast, so pip install -e . resolves it and installs a plain copy into site-packages — silently replacing an editable install done earlier. Editing src/robovast_client/ then has no effect, and nothing says so. make venv installs it after everything that depends on it for exactly this reason; if you install by hand, do the same, and check with:

python -c "import robovast.client.cli as m; print(m.__file__)"

A path under src/robovast_client/ is editable; one under site-packages is not.

Entry points live in installed metadata, not in ``pyproject.toml``. Adding, moving or removing one does nothing until the owning distribution is reinstalled, and an editable checkout looks entirely normal in the meantime. The failure mode depends on which group:

  • robovast.cli_plugins degrades loudly — load_plugins() prints Warning: Failed to load plugin '<name>' and carries on.

  • robovast.cli_startup refuses rather than running on, naming the reinstall. Degrading silently there costs real damage: with the core installed but its .env hook unregistered, no ./.env is read, and vast service upgrade — which reconciles Secrets from the environment — concludes the registry and git credentials are gone and deletes both.

The rule that follows: after touching any [tool.poetry.plugins."..."] block, reinstall before you conclude anything from a test run. make venv re-runs when a manifest or the Makefile changes, so it is the safe way to do it.

A distribution’s own verbs are not entry points. An entry point is for crossing a distribution boundary; a verb defined in the same module as its group stays an ordinary @group.command(). Declaring your own in installed metadata means a pyproject edit without a reinstall makes the verb vanish, silently. Moving a verb between distributions carries the mirror hazard — the old declaration is live until every distribution is reinstalled, and two providers of one name resolve unpredictably; tests/test_editable_installs.py asserts exactly one provider per name. When vast reports has no attribute after a pull, this is it, and make venv is the fix.

A client install must stay a working install. robovast-client ships without the core, and every leak found so far has been a deferred import of it — the module imports perfectly and the command dies at call time, in exactly the install the distribution advertises. An import check cannot see that; tests/service/test_client_needs_no_core.py drives the commands with the core made not importable. Anything the client needs must live in the client: a wire constant like COMMAND_LIMIT_S belongs in interface.py, not in the server module that enforces it.

Releasing to PyPI

The five distributions are released as one set at one version, because robovast requires robovast-client and robovast-sim-roqsim at exactly the version being released, and robovast-nav and robovast-cluster require robovast. A version that exists for some of them and not the others is a set nobody can install, so .github/workflows/publish.yml stamps the tag’s version into every manifest before it builds a wheel. The tree carries the version being released too, set in all five pyproject.toml files in the release commit, because the images are built from the tree and report the version it names.

The version’s section of ``CHANGELOG.md`` comes first, written with the changelog skill (skills/changelog/SKILL.md) from the merges since the previous tag and reviewed before it is committed: a few short entries, never more than fifteen, on what a user of the previous version would want to hear about, each citing its pull requests as links. skills/changelog/changelog.py check holds it to that shape. That section is the release’s notes; the rest is in the git history.

Every release goes to TestPyPI first, and is tested by hand there. An upload cannot be taken back — a version can be yanked, never replaced, and pip still installs a yanked wheel when pinned exactly — so the only cheap place to find a packaging mistake is an index nothing depends on. The two triggers of the workflow are the two halves of that rule:

workflow_dispatch → TestPyPI

gh workflow run publish.yml -f version=2.1.0rc1 publishes the whole set as that version to TestPyPI. Pre-release numbers (rc1, rc2, …) are the convention, since pip does not pick them up unless pinned. Install it in a fresh venv and try it:

pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ \
    "robovast[nav,roqsim]==2.1.0rc1" "robovast-cluster==2.1.0rc1"
vast cluster setup minikube   # the web UI must come up, not "API only"

Something wrong is a fix on main and rc2; nothing has been consumed.

a v* tag → PyPI

git tag v2.1.0 && git push origin v2.1.0, on the commit the release candidate was built from. The same workflow, the same wheels, the real index. The tag also produces the versioned container images (image.yml), so the wheels and the images a deployment runs carry the same number.

What the workflow does, and why in that order. One build job makes and checks all five wheels before anything is uploaded: it verifies the lock against the manifest, builds both frontend bundles (make frontend) and stages them (make ui-stage), stamps the version, rewrites the root’s path dependencies to ==<version> (tools/pin_released_siblings.py — a path dependency reaches the metadata as a direct reference, which the index refuses), and then opens every wheel to confirm it carries no such reference and does carry its package data (robovast/_ui/index.html, robovast_nav/web/dist/remoteEntry.js). Everything that can fail for a reason inside this repository fails there, at no cost. The publish jobs then upload the artifacts in dependency order — client and sim-roqsim, then robovast, then nav and cluster — with skip-existing so that a re-run of the same tag after a transient failure continues past the wheels already on the index. A last job installs the set from the index it went to and checks the same things in what was installed, so a break that reached the index is named by the workflow rather than by a user.

Authentication is trusted publishing. Each index holds one publisher per distribution, naming this repository, publish.yml and the pypi or testpypi environment; the jobs mint a short-lived OIDC token and no API token exists anywhere. A distribution not yet on an index is created by its first upload through a pending publisher, registered the same way.

Rehearsing without CI. make publish-test uploads the set to TestPyPI from your checkout, stamped with a post-release of the tree’s version that is free on all five histories at once (tools/next_testpypi_version.py), and make publish-test-venv installs it into a fresh venv and checks the bundles, the entry points and the CLI. Every manifest is restored afterwards, including on failure. DRY_RUN=1 builds without uploading.

Container Image Compatibility Version

RoboVAST enforces a compatibility version between the host Python code and the Docker container image. This prevents cryptic runtime failures when the two sides are out of sync (e.g. after updating one without the other).

How it works

The host declares a window of protocol versions it can drive: MIN_IMAGE_COMPAT .. COMPAT_VERSION, both in src/robovast/common/execution.py. COMPAT_VERSION is baked into the container images as the label org.robovast.compat-version — one marker, and the only one.

A window rather than a single value, because an exact-match comparison orphans every image already published on the first bump: a campaign whose results pin an image by digest could never be re-run again — even with those exact bytes still in the registry. Bumping the maximum is harmless; dropping support is a separate, deliberate act of raising the minimum.

A label, read from the registry’s config blob without pulling a layer or starting a container, so it answers for an image that exists only in the registry.

  • Cluster execution: the submitter checks, host-side, immediately after the campaign’s refs are pinned to digests and before any manifest is written — so the verdict is about the exact bytes the pods will run, and a refusal creates no pods. It is deliberately not asked of the image itself: a workload inspecting its own image is not how admission is decided anywhere else, and could only report a mismatch by failing one init container per job in the batch.

  • Retrigger pre-flight: the service reads the label of each recorded image from its registry; an image the registry will not answer for is reported as unknown.

  • Postprocessing: docker_exec.sh checks before docker run.

The cluster check fails closed: if the registry will not say what protocol the image speaks, the campaign is refused rather than started. Pinning, right beside it, does the opposite and is right to — pinning is an optimisation, and not applying it leaves exactly what would have run anyway, whereas a compat check that could not read the image has established nothing. Set ROBOVAST_SKIP_IMAGE_COMPAT_CHECK=1 to run anyway; that is the right move only when the registry is unreachable and you know the image.

Only the campaign’s own image is checked. The sidecar sets no label and a system-under-test image is not a robovast image at all, so an absent label on those means “not applicable” rather than “unreadable” — failing closed on them would refuse every campaign that has a SUT.

An image built before the label existed now reports nothing and is refused. Those images predate protocol 2, which is also MIN_IMAGE_COMPAT, so a refusal is the right answer for them regardless — but the message says what to do about it (rebuild from the recorded revision, or re-tag with the label) rather than only that it could not tell.

Outside the window, execution fails immediately, and the message differs by direction because the fixes are not interchangeable: an image older than the window means “check out the revision the campaign recorded (_execution/execution.yaml) and run it there”, while one newer means “upgrade robovast”. Neither ever advises pulling a newer image — a re-run needs the bytes the campaign recorded, not today’s.

Warning

The window is a claim. Raise MIN_IMAGE_COMPAT when support is genuinely dropped, or it replaces a safe refusal with a broken run. configs/examples/camera_smoke is the cheap way to keep the claim true.

When to bump the version

Bump COMPAT_VERSION when the contract between host scripts and the container changes:

  • A new Python or system package is required inside the container

  • The ROS distribution changes

  • The interface of mounted scripts changes (e.g. ros2_exec.sh, entrypoint.sh)

  • A postprocessing script requires a new ROS package

How to bump the version

  1. Increment COMPAT_VERSION in src/robovast/common/execution.py

  2. Rebuild and push the container images

That is the whole procedure, because there is only one copy. The images take the value as a build arg that container/image_stamp.sh derives from that constant — the same way they take the git revision and the build date, and for the same reason: a constant copied by hand into each Dockerfile drifts, which is why it used to need a CI gate comparing the copies. image_stamp.sh refuses to build if it cannot read the constant, rather than producing an image that carries no protocol at all.

image.yml still guards the shape: each Dockerfile must declare ARG ROBOVAST_COMPAT_VERSION and label from it, so a literal cannot creep back.

Remembering to bump is the part nothing could check — the version is a claim, and it fails by being forgotten. make check-compat-version (run in CI on every pull request, beside the config-version check) compares the change against the files that define the contract — the two Dockerfiles, entrypoint.sh, secondary_entrypoint.sh, ros2_exec.sh — and fails if one of them moved and the version did not.

It is a prompt, not a proof: it cannot tell a contract change from a comment edit in the same file, and it cannot see a change made somewhere it is not looking. When nothing the host relies on actually moved, say so in the commit message and it passes:

Compat-Unchanged: moved a comment; nothing the container must provide changed

A trailer rather than a flag, so the answer lands in history next to the change it is about. If it turns out to be used on most changes, the check is crying wolf and the path list is too broad — that ratio is the signal for whether it earns itself.

Extending RoboVAST

Contribution hooks: the shared contract

Several seams below follow one shape: a plugin contributes a typed fact about something the caller already has, through an optional classmethod. config_view_data (geometry for the config view), collect_prov_metadata (provenance nodes) and get_required_container (an auxiliary container a variation needs) are the three that exist. A new hook should look like them rather than invent its own habits:

  1. An optional classmethod on the plugin class. It can be asked without instantiating the plugin or composing a campaign, so a caller holding only the class gets an answer.

  2. Answered from data the caller already holds — a resolved configuration, a config record, the plugin’s own parameters. No filesystem, no container, no network: that is what keeps a hook callable inside a request, and what makes its answer reproducible.

  3. Silence is valid, and it is the default. The base implementation returns empty or None, so a plugin says nothing until it has something to say.

  4. A typed contribution, not a bare dict (ConfigViewContribution, ProvContribution, ContainerSpec), so the collector, the wire and these docs can describe it.

  5. Attribution travels with the contribution. A marker’s group defaults to the contributing class’s name; a PROV node is named per variation type. A reader can always ask which plugin said this.

  6. The failure policy follows the consequence. This is the one axis the three genuinely differ on, and it is a decision to make rather than a default to inherit:

    • the missing answer changes what runs -> let it propagate. get_required_container does: without an answer we cannot know whether the variation needs a helper image, so composition fails rather than running it without one it may have declared.

    • it changes what a record claims -> record the gap in the record. collect_prov_metadata does: the graph gains a ProvenanceGap entity naming the contribution that raised, because a published provenance record whose incompleteness is invisible is worse than a loud failure.

    • it changes only a view -> report it in the view. config_view_data does: the hook’s error lands in the contribution’s errors and is shown beside the markers that did arrive, since a view missing one variation’s geometry is otherwise indistinguishable from a variation that placed nothing.

Add an interface operation

Every remote operation is a method on robovast.service.interface.RobovastInterface, and each client is a thin caller of it (One interface, one implementation, many clients). Adding one means threading it through the whole set:

  1. The interface — the abstract method, its Routes entry and the request/response models. They live in robovast-client, so a client install has them without the core.

  2. The service and the HTTP transport — in robovast.service.service_base.ServiceBase when the body is correct for any implementation, otherwise an abstract hook there and a body in ClusterService and in the suite’s NullService (One base, the implementation’s hooks); and HTTPTransport over the route. An implementation may decline an operation, and then it raises UnsupportedOperation naming the operation and itself, in its own class; never a default on the base, and never a ValueError that reads as bad input. The refusal crosses HTTP as a 501 and reaches every client as the one sentence (HTTP API, “Status codes”).

  3. The HTTP route in robovast.service.app.

  4. The surfaces — the vast CLI, the MCP tools, and, where the web UI shows the operation, frontend/ui/src/lib/robovastClient.ts (Web UI internals).

An operation implemented inside one client — an MCP tool calling a local function directly, say — is reachable from that client alone, and the CLI, the web UI and the HTTP API have no way to it. Putting it on the interface is what gives every client the same answer, over either transport.

Anything the executing pod needs travels the same way: a value the controller reads must be passed through ClusterService (cluster_service.py into the pod’s environment), or the operation is accepted by the service and does nothing in the pod. The hooks are what keep that failure loud rather than silent: a hook ClusterService has not answered refuses to construct, and a body on the base is one that is correct for any implementation, so the only way a cluster path can be missing is for ClusterService itself to leave it out.

Keep disk and database I/O off the event loop

The service answers every request, /healthz included, from one event loop. A route or MCP tool declared def runs on a worker thread and may block; one declared async def runs on the loop, so anything it does that waits on a disk or on the index stalls every other request for as long as that takes. A disk that is filling makes every write slow at once, and a stalled loop fails the liveness probe, so the pod is restarted for being short of disk.

  • Declare a route or tool async only when it has something to await; otherwise make it a plain def.

  • Inside async code, hand blocking calls to a worker with anyio.to_thread.run_sync, and write files through anyio.open_file.

  • Work that must not compete with the routes’ threads gets its own anyio.CapacityLimiter, as the SSE pulls and the refusal records in robovast.service.app do.

Add Variation Plugin

Provide your custom variation type by creating a class that inherits from robovast.common.variation.Variation.

To your pyproject.toml, add an entry under [tool.poetry.plugins.”robovast.variation_types”] to register your variation type. The key is the name used in the RoboVAST configuration file, and the value is the import path to your variation class.

[tool.poetry.plugins."robovast.variation_types"]
"YourVariation" = "robovast_<yourplugin>.your_variation:YourVariation"

A variation can also be loaded from a local file relative to the .vast without packaging it — reference it as <path>.py:<Class> wherever a variation name is expected (in a configuration[].variations list or a search.variations template). This is the same ./path.py:Class convention used by search strategies, extractors and postprocessing plugins:

variations:
- variations/wind.py:WindFieldVariation:
    wind_speed: 5.0

See configs/examples/quadrotor_landing/variations/wind.py for a runnable example (a wind model that derives the simulator’s wind_strength), wired into the quadrotor search vasts.

The referenced module travels with the campaign: it is archived into <campaign>/_config/ alongside the .vast, so a retrigger — which re-composes from that snapshot rather than replaying recorded configurations — can resolve it, and it is content-hashed into the campaign’s config_identifier, because the source that decides which configurations exist is part of what the experiment is. Editing it therefore gives the campaign a new identity, exactly as editing a packaged variation’s module does.

Important

A file-referenced plugin must be one self-contained module. It is imported by path, with nothing added to sys.path, so it cannot import a sibling module in its own directory. Nor can anything discover a data file it opens itself — that dependency lives in the plugin’s code rather than in the config, so nothing archives it and a retrigger finds it missing. If you need more than one file, package the plugin and declare it under plugins:; if you need a data file, name it in execution.run_files.

How a failure in your plugin is reported. A refusal your plugin means — a VariationConfigError for parameters outside what it accepts, a VariationInfeasibleError for a draw no arrangement realizes — is reported as its message alone, naming the plugin and the config block. Any other exception escaping variation() is a bug, and is reported as Variation failed. <Class>: <exception> followed by the tail of the traceback: the file, line and source of the frame it was raised in. Either report is prefixed with the .vast line the variation is written on and the config block that ran it (campaign.vast:10: config 'cell1': …), so the entry to edit is named as well as the code. That text is the same on the CLI, in the MCP tools (validate_project, preview_configurations) and in the web UI’s config editor, so the line to fix is in front of whoever is authoring the plugin.

Note

Packaging a variation plugin as its own distribution. If your variation types live in a separate installable package (as robovast-nav does for FloorplanVariation, PathVariation*, ObstacleVariation*), it must be importable everywhere scenario variations get composed — not just where scenarios run. Composition happens in the process that expands the variations: cross-product: the vast CLI on your host for vast workspace run; and the robovast-service for every service/cluster campaign — the service drives the campaign in-process, so composition (which for search runs once per generation) happens in the service, not in a separate pod.

There is one mechanism for all of these — the .vast’s plugins: list (next section) — and it is uniform across the CLI and the MCP/service paths:

  • On your host (``vast`` CLI), a plugin you have already installed is detected and used as-is. config_plugins.ensure_workspace_plugins checks whether each declared distribution is importable; if you ran pip install / make venv yourself, nothing is re-fetched. Only the declared specs that are missing are installed — into a virtual environment under the project’s .robovast_plugins/ directory (never your active venv).

  • For a service/cluster campaign, the service installs each declared plugin into the workspace’s ``.robovast_plugins/`` and imports it off ``sys.path`` when it composes (config_plugins.ensure_workspace_plugins). The install runs once, on the credentialed service, so a private-repo clone needs credentials only there; the driver then uses the installed plugin directly — there is no staging round-trip and no separate pod.

Dependencies come from each package’s own metadata into that same environment, so the one real constraint is that your plugin must declare its dependencies correctly (e.g. scenario_mt’s shapely and its fpm @ git+…). There is no separate host-venv “injection” step and no wheel-shipping; a plugin installed only in your host venv but not declared in plugins: will not reach the service.

Do not declare a dependency on robovast itself. Your plugin is loaded into robovast’s process, so the host always provides it; the declaration is redundant, and it makes your package unnecessarily expensive to resolve anywhere the host is not already present. validate_project reports it once the plugin is installed.

Declaring a plugin in the ``.vast`` (CLI, MCP, and ``robovast-service``). Declare the plugin inside the ``.vast`` with a top-level plugins: list of pip requirement specs. This is the single portable mechanism: it works for the CLI (where you may instead just install the plugin yourself — it is detected) and for an MCP/service campaign, where the LLM only authors inputs into a workspace and .vast / .osc are the only file types it can write inline:

plugins:
  - scenario_mt @ git+https://github.com/secorolab/metamorphic_testing@main
  - some_published_plugin==1.2.3
  - ./plugins/my_plugin-1.0.0-py3-none-any.whl   # a wheel you uploaded

These are installed into the workspace. Before variation types are resolved from entry points, config_plugins.ensure_workspace_plugins (called once at the top of generate_scenario_variations — the sole convergence for local and service composition) installs the declared specs (with dependencies) into a venv under <workspace>/.robovast_plugins/ and puts that venv’s site-packages on sys.path so the entry points resolve. A .installed marker (a hash of the specs and of the environment they were resolved against) makes it idempotent.

Why a venv rather than pip install --target. A plugin is resolved against the host, not in isolation from it: inside a venv pip treats what the host already provides as satisfied, and declines to uninstall anything living outside the environment it targets. --target cannot do either — pip forces --ignore-installed whenever it is given, so every dependency is re-materialized, and a plugin depending on robovast got a second robovast installed beside it. Since importlib.metadata deduplicates distributions by name and keeps the first on sys.path, that copy’s entry points became the only ones the process could see. The same mechanism, for the same reason, backs the experiment image build (image_build._VENV_SETUP).

The key property: .robovast_plugins/ lives in the workspace the driver composes from, so the plugin is imported straight off sys.path with no staging step. The install runs where the workspace lives:

  • the robovast-service runs it in create_campaign (and for validate/preview) — its environment is what reaches the source and holds any git credentials;

  • on your host (vast CLI), a declared plugin already installed in the active venv is detected and used as-is — install it yourself and it is not re-fetched. Only missing specs are installed, and always into the workspace venv (never your site-packages), so a run is non-invasive.

Sources. An index pin needs no source access. A git URL works when the install environment can reach it — for a private repo in the service, provide a GitHub token at vast cluster setup (below). A workspace-relative wheel you create_uploaded is the fully offline path (no git, no credentials) for a private or unpublished plugin. On a failed install the error is raised synchronously from the service call (create/validate/preview), so an MCP user sees an actionable message instead of an Unknown variation class surfacing later on the campaign’s status.

Because the robovast-service is one long-lived process serving many workspaces and Python cannot unload a module, a plugin already loaded (from another workspace) cannot be swapped; ensure_workspace_plugins logs a warning in that case (the already-loaded version wins until the service restarts).

Private-repo credentials (``vast cluster setup``). When a git+https plugin points at a private repo, provide a token via ROBOVAST_GIT_TOKEN (or GITHUB_TOKEN / GH_TOKEN) — either exported in the environment or, more conveniently, set in the project’s .env file (every vast command loads ./.env before it runs). deploy_service stores it in a robovast-git-credentials Secret and mounts it read-only as a file into the service pod (/var/run/secrets/robovast-git/token). The token is handled so it is not accessible to any workspace or command:

  • it is never put in an environment variable (which every child process / command would inherit), never written to ~/.gitconfig, and never placed on a command line (ps-visible);

  • it is supplied to git only for the one pip install subprocess, via a throwaway GIT_ASKPASS helper in an owner-only temp dir that is removed immediately after;

  • it never enters a workspace or a campaign’s inputs (the driver imports the already-installed .robovast_plugins/ from the workspace and needs no credentials at composition time).

Composition moves to a subprocess when ``plugins:`` is declared. The plugin and its pinned dependencies are then imported only there, never in the long-lived service process, so a plugin needing a different version of something RoboVAST also uses cannot disturb it. An operator who needs hard isolation from untrusted plugin code should still prefer the uploaded-wheel source, which needs no credential at all.

That subprocess is also why a variation’s auxiliary container needs a hand. The execution backend publishes its runner factory in a ContextVar, which does not cross a process boundary — so a campaign declaring plugins: and a variation with get_required_container (floorplan generation, say) once could not compose at all on a cluster backend, having neither the backend’s factory nor a local docker to fall back on.

robovast.common.container_runner_proxy closes that: the parent serves its own live factory on a Unix socket beside the job file, and the worker installs a factory whose runners forward run / close / expose back across it. The runner — and with it the Kubernetes client, the storage client and the credentials both authenticate with — stays in the parent; only the four calls of the ContainerRunner contract cross, plus workspace, which is a path both sides can already see. Command output is streamed frame by frame, so a plugin’s progress still reaches the campaign log while the command runs, and a failed command still raises subprocess.CalledProcessError in the worker with its output intact.

Nothing about a backend is serialized, so this works for every backend without any of them describing itself — including the local docker one and any added later. When no factory is active (a plain CLI run) no socket is served and the worker uses its own docker, exactly as before.

For a single dependency-free variation you can skip packaging entirely and use a <path>.py:<Class> file reference resolved relative to the .vast directory (see below).

Two pitfalls when exposing the package as a poetry extra of this repo (e.g. nav = ["robovast-nav"]) rather than as an independent third-party plugin:

  • The extra’s package name must also be declared as an optional dependency in [tool.poetry.dependencies] (e.g. robovast-nav = {path = "src/robovast_nav", optional = true} for an in-repo sibling package) — poetry check catches the mismatch if not.

  • For a cluster run, the built-in robovast / robovast-nav code comes from the service image (vast cluster setup deploys it), so dev changes to those sources reach a run by rebuilding/redeploying that image — publish your own set (make release-images PROJECT=docker.io/<you> PUSH=1) and point ROBOVAST_PROJECT at it to iterate. This applies only to the built-in sources; independent plugins are handled by discover_plugin_installs above (installed into the workspace, no image rebuild needed).

If your plugin’s package pulls in a dependency that itself needs system shared libraries (e.g. robovast-nav hard-depends on pyside6-essentials, whose bundled Qt6 libs need libGL.so.1 and friends to even import, regardless of whether any GUI is ever shown), the controller image needs those apt packages too — see the container/controller/Dockerfile apt-get block for the list verified against robovast-nav. A missing system lib shows up the same way as a missing extra: the plugin’s entry point fails to load and the variation type is reported as unknown.

Add Command-line Plugin

To create a plugin for the vast CLI:

  1. Create a Click group or command in your package

  2. Register it in your pyproject.toml under [tool.poetry.plugins.”robovast.cli_plugins”]

  3. The plugin will be automatically discovered and added to the vast command

Example plugin registration:

[tool.poetry.plugins."vast.plugins"]
variation = "variation_utils.cli:variation"

Add a config format for the system under test

The sut: channel addresses values inside the stack’s own configuration files. yaml, json and xml ship built in; a stack configured by something else — a bespoke .ini, a proprietary descriptor — needs a format, not a change to RoboVAST.

Subclass robovast.common.sut_formats.SutConfigFormat and register it under [tool.poetry.plugins."robovast.sut_formats"]. The built-ins register through that same entry point, so the route your package takes is the one this repository exercises on every run.

class IniFormat(SutConfigFormat):
    EXTENSIONS = (".ini", ".cfg")

    def load(self, path): ...
    def can_address(self, doc, path) -> bool: ...
    def set(self, doc, path, value): ...
    def remove(self, doc, path): ...
    def dump(self, doc, path): ...

Four things decide whether a format behaves like the built-ins:

  • You own the path syntax. RoboVAST splits a destination once, on the first ., to find the source; the rest arrives at your format verbatim. Use dotted keys, XPath, JSON Pointer — whatever addresses your documents honestly.

  • ``can_address`` is the pre-check, and it is yours to answer. It asks whether a path is writable, not whether a value is already there: a factor may set a key the file leaves at its default, and refusing that would reject a correct campaign. What must exist is the parent. Raise CannotAnswer when you genuinely cannot decide — that leaves the destination unchecked and reported, which is different from “no”, and keeping them distinct is what stops an unimplemented check from looking like a rejection.

  • ``set`` creates or replaces, and value may be a scalar, a mapping, a list or a fragment. A factor swapping a whole subtree must be the same call as one changing a number.

  • ``remove`` is separate from ``set``, because a configuration block that is present and empty is not one that is absent, and stacks tell them apart. It is what {$absent: true} reaches.

addresses is optional and may return None: it feeds error messages and editor completion, never the check. A format whose address space is not enumerable — any XPath expression is a potential address — loses only the listing.

Add Metadata Processing Plugin

Metadata processing plugins run after the generic and variation-plugin metadata phases and can modify the metadata.yaml produced for each campaign. They are configured in the .vast file under results_processing.metadata_processing:

results_processing:
  metadata_processing:
    - my_metadata_plugin
    - my_metadata_plugin:
        param1: value1
        param2: value2

Each plugin must subclass robovast.common.metadata.MetadataProcessor and implement the process_metadata method:

from pathlib import Path
from robovast.common.metadata import MetadataProcessor

class MyMetadataPlugin(MetadataProcessor):

    def process_metadata(self, metadata: dict, campaign_dir: Path) -> dict:
        # Modify metadata as needed
        metadata["custom_field"] = "custom_value"
        return metadata

Register the plugin in your package’s pyproject.toml:

[tool.poetry.plugins."robovast.metadata_processing"]
my_metadata_plugin = "my_package.metadata:MyMetadataPlugin"

Add Variation Plugin Metadata Hook

The Variation base class defines an overridable classmethod that returns an empty dict by default. Subclasses implement it to attach domain-specific metadata to each configuration entry in metadata.yaml:

from pathlib import Path
import yaml
from robovast.common.variation import Variation

class MyVariation(Variation):

    @classmethod
    def collect_config_metadata(cls, config_entry, config_dir: Path,
                                 campaign_dir: Path) -> dict:
        """Load extra metadata from a YAML sidecar in _config/."""
        data_file = config_dir / "_config" / "my_data.yaml"
        if data_file.exists():
            with open(data_file) as f:
                return {"my_data": yaml.safe_load(f)}
        return {}

collect_config_metadata is called once per configuration that used the variation and returns a dictionary that is merged into the configuration’s metadata entry.

Add PROV-O Provenance Hook to a Variation Plugin

Variation plugins can contribute domain-specific nodes to the campaign’s PROV-O provenance graph by overriding collect_prov_metadata on the Variation base class. The default implementation returns None (no contribution). It follows the Contribution hooks: the shared contract contract, and its failure policy is the “record the gap in the record” case: a hook that raises is logged and named in the graph as a ProvenanceGap, so an incomplete provenance record says that it is incomplete.

This hook is the right place for provenance that is tightly coupled to a specific variation — for example, a floorplan generation variation knows which map and mesh files it produced and can declare their lineage in the graph.

Return type: ProvContribution (or None to contribute nothing):

from robovast.common.variation import Variation, ProvContribution

class MyVariation(Variation):

    @classmethod
    def collect_prov_metadata(
        cls,
        config_entry: dict,
        campaign_namespace,   # rdflib.Namespace for the campaign
        config_namespace,     # rdflib.Namespace for this config
        gen_activity_id: str, # IRI of the config-generation activity
        vast_id: str, # IRI of the vast file that contains it
    ):
        """Contribute domain-specific PROV-O nodes."""
        from rdflib import PROV, Namespace

        _ID, _TYPE = "@id", "@type"
        MY_NS = Namespace("https://example.org/metamodels/")

        config_cfg = config_entry.get("config", {})
        my_file = config_cfg.get("my_output_file", "")
        if not my_file:
            return None

        file_iri = config_namespace[my_file]

        return ProvContribution(
            # Extra graph nodes (entities, activities) appended to @graph
            graph_nodes=[{
                _ID: file_iri,
                _TYPE: PROV["Entity"],
                "wasGeneratedBy": gen_activity_id,
                MY_NS["someProperty"]: "value",
            }],
            # Properties merged onto the concrete scenario node
            scenario_properties={MY_NS["outputCount"]: 1},
            # IRIs that each run activity should declare as "used"
            run_used_iris=[file_iri],
        )

ProvContribution fields:

graph_nodes

List of JSON-LD node dictionaries appended to the PROV @graph. Use rdflib.PROV, rdflib.DCTERMS, or your own Namespace objects as keys/values.

scenario_properties

Dict merged onto the concrete scenario entity node for this configuration. Useful for adding counts or classification properties (e.g. number of goals, number of obstacles).

run_used_iris

List of IRIs that every run activity in this configuration will declare as prov:used. Typically the IRIs of entities generated by this variation that are consumed at runtime (e.g. a map file, a mesh file).

Note

collect_prov_metadata receives rdflib.Namespace objects (campaign_namespace, config_namespace) so you can construct campaign-relative IRIs with campaign_namespace["some/path"]. rdflib is a required dependency of the core robovast package.

Add a Simulator Backend

A backend supplies what a campaign would otherwise restate for every simulator run: the image, the packages, the environment, and how the simulator is started. Registered in the robovast.simulators entry-point group; implementation: robovast.common.simulators. The user-facing side, the two shapes, and a worked example are in Simulators.

Two constraints that are not obvious from the base class:

It must import without its simulator installed. apply_backend runs in the long-lived service process, which has no reason to carry a MuJoCo or an Isaac runtime. A backend declares strings and container specs; anything genuinely needing the simulator returns a ContainerSpec from input_files and runs inside the simulator’s image.

It can arrive through plugins:, but not via the driver’s plugin-install phase. That phase (controller._install_plugins) is deliberately add_to_path=False and in any case runs long after the image and environment hooks. What makes a packaged backend work is a separate, earlier install: ServiceBase._build_specs_for resolves the campaign’s plugins: before extracting the build specs, precisely because the container plan depends on the backend. The ordinary answer is still an installed distribution of the service environment, like robovast_nav, or a .vast-relative file ref — a wheel per campaign is a real cost. See Simulators for the three options in preference order.

Note

That two install sites decide this, and only one of them is documented where a backend author would look, is worth revisiting: whether a backend should be shippable per campaign is a design question, and today the answer is an accident of call order.

Where it runs in composition. apply_backend() is called once at the top of the execution extraction in generate_scenario_variations(), so the container plan, the image builds and the run environment all read one already-merged mapping instead of each re-asking the backend and risking different answers. The campaign always wins: a backend fills in keys the author left out and never overrides one they set.

Add Input Generator Plugin

Input generators produce a campaign’s derived inputs before it is composed — the mirror image of postprocessing, at the other end of the campaign. They are declared as execution.generate and registered in the robovast.input_generators entry-point group. Implementation: robovast.common.input_generation.

Where it runs in composition. run_input_generators() is called from generate_scenario_variations() before collect_filtered_files(), and each entry’s outputs are appended to run_files. That position is the whole design, and everything else follows from it: the outputs reach hash_run_files() (so they enter the configuration identity), prepare_campaign_configs() copies them into <campaign>/_config/, and the run container bind-mounts them at /config/<path>. There is no second code path for generated files anywhere downstream. A per-configuration artifact — one a variation produces per cell — is staged at that same /config/<path> instead of the campaign’s copy, so a generator and a variation put a file in the same place and the campaign-wide one simply loses the path when a cell owns it. It also means generation is host-side, before publication, so the cluster gets the artifacts with no extra work.

Generation vs. variation. Both can produce artifacts, and the test for which you want is: does it produce configurations, or only files? A generator produces files, once per campaign. Something that multiplies the configuration set — one config per floorplan, with the generated paths bound to scenario parameters — is a variation even when it also generates, because execution.generate has no way to emit configurations.

Return value: (success: bool, message: str), or None for success. Raising and returning False are reported identically, so use whichever carries the better message.

Reserved key. out belongs to RoboVAST, not the plugin: it is validated, created as a temporary directory, and swapped into place only on success. So outputs can be expanded into run_files without importing the plugin (which the isolated compose subprocess relies on), and a half-written artifact can never be mistaken for a finished one.

Staleness. Declare what was read by calling write_manifest(out_dir, paths), which writes .generated.json into the output directory; the next composition hashes those paths and skips the generator when nothing moved. Report the real set — for anything compiled from a description that can reference others, that includes the transitive ones. A generator that reports nothing is never cached, and a cached result is honored only while its outputs are still on disk unchanged: staleness fails towards doing the work, never towards serving a stale artifact.

Creating an Input Generator:

from robovast.common.input_generation import BaseInputGenerator, write_manifest


class MyGenerator(BaseInputGenerator):
    """Compile <thing> into a campaign input."""

    #: Bump when the OUTPUT FORMAT changes, so an upgraded plugin regenerates
    #: even though none of its inputs moved.
    FORMAT_VERSION = 1

    @classmethod
    def get_required_container(cls, parameters):
        """Optional: an aux image, when the tool is not installed alongside RoboVAST.

        Same contract as a variation's — a container in an aux pod (the campaign's
        during a run; one held by the container-exec manager when a preview
        composes), or an ephemeral ``docker run`` under ``vast configuration
        generate`` on a development machine. Reached via
        ``self.container_runner``,
        whose ``workspace`` is visible at the same path on both sides (use
        ``stage_for_container`` / ``collect_from_container``).
        """
        return None

    def __call__(self, vast_dir, out_dir, source=None, **params):
        sources = compile_thing(os.path.join(vast_dir, source), out_dir)
        write_manifest(out_dir, sources)
        return True, f"compiled {source}"

Registration:

[tool.poetry.plugins."robovast.input_generators"]
my_generator = "your_package.generators:MyGenerator"

A project can also skip packaging entirely and reference a local file: - ./tools/gen.py:MyGenerator: {out: files/thing}.

Usage in .vast config:

execution:
  generate:
  - my_generator:
      out: files/thing
      source: things/input.yaml

Note

The generator must be importable by the process that composes the campaign, which for a running service is vast serve — not the shell the user typed in. The unresolved-name error prints sys.prefix for exactly this reason. An aux container sidesteps the question entirely, which is the main argument for declaring one.

Add Postprocessing Command Plugin

Postprocessing plugins are Python functions that process run result directories (e.g., convert rosbag data to CSV). They are registered as entry points and executed before analysis.

Return value: A plugin must return (success: bool, message: str). It may optionally return a third value, a list of provenance entries, so that each produced file is recorded (e.g. which CSV was created from which rosbag). Each entry is a dict with keys: output (path relative to results_dir), sources (list of paths), plugin (plugin name), params (optional dict). If returned, these entries are merged and written into postprocessing.yaml in each run folder (<campaign-name>-<timestamp>/<config>/<run-number>/).

Steps that work run by run: a plugin whose output for a run depends only on that run, the scenario jobs it links, the other runs of those jobs and the campaign-level inputs (_config, _execution, _transient) declares scope = "run". A cluster may then run it on parts of the campaign in parallel (Splitting a campaign’s postprocessing): every part is a set of whole scenario jobs with their runs, and the step must write only into those. The default, scope = "campaign", runs the step once over the whole tree. Only the leading run of run-scoped steps is split – a run-scoped step listed after a campaign-scoped one runs after it, over the whole tree.

Steps that run in the campaign’s execution image: some work can only happen in the image the runs used — deserializing a bag needs the message definitions of the custom types it recorded. Such a plugin derives from ExecutionImagePlugin and names the command to run there instead of implementing __call__:

import os
from robovast.results_processing.postprocessing_plugins import ExecutionImagePlugin

class DecodeCamera(ExecutionImagePlugin):
    def image_command(self, ctx, topic="/camera"):
        # A file in the scripts directory, then its arguments. ctx says where the
        # campaign is inside the container, and whether to force or be verbose.
        argv = ["decode_camera.py", "--topic", topic]
        if ctx.provenance_file:
            argv += ["--provenance-file", ctx.provenance_file]
        return argv + [ctx.campaign_dir]

    def image_files(self):
        # Shipped beside the conversion scripts, under their own names.
        return [os.path.join(os.path.dirname(__file__), "decode_camera.py")]

The postprocessing Job runs exactly that command from /scripts through ros2_exec.sh in its image container; vast results postprocess on a development machine runs it with docker_exec.sh against the campaign’s execution image. A parameter the command does not take fails the step. The script runs without robovast — only the standard library, what the image provides, and the files beside it (rosbags_common among them). It writes its outputs into the campaign tree; on the cluster, every file it writes is delivered back, whether or not it records provenance. To record provenance it appends {"output", "sources", "plugin", "params"} entries (paths relative to the campaign) to --provenance-file with rosbags_common.write_provenance_entry.

Creating a Postprocessing Plugin:

from typing import Tuple, Optional, List

def my_postprocessing_command(
    results_dir: str,
    config_dir: str,
    custom_param: Optional[str] = None,
    provenance_file: Optional[str] = None,
) -> Tuple[bool, str]:
    """Convert custom data to CSV.

    Args:
        results_dir: Path to the <campaign-name>-<timestamp> run directory to process
        config_dir: Config file directory (for resolving relative paths)
        custom_param: Optional custom parameter
        provenance_file: Optional path for provenance JSON (for container scripts)

    Returns:
        Tuple of (success, message) or (success, message, provenance_entries)
    """
    import subprocess
    import os

    script = os.path.join(config_dir, "tools/script.sh")
    cmd = [script, results_dir]
    if custom_param:
        cmd.extend(["--param", custom_param])

    result = subprocess.run(cmd, capture_output=True, text=True)

    if result.returncode != 0:
        return False, f"Failed: {result.stderr}"
    return True, "Success"

Register in pyproject.toml:

[tool.poetry.plugins."robovast.postprocessing_commands"]
my_postprocessing_command = "your_package.postprocessing_plugins:my_postprocessing_command"

Usage in .vast config:

analysis:
  postprocessing:
    - my_postprocessing_command:
        custom_param: value

Add Publication Plugin

Publication plugins package or distribute the results directory after postprocessing. They are plain callables (functions or class instances) that operate on the full results directory.

Return value: A plugin must return (success: bool, message: str).

Creating a Publication Plugin:

from typing import Optional, Tuple

def my_publication_plugin(
    results_dir: str,
    config_dir: str,
    destination: Optional[str] = None,
) -> Tuple[bool, str]:
    """Upload results to a remote storage location.

    Args:
        results_dir: Path to the results directory (parent of campaign directories).
        config_dir: Directory containing the .vast config file; relative
            paths should be resolved from here.
        destination: Remote destination URL or path.

    Returns:
        Tuple of (success, message).
    """
    import subprocess
    dest = destination or "s3://my-bucket/results/"

    result = subprocess.run(
        ["aws", "s3", "sync", results_dir, dest],
        capture_output=True, text=True,
    )
    if result.returncode != 0:
        return False, f"Upload failed: {result.stderr}"
    return True, f"Uploaded results to {dest}"

Register in pyproject.toml:

[tool.poetry.plugins."robovast.publication_plugins"]
my_publication_plugin = "your_package.publication_plugins:my_publication_plugin"

Usage in .vast config:

results_processing:
  publication:
    - my_publication_plugin:
        destination: s3://my-bucket/results/

Add Cluster Config Plugin

To add a new cluster configuration option for RoboVAST, create a class that inherits from robovast.execution.cluster_config.base.BaseConfig. Register your cluster config in your pyproject.toml under [tool.poetry.plugins.”robovast.cluster_configs”]. The key is the name used to select the configuration, and the value is the import path to your configuration class.

[tool.poetry.plugins."robovast.cluster_configs"]
"YourClusterConfig" = "robovast_<yourplugin>.your_cluster_config:YourClusterConfig"

To test your cluster configuration, call prepare_setup_cluster() with an output directory. It writes the files and the instructions for performing the setup steps by hand, without applying anything. There is no CLI verb for it — vast cluster setup is the applying path.

Staging for experiment-image builds. A build’s context (the project dir plus the generated Dockerfile) is a scratch tree on the service’s own disk, under the results root’s _staged/image-builds/<build-id>/ slot (cluster_image_build.context_slot, written by stage_context). The build Job’s init container fetches it from the service’s data plane with a token scoped to that slot (pod_access), so a cluster config contributes nothing here: no storage backend and no credential reaches the Job, only a token that opens one slot.

A staged context is deleted again when the build ends (ClusterService calls its own discard_staged on the slot; cluster_image_build.staged_context_build_ids lists what is still staged for the sweep of contexts whose Job is gone).

Pod DNS for unresolvable hosts. get_host_aliases() parses ROBOVAST_EXTRA_HOST_ALIASES (<host>=<ip>, comma-separated, grouped by IP into the shape of the k8s field) and is spliced into the build Job (cluster_image_build.build_job_manifest) and campaign Jobs (kubernetes_backend). Override it if a deployment knows its aliases from somewhere other than the environment. A malformed entry raises rather than being skipped: a dropped alias reappears as an unexplained no such host inside a pod, far from its cause.

The boundary is worth keeping in mind when adding pod specs: hostAliases writes /etc/hosts in the pod, so it governs what the pod’s own processes resolve — the BuildKit push, or a node inside the scenario. The image pull is performed by the node’s container runtime before the pod exists, so no pod-level field can influence it; that stays node configuration, exactly like registry TLS trust.

Add Share Provider Plugin

Share providers are discovered as entry-point plugins under the robovast.share_providers group. They determine where a finished campaign is uploaded (see Sharing Results). To add a new provider:

  1. Create a provider class that inherits from BaseShareProvider and implements the three abstract methods:

    import os
    
    from robovast.execution.share_providers.base import (
        BaseShareProvider,
        StreamProgressReader,
        UploadProgressReader,
    )
    
    class MyShareProvider(BaseShareProvider):
        SHARE_TYPE = "myshare"
    
        def required_env_vars(self) -> dict[str, str]:
            return {
                "ROBOVAST_SHARE_URL": "URL of the target folder",
                "MY_SHARE_TOKEN":     "API token for the share service",
            }
    
        def build_pod_env(self) -> dict[str, str]:
            return {
                "MY_SHARE_URL":   os.environ["ROBOVAST_SHARE_URL"],
                "MY_SHARE_TOKEN": os.environ["MY_SHARE_TOKEN"],
            }
    
        def upload_archive(self, archive_path, object_name, progress_callback=None):
            total = os.path.getsize(archive_path)
            with open(archive_path, "rb") as fh:
                body = UploadProgressReader(
                    fh, total, progress_callback=progress_callback)
                ...  # PUT/stream `body` to the share, raising on failure
    
        def upload_archive_stream(self, fileobj, object_name, progress_callback=None):
            # The launch-time upload-to-share path: the campaign is tarred + gzipped
            # on the fly, so `fileobj` is a readable stream of UNKNOWN length. Use a
            # chunked/streaming transfer (no Content-Length) and wrap `fileobj` in
            # StreamProgressReader to report bytes-sent (total is 0/unknown).
            reader = StreamProgressReader(fileobj, progress_callback=progress_callback)
            ...  # stream `reader` to the share with chunked transfer, raising on failure
    
  2. Implement both upload methods on BaseShareProvider. They run in-process in the driver (no sidecar, no subprocess) and read credentials from os.environ (populated by build_pod_env()):

    • upload_archive() uploads a local archive_path (the download_archive counterpart; known size, resumable). Wrap the body in UploadProgressReader.

    • upload_archive_stream() uploads a streamed archive of unknown length (the launch-time upload-to-share, which never writes a tar.gz to disk — decisive for ~1TB campaigns). Use chunked transfer (no Content-Length; resume is not available) and wrap the body in StreamProgressReader.

    Optionally override verify_access() with a cheap authenticated check so a bad configuration fails the pre-flight credential check before any batches run.

    Raise ShareError for everything the remote or the configuration refuses, and put the setting to change in the message. The share step is best-effort — the campaign keeps its data and finishes either way — so that message is the whole report: it is what the campaign log shows, what share_error carries into the durable outcome, and what vast share prints. A ShareError is recorded without a stack trace for that reason, so a message that only quotes a status code leaves the reader with nothing to act on.

  3. Register the provider in your package’s pyproject.toml:

    [tool.poetry.plugins."robovast.share_providers"]
    myshare = "mypackage.myshare:MyShareProvider"
    
  4. Re-install the package (pip install -e .) so the entry point is registered.

After that, ROBOVAST_SHARE_TYPE=myshare in .env will select your provider automatically.

Share provider API reference

class robovast.execution.share_providers.base.BaseShareProvider

Base class for all share providers.

A share provider encapsulates everything needed to upload a tar.gz archive to a remote storage service (Nextcloud, Google Drive, …) and to download it back. Both directions run in-process in the controller pod (no archiver sidecar, no kubectl exec): see robovast.execution.cluster_execution.in_pod_upload.

Subclasses must:

  • Set SHARE_TYPE to the provider name (matching the entry-point key).

  • Declare all required environment variables in required_env_vars().

  • Implement upload_archive() (the in-process transfer).

  • Provide environment variables for the controller pod via build_pod_env() (the launcher resolves host credentials — e.g. a key file into inline JSON/PEM — and injects them into the pod environment).

The constructor automatically validates that all required env vars are present; it raises ShareError if any are missing. Values are read from os.environ (which is already populated by python-dotenv before the provider is instantiated).

To add a new provider:

  1. Create a new file in this package (e.g. myshare.py).

  2. Subclass BaseShareProvider, fill in the abstract members.

  3. Register the provider in pyproject.toml under [tool.poetry.plugins."robovast.share_providers"].

SHARE_TYPE: str = ''

Short identifier for the provider, e.g. "nextcloud" .

archive_exists_on_share(object_name: str) → bool

Return True if object_name already exists on the share.

Used to skip an upload when the archive is already present (unless --force is given). Only meaningful for providers that support remote listing or HTTP HEAD checks.

The default implementation always returns False (no skip), so providers that do not override this method will always re-upload.

Parameters:

object_name – Filename of the archive on the server (e.g. campaign-2025-02-27-123456.tar.gz).

Returns:

True if the archive already exists on the share, False otherwise.

archive_url(object_name: str) → str | None

Return a full, human-shareable link to object_name on the share.

object_name is a value as returned by list_archives() (a bare object key). The link is for display — so users can copy it or hand it to a browser / curl. It need not be pre-authenticated: for private shares it identifies the object’s location, not a signed download.

Returns None (the default) when the provider has no meaningful URL form for an object.

abstract build_pod_env() → dict[str, str]

Return environment variables to inject into the pod exec call.

These variables will be set for the upload script executed inside the archiver container. Include everything the script needs: URLs, tokens, credentials, etc.

The return value is merged into the pod’s environment via the --env flag of kubectl exec.

Returns:

Mapping of variable name to value.

Return type:

dict[str, str]

download_archive(object_name: str, dest_path: str, progress_callback=None, resume_offset: int = 0) → None

Download object_name from the share to the local dest_path.

Parameters:
  • object_name – The object/file name on the share (as returned by list_archives()).

  • dest_path – Absolute local path to write the downloaded file to.

  • progress_callback – Optional callable (bytes_received, total_bytes) called periodically during the download.

  • resume_offset – Byte offset to resume downloading from. When non-zero the provider should skip the first resume_offset bytes and append to dest_path.

Raise NotImplementedError if the provider does not support downloading (default).

list_archives() → list[str]

Return the object names of every archive this system wrote to the share.

Both kinds: a campaign’s results and a workspace’s project files. An implementation keeps the names is_share_archive_name() accepts and drops everything else – a share is somebody’s storage and holds other things, and classifying a name is the caller’s job, not the provider’s.

Raise NotImplementedError if the provider does not support downloading (default). Implementations should return bare object names (keys), not full URLs.

The default implementation delegates to list_archives_with_size() and discards the size. Override list_archives_with_size() to provide sizes.

list_archives_with_size() → list[tuple[str, int]]

Return (object_name, size_in_bytes) for each archive on the share.

The listing list_archives() is built on; same membership rule.

size_in_bytes is -1 when the provider cannot determine the file size. Raise NotImplementedError if the provider does not support listing at all (default).

Implementations should return bare object names (keys), not full URLs.

remove_archive(object_name: str) → None

Remove object_name from the share.

Two callers: vast share remove, and the discard of what an interrupted upload left (KubernetesBackend.discard_partial_share). The second is why raising for an object that is not there stays correct — an operator naming a missing archive needs to be told, and the discard path words its own uncertainty rather than having this one guess for it.

Parameters:

object_name – The object/file name on the share (as returned by list_archives()).

Raise NotImplementedError if the provider does not support removal (default).

abstract required_env_vars() → dict[str, str]

Return a mapping of environment-variable name → human-readable description.

All listed variables must be non-empty strings in the environment when the provider is instantiated. The base class validates them automatically and raises ShareError if any are missing.

Example:

return {
    "ROBOVAST_SHARE_URL": "Public share URL of the target folder",
}
abstract upload_archive(archive_path: str, object_name: str, progress_callback=None) → None

Upload the local archive_path to the share as object_name.

Runs in-process in the controller pod. Credentials and target settings are read from os.environ (populated by build_pod_env() at launch). Implementations should be resumable where the backend allows it.

Parameters:
  • archive_path – Absolute path to the local <campaign>.tar.gz.

  • object_name – Destination object/file name on the share (the archive basename, optionally with a provider prefix).

  • progress_callback – Optional (bytes_sent, total_bytes) callable, invoked periodically during the transfer — the same shape as the download_archive() callback. Use UploadProgressReader to drive it from a streamed request body.

Raise on failure (the caller treats any exception as a failed upload and keeps the controller alive for a retrigger).

upload_archive_stream(fileobj, object_name: str, progress_callback=None) → None

Upload a streamed archive (unknown length) to the share as object_name.

The on-the-fly counterpart of upload_archive(): fileobj is a readable binary stream of the tar.gz (produced by campaign_archive) that must never be materialised to disk — decisive for the ~1TB campaigns this project runs. Because the length is unknown, this path uses chunked transfer and does not support resume; progress_callback(sent, 0) reports bytes sent.

Default raises NotImplementedError; every built-in provider overrides it. Wrap fileobj in StreamProgressReader (for a file-like body) to drive the callback.

verify_access() → None

Verify the share is reachable with the configured credentials.

Called by the in-cluster controller before any batches start, so a campaign that could never be delivered fails fast instead of wasting compute. Implementations should perform the cheapest authenticated operation that proves write access (a HEAD/PROPFIND, a token exchange, an SFTP stat) and raise on failure.

The default is a non-blocking warning: providers that cannot cheaply check access do not gate the campaign.

class robovast.execution.share_providers.base.ShareError(message: str, ctx: Context | None = None)

A share operation refused by the remote, or misconfigured locally.

Everything a provider raises is of this shape: the credentials are wrong, the URL points somewhere that is not the share, the remote said no. The message names the setting to change, so it is complete on its own.

A click.UsageError so vast share keeps printing it as one line without a stack; the include_traceback flag carries the same promise to the callers that are not the CLI – the controller logs it and failure_detail() records it – because the frames through requests name nothing the sentence does not.

class robovast.execution.share_providers.nextcloud.NextcloudShareProvider

Upload/download campaign archives to a public Nextcloud share (WebDAV).

The share must be a public link that allows file uploads without a password. In the Nextcloud web UI, create a share with “Allow upload and editing” enabled and copy the link.

Required .env variables:

Variable

Description

ROBOVAST_SHARE_TYPE

Must be nextcloud

ROBOVAST_SHARE_URL

Public share URL (e.g. https://cloud.example.com/s/AbCdEfGhIjKlMn)

SHARE_TYPE: str = 'nextcloud'

Short identifier for the provider, e.g. "nextcloud" .

archive_url(object_name: str) → str

Return the public download link for object_name on the share.

Nextcloud serves a single file from a public folder share via https://<host>/s/<token>/download?files=<name> — a browser- and curl-friendly link that needs no separate authentication when the share is public.

build_pod_env() → dict[str, str]

Return environment variables to inject into the pod exec call.

These variables will be set for the upload script executed inside the archiver container. Include everything the script needs: URLs, tokens, credentials, etc.

The return value is merged into the pod’s environment via the --env flag of kubectl exec.

Returns:

Mapping of variable name to value.

Return type:

dict[str, str]

download_archive(object_name: str, dest_path: str, progress_callback: Callable[[int, int], None] | None = None, resume_offset: int = 0) → None

Download object_name from the Nextcloud share to dest_path.

Parameters:
  • object_name – Filename of the archive on the share.

  • dest_path – Local destination path.

  • progress_callback – Optional (bytes_received, total_bytes) callable.

  • resume_offset – Byte offset to resume downloading from.

list_archives() → list[str]

Return a list of campaign *.tar.gz filenames on the share.

list_archives_with_size() → list[tuple[str, int]]

Return (filename, size_in_bytes) for each campaign *.tar.gz on the share.

Uses WebDAV PROPFIND Depth: 1 against the Nextcloud public.php/webdav/ endpoint, authenticated with the share token as the HTTP Basic Auth username. size_in_bytes is -1 when the server does not return a getcontentlength value.

remove_archive(object_name: str) → None

Delete object_name from the Nextcloud share via WebDAV DELETE.

Parameters:

object_name – Filename of the archive on the share (as returned by list_archives()).

required_env_vars() → dict[str, str]

Return a mapping of environment-variable name → human-readable description.

All listed variables must be non-empty strings in the environment when the provider is instantiated. The base class validates them automatically and raises ShareError if any are missing.

Example:

return {
    "ROBOVAST_SHARE_URL": "Public share URL of the target folder",
}
upload_archive(archive_path: str, object_name: str, progress_callback=None) → None

Upload archive_path to the public Nextcloud share via WebDAV PUT.

upload_archive_stream(fileobj, object_name, progress_callback=None) → None

Stream fileobj to the public Nextcloud share via a chunked WebDAV PUT.

No Content-Length is sent (the archive length is unknown), so http.client falls back to Transfer-Encoding: chunked. Wrapping the stream in StreamProgressReader (which exposes only read) keeps the client from inferring a length.

verify_access() → None

Confirm the public Nextcloud share accepts authenticated WebDAV access.

Issues a PROPFIND Depth: 0 against the share’s public.php/webdav/ collection; 207 means the share token resolves and uploads are permitted.

class robovast.execution.share_providers.gcs.GcsShareProvider

Upload campaign archives to a Google Cloud Storage bucket.

Authentication uses a service-account key file. Create a service account with the Storage Object Creator role on the target bucket, generate a JSON key, download the file, and point ROBOVAST_GCS_KEY_FILE at it.

Required .env variables:

Variable

Description

ROBOVAST_SHARE_TYPE

Must be gcs

ROBOVAST_GCS_BUCKET

Target GCS bucket name (e.g. my-robovast-results)

ROBOVAST_GCS_KEY_FILE

Path to the service-account JSON key file (required for writing only – an upload or a remove; not needed for results download on public buckets)

Optional .env variables:

Variable

Description

ROBOVAST_GCS_PREFIX

Object-name prefix inside the bucket (e.g. results/). Defaults to the bucket root.

SHARE_TYPE: str = 'gcs'

Short identifier for the provider, e.g. "nextcloud" .

archive_url(object_name: str) → str

Return the public storage.googleapis.com URL for object_name.

object_name is the full object key (including any prefix) as returned by list_archives(). The URL downloads directly for a publicly-readable bucket — the same URL download_archive() fetches.

build_pod_env() → dict[str, str]

Return environment variables to inject into the pod exec call.

These variables will be set for the upload script executed inside the archiver container. Include everything the script needs: URLs, tokens, credentials, etc.

The return value is merged into the pod’s environment via the --env flag of kubectl exec.

Returns:

Mapping of variable name to value.

Return type:

dict[str, str]

download_archive(object_name: str, dest_path: str, progress_callback: Callable[[int, int], None] | None = None, resume_offset: int = 0) → None

Stream object_name from the public GCS bucket to dest_path.

Uses chunked streaming so that archives of any size (including 100 GB+) are written incrementally without loading the file into memory.

Parameters:
  • object_name – GCS object key (as returned by list_archives()).

  • dest_path – Local file path to write the downloaded content to.

  • progress_callback – Optional (bytes_received, total_bytes) callable called after each chunk. total_bytes is 0 if unknown.

  • resume_offset – Byte offset to resume downloading from.

list_archives_with_size() → list[str]

List all campaign *.tar.gz objects in the configured GCS bucket.

Recognizes archives whose base name (without .tar.gz) matches the campaign naming convention (<campaign-name>-YYYY-MM-DD-HHMMSS). Uses the public GCS XML API (no credentials required for public buckets). Handles GCS list pagination via the NextContinuationToken marker.

Returns:

List of (object_name, size_in_bytes) tuples.

remove_archive(object_name: str) → None

Delete object_name from the GCS bucket.

Requires ROBOVAST_GCS_KEY_FILE to be set to a service-account key file with at least Storage Object Admin (or Storage Object Viewer + Storage Object Creator + delete permission) on the bucket.

Uses the GCS JSON API DELETE endpoint with a Bearer token.

Parameters:

object_name – GCS object key (as returned by list_archives()).

required_env_vars() → dict[str, str]

Return a mapping of environment-variable name → human-readable description.

All listed variables must be non-empty strings in the environment when the provider is instantiated. The base class validates them automatically and raises ShareError if any are missing.

Example:

return {
    "ROBOVAST_SHARE_URL": "Public share URL of the target folder",
}
upload_archive(archive_path: str, object_name: str, progress_callback=None) → None

Upload archive_path to the bucket as <prefix><object_name>.

Uses a GCS resumable upload so an interrupted transfer can continue: the session URI is persisted next to the archive and, on a later attempt, GCS is asked how many bytes it already holds so the upload resumes from there. Expired sessions (HTTP 404) restart cleanly.

upload_archive_stream(fileobj, object_name, progress_callback=None) → None

Stream fileobj to the bucket via a chunked resumable upload (no local file).

The archive length is unknown, so each chunk is PUT with Content-Range: bytes X-Y/* (GCS replies 308 Resume Incomplete) until EOF, when the final chunk carries the now-known total. Not resumable across attempts (there is no on-disk archive to seek back into).

verify_access() → None

Confirm the service-account credentials can reach the target bucket.

Exchanges the service-account key for an access token (proving the key is valid) and GETs the bucket metadata over the GCS JSON API (proving the account can see the bucket). Accepts the key as inline JSON (ROBOVAST_GCS_KEY_JSON, as injected into the controller) or, on the host, via ROBOVAST_GCS_KEY_FILE.

Load and pre-flight the configured share provider, in-process on the driver.

The actual delivery is a streamed upload: the campaign is tarred + gzipped on the fly from the driver’s scratch straight into provider.upload_archive_stream (see KubernetesBackend.share_campaign()), so no compressed copy ever lands on disk. This module stays generic and just resolves the provider and checks its credentials.

Share credentials come from the service’s own environment (its Deployment env), so they are already present in os.environ here. load_provider_from_env() reads them, with optional overrides supplied per call – it lives in core (robovast.execution.share_providers) and is re-exported here, because the service resolves the same provider for an import and cannot import this package.

robovast.execution.cluster_execution.in_pod_upload.load_provider_from_env(overrides: dict | None = None)

Instantiate the configured share provider from the environment, or None.

Here rather than in the Kubernetes backend: the service itself talks to the share (importing a campaign from it, listing it for the web UI), and core must never import a backend – that edge is what would make the dependency graph cyclic.

Parameters:

overrides – Optional {ENV_VAR: value} applied to os.environ before the provider is built – used to supply corrected credentials without relaunching.

Returns:

A BaseShareProvider, or None when ROBOVAST_SHARE_TYPE is unset.

Raises:
  • ValueError – the configured share type has no registered provider.

  • ShareError – required provider env vars are missing (from the constructor).

robovast.execution.cluster_execution.in_pod_upload.share_type_configured() → bool

True when a share provider is configured in the environment.

robovast.execution.cluster_execution.in_pod_upload.verify_share_access(provider) → None

Run the provider’s pre-flight credential check (raises on failure).

Add a MCP Plugin

Contribute data, not read tools. A domain package (a robot type, a stack) makes its results readable through core rather than through tools of its own. A per-domain read tool is a second reader of facts the index already holds: it drifts from the first, it exists only where its package is installed, and every domain that adds one spends the tool budget every client pays for. The seams that make a domain’s facts readable everywhere:

  • A table that follows a contract. A pose table that follows the pose contract appears in pose_track_view with no registration. A measurement outside a contract is still a table, and its meaning goes into column notes.

  • Scene markers. A variation’s config_view_data places neutral markers (a planned path, goals, obstacles). The config view draws them, get_config_contribution serves them, get_track_deviation measures against a path marker, and draw_config draws them.

  • A config panel’s plot. A package that renders a contributed file in the browser draws it for draw_config too (Web UI internals).

An MCP plugin is for what these cannot carry, such as an action a domain offers.

Create a class with a name property and a register(mcp) method:

# my_package/mcp_plugin.py
from fastmcp import FastMCP

class MyMCPPlugin:
    @property
    def name(self) -> str:
        return "my_plugin"

    def register(self, mcp: FastMCP) -> None:
        @mcp.tool()
        def my_tool() -> str:
            """A custom tool."""
            return "hello"

Then register the class as an entry point in pyproject.toml:

[tool.poetry.plugins."robovast.mcp_plugins"]
my_plugin = "my_package.mcp_plugin:MyMCPPlugin"

The plugin is picked up automatically the next time the server starts.

A call the tool rejects — an unknown or missing argument — is answered with the arguments the tool does take. Where a tool lacks an argument a caller will reach for on purpose, declare why with robovast.mcp_server.lacks.lacks(), and the reason is added to that answer instead of costing description tokens on every request:

from robovast.mcp_server.lacks import lacks

@lacks(timeout="the bound follows from what is run")
def my_tool(command: str = "") -> dict: ...

@lacks("there is only one, and it belongs to the service")
def stop_it() -> dict: ...

Where the plugin’s tools answer a question an agent would otherwise put to a core tool, give the class an instructions attribute: one short paragraph appended to the server’s MCP instructions, the only text a client shows before any tool is chosen. Core text names no plugin tool, so the routing exists exactly where the plugin is installed. The server refuses to start when the instructions with every plugin’s paragraph exceed what a client shows (the instructions limit); keep the paragraph to a sentence or two.

Add Search Strategy Plugin

A search strategy drives the closed-loop search (see Iterative Search): it proposes parameter sets, is told their evaluations, and produces a final report. Strategies are algorithm-agnostic and share one config schema; only the per-strategy strategy_parameters block differs.

Subclass robovast.search.strategy.SearchStrategy and implement the four abstract methods. Optionally set PARAMS_MODEL to a Pydantic model — the framework validates search.strategy_parameters against it and passes the parsed object as params:

from pydantic import BaseModel
from robovast.search.strategy import SearchStrategy
from robovast.search.types import ParamSet, SearchReport

class MyParams(BaseModel):
    step: float = 0.1

class MyStrategy(SearchStrategy):
    PARAMS_MODEL = MyParams  # optional; omit (None) for no parameters

    def ask(self, n: int) -> list[ParamSet]:
        """Propose n parameter sets (keys match search_space dims)."""
        ...

    def tell(self, evaluations) -> None:
        """Ingest the evaluations of the batch just run."""
        ...

    def is_done(self) -> bool:
        """True when the budget is exhausted / converged."""
        ...

    def report(self) -> SearchReport:
        """Return the deliverable (ranked best, archive, Pareto front)."""
        ...

self.search_space, self.objectives and the validated self.params are available on the instance. For single-objective strategies, self.objective_value(ev) returns the sole objective sign-oriented so that higher is always better.

Register the class under robovast.search_strategies; the key is the search.strategy name:

[tool.poetry.plugins."robovast.search_strategies"]
my_strategy = "your_package.strategies:MyStrategy"

A strategy can also be loaded from a local file relative to the .vast without packaging, using strategy: ./search/my_strategy.py:MyStrategy (the same load_ref mechanism used for extractors and search postprocessing).

Add Extractor Plugin

The extractor is the single, SUT-specific scoring step: it reads a parameter set’s per-config result directory and returns named objectives (optimized) and measures (quality-diversity behavior axes; ignored by non-QD strategies). This is the one place system-under-test logic lives.

Subclass robovast.search.extractor.Extractor. It is constructed with the extract.params from the .vast (so thresholds / column names can be swept without editing code), and aggregation over the config’s runs is its responsibility:

from pathlib import Path
from robovast.search.extractor import (Extractor, ExtractResult,
                                       completed_run_dirs)

class MyExtract(Extractor):
    # __init__(self, **params) is inherited; params land on self.params

    def extract(self, config_dir: Path) -> ExtractResult:
        runs = completed_run_dirs(config_dir)        # helper: finished runs
        failures = sum(1 for r in runs if _failed(r))
        return ExtractResult(
            objectives={"failure_rate": failures / max(len(runs), 1)},
            measures={},                              # {} when unused
        )

objectives and measures are named dicts, so single- and multi-objective use the same shape. The framework records how many runs backed each result.

Register under robovast.extractors (referenced by search.extract.plugin), or load from a local file with extract.plugin: ./search/extract.py:MyExtract:

[tool.poetry.plugins."robovast.extractors"]
my_extract = "your_package.extractors:MyExtract"

The extractor reads what a postprocessing plugin produced (e.g. per-run metrics.csv) — it does not compute raw metrics itself. Pair it with a postprocessing plugin (below) that writes metrics.csv from raw artifacts: list that plugin in search.postprocessing (run before extract) and/or in results_processing.postprocessing (analysis), so search and the analysis notebooks read the same metrics. Postprocessing plugins load identically in both lists — by entry-point name or a local ./path.py:Class file reference.

Campaign Store and Results Indexing

Every campaign — batch or search — is described by a single sqlite store, campaign.db (robovast.common.store.STORE_FILENAME), written at the root of the campaign directory. It is the single source of truth the results GUI reads, and the seam an in-cluster controller or web UI can later read/stream.

A campaign runs one or more batches: a batch-mode campaign (no search: block) has exactly one batch of the enumerated configs; a search campaign has one batch per ask/tell round.

Schema

robovast.common.store.CampaignStore is a thin wrapper over five tables:

campaign (1) --< batch (1) --< unit (one per param set / config) (1) --< run (one per repetition)
campaign (1) --< job  (one per execution job)  ...............<  run (via run.job_id)
  • campaign — mode (batch/search), config_dir (base directory against which visualization.results.explorer.notebooks notebooks resolve), config_json (the full config), an opaque strategy_state blob for resumable strategies, and the campaign’s execution provenance: robovast_version, execution_type (local/cluster), image, image_revision (the repo@sha256 the runs actually used), execution_started_at, elapsed_s, plus execution_json holding the rest of _execution/execution.yaml. Those are the fields one compares across campaigns, and the SQL interface can attach several campaigns at once — see What goes in the database, and what stays a file. They are written by record_execution once the backend has produced execution.yaml, so they are NULL for a campaign that died before execution began. Note execution.yaml’s own execution_time is a start timestamp, not a duration, hence the column name. What the campaign was asked for is not here: it lives in _execution/launch.yaml, because the service can write that before the run starts while these columns can only be filled after it (see the launch record).

  • job — one row per execution job, holding that job’s sysinfo.yaml verbatim in sysinfo_json. It is a table rather than a column on run because sysinfo is written once per job: a packed multi-config job runs several (config, run) pairs which reach the same file through each run dir’s job symlink. A per-run copy would repeat the blob and destroy the fact that those runs shared a machine — which is what makes “did the slow runs land together?” answerable. job_dir is campaign-relative (_jobs/batch-0/job-3), or the run’s own directory for an older layout that wrote sysinfo beside the run.

  • batch — one ask/tell round (search), or the single batch (idx=0) of a batch-mode campaign.

  • unit — one evaluated parameter set (search) or one configuration (batch): the sampled params, objectives/measures (JSON; {} for batch), and the result_dir. n_samples and the aggregate status are roll-ups of the unit’s run rows, kept for convenience.

  • run — one repetition of a unit (schema v2+). Mirrors that run’s test.xml: status (passed/failed/error/unknown), passed (0/1), errors/failures/tests, duration_s, start_time and failure_message. run_id is the numeric run index within the config dir — so it is not unique on its own; config_name lives on unit. job_id points at the job it ran in. A run whose test.xml is missing or unparseable is still recorded, as unknown — never dropped.

Two definitions of the schema, on purpose

_SCHEMA is the full current layout, applied in one step to a fresh database, so a reader can see what a table looks like without replaying history. _MIGRATIONS is the append-only ladder that upgrades an existing store; entry i takes user_version i to i+1 and is never edited once shipped, because some database on disk has already applied it. Note migration 0→1 is a frozen copy of the v1 layout rather than _SCHEMA — reusing _SCHEMA there would jump an old store to today’s tables and the later ALTER TABLE steps would then fail on columns that already exist.

Adding a column therefore means touching both, in the same position. test_fresh_and_migrated_schemas_match builds a store each way (from fresh, and from every older user_version) and compares the resulting sqlite_master column by column, so the two cannot drift apart.

Why run exists — the data-model layering

run is the operational source of truth for per-run outcomes, and it is why listing campaigns is cheap. test.xml (JUnit, one per run) is the runner’s on-disk contract; the controller already parses it at record time, so capturing a run row there is free. Pass/fail counts are then one GROUP BY status over run — no filesystem walk — and are available live, before postprocessing. The central index’s run_view is the analytics-wide view over these rows (joining sysinfo and the unit’s params); the ingest mirrors campaign.db rather than re-parsing every test.xml. Heavy per-run measurement data (metric time-series) is streamed into the index — campaign.db remains the lightweight live store.

test.xml         runner artifact (per run) — the on-disk contract
  -> captured live at record time (data already in hand)
campaign.db.run  operational source of truth — queryable DURING the run
  -> postprocessing joins sysinfo/params/metrics
index runs view  analytics-ready wide view (param_* columns, metrics) — DERIVED

Note

Schema v1 stores (written before the run table) migrate forward on open to an empty run table. robovast.common.campaign_index.backfill_run_rows() fills them from disk test.xml; the service does this lazily for finished campaigns, and the summary path falls back to the test.xml walk (get_vast_configuration_info()) until it is backfilled, so counts are never under-reported.

Who writes it

  • The controller (robovast.execution.controller.CampaignController) writes the store live for both modes as each batch is evaluated, so progress is queryable while a campaign runs. It owns the campaign id, the flat results layout (<campaign>/<config>/<run>/) and the batch loop; an ExecutionBackend (KubernetesBackend) only dispatches one batch’s jobs.

  • The post-hoc indexer robovast.common.campaign_index.build_campaign_store(campaign_dir) reconstructs the same store by scanning a finished results tree (reusing the campaign_data readers). It is used for campaign dirs not produced by the controller — e.g. one unpacked from an archive somebody else made — and is idempotent (mtime-guarded; force=True to rebuild). Controller-written stores are left untouched. It has no CLI entry point, so a tree that has no store keeps none. The function itself is the register step of vast campaign import: an archive somebody else produced usually has no store, and extraction alone leaves the campaign invisible to every store-driven view — which is what keeps the primitive alive, and out of a routine dead-code sweep.

Taking a campaign in

robovast.service.ingest owns both halves. import_archive() unpacks an archive into a results root; ingest_campaign() registers what came out and reports per stage, since a campaign archive carries three version surfaces of its own (the .vast’s, campaign.db’s and the analysis DB’s) which can independently be older, newer, absent or corrupt. Neither re-implements a migration – the config ladder is applied in memory and the store migrates on open, so this module observes and reports.

Three entry points, one implementation: vast campaign import (locally, or streamed to a reachable service), POST /campaigns/import behind the web UI’s upload button, and the import_campaign MCP tool. Only the first two ever move bytes; both do it through the archive side channel documented in HTTP API, and the import itself always takes a path on the host that will hold the campaign.

Two refusals in import_archive are load-bearing rather than defensive. The archive must hold exactly one top-level entry, computed with a ./ root ignored – reading the first path segment literally makes tar czf x.tar.gz -C <results> . look like one entry named ., which resolves to the results root, so a forced import would have deleted every campaign to make room for itself. And that entry must satisfy is_campaign_dir(), because the local listing and the delete guard both filter on that shape: a differently-named directory would import, report every stage ok, and then be invisible and undeletable.

Store-driven results views

The results GUI (RunResultsAnalyzer) discovers campaigns by scanning <results_dir>/*/campaign.db — there is no filesystem-walk or depth-based heuristic. It reads the campaign/batch/unit rows to build the tree (campaign → batch, search only → config), resolves notebook workloads from config_json against config_dir, and enumerates only the run-level leaves from each unit’s result_dir.

Campaign state and control

A campaign is driven by a CampaignController running in the driving process — the robovast-service, one worker thread per campaign. There is no separate controller pod and no in-pod HTTP control channel: monitoring and control are ordinary RobovastInterface calls the service answers directly, so the CLI, MCP and web UI all use one path.

Live state

ControllerState is the thread-safe holder the controller writes and readers snapshot. The controller calls update / set_phase at each batch boundary; a reader takes a consistent snapshot (deep copy), which is exactly what get_status returns. The one status model, Status, is reused verbatim by the interface, the MCP tools and the TS client.

Because the service drives many campaigns as threads in one process rather than one pod each, per-campaign state is isolated explicitly: the container-runner factory is a ContextVar, controller.log is filtered to its worker thread, and each aux pod / result prefix is keyed by campaign id.

Status: phase and stage

phase is an open string advanced through a documented vocabulary, with stage carrying finer markers so new states slot in without a schema change:

phase

Meaning

initializing

Accepted: registered, listed, and addressable by id, with the service’s pre-flight (project push, registry/base-image resolution) still to do. The first phase every campaign has, and the one that guarantees a caller can always find what it just started — see A started campaign is findable.

building

Building the experiment image (only when the .vast has a build: section).

starting

Image settled; handing off to the worker (campaign context, backend construction).

plugin install

Installing the declared plugins: (only when any are declared). pip’s output streams live into _execution/plugin_install.log.

variation

Expanding the config variations (batch mode); its log is variation.log.

running

Batch loop executing.

finishing

Search stop condition met (or a stop was requested); winding down.

sharing

Streaming the raw pre-postprocessing archive to the configured share (only when upload_to_share was set).

postprocessing

Chained analysis postprocessing (rosbag→CSV Job + index ingest) running.

finished

Done; every run’s output is on the service’s results volume. A post-run step may still have failed: the runs are the deliverable, so a failed upload-to-share or postprocessing keeps the phase finished and records the reason on share_error / postprocessing_error (durable, re-triggerable) rather than failing the campaign.

failed

The runs were aborted. stage/error say why (e.g. a bad config_filter lists the available config names). Recorded to _execution/outcome.json so it is readable after the fact.

stopped / crashed / unknown

Terminal: cooperatively stopped; the driver died without recording an outcome; or (after a restart) not reconstructable from disk.

phase_since records when the current phase was entered (and only moves on an actual change, so a defensive re-set does not restart the clock). It exists because a phase name alone cannot separate slow from wedged: an image build in progress and one that will never finish both read building. Readers render it as an age — the MCP status returns phase_age_s, and the web Monitor shows it beside the phase dot while a pre-run phase is in effect, where there is no progress bar to watch instead.

A started campaign is findable

Invariant: from the moment the service accepts a campaign, its id resolves on every read path — get_status and list_campaigns (also the MCP’s running_only filter). This is what makes the standing advice (“a timed-out start is not a failed start; check, never retry”) actually true: retrying a start that in fact succeeded creates a second campaign, and the only defense is that the first one can be found.

Two things back it. Registration happens before the slow work — create_campaign records the campaign in the service’s registry and returns, and the driver builds the image — so the campaign is live from t=0 rather than from whenever its results directory appears (see A campaign waits for its image; the request does not). list_campaigns unions that registry into its id set beside the disk scan, so a campaign is listed for the whole length of its pre-flight — without that it would be registered and addressable by id, but undiscoverable by anyone who did not already know the id, which is precisely the caller whose start response was lost.

Discovering campaigns across a service restart

list_campaigns builds its id set from two sources: the results root on disk, and the in-memory registry of what is being driven. The first is what survives the process, and it is the whole picture — a campaign is a directory under <results_root>/<campaign_id>/ on the service’s results volume, so a service that restarts finds every campaign from every previous life by the same iterdir. There is no second home to enumerate and no index of campaigns to keep in step with the tree.

That is also what makes the records cheap to read. _summary_for, _started_at_for, _description_for and _status_from_disk all resolve the campaign through one seam, campaign_dir, and read campaign.db and _execution/outcome.json from it as local files — no fetch, no per-row round trip on a listing, and nothing that can be reachable for one reader and not another.

The order is computed over every id before the page is cut, so each sort key has to be answerable without building a summary per campaign: the recency keys come from the memoized _started_at_for / _finished_at_for, and sort="size" from _results_bytes_for, which reads the same outcome.json figure the row shows and memoizes it — None included — once a terminal record says it. Only the page’s own campaigns get a _summary_for.

A campaign waits for its image; the request does not

create_campaign is fire-and-forget, so the image build runs on the campaign’s worker, not on the caller’s thread. Awaiting it inline meant a cluster start for a project with a build: section died on the HTTP client’s 30 s read timeout while the server kept going and the campaign succeeded — a reported failure for work that worked — and, since the campaign was created only after the build, none of that work was observable while it ran.

What the caller gets instead: the campaign id immediately, and a campaign in phase building whose stage reads waiting for image <tag>. list_campaigns(running_only=True) needed no change for it to appear there — a building campaign is a campaign, and the listing already unions the in-memory registry. (Teaching that tool to enumerate builds would have needed a listing endpoint that does not exist, returned entries that are not campaigns, and created a second in-flight registry to keep consistent with the first.)

Builds are shared — build_hash is content-addressed over the spec and context, so two campaigns needing the same image both wait on one build. Two consequences:

  • The phase means waiting for its image, not performing the build; otherwise two campaigns each appear to be building the same one. The stage and the build.log header both say so.

  • Stopping a building campaign detaches it. _await_build_image raises CampaignStopped and touches neither the build Job nor the local build thread: a sibling may be waiting on that build, and the image is a cache entry rather than this campaign’s property. Nothing cancels a build today — the cluster teardown is label-scoped to jobgroup=scenario-runs and cannot reach a jobgroup=image-builds Job, and the local docker rm -f robovast cannot reach a buildx thread — and it must stay that way.

One wait loop: _await_build_image drives get_image_build_status and get_image_build_log, which are interface operations each transport implements, so the in-cluster BuildKit Job is waited on by the base’s code rather than by a loop of ClusterService’s own.

The loop also tees the build log into the campaign’s own _execution/build.log, which INFRA_PHASES serves as a leading BUILD section (see the MCP build workflow). That is the campaign’s only durable record of the image it ran on: the live source dies with the build, since a build Job is reaped at ttlSecondsAfterFinished, and a failed build is exactly when someone comes looking. It is also why controller._record_controller_outcome uploads build.log alongside outcome.json — a campaign that died waiting for its image never reaches finalize_campaign at all.

This changes an error path on purpose: a failed build is now an inspectable failed campaign — reason in its status, output in its own log — rather than a 500 and no campaign. Building is part of the campaign’s driven work, not a precondition of its existence.

Control operations

  • stop (client.stop) — sets a cooperative flag on the campaign’s ControllerState (request_stop); in the service this is a direct in-process call. Every wait the campaign’s thread sits in ends on that flag (wait_for_stop), and every boundary between two steps gives up on it (raise_if_stopped), so the phase a campaign happens to be in does not decide how long the stop takes. Which work a stop lands on, and what each one leaves behind, is the per-phase table in Client–Server Architecture.

  • get_campaign_logs — serves controller.log from a byte offset, the same file on the service’s results volume while the campaign runs and after. The web UI polls it to stream the log; vast … monitor renders live status from get_status.

  • upload_to_share (launch flag on create_campaign) — when set, the driver streams a raw, pre-postprocessing archive of the campaign to the configured share the moment the runs finish, before analysis postprocessing. A share failure never loses the campaign: it stays finished and the reason is recorded on share_error (durable). The backend streams it to the share provider with no on-disk copy. The download counterpart is the /data/campaigns/{id}/archive stream (the campaign as the service holds it, tarred on the fly off the results volume).

  • run_share (client.run_share) — re-triggers the upload-to-share on a finished campaign, from the stored campaign alone (works after a service restart, no live entry). The provider comes from the environment, so adjusting ROBOVAST_SHARE_TYPE and re-triggering uploads to a different destination. Mirrors run_postprocessing.

API reference

Live campaign state holder (ControllerState).

Every campaign is driven by a CampaignController that runs in the driving process — the robovast-service. The controller advances a shared ControllerState; the service reads its snapshot() directly to answer GET /campaigns/{id}/status (no separate control server, no pod-IP hop — those existed only when the controller lived in its own pod).

The status contract it carries (Phase, Status and the phase-group predicates) now lives in robovast.client.status, so foundational common modules can depend on it downward instead of common reaching up into execution. It is re-exported here verbatim, so from robovast.execution.control_server import Status (and Phase etc.) keeps working everywhere. fastapi/uvicorn are not needed here.

class robovast.execution.control_server.ControllerState(**initial)

Thread-safe holder the controller writes and the service reads.

The controller calls update() / set_phase() at each batch boundary (and update() for run-level progress within a batch); the reader takes a consistent snapshot(). request_stop() / stop_requested back the cooperative stop (now an in-process call from client.stop rather than an HTTP command).

property postprocessing_stop_requested: bool

Whether postprocessing was asked to end (see stop_checker()).

property progress_suspended: bool
property project_released: bool

Whether the campaign has stopped reading the project it was launched from.

raise_if_stopped(reason: str, scope: str = 'runs') → None

End the caller with CampaignStopped when scope was asked to stop.

For the boundaries between steps, where the wait primitive has nothing to wait on: a campaign staging its project, installing plugins or composing its sweep is doing bounded work in sequence, and the cheapest honest place to give up is before the next step rather than in the middle of one. reason names the step, because it becomes the campaign’s recorded stop message.

release_project() → None

Record that the campaign no longer reads the project it was launched from.

Set once the campaign has staged its configurations, which for a batch campaign is the last thing it does with that project (see CampaignController._on_configs_staged). Never set for a search campaign, which composes out of the project again for every generation. Until it is set, editing that project changes what the campaign is about to run.

request_stop(scope: str = 'runs') → None

Ask scope’s work to end cooperatively.

Defaulting to STOP_RUNS rather than stopping everything: a caller that does not say what it is stopping means the campaign’s runs, which is what a stop meant when there was only one of them.

resume_progress() → None
set_phase(phase: str, stage: str | None = None) → None

Advance to phase, stamping when it started.

phase_since moves only on an actual change, so re-setting the current phase (some paths do, defensively) does not keep resetting the clock a reader uses to tell “slow” from “wedged”.

Reaching a new phase is itself forward movement, so it also restarts progress_since. Without that, a campaign that spent ten minutes in variation would enter running already carrying a ten-minute-old progress clock, and be reported as stalled before its first run had a chance to finish.

A stage belongs to the phase that set it, so a phase change clears it unless the new call names one. Without that, stage outlived its phase for the rest of the campaign’s life: a campaign that had waited for an image reported “waiting for image(s) simulation, sut” while it ran, while it postprocessed, and after it finished – a sentence that was true once and then read as a live statement about a campaign that was already over. The one call site that cleared it by hand (stage="" on entering starting) is the workaround this replaces.

Re-setting the same phase keeps the stage: some paths do that defensively, and clearing there would wipe a stage the same phase had just set.

property share_stop_requested: bool

Whether the share upload was asked to end (see UploadProgress).

snapshot() → Status
property stop_requested: bool

Whether the campaign’s runs were asked to end.

Deliberately still the runs and nothing else. A stop of the analysis or of the upload must not read as “the campaign was stopped”: that is what made a stopped run also discard its analysis, because one bit answered both questions.

stop_requested_for(scope: str) → bool

Whether scope’s work was asked to end.

suspend_progress() → None

Pause the run-progress poller (see CampaignController._poll).

The poller lists the whole campaign prefix every few seconds. During a batch’s result download over the same (off-cluster) storage tunnel, those extra list calls compound tunnel contention; suspending the poller for the download keeps the single transfer uncontended.

update(**fields) → None
update_runs(**fields) → None

Merge fields into the runs sub-model, atomically.

update() assigns whole attributes, so update(runs={...}) would drop every counter the dict omits. The counters are written from different places (the progress poller, the batch-completion tally, the outcome tally) and must not clobber each other, and merging under the lock keeps that safe without each caller having to read-modify-write.

wait_for_stop(timeout: float, scope: str = 'runs') → bool

Wait up to timeout seconds, returning early and True if scope is stopped.

The replacement for time.sleep in every poll loop the campaign’s own thread runs: a sleeping loop only notices a stop at the end of the interval it happened to be in, so the interval that makes a poll cheap is also the delay a stop pays. Waiting on the event instead makes the two independent – the poll stays as rare as it should be, and the stop lands the instant it is requested.

Returns what stop_requested_for() would answer, so a caller can wait and decide in one step.

class robovast.execution.control_server.Status(*, phase: str = Phase.INITIALIZING, phase_since: float = <factory>, progress_since: float = <factory>, progress_deadline_s: int | None = None, waiting_for_capacity: bool = False, batch_since: float | None = None, search_since: float | None = None, stage: str | None = None, mode: str | None = None, campaign_id: str | None = None, batch: int = 0, batches_done: int = 0, budget: list[~robovast.client.status.BudgetItem] = <factory>, runs: ~robovast.client.status.RunProgress = <factory>, best_objective: float | None = None, stop: dict | None = None, error: str | None = None, postprocessed: bool = False, postprocessing_error: str | None = None, share_error: str | None = None, results_bytes: int | None = None, extra: dict = <factory>, health: list[~robovast.client.status.HealthFinding] = <factory>, health_skipped: list[str] = <factory>, nodes_skipped: dict = <factory>, updated_at: float = <factory>)

The controller’s live state, served by GET /campaigns/{id}/status.

phase is an open string the controller advances through a documented vocabulary (initializing → building → starting → variation → running → finishing → importing → postprocessing → sharing → finished / failed); stage and extra exist so future markers (e.g. "upload-to-share-done") slot in without a schema change.

What is on the share is deliberately not here. It was, as share_provider, and nothing ever wrote it – which is the shape of the mistake: the share is another system’s state, so a copy of it on a campaign goes stale the first time somebody deletes an archive out of band. list_share_archives asks the share.

Before adding a field: this is a hot fan-out payload. The web UI renders every campaign in the list as a card and each card polls GET /campaigns/{id}/status every 1.5 seconds, so anything here is paid for once per campaign on screen, per poll, per open tab – and some of it is computed per read (health). What belongs here is therefore BOUNDED state and cheap cursors: a phase, a counter, a best-so-far, a batch index. A SERIES does not, however small it looks: batch_history was one (a row per batch, growing all run long), nothing ever read it, and it was removed in favour of GET /campaigns/{id}/search/history – a route fetched only while something is displaying it, keyed on the batches_done cursor that lives here. The rule, and where a live run view lands, is in docs/http_api.rst.

batch: int
batch_since: float | None
batches_done: int
best_objective: float | None
budget: list[BudgetItem]
campaign_id: str | None
error: str | None
extra: dict
health: list[HealthFinding]
health_skipped: list[str]
mode: str | None
model_config: ClassVar[ConfigDict] = {'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

nodes_skipped: dict
phase: str
phase_since: float
postprocessed: bool
postprocessing_error: str | None
progress_deadline_s: int | None
progress_since: float
results_bytes: int | None
runs: RunProgress
search_since: float | None
share_error: str | None
stage: str | None
stop: dict | None
updated_at: float
waiting_for_capacity: bool

Per-Cluster Resource Resolution

The resolution logic for per-cluster resources (the {context-name: value} mappings documented under Per-Cluster Resource Limits in Cluster Execution) lives in robovast.execution.cluster_execution.cluster_context:

Kubernetes context awareness and per-cluster resource resolution.

Resource values in the .vast config file may be given as a per-cluster list keyed by the real Kubernetes context name instead of a scalar:

resources:
  cpu:
    - gke_my-project_us-central1_my-cluster: 4
    - minikube: 8
  memory:
    - gke_my-project_us-central1_my-cluster: 10Gi
    - minikube: 20Gi

Scalars always work and are the recommended default when a single cluster is used. Pass the matching context name via --context/-x when running commands against a specific cluster.

robovast.execution.cluster_execution.cluster_context.get_active_kube_context() → str | None

Return the name of the currently active Kubernetes context.

Reads the active context from the local kubeconfig (~/.kube/config or KUBECONFIG). Returns None when the context cannot be determined (e.g. kubeconfig is absent).

robovast.execution.cluster_execution.cluster_context.get_config_context_names(config_path: str) → set[str]

Extract all context names used in per-cluster resource lists.

Scans a .vast YAML config for any field that uses the per-cluster list syntax ([{context-name: value}, …]) and returns the union of all keys.

Parameters:

config_path – Absolute path to a .vast YAML config file.

Returns:

Set of context name strings. Empty when no per-cluster lists are found.

robovast.execution.cluster_execution.cluster_context.list_all_contexts() → list[tuple[str, str]]

List all available (label, kube_context_name) pairs from the kubeconfig.

Returns:

List of (label, kube_context_name) tuples sorted by name. Returns an empty list when no kubeconfig is available.

robovast.execution.cluster_execution.cluster_context.require_context_for_multi_cluster(kube_context: str | None, config_path: str | None = None) → None

Raise ValueError when a multi-cluster config is used without --context.

Scans config_path for per-cluster resource lists and raises an informative error when more than one context name is present and no kube_context was specified.

Which config to scan is the caller’s decision, never discovered here: this used to find a .robovast_project by walking up to the filesystem root, so a project far above the CWD could demand --context from a command that was never told about it. There is no ambient project at all now – callers pass the config the operator named with --vast, or None.

This is a no-op when:

  • kube_context is already set (the user supplied --context).

  • config_path is None — nothing named a config, so there is nothing to scan.

  • The config uses only a single context name (or only plain scalars).

Parameters:
  • kube_context – The Kubernetes context name (None when the user did not pass --context).

  • config_path – The .vast to scan, or None when none was named.

Raises:

ValueError – When multiple context names are found and kube_context is None.

robovast.execution.cluster_execution.cluster_context.resolve_resource_value(value: Any, context: str | None) → Any

Resolve a resource value for the active Kubernetes context.

Handles two forms:

  • Scalar (int, float, or str): returned as-is.

  • Per-cluster list ([{context-name: value}, …]): the entry whose key matches context is returned.

Raises:

ValueError – When the value is a per-cluster list but context is None, or when the context has no entry in the list.

Parameters:
  • value – Raw resource value (scalar or per-cluster list).

  • context – Active Kubernetes context name, or None.

Returns:

Resolved scalar value, or None when value is None.

robovast.execution.cluster_execution.cluster_context.resolve_resources(resources: dict, context: str | None) → dict

Resolve all resource fields in a resources dict for the active cluster.

Calls resolve_resource_value() for every key in resources and returns a new dict with all per-cluster lists replaced by their resolved scalar values.

Raises:

ValueError – Propagated from resolve_resource_value() when a per-cluster list has no entry for context.

Parameters:
  • resources – Raw resources dict (e.g. {'cpu': 15} or {'cpu': [{'gke_my-project_…_cluster': 4}, {'minikube': 8}]}).

  • context – Active Kubernetes context name, or None.

Returns:

New dict with resolved scalar values (None entries removed).

Job admission internals

User-facing behaviour is in Cluster Execution; this is the machinery behind it.

RoboVAST admits its own jobs. A campaign’s Jobs are created by AdmissionController, one at a time, each only once the cluster has room for it — so nothing is ever submitted to a queue and left to wait. runs_per_job defaults to 1, so a typical campaign’s plan is upwards of a thousand Jobs, and none is created before the cluster can hold it.

Why creation is the admission point. A Job’s spec.template is immutable: patching it is rejected, so the request a Job is created with is permanent and anything that wants to size a Job per node has to decide before it exists. Holding the plan in RoboVAST’s own store and creating on admit makes that immutability irrelevant rather than something to work around.

Ordering is a property of the queue, not of thread scheduling. One global queue drains in (priority, campaign rank, campaign start) order, and any campaign thread that wakes drains it — so an older campaign’s later batch still outranks a younger campaign’s first, which submission order alone would get backwards. A search submits its batches one after another, so ordering by submission time makes an older campaign look younger with every batch and the two take turns instead of the older one finishing.

Capacity, and the double-counting hazard. Free capacity is allocatable − committed requests − headroom, per node — a pod runs on one machine, so a cluster-wide figure cannot see fragmentation and admitting against it is how jobs end up Unschedulable while the cluster reports room. A Job created moments ago has no pod bound yet, so the next measurement cannot see its requests; handing the same cores out twice is exactly the over-admission this has to avoid. The controller keeps an in-flight ledger and stops subtracting a reservation the instant the measurement starts counting the real pod, which needs no timer and leaves no window where the same cores are counted twice or not at all.

A pin narrows, and only a probe claims its node. A WorkItem may carry a pin (the one node id it may use) and an accepts_node gate (whether its owner’s work may go to a node yet); may_use requires both, and a pinned item never takes a node without an identity label. Two kinds of work are pinned. A calibration probe is pinned to the node it measures and, when it does not fit, claims that node for the rest of the drain pass, so smaller lower-ranked work cannot keep taking the room it is waiting for. A campaign confined with execution.kubernetes.jobs.node is pinned to its node with reserves=False: it always has another job queued, so a claim would renew every pass for its whole life and shut every lower-ranked campaign out of that node. It waits for room like any other work, and preflight judges it against its own node’s capacity instead, since no claim stands behind it. The alias is resolved inside the job node pool before a Job of the campaign exists, and _pin falls back to the campaign’s node wherever the queue granted none. Postprocessing is not confined.

Waiting for capacity is not a failure, and is not a stall. A job that has not been created yet is PLANNED in the controller — a fact it holds, not something inferred from a pod. list_campaign_jobs reports it waiting; the no-progress deadline is suppressed while a batch is entirely queued, because that deadline is a per-run budget and no run of the campaign is running. Getting this wrong is what made a healthy third campaign report as wedged behind two others.

Web UI internals

The web frontend (user guide: Web UI) is a plain Vite + React + TypeScript app (MUI + TanStack Query) living at frontend/ui/. It has its own package.json / node_modules and is independent of the Python build; npm run build emits frontend/ui/dist.

It is deliberately not a plugin framework or a shared UI shell — most of what a bespoke shell would provide maps to stable, widely-used OSS (server-state polling → TanStack Query, charts → Plotly, forms → Monaco/rjsf), so the app is an ordinary React app that reuses those libraries directly.

Thin client of the interface. All service access goes through one seam, frontend/ui/src/lib/robovastClient.ts, which mirrors robovast.service.interface.RobovastInterface — the Routes paths and the request/response models (including Status) — 1:1, exactly as the Python HTTPTransport does. When the interface changes, update this file. Pages call the client via TanStack Query (refetchInterval drives the Monitor’s live polling; mutations drive create/stop). Current pages: pages/Monitor.tsx (the launch bar merged into it — there is no separate Launcher), pages/config/ConfigPage.tsx, pages/results/ResultsPage.tsx and pages/admin/AdminPage.tsx, sharing components/StatusView.tsx for the live Status render.

Two things on the Admin page are worth knowing before changing them.

NavTopic.footer renders a topic at the foot of the sidebar rather than in the main list, and Admin is the only user of it: it is about the service, so it belongs beside the usage meters reporting that same service, and listing it with Campaigns and Results would rank it as their peer. It is a field on the one TOPICS declaration in App.tsx rather than a second prop, so navigation still resolves from one source and hashNav needs no change — a leaf topic already falls out of the existing grammar.

components/LogPanel.tsx is the one live-log renderer, for the campaign log, the job log and the service log alike. It is its own module rather than part of StatusView.tsx: reaching it through there would drag BatchObjectiveChart, DetailsBox and the ETA maths into a lazily-loaded page that needs none of them. That the same component fits all three is not a coincidence: every live log here is a fetch(offset) -> LogChunk behind one SSE loop.

A log panel is a tail, and both ends of it are bounded. A campaign’s assembled infrastructure log reaches tens of megabytes, while the pane it is read through is a few hundred pixels tall — so the stream serves an over-long frame from its end (_sse_log_frame_cap in service/app.py, which leaves a note naming vast campaign log for the whole thing), and the panel keeps only a bounded tail of what it has received (trimHead). The second bound is not about memory: the body is rebuilt from the whole buffer on every delta, so an unbounded one re-renders a span per line of the entire log twice a second. Dropping the head costs the offset protocol nothing — next_offset is where the log continues, not how much was sent — so a resumed connection still resumes at the right byte.

An open stream with nothing in it is ambiguous, and the panel must not resolve it by guessing: these servers flush the response headers before their first pull, and that pull is network I/O. So useLiveStream reports received — whether a frame has actually arrived, a delta or the heartbeat that means “read it, there was nothing” — and only then does the panel call the log empty rather than still loading (logFooter).

The usage recording and the service-log ring both live in the serving layer (service/app.py and service/service_log.py), not on RobovastInterface. The precedent is _sse_campaign_list and /healthz: they describe the process that is serving rather than the campaigns it drives, and there is nothing a transport would implement differently. The usage ring is per-app (on app.state) so two apps built in one process record separately; the log ring is process-global, because a logging handler cannot sensibly be otherwise.

The service serves the SPA. robovast.service.app.build_app() mounts frontend/ui/dist at / (_mount_ui / _ui_dist) after registering the API routes, so the interface routes win and the SPA is served from everything else. The UI is therefore same-origin with the API (no CORS in production) and starts with the service. The build location is frontend/ui/dist relative to the repo, overridable with ROBOVAST_UI_DIST; package the built assets into the service/controller image so the in-cluster service ships the UI. If no build is present the service runs API-only.

Dev loop. cd frontend/ui && npm run dev runs Vite (:5173) and proxies the API path prefixes to a running vast serve (ROBOVAST_SERVICE_URL to retarget), keeping the browser same-origin for hot-reload development.

Navigation is the URL hash, and nothing else holds it. frontend/ui/src/lib/hashNav.ts is the whole grammar — pure, React-free, and the only place a hash is parsed or spelled:

#/<topic>                                     a leaf topic (Config)
#/<topic>/<view>[/<campaign>]                 a topic with views (Results)
#/results/<view>/<campaign>/<config>[/<run>]  ...down to one node of that campaign
#/results/<view>/<campaign>?batch=<i>         ...a search campaign's round
#/results/explorer/<campaign>/…?tab=<name>    ...and which of its notebook tabs
#/config/campaign/<campaign>                  the frozen ``_config/`` of one campaign
#/execution?import=<search>                    the campaign view, share-import dialog on <search>

navFromHash parses into Nav; hashFor spells one back; nextNav is the transition behind a sidebar click. Three rules live there rather than in a page, and each is pinned by hashNav.test.ts:

The last form is the odd one out and says so in its own comment: ?import= is a request carried by a link, not an address. It is parsed for the execution topic only — every page stays mounted (KeepAlive), so a #/config?import=x that resolved would have the campaign view open a dialog over a page nobody is looking at, whereas a ?tab= a view ignores is simply inert. The campaign view clears it once taken, so the address stops claiming a dialog that has been closed. It is also the one URL form with a reader outside the browser: vast share import parses the same fragment (share_cli.campaign_from_ui_link), parse-only, so the copy button produces something that works in both places a campaign is imported.

  • The node address is positional, matching the on-disk and /results spelling of a run (One address space for files) rather than inventing cfg/run markers. ResultsSel is a discriminated union over the four tree levels, so a config name and a batch index cannot both be held — an impossible URL is not representable rather than forbidden in prose.

  • A view’s hash carries only what that view acts on. hashFor gives the Explorer the whole selection plus its tab, the Run view a run-level selection only (it cannot replay a batch or a config), and the Data browser the campaign alone. A hash is a view’s address, not a store for state the view ignores.

  • ``configCampaignId`` is dropped by every transition. A campaign’s frozen config is reachable only from its card, and that guarantee is nextNav dropping the field — a mistake there does not misplace a view, it exposes one.

App owns the Nav state and writes the hash two ways, and the difference is load-bearing: select assigns location.hash (pushes a history entry, as do the cross-page jumps in lib/nav.ts), while setResults uses replaceState — a selection made inside a view must not cost a Back press. It is guarded on hashFor(next) === hashFor(nav), which is total by construction and cannot go stale as Nav grows a field.

Turning a selection into the tree’s node id, and back, is resultsTree.ts’s selectionNodeId / selectionOf, beside the id builders themselves — the URL is a third builder of node ids, so it must not spell one itself. resolveSelection answers “does this campaign have this node, and which round is it in” from the rows the tree already loaded; because a finished campaign’s runs are fixed, it is a useMemo and not a watcher.

Saying something short-lived: useToasts. frontend/ui/src/components/ToastProvider.tsx is the app-wide way to state a passing fact — mounted once in main.tsx, used as const { notify } = useToasts(). It is the sibling of DialogProvider: that one asks a question and blocks on the answer, this one states a fact and gets out of the way. Three rules travel with it.

Failures do not go through it. The Severity union in lib/toasts.ts has no error member, which is the rule expressed as a type: a refusal or an error carries backend text worth reading twice and keeps its inline Alert with ErrorText. Only successes and dispatches become transient.

It is not a notifier. The provider draws a rectangle and nothing else. A caller that also wants an OS-level notification — the campaign lifecycle watcher in CampaignStreamProvider, and the upgrade handover in AdminPage — calls lib/browserNotify.post itself, beside its notify. That keeps the Notification API out of a React component, and out of a flag threaded through the queue that only some callers would set. post enforces its own preconditions, including that the tab is actually hidden, so a later caller cannot forget them.

Both callers have the same shape, which is what the sink is for: work that outlives the view that started it. A campaign runs on the service; a service roll keeps polling from a KeepAlive-mounted Admin page. In each case the thing that finishes has no reason to expect the user to still be looking at the page that started it.

The opt-in is asked once, in context. components/NotificationAsk.tsx is a banner above every view, raised the first time useCampaignStream shows a running campaign and browserNotify.shouldAsk() is still true — a permission ask needs a reason on screen, and arriving is not one. It is deliberately not a toast: a toast clears itself after ten seconds by contract, and an ask nobody was looking at would be spent for nothing. Either button stores an answer (setOptedIn), which is what makes the key in lib/browserNotify.ts three-state — absent means never asked, and a decline that stored nothing would be re-asked on every campaign. The standing switch lives in AdminPage’s This browser paper; it is kept out of ServiceConfigPanel on purpose, since that panel reports one shared service environment and this preference is per browser on an instance several people share.

The prompt itself can only be raised from a real click: Firefox and Safari resolve a gesture-less requestPermission() to default without ever showing it. That constraint is what shapes both surfaces — the banner exists to produce the click, and the Admin control is a switch rather than a setting read at start-up.

It is not a route to ntfy. Campaign lifecycle already reaches a phone from the server (robovast.execution.notify), once per campaign. Fanning out from the browser would send one push per open tab and would put the ntfy token in the client.

The queue itself (lib/toasts.ts) and the phase diff behind the lifecycle notices (lib/campaignEvents.ts) are pure and unit-tested; the provider around them is not. Note that remote panels (frontend/panel-kit/) share only react/react-dom with the host, so a federated panel cannot reach this context — one that needs it must be handed a builtin.

Frontend tests stay minimal. npm run test runs vitest over the pure modules in frontend/ui/src/lib/ — and nothing else. The checks that matter most here are still npm run lint (tsc --noEmit) and npm run build; a spec is the exception, not the habit.

A spec earns its place only where the logic is non-obvious and regression-prone and checking it by hand means clicking through the UI: node-id and route stability, grouping, rollups, formatting. src/lib/resultsTree.test.ts is the model — the Results tree’s node ids are a contract between two components that build them independently (the tree renders them, the Run view reconstructs one to highlight the current run), so a drift shows up only as “the picker stopped opening on the selected run”, which no compiler catches.

What not to do, because each of these costs far more than it protects here:

  • no component or DOM tests, no jsdom, no @testing-library, no snapshots of rendered output — the UI is a thin client of RobovastInterface, so behaviour belongs in the Python tests, which is where it is enforced;

  • no network mocking — if a test needs the service, it is testing the service;

  • nothing tsc already proves (a required field being present, a union being exhaustive);

  • no test framework stack beyond vitest’s defaults over the existing vite.config.ts (which is also where the @ alias comes from, so a spec needs no path config of its own).

Note that CI does not build frontend/ui at all, so these run locally and in agent sessions rather than as a gate — another reason to keep them few and fast. They are npm run test inside frontend/ui; the Python suite is make test at the repo root (or make test-<subtree>, e.g. make test-service, to re-run only what you are editing). CI calls make test too, so the command a developer runs and the one that gates a merge cannot drift apart.

The Python suite needs a Postgres, because campaign results live in one, and it provides its own rather than asking you for one: a container named robovast-test-pg (see tests/pg_provision), started on the first run that finds it absent and left running afterwards so later runs cost a CREATE DATABASE instead of a container start. Each session works in its own database inside it, so two suites can run at once, and drops it on the way out. export ROBOVAST_TEST_PG_DSN=... to use a server you already have instead, and docker rm -f robovast-test-pg to take the suite’s back. Without a Docker daemon and without that variable, the tests that need a database skip and say so in their reason.

One colour scheme. Every colour the UI paints that is not a one-off comes from frontend/ui/src/colors.ts. A Style object holds them; buildTheme(style) in src/theme.ts maps one onto MUI’s palette roles, so a component reaching for 'primary.main' or 'error.main' resolves there. The module keeps four groups apart deliberately, because which group a colour is in decides whether it may move when the brand does:

  • surface — the ground, a neutral dark grey, and deliberately not the marketing site’s near-black green. This is a tool that stays open for hours in front of long log tails, 3D scenes and charts, and on a tinted ground every one of those picks up the tint; a colourless ground leaves the accent and the status colours as the only hues on screen, which is what makes them read as meaning something;

  • brand — the accent (mint), transcribed from the site’s CSS custom properties rather than imported (it is a static page with no build step and no package to depend on, and a shared palette package would couple a deployed service’s bundle to a website). The favicon, frontend/ui/public/favicon.svg, is the same mark in the same mint;

  • status — phase chips, capacity meters, scenario-tree nodes, log severities. Semantics, not identity: they must stay put when the brand moves, or a re-brand silently re-labels every campaign. Held to one lightness band so no single status shouts and all of them sit clearly below the accent, with the hues kept meaningful (green good, amber attention, rose bad, blue working). The site has no equivalent, having no notion of a run that failed;

  • data — the categorical series scale, shared by the Vega eval charts, the run-view time series and the variation previews so a series keeps its colour between them. Kept in a brighter register than status on purpose: status is painted as filled chips and bars where a deep tone reads well, while a series is a 1px line that has to survive being thin. An encoding, so its order is load-bearing — append rather than reorder.

One thing is pointedly not in the status group: src/components/runLog/ansi.tsx maps ANSI SGR codes to colours, and those describe what the program printed. Harmonising them into the scheme would misreport a tool’s own output, so that table keeps its own values.

A second style is a new object, not a refactor. ACTIVE names the one style there is (midnightMint); implementing Style and passing it to buildTheme is the whole of adding another. What is deliberately not built is runtime switching: the flat tokens are module constants resolved at build time, which is what lets a canvas-painting panel read a colour without a hook. Switching while the app runs needs those panels to read from useTheme() instead, and every token a panel uses to live on the theme — that is the entire delta, and nothing in the current shape blocks it.

Import a token instead of writing a hex. Before this module the surfaces were spread across the theme and two components, the panel-canvas colour was copied into six files, the series scale existed twice — with a comment on one copy asking the other to please stay in sync — and the scenario tree and the log view each had their own private green and red.

Extending. Add an operation by giving robovastClient.ts a method mirroring the new interface op, then a page/tab that queries it.

Resource usage (``/usage``). resource_usage is a backend-agnostic interface op returning ResourceUsage (CPU cores + memory bytes, capacity vs. used, and a parallel_runs flag). How the service measures lives entirely in its _compute_resource_usage: ClusterService sums node allocatable (capacity, reusing kube_client.parse_resource) and the requests of the non-terminal pods bound to those same nodes (used) — so callers (the top-bar chip, the resource_usage MCP tool) never branch on backend. Summing both sides over one node set is what keeps used <= capacity: a pod still queued for a node requests cores nothing has granted, and counting it reported “29.7/24” on a 24-core cluster. Pending work is jobs_pending, not usage.

That scenario-run tally is the other half of the op, and it is counted from Jobs, not pods: running = executing, pending = accepted but not executing. A Job waiting its turn has no pod at all — the state every cluster batch starts in, because the admission queue has not created it yet — so a pod-based count reports a freshly launched sweep as 0/0 with its whole queue waiting for capacity. ClusterService._scenario_job_tally therefore delegates to cluster_execution.list_jobs_with_phase (the single place Jobs + pods become a phase, so no consumer re-derives it and drifts) and folds waiting/blocked into pending, namespace- scoped because it answers “what is this service running” while CPU/memory must stay cluster-wide. The UI’s sidebar meter reads running/(running+pending) and hides itself when both are zero.

The reading goes through one TTL-cached path on the base class (ServiceBase.resource_usage memoizes for _USAGE_CACHE_TTL under a lock), so many polling clients cost one sampling per window. The cluster read needs cluster-scoped RBAC (nodes are not namespaced): setup grants the service ServiceAccount a read-only ClusterRole over nodes/pods (service_deploy._service_rbac_manifests), so upgrading an already-deployed service to this version requires setting it up again (vast cluster cleanup then setup, or setup --force) to add the grant.

Config editor (frontend/ui/src/pages/config/) is where a .vast is authored: a Monaco editor (frontend/ui/src/lib/monaco.ts bundles the editor + YAML workers and, via monaco-yaml, drives completion/inline-validation from get_config_schema()). It edits a workspace’s .vast (workspace = the project, since a browser has no CWD), autosaves + debounce-validates through validate_project, and previews resolved configurations through preview_configurations. These four ops — validate_project / preview_configurations / get_config_schema / list_variation_types — live on RobovastInterface, so every client reaches them; ServiceBase wraps validate_project_file / generate_scenario_variations / ConfigV1.model_json_schema() / the robovast.variation_types entry points, and validate_project swallows unexpected errors into a structured problem so the editor’s live validation never 500s on in-progress YAML.

Per-variation previews. preview_configurations returns, per configuration, a previews list ({variation_type, params, remote}) read from the config’s declared variations. The editor (frontend/ui/src/preview/) renders built-in variation types host-native — builtins.tsx maps a variation-type name to a Plotly component (distribution curves for ParameterVariationDistribution*, a value list for ParameterVariationList). An external variation plugin may instead ship a web preview: it sets Variation.WEB_PREVIEW to a package-relative dir holding a built Module Federation remote (remoteEntry.js exposing a ./preview React component); the service serves it at GET /variation_types/{name}/assets/{path} and preview_configurations fills in the remote descriptor, which RemotePreview.tsx loads at runtime (sharing the host’s React singletons). Built-ins ship no assets; a type with neither shows just the resolved parameters.

Results viewer (frontend/ui/src/pages/results/): pick a campaign, browse its results schema, run read-only SQL, and chart the result with Vega-Lite (frontend/ui/src/preview/VegaLiteChart.tsx — rows bound in as data.values). The two data-query ops — describe_campaign_data / query_campaign_data_sql — were promoted onto RobovastInterface; the actual SQL lives in one shared, directory-based helper, robovast.results_processing.data_query (mode=ro + a sqlite3 authorizer, campaign.db attached as schema campaign). Both callers reuse it: the service methods resolve the campaign dir the same way (campaign_dir under the results root), and the MCP run_data plugin resolves it via results_resolver or delegates to a configured service — so CLI, MCP, and the web UI query results identically, local or cluster. User-declared plots (visualization.results.data_browser.plots in the .vast, robovast.common.config.PlotSpec) are surfaced by list_campaign_plots and rendered by the same Vega-Lite component.

Run view panel framework (frontend/ui/src/lib/dashboard/) — the Run view (user guide: Web UI) is a small plugin framework with three deliberate seams, all designed so a future live view (watching a running system instead of replaying a rosbag) slots in without touching any panel:

  • Panels are plugins with two delivery mechanisms behind one contract. A PanelPlugin = manifest (type name + layout defaults) + React component implementing PanelProps ({spec, clock, data}). Core built-in panels self-register via registerPanel (registry.ts) by importing frontend/ui/src/panels/run_view/index.ts. Remote panels — package-provided or user-authored — are loaded at runtime as Module-Federation remotes (see below) and never touch the static registry. The .vast’s visualization.results.run_view.panels specs are normalized by parsePanels.ts (single-key shorthand → {type, ...fields}; the same shorthand is accepted by the Pydantic schema, see PanelConfig._flatten_shorthand). Valid type values are the core built-ins (BUILTIN_PANEL_TYPES) ∪ installed robovast.panel_types entry-point names ∪ custom — validated in PanelConfig._known_type in robovast.common.config (so a package panel is valid only when its plugin is installed). Adding a core built-in is still one file + one BUILTIN_PANEL_TYPES entry.

    Two surfaces, one framework. The Config tab’s column (The config view) uses the same registry factory, the same spec parsing and the same panel frame; what differs is the layout grammar (one column, so a panel declares only a height — ColumnHost.tsx, not PanelHost.tsx) and the props (ConfigPanelProps: a resolved configuration, and no clock). A config panel lives in frontend/ui/src/panels/config/, registers with registerConfigPanel, and its type joins BUILTIN_CONFIG_PANEL_TYPES. Package-provided config panels use the same robovast.panel_types entry-point group and the same asset route; the panel class’s SURFACE ("run" by default, "config" otherwise) is what tells them apart, so naming a run panel in visualization.config.panels is refused – naming the surface the panel does belong to and the block to declare it under, rather than reporting it as unknown. A config panel must set SURFACE: the default is "run", so a class that forgets it registers on the wrong surface and its own campaigns then read as “unknown config-view panel type”.

    A panel type may also declare what a .vast may say to it, as CONFIG_CLASS – the same attribute a variation type uses, so get_plugin_details describes a panel’s fields without knowing anything about panels, and the editor completes them from the served schema. Fields take a Binding: a literal, {param: ...} (a scenario parameter), {internal: ...} (a key a variation left on the configuration) or {role: ...} (a contributed file), resolved by one function in panel-kit (resolveBinding) so no panel parses its own bindings. Declaring the model is opt-in: a type that declares none keeps the free-form extra rule, which is what lets the run view’s vega_lite/layers/series bindings stay free-form. A type that declares one gets its keys checked, and a misspelled binding becomes an error naming the valid fields instead of a panel that silently draws nothing.

    A config panel that renders a contributed file may also draw it without a browser, for the draw_config MCP tool: it declares FILE_ROLE (the contribution.files role it reads) and plot(ax, path, projection, read), which draws the file onto a matplotlib Axes. read(path) returns a campaign file’s bytes by its campaign-relative path, so a format that names sibling files (a map YAML and its image) fetches them the same way. plot returns False for a projection it does not draw and raises when the file cannot be drawn; it imports matplotlib inside the function, so the panel module still imports without it. The package that renders a format in the browser is then the one that renders it here, and core never learns the format. Reference: Map2DPanelType in robovast_nav/panels.py.

  • Data comes only through DataProvider (dataProvider.ts): rows by table+time, nearest-sample lookups, a generic run-scoped fetchRun(endpoint, params) (GET a campaign endpoint with config_name``+``run_id applied — how a panel reaches a specialized endpoint without the generic seam knowing about it, e.g. the costmap panel’s nav grids), and runFileUrl for per-run artifact files — today implemented over the read-only data-query endpoints (dbDataProvider); a live implementation would wrap a rosbridge buffer. timeSeries.ts wraps one table as a time-indexed TimeSeriesSource (at(t)/upTo(t)).

  • Time comes only from the shared PlaybackClock (clock.ts), an external store (not React state) so display-rate updates don’t re-render the tree; canvas panels subscribe imperatively via useCanvasClock.

Package-provided & user-authored panels (Module Federation). A remote panel is loaded at runtime by exactly the seam the variation-preview path uses (above) — the two now share their machinery. Server side, _resolve_plugin_asset(group, name, rel, asset_attr) (in service/app.py) and _plugin_remotes(group, asset_attr, url_builder, module_attr) (in service/client.py) are the generalized forms of the old variation-only helpers; _variation_remotes and _panel_remotes are thin wrappers. A package panel is a class in the robovast.panel_types entry-point group declaring WEB_PANEL (its built bundle dir, shipped as package data) + PANEL_MODULE; the service serves it at GET /panel_types/{name}/assets/{path} and list_campaign_panels attaches its remote descriptor. A user-authored custom panel names a bundle path relative to the .vast; _collect_analysis_input_files stages the bundle into the campaign’s _config/ and it is served at GET /campaigns/{id}/panel_assets/{path} (resolve_campaign_panel_asset, path-confined). Both descriptors are {name, remote_entry_url, module}; on the client, useRemoteComponent (frontend/ui/src/lib/remote.ts, factored out of RemotePreview.tsx) loads the module — seeding the host’s React singletons — and PanelHost mounts it with the full PanelProps. Reference example: the costmap panel, relocated out of the core UI into robovast_nav — Python descriptor in robovast_nav/panels.py, build sources in src/robovast_nav/web (a @module-federation/vite remote exposing ./costmap, built into robovast_nav/web/dist). It is the first real remote, and validates the variation-preview loading path too. Its data endpoint is likewise package-provided — see Package-provided service data endpoints below — so both halves of the costmap panel live in robovast_nav.

Write a remote against frontend/panel-kit (@robovast/panel-kit), never against a copy of the host’s types. Sharing only react/react-dom at runtime is a runtime constraint, not a source one: the contract (PanelProps, DataProvider, PlaybackClock) and the clock-driven scaffolding (useCanvasClock, the time-index binary search, keyframes for samples too large to preload) are in-tree source that both the host and each remote compile into their own bundle. Resolve it with a tsconfig paths entry plus a matching vite resolve.alias — see src/robovast_nav/web for the two lines. Do not add it to the MF shared map: bundling a private copy is what keeps a version skew between an installed package and a newer host UI an ordinary build rather than a remote-load failure. The earlier arrangement — a hand-maintained contract.ts mirror plus a re-implementation of the host’s canvas/clock scaffolding — is what let a fetch-staleness bug exist in the costmap panel and nowhere else.

A panel type may also declare an optional REMOTE_NAME — the Module-Federation container name, defaulting to the entry-point name (one container per type). Panels that share a single built bundle set the same REMOTE_NAME so _plugin_remotes emits that shared name while each type keeps its own asset URL (/panel_types/<name>/assets/... all resolve to the one bundle). robovast_nav uses this to host its panels in one robovast_nav container (see robovast_nav/web/vite.config.ts), sharing React/vendor chunks.

Nav2 behavior tree (when a package should ship no panel at all). nav2’s internal behavior tree is shown in the run view, and robovast_nav contributes none of the rendering. nav2’s /behavior_tree_log is the only generic, always-on source of BT state, but it is topology-free — a flat stream of status transitions keyed by node_name. The data half splits across the two extension seams the same way costmap’s does: the core rosbag handler nav2_bt_to_csv (rosbags_process.py; core because HANDLER_REGISTRY isn’t plugin-extensible and it needs the in-container rosbags_process step) writes the raw nav2_behavior_tree transitions table, and robovast_nav’s nav2_bt_tree postprocessing plugin (a robovast.postprocessing_commands entry point) reconstructs structure from the BT XML nav2 ran and joins it into a nav2_behaviors table.

That table uses the same schema as scenario_execution’s behaviors table, and that is the whole point: the built-in scenario_tree panel renders any tree expressible in it. So robovast_nav ships the nav2_behavior_tree type but no renderer — its panel module is a handful of lines that render the built-in one with nav2’s table, title and empty-state hint.

Deriving a panel is what makes that possible. PanelProps.builtins carries the host’s own panel components to a remote (PanelHost’s RemotePanel reads them from the registry at mount), because a Module-Federation remote shares only react/react-dom and cannot import the host’s modules. A derived panel inherits every later improvement to the built-in — child ordering, node-kind glyphs, feedback, source lines, scrolling — instead of needing them implemented twice.

The package did once ship a full port of the built-in panel in plain React; it was deleted for exactly that reason. Deleting the type along with it was a mistake worth recording: a type name is not a duplicate. Configs name it (frozen _config copies of past campaigns included), and PanelConfig._known_type validates against installed entry points, so removing the entry point turned every one of those configs into a validation error and an “Unknown panel type” box.

The rule, then: a package needs a renderer only when the host cannot draw its data at all. costmap qualifies — binary grids need their own service endpoint. A package whose data is a table in an existing schema still gets a type of its own, but derives the panel.

Package-provided service data endpoints (robovast.service_endpoints, robovast.service.endpoint_plugin) — an installed package contributes a run-scoped data endpoint served at GET /campaigns/{id}/<name>?config_name=…&run_id=…&… → JSON, with no core edit and no frontend change (the run view already reaches any such endpoint via data.fetchRun(name, params), frontend/ui/src/lib/dashboard/dataProvider.ts). This closes the last core-coupling for a self-contained analysis package: it ships a postprocessing plugin (writes an index table), a service endpoint (serves it), and a panel (renders it) — all via entry points. The mechanism mirrors the MCP-plugin loader: a ServiceEndpoint Protocol (name + handle(ctx)) and load_service_endpoints(); build_app registers one route per plugin (before the SPA mount) and dispatches to handle with a ``RunDataContext`` facade — ctx.open_db() (a read-only connection to the central index, from the public data_query.open_data_db), ctx.run_dir(config, run), ctx.params. Handlers raise KeyError/ValueError/ DataQueryError → 404/400 via the shared _guard. Cluster-transparent because dispatch resolves the campaign dir through the public impl.campaign_dir(campaign_id) seam, which is a directory under the results root — so a plugin endpoint works on every deployment unchanged. Endpoint names should be package-namespaced (nav/foo) to avoid collisions; core route names are reserved (RESERVED_CAMPAIGN_ENDPOINTS). Scope: run-scoped GET→JSON only — binary/large per-run artifacts are already served by the file address space, GET /results/<campaign>/<config>/<run>/<path> (DataProvider.runFileUrl), and producing data is a postprocessing plugin’s job. Reference: robovast_nav’s CostmapEndpoint (robovast_nav/service_endpoints.py), relocated verbatim from core’s old read_costmap_frame — it reads the costmaps table via ctx.open_db() and returns the frame dict the costmap panel decodes.

3D scene viewer core (frontend/ui/src/lib/scene3d/) — renders the browser scene descriptor (scene.json/scene.bin), which a simulator backend’s exporter produces; the format is RoboVAST’s, see Replay artifacts: the scene and the capture. sceneLoader.ts builds a three.js Group and returns an imperative animation API (jointMap / basePose); viewport.ts is a plain-three viewport (renderer/camera/lights/grid/orbit controls + the Z-up wrapper). The wheel is the viewport’s own, not the orbit controller’s (cursorDolly.ts): scaling the orbit radius toward a fixed pivot makes the travel a geometric series converging on that pivot, so a notch moves millimeters once you are close and the pivot can never be passed through — flying the eye and the pivot along the cursor ray keeps the radius, and with it the step size, constant. The same change makes a fixed far plane visible, so viewport.ts sizes the frustum each frame to enclose the world’s bounding sphere — measured from the scene, not from the pivot, which the wheel now carries along and which is therefore constant by design. Extractability rule: files in this directory import only ``three`` — never ``@/…`` (see its README) — it is shared-candidate code, so all robovast-specific wiring lives in the consumer, frontend/ui/src/panels/Scene3DPanel.tsx, which binds the vast spec, fetches the descriptor via DataProvider.runFileUrl (GET /results/<campaign>/<config>/<run>/<path> — the address is the run’s real directory, so the loader’s relative sibling fetches, scene.bin/textures, stay in it), and drives basePose from the clock.

Deferred: a bundle code-split (Monaco + Plotly + Vega + Module Federation make the SPA large).

Analysis postprocessing: local vs cluster

The rows query_campaign_data_sql reads are produced by analysis postprocessing. It splits along a natural seam: only the batched ``rosbags_*`` → CSV step needs ROS2; everything after it (the index ingest, metadata) is plain Python.

What the index ingest reads. One table per data file, discovered per run directory: *.csv, and *.jsonl for producers that need more than a flat table can express. A JSONL file declares its layout in the format key of its first record, and _JSONL_READERS maps that to the function turning its records into rows — dispatch on the file’s own declaration rather than on its name, so adding a producer does not mean hardcoding a filename in the ingest. Rows are then typed and inserted through exactly the same path as a CSV’s, which is why column types are inferred per column and a JSONL file whose records have differing keys still gets a column for each (the column list is the union over rows, not the first row’s keys).

The one such producer today is behaviors.jsonl, written by scenario_execution’s --bt-log, on for every run. Because the runner writes the file itself — rather than the table being converted from a recorded /scenario_execution/snapshots topic, which only works for ROS runs and would leave mode: base campaigns with no behaviour-tree data at all — both kinds of run produce the behaviors table by the same path, and the snapshots topic need not be in the bag. Its first seven columns are the ones nav2_behaviors (see above) shares, so the same panel renders both; to them it adds what the JSONL carries: child_index, type, additional_detail, feedback_message, is_active, tip_id, osc_file, osc_line, osc_column and removed. The numeric status column is re-derived in the reader from the status name, because those 1–4 codes come from py_trees_ros_interfaces — plain py_trees Status values are strings.

Both backends run the rosbag conversion in the campaign’s own execution image — the system-under-test’s image, recorded in <campaign>/_execution/execution.yaml — because rosbags carry the SUT’s custom ROS2 message types and only deserialize there. On the cluster backend the image is pinned to the immutable digest the run pods used (captured from the pods and stored as image_revision; campaign_execution_image prefers it), so a re-postprocess months later deserializes against the exact image the runs recorded the bags with — not whatever a floating :latest resolves to then.

Local. run_postprocessing reads that image: and passes it to docker_exec.sh --image, which bind-mounts the campaign dir (-v $INPUT_DIR:/input) — so the container writes CSVs in place. Nothing to sync. The scripts are bind-mounted from the driver’s own package (-v $SCRIPT_DIR:/scripts:ro, $SCRIPT_DIR = robovast.results_processing.data), so the script version always matches the driver.

Cluster. A pod cannot bind-mount the caller’s filesystem, so all of postprocessing runs as one Job (robovast.execution.cluster_execution.postprocess_job), with one copy of the campaign in it. A campaign emptyDir mounted at /campaign in every container carries the tree, and Kubernetes alone orders the containers — initContainers run sequentially to completion in declaration order:

  • stage (initContainer, sidecar image) fetches the campaign as one tar stream from the service’s data plane — curl | tar of GET /data/campaigns/<id>/archive, narrowed by stage_include() — and lands it on that mount. Nothing of ours sits between the socket and the disk, so its memory is the pipe. Calibration probes and the log this Job will write are never staged, and where no conversion is configured neither are the rosbags — nothing else in the pod opens one.

  • convert (initContainer, the campaign’s execution image) runs every step that needs that image — the rosbags_* → CSV conversion and any other ExecutionImagePlugin — in order, each as its plugin’s own image_command, against the same mount. It touches a marker beside the campaign first; every file changed after it is this container’s output. It exists only where the campaign has such a step. It exits 0 whatever its steps returned and hands their status to host in a file beside the marker: a failed initContainer starts nothing after it, so a step that refused a few bags would otherwise take every converted one down with the pod. host then runs, delivers, and fails the Job with its own exit code 3, which the failure reason names as the conversion’s.

  • host (container, controller image, postprocess_host) runs everything else — the other plugins, the index ingest and metadata — and is what delivers: one tar of what the Job derived, PUT to the data plane, which writes it into the campaign’s directory on the results volume. It is the only container given the campaign’s token.

The conversion container holds no credentials of any kind — not the campaign’s data-plane token, not the index DSN: it is an arbitrary user image and reads and writes the shared mount only. Further:

  • image = the campaign’s execution image (never a default — a missing image: is an error, not a silent wrong-image conversion);

  • the conversion scripts, and the files each image step ships, are delivered as a per-campaign ConfigMap built from the driver’s own robovast.results_processing.data and mounted read-only at /scripts — the K8s analog of the local -v $SCRIPT_DIR:/scripts:ro bind-mount. This is deliberate: sourcing the scripts from the driver rather than from a separately-versioned controller image guarantees the in-cluster scripts match the driver that generates the conversion command, so an off-cluster/dev driver running ahead of a published image cannot skew them (a skew surfaces as a spurious --output-root error). The scripts are self-contained (stdlib + ROS2 libs, no robovast import) and small; nothing is ever baked into the user’s image;

  • each image step runs over the shared campaign tree itself, so every output lands at its campaign-relative path and goes to its canonical key with no mapping step.

The host stage is pure Python and reuses the normal pipeline with the image steps skipped (run_host_postprocessing), so there is no second copy of the postprocessing sequence — the same function vast results postprocess calls, run beside the data instead of fetching it.

Two entry points share that one implementation (postprocess_campaign):

  • auto-chain — the campaign driver, when create_campaign(postprocess=True) sets RunOptions.postprocess (an option, not a process-global env var, so concurrent campaigns in the one service process stay distinct). It runs after ``store.close()`` (campaign.db must be flushed — the index’s runs table is built from it) and before ``_finalize``, so the results ride the campaign’s existing upload rather than needing one of their own.

  • explicit re-run — run_postprocessing, the cluster’s body for the interface operation: the service has no ROS runtime, so it submits the Job, whose pod is what holds the campaign, and materialises only the few small status objects it edits and publishes back — before and again after the pod, since the pod is what wrote the provenance the reconstruction reads. This backs the web Retrigger postprocessing dialog, the MCP run_postprocessing tool, and the CLI. It reads the campaign’s own _config/<name>.vast (which the edit dialog overwrites in place — the single source of truth resolved by common.results_utils.campaign_vast), refreshes the durable outcome (clearing/setting postprocessing_error), and is dispatched in the background via ServiceBase._dispatch_background, which registers a tracked campaign entry set to the postprocessing phase (busy-guarded against a second concurrent op) and returns at once, so the campaign view shows it live. A minutes-to-hours re-run therefore never blocks the caller.

The upload-to-share step mirrors this: a failure records share_error (durable) instead of being swallowed, and run_share() re-triggers it (web Retrigger upload-to-share, MCP run_share, POST /campaigns/{id}/share/run) — also via _dispatch_background (sharing phase). Both re-triggers need no live in-memory campaign entry, so they work after a service restart.

A post-run step failure is deliberately not a campaign failure: the phase stays finished and the reason lives on postprocessing_error / share_error. After a restart the cluster service reconstructs a campaign’s status from the _execution/outcome.json in its directory, falling back to the on-disk run artifacts (reconstruct_status_from_disk) when no outcome was recorded — so a finished campaign reads as finished, never a bare unknown.

Querying RoboVAST campaigns

Using [rdflib](https://rdflib.readthedocs.io/), you can query the generated metadata graph using [SPARQL](https://www.w3.org/TR/sparql11-query/).

Load the metadata graph

from rdflib import Graph

g = Graph()
g.parse("metadata.prov.json)

Loading SPARQL queries

To do so, you can load any of the queries below as text, and use the query method for any graph g:

with open("query-file.rq", "r") as f:
  query_string = f.read()

qres = g.query(query_string)
for row in qres:
  # Process your results
  print(row)

Below are a few example queries demonstrating the PROV relationships in the metadata graph.

Scenario inputs

FloorPlan models:

 SELECT ?floorplan ?creator ?date
 WHERE {
      ?floorplan rdf:type env:FloorPlanModel .
      OPTIONAL {?floorplan prov:wasAttributedTo ?creator .}
      OPTIONAL {?floorplan dcterms:modified ?date .}
}

Vast file:

SELECT ?vast_file ?creator ?date ?abstract_scenario
WHERE {
    ?vast_file rdf:type robovast:VastConfiguration .
    ?vast_file dcterms:references ?abstract_scenario .
    ?abstract_scenario rdf:type scenarios:AbstractScenario .
    OPTIONAL {?vast_file prov:wasAttributedTo ?creator .}
    OPTIONAL {?vast_file dcterms:modified ?date .}
}

Generation

FloorPlan Model-to-Model Transformation:

SELECT ?floorplan ?activity ?jsonld_file ?agent
WHERE {
    ?floorplan rdf:type env:FloorPlanModel .
    ?activity rdf:type robovast:FloorPlanTransformation .
    ?activity prov:used ?floorplan .
    ?jsonld_file prov:wasGeneratedBy ?activity .
    OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
}

FloorPlan Artefact Generation:

SELECT ?source_files ?activity ?gen_file ?agent
WHERE {
    ?activity rdf:type robovast:FloorPlanGeneration .
    ?activity prov:used ?source_files .
    ?gen_file prov:wasGeneratedBy ?activity .
    OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
}

Generation of Concrete Scenario

SELECT ?vast_file ?ref_file ?activity ?agent ?gen_file
WHERE {
    ?vast_file rdf:type robovast:VastConfiguration .
    ?activity prov:used ?vast_file .
    ?vast_file dcterms:references ?ref_file .
    ?gen_file prov:wasGeneratedBy ?activity .
    OPTIONAL {?activity prov:wasAssociatedWith ?agent .}

Test Execution

Test results generated from a test run:

SELECT ?scenario ?config_files ?activity ?agent ?gen_file ?start_time ?end_time
WHERE {
    ?scenario rdf:type smm:ConcreteScenario .
    ?activity prov:used ?scenario .
    ?activity prov:used ?config_files .
    OPTIONAL{?activity prov:startedAtTime ?start_time .}
    OPTIONAL{ ?activity prov:endedAtTime ?end_time . }
    ?gen_file prov:wasGeneratedBy ?activity .
    OPTIONAL {?activity prov:wasAssociatedWith ?agent .}

Postprocessing

Postprocessing of a bagfile:

SELECT ?bag_file ?activity ?agent ?gen_file ?start_time ?end_time
WHERE {
    ?bag_file rdf:type robovast:ROSBag .
    ?activity prov:used ?bag_file .
    ?gen_file prov:wasGeneratedBy ?activity .
    OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
    OPTIONAL{?activity prov:startedAtTime ?start_time .}
    OPTIONAL{ ?activity prov:endedAtTime ?end_time . }
}

Metadata and Graph postprocessing:

SELECT ?metadata_file ?graph_file ?md_activity ?graph_activity ?agent ?start_time ?end_time
WHERE {
    ?md_activity rdf:type robovast:PostprocessingMetadata .
    ?graph_activity rdf:type robovast:PostprocessingGraph .
    ?metadata_file prov:wasGeneratedBy ?md_activity .
    ?graph_file prov:wasGeneratedBy ?graph_activity .
    ?graph_activity prov:used ?metadata_file
    OPTIONAL {?md_activity prov:wasAssociatedWith ?agent .
    ?graph_activity prov:wasAssociatedWith ?agent .}
    OPTIONAL{?md_activity prov:startedAtTime ?start_time .}
    OPTIONAL{ ?md_activity prov:endedAtTime ?end_time . }
}

Analysis

Identifying which variation types were used on each config:

SELECT ?config ?variation_type
WHERE {
    ?config rdf:type smm:ConcreteScenario .
    ?config robovast:variations/rdf:rest*/rdf:first ?variation .
    ?variation rdf:type ?variation_type .
    FILTER (?variation_type != prov:Activity)
    FILTER (?variation_type != robovast:Variation)
}

Getting the failure rate by environment:

SELECT  ?env_model (SUM(?failures)/COUNT (?activity) * 100 AS ?result) (COUNT (?activity) AS ?total)
WHERE {
    ?conf rdf:type smm:ConcreteScenario .
    ?activity prov:used ?conf .
    ?activity rdf:type robovast:TestExecution .
    ?activity robovast:success ?success .
    BIND(IF(?success=true, 0, 1) AS ?failures) .
    ?conf dcterms:references ?env_model .
    ?env_model rdf:type env:FloorPlanModel .
} GROUP BY ?env_model