Developer Guide
Test your Robotic Software with RoboVAST
RoboVAST is designed to facilitate testing and validation of robotic software systems by generating diverse scenarios and executing them in simulation environments. This guide provides an overview of how to utilize RoboVAST for testing your robotic applications.
1. Containerize your Software
As RoboVAST relies on containerization to ensure consistent and reproducible environments, the first step is to create a Docker container for your robotic software.
There are some requirements your container image must fulfill to be compatible with RoboVAST: - the image must contain scenario-execution package installed in /ws/install (which currently is available for ROS2 jazzy) - the image must be accessible by Kubernetes, e.g. by pushing it to a container registry.
2. Define a Test Scenario
Use the examples and the documentation of scenario-execution to create a scenario that tests your robotic software.
Keep in mind, that variations are currently supported for all overwritable scenario parameters as described here.
To test your scenario locally, you can run:
ros2 run scenario_execution_ros scenario_execution_ros <scenario-file> -t -d
3. Create Initial RoboVAST Configuration
Create a RoboVAST configuration file, based on the existing examples in the configs/ directory. Do not set any configuration, as this will be done in the next step.
Push the directory to a service — the one-node deployment on your own machine (On this machine) is the development loop — and run one configuration through it. This is the same path every campaign takes, differing only in which service answers.
vast workspace init . # upload the project
vast workspace validate <workspace> my.vast # check it
vast workspace run <workspace> my.vast --filter config1 --runs 1
vast workspace list names the workspace; vast workspace update . <workspace> pushes
an edit. Before a campaign, vast container run (or the MCP exec_in_container) runs
a command or one configuration’s scenario in the image, which is where an import error or
a missing package shows up in seconds rather than after a pull.
Next, it is important to verify that the output (e.g. ROS bag) is stored correctly.
vast workspace run <workspace> my.vast --filter config1
vast campaign wait <campaign-id>
vast campaign download <campaign-id> # <campaign-id>/<config-name>/<run_number>/
Once you are satisfied that the scenario and configuration work as expected, you can proceed to the next step.
4. Define Configurations
Define configurations in your .vast file.
A good procedure is to add configurations one-by-one and analyze the result.
# 1. add configuration in config file
# 2. list created configurations
vast config list my.vast
# 3. try local execution with one of the created configurations
vast workspace run <pinned-workspace> my.vast --filter <config-name> --runs 1
5. Execute in Cluster
Once you have defined your configurations and verified local execution, you can run the tests in a Kubernetes cluster.
A good practice is, to first run a single configuration to verify that everything works as expected.
# 1. run single configuration in cluster, once
vast workspace run <workspace> my.vast --filter config1 --runs 1
# 2. fetch the campaign's archive (or publish it to the share for someone else)
vast campaign download <campaign-id> # -> ./<campaign-id>.tar.gz
vast share export -i <campaign-id> # -> the configured share
# Inside the archive: <campaign-name>-<timestamp>/<config-name>/<run_number>/
vast workspace run is fire-and-forget: it starts the campaign on the
robovast-service, which drives it in-process, and returns immediately. The
campaign runs in the background in the cluster:
# Monitor job status (shows progress per run when multiple runs are active)
vast cluster monitor
# Clean up after jobs complete (all campaigns, or use --campaign for a specific campaign)
vast cluster jobs-cleanup
Campaigns run in parallel and a new one never removes an older one’s Jobs. Clear
them when you are done with vast cluster jobs-cleanup (add --campaign for
one campaign). It touches Jobs and pods only; the campaign’s results live with the
service, and vast campaign delete is what removes a campaign.
Running local container images in minikube
To test local container images in a minikube cluster, you can load the image into minikube’s Docker environment.
# first terminal
docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000"
# second terminal -- publish the whole family into the local registry
make release-images PROJECT=localhost:5000 PUSH=1
# third: point runs at it (a .vast names no RoboVAST image; see docs/images.rst)
export ROBOVAST_PROJECT=localhost:5000
6. Analysis
RoboVAST provides a GUI for analyzing run results, which is based on user-provided Jupyter notebooks.
To develop the notebooks, it is recommended to use e.g. VSCode. For the RoboVAST GUI to work, it is expected to contain a DATA_DIR definition. The RoboVAST GUI will replace this line with the actual path to the results directory. During development you can set this variable manually to point to your results directory.
# for single-run (specific run of a configuration)
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>/<config-name>/<run_number>'
# for configuration (all configurations)
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>/<config-name>'
# for complete run
DATA_DIR = '<path-to-your-results-directory>/<campaign-name>-<timestamp>'
In case you are using ROS bags as output format, it is recommended to postprocess the results before analysis. This can be done with the postprocessing commands defined in the configuration file. RoboVAST provides several conversion scripts for common use-cases.
Postprocessing is cached based on the results directory hash. To bypass the cache and force postprocessing (e.g., after updating postprocessing scripts), use the --force or -f flag:
Afterwards, read the results in the browser (vast ui, against a running service):
vast campaign postprocess <campaign-id>
# or, to force postprocessing even if results are unchanged:
vast campaign postprocess <campaign-id> --force
Note
The Results views discover campaigns exclusively from a per-campaign ``campaign.db`` store — they do not walk the results filesystem. Search campaigns write this store live; a batch campaign’s is written by the controller that ran it.
A results tree with no store is therefore invisible to the UI, and nothing rebuilds one. That only affects a tree produced outside a controller — a hand-copied or pre-store campaign.
The store also carries the campaign mode (batch/search), so the Explorer
renders the search batch level and resolves the
visualization.results.explorer.notebooks notebooks from the recorded config_dir. See
Campaign Store and Results Indexing for the schema and internals.
Working across the distributions
RoboVAST is four packages in one checkout (see Four distributions, layered by audience), which changes two things about the development loop. Both have bitten; both are silent.
Install the client last. robovast-client is a non-optional path dependency of
robovast, so pip install -e . resolves it and installs a plain copy into
site-packages — silently replacing an editable install done earlier. Editing
src/robovast_client/ then has no effect, and nothing says so. make venv installs it
after everything that depends on it for exactly this reason; if you install by hand, do
the same, and check with:
python -c "import robovast.client.cli as m; print(m.__file__)"
A path under src/robovast_client/ is editable; one under site-packages is not.
Entry points live in installed metadata, not in ``pyproject.toml``. Adding, moving or removing one does nothing until the owning distribution is reinstalled, and an editable checkout looks entirely normal in the meantime. The failure mode depends on which group:
robovast.cli_pluginsdegrades loudly —load_plugins()printsWarning: Failed to load plugin '<name>'and carries on.robovast.cli_startuprefuses rather than running on, naming the reinstall. Degrading silently there costs real damage: with the core installed but its.envhook unregistered, no./.envis read, andvast service upgrade— which reconciles Secrets from the environment — concludes the registry and git credentials are gone and deletes both.
The rule that follows: after touching any [tool.poetry.plugins."..."] block, reinstall
before you conclude anything from a test run. make venv re-runs when a manifest or
the Makefile changes, so it is the safe way to do it.
A distribution’s own verbs are not entry points. An entry point is for crossing a
distribution boundary; a verb defined in the same module as its group stays an ordinary
@group.command(). Declaring your own in installed metadata means a pyproject edit
without a reinstall makes the verb vanish, silently. Moving a verb between distributions
carries the mirror hazard — the old declaration is live until every distribution is
reinstalled, and two providers of one name resolve unpredictably;
tests/test_editable_installs.py asserts exactly one provider per name. When vast
reports has no attribute after a pull, this is it, and make venv is the fix.
A client install must stay a working install. robovast-client ships without the
core, and every leak found so far has been a deferred import of it — the module imports
perfectly and the command dies at call time, in exactly the install the distribution
advertises. An import check cannot see that;
tests/service/test_client_needs_no_core.py drives the commands with the core made
not importable. Anything the client needs must live in the client: a wire constant like
COMMAND_LIMIT_S belongs in interface.py, not in the server module that enforces it.
Releasing to PyPI
The five distributions are released as one set at one version, because robovast
requires robovast-client and robovast-sim-roqsim at exactly the version being
released, and robovast-nav and robovast-cluster require robovast. A version
that exists for some of them and not the others is a set nobody can install, so
.github/workflows/publish.yml stamps the tag’s version into every manifest before it
builds a wheel. The tree carries the version being released too, set in all five
pyproject.toml files in the release commit, because the images are built from the tree
and report the version it names.
The version’s section of ``CHANGELOG.md`` comes first, written with the changelog
skill (skills/changelog/SKILL.md) from the merges since the previous tag and reviewed
before it is committed: a few short entries, never more than fifteen, on what a user of the
previous version would want to hear about, each citing its pull requests as links.
skills/changelog/changelog.py check holds it to that shape. That section is the release’s
notes; the rest is in the git history.
Every release goes to TestPyPI first, and is tested by hand there. An upload cannot be
taken back — a version can be yanked, never replaced, and pip still installs a yanked
wheel when pinned exactly — so the only cheap place to find a packaging mistake is an index
nothing depends on. The two triggers of the workflow are the two halves of that rule:
workflow_dispatch→ TestPyPIgh workflow run publish.yml -f version=2.1.0rc1publishes the whole set as that version to TestPyPI. Pre-release numbers (rc1,rc2, …) are the convention, sincepipdoes not pick them up unless pinned. Install it in a fresh venv and try it:pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ \ "robovast[nav,roqsim]==2.1.0rc1" "robovast-cluster==2.1.0rc1" vast cluster setup minikube # the web UI must come up, not "API only"
Something wrong is a fix on
mainandrc2; nothing has been consumed.- a
v*tag → PyPI git tag v2.1.0 && git push origin v2.1.0, on the commit the release candidate was built from. The same workflow, the same wheels, the real index. The tag also produces the versioned container images (image.yml), so the wheels and the images a deployment runs carry the same number.
What the workflow does, and why in that order. One build job makes and checks all
five wheels before anything is uploaded: it verifies the lock against the manifest, builds
both frontend bundles (make frontend) and stages them (make ui-stage), stamps the
version, rewrites the root’s path dependencies to ==<version>
(tools/pin_released_siblings.py — a path dependency reaches the metadata as a direct
reference, which the index refuses), and then opens every wheel to confirm it carries no
such reference and does carry its package data (robovast/_ui/index.html,
robovast_nav/web/dist/remoteEntry.js). Everything that can fail for a reason inside
this repository fails there, at no cost. The publish jobs then upload the artifacts in
dependency order — client and sim-roqsim, then robovast, then nav and cluster — with
skip-existing so that a re-run of the same tag after a transient failure continues past
the wheels already on the index. A last job installs the set from the index it went to and
checks the same things in what was installed, so a break that reached the index is named
by the workflow rather than by a user.
Authentication is trusted publishing. Each index holds one publisher per distribution,
naming this repository, publish.yml and the pypi or testpypi environment; the
jobs mint a short-lived OIDC token and no API token exists anywhere. A distribution not yet
on an index is created by its first upload through a pending publisher, registered the
same way.
Rehearsing without CI. make publish-test uploads the set to TestPyPI from your
checkout, stamped with a post-release of the tree’s version that is free on all five
histories at once (tools/next_testpypi_version.py), and make publish-test-venv
installs it into a fresh venv and checks the bundles, the entry points and the CLI. Every
manifest is restored afterwards, including on failure. DRY_RUN=1 builds without
uploading.
Container Image Compatibility Version
RoboVAST enforces a compatibility version between the host Python code and the Docker container image. This prevents cryptic runtime failures when the two sides are out of sync (e.g. after updating one without the other).
How it works
The host declares a window of protocol versions it can drive:
MIN_IMAGE_COMPAT .. COMPAT_VERSION, both in
src/robovast/common/execution.py. COMPAT_VERSION is baked into the
container images as the label org.robovast.compat-version — one marker, and
the only one.
A window rather than a single value, because an exact-match comparison orphans every image already published on the first bump: a campaign whose results pin an image by digest could never be re-run again — even with those exact bytes still in the registry. Bumping the maximum is harmless; dropping support is a separate, deliberate act of raising the minimum.
A label, read from the registry’s config blob without pulling a layer or starting a container, so it answers for an image that exists only in the registry.
Cluster execution: the submitter checks, host-side, immediately after the campaign’s refs are pinned to digests and before any manifest is written — so the verdict is about the exact bytes the pods will run, and a refusal creates no pods. It is deliberately not asked of the image itself: a workload inspecting its own image is not how admission is decided anywhere else, and could only report a mismatch by failing one init container per job in the batch.
Retrigger pre-flight: the service reads the label of each recorded image from its registry; an image the registry will not answer for is reported as unknown.
Postprocessing:
docker_exec.shchecks beforedocker run.
The cluster check fails closed: if the registry will not say what protocol the
image speaks, the campaign is refused rather than started. Pinning, right beside
it, does the opposite and is right to — pinning is an optimisation, and not applying
it leaves exactly what would have run anyway, whereas a compat check that could not
read the image has established nothing. Set
ROBOVAST_SKIP_IMAGE_COMPAT_CHECK=1 to run anyway; that is the right move only
when the registry is unreachable and you know the image.
Only the campaign’s own image is checked. The sidecar sets no label and a system-under-test image is not a robovast image at all, so an absent label on those means “not applicable” rather than “unreadable” — failing closed on them would refuse every campaign that has a SUT.
An image built before the label existed now reports nothing and is refused. Those
images predate protocol 2, which is also MIN_IMAGE_COMPAT, so a refusal is the
right answer for them regardless — but the message says what to do about it
(rebuild from the recorded revision, or re-tag with the label) rather than only
that it could not tell.
Outside the window, execution fails immediately, and the message differs by
direction because the fixes are not interchangeable: an image older than the
window means “check out the revision the campaign recorded
(_execution/execution.yaml) and run it there”, while one newer means
“upgrade robovast”. Neither ever advises pulling a newer image — a re-run needs the
bytes the campaign recorded, not today’s.
Warning
The window is a claim. Raise MIN_IMAGE_COMPAT when support is genuinely
dropped, or it replaces a safe refusal with a broken run.
configs/examples/camera_smoke is the cheap way to keep the claim true.
When to bump the version
Bump COMPAT_VERSION when the contract between host scripts and the
container changes:
A new Python or system package is required inside the container
The ROS distribution changes
The interface of mounted scripts changes (e.g.
ros2_exec.sh,entrypoint.sh)A postprocessing script requires a new ROS package
How to bump the version
Increment
COMPAT_VERSIONinsrc/robovast/common/execution.pyRebuild and push the container images
That is the whole procedure, because there is only one copy. The images take the
value as a build arg that container/image_stamp.sh derives from that constant
— the same way they take the git revision and the build date, and for the same
reason: a constant copied by hand into each Dockerfile drifts, which is why it
used to need a CI gate comparing the copies. image_stamp.sh refuses to build
if it cannot read the constant, rather than producing an image that carries no
protocol at all.
image.yml still guards the shape: each Dockerfile must declare
ARG ROBOVAST_COMPAT_VERSION and label from it, so a literal cannot creep back.
Remembering to bump is the part nothing could check — the version is a claim,
and it fails by being forgotten. make check-compat-version (run in CI on every
pull request, beside the config-version check) compares the change against the
files that define the contract — the two Dockerfiles, entrypoint.sh,
secondary_entrypoint.sh, ros2_exec.sh — and fails if one of them moved and
the version did not.
It is a prompt, not a proof: it cannot tell a contract change from a comment edit in the same file, and it cannot see a change made somewhere it is not looking. When nothing the host relies on actually moved, say so in the commit message and it passes:
Compat-Unchanged: moved a comment; nothing the container must provide changed
A trailer rather than a flag, so the answer lands in history next to the change it is about. If it turns out to be used on most changes, the check is crying wolf and the path list is too broad — that ratio is the signal for whether it earns itself.
Extending RoboVAST
Add an interface operation
Every remote operation is a method on
robovast.service.interface.RobovastInterface, and each client is a thin caller of it
(One interface, one implementation, many clients). Adding one means threading it through the whole set:
The interface — the abstract method, its
Routesentry and the request/response models. They live inrobovast-client, so a client install has them without the core.The service and the HTTP transport — in
robovast.service.service_base.ServiceBasewhen the body is correct for any implementation, otherwise an abstract hook there and a body inClusterServiceand in the suite’sNullService(One base, the implementation’s hooks); andHTTPTransportover the route. An implementation may decline an operation, and then it raisesUnsupportedOperationnaming the operation and itself, in its own class; never a default on the base, and never aValueErrorthat reads as bad input. The refusal crosses HTTP as a501and reaches every client as the one sentence (HTTP API, “Status codes”).The HTTP route in
robovast.service.app.The surfaces — the
vastCLI, the MCP tools, and, where the web UI shows the operation,frontend/ui/src/lib/robovastClient.ts(Web UI internals).
An operation implemented inside one client — an MCP tool calling a local function directly, say — is reachable from that client alone, and the CLI, the web UI and the HTTP API have no way to it. Putting it on the interface is what gives every client the same answer, over either transport.
Anything the executing pod needs travels the same way: a value the controller reads must be
passed through ClusterService (cluster_service.py into the pod’s environment), or
the operation is accepted by the service and does nothing in the pod. The hooks are what
keep that failure loud rather than silent: a hook ClusterService has not answered
refuses to construct, and a body on the base is one that is correct for any
implementation, so the only way a cluster path can be missing is for ClusterService
itself to leave it out.
Keep disk and database I/O off the event loop
The service answers every request, /healthz included, from one event loop. A route or MCP
tool declared def runs on a worker thread and may block; one declared async def runs
on the loop, so anything it does that waits on a disk or on the index stalls every other
request for as long as that takes. A disk that is filling makes every write slow
at once, and a stalled loop fails the liveness probe, so the pod is restarted for being short
of disk.
Declare a route or tool
asynconly when it has something toawait; otherwise make it a plaindef.Inside
asynccode, hand blocking calls to a worker withanyio.to_thread.run_sync, and write files throughanyio.open_file.Work that must not compete with the routes’ threads gets its own
anyio.CapacityLimiter, as the SSE pulls and the refusal records inrobovast.service.appdo.
Add Variation Plugin
Provide your custom variation type by creating a class that inherits from robovast.common.variation.Variation.
To your pyproject.toml, add an entry under [tool.poetry.plugins.”robovast.variation_types”] to register your variation type. The key is the name used in the RoboVAST configuration file, and the value is the import path to your variation class.
[tool.poetry.plugins."robovast.variation_types"]
"YourVariation" = "robovast_<yourplugin>.your_variation:YourVariation"
A variation can also be loaded from a local file relative to the .vast
without packaging it — reference it as <path>.py:<Class> wherever a variation
name is expected (in a configuration[].variations list or a search.variations
template). This is the same ./path.py:Class convention used by search
strategies, extractors and postprocessing plugins:
variations:
- variations/wind.py:WindFieldVariation:
wind_speed: 5.0
See configs/examples/quadrotor_landing/variations/wind.py for a runnable
example (a wind model that derives the simulator’s wind_strength), wired into
the quadrotor search vasts.
The referenced module travels with the campaign: it is archived into
<campaign>/_config/ alongside the .vast, so a retrigger — which re-composes
from that snapshot rather than replaying recorded configurations — can resolve it, and it
is content-hashed into the campaign’s config_identifier, because the source
that decides which configurations exist is part of what the experiment is. Editing it
therefore gives the campaign a new identity, exactly as editing a packaged variation’s
module does.
Important
A file-referenced plugin must be one self-contained module. It is imported by path,
with nothing added to sys.path, so it cannot import a sibling module in its own
directory. Nor can anything discover a data file it opens itself — that dependency
lives in the plugin’s code rather than in the config, so nothing archives it and a
retrigger finds it missing. If you need more than one file, package the plugin and
declare it under plugins:; if you need a data file, name it in
execution.run_files.
How a failure in your plugin is reported. A refusal your plugin means — a
VariationConfigError for parameters
outside what it accepts, a
VariationInfeasibleError for a draw no
arrangement realizes — is reported as its message alone, naming the plugin and the config
block. Any other exception escaping variation() is a bug, and is reported as
Variation failed. <Class>: <exception> followed by the tail of the traceback: the file,
line and source of the frame it was raised in. Either report is prefixed with the .vast
line the variation is written on and the config block that ran it
(campaign.vast:10: config 'cell1': …), so the entry to edit is named as well as the
code. That text is the same on the CLI, in the MCP tools (validate_project,
preview_configurations) and in the web UI’s config editor, so the line to fix is in
front of whoever is authoring the plugin.
Note
Packaging a variation plugin as its own distribution. If your variation
types live in a separate installable package (as robovast-nav does for
FloorplanVariation, PathVariation*, ObstacleVariation*), it must
be importable everywhere scenario variations get composed — not just where
scenarios run. Composition happens in the process that expands the
variations: cross-product: the vast CLI on your host for
vast workspace run; and the robovast-service for every service/cluster
campaign — the service drives the campaign in-process, so composition (which for
search runs once per generation) happens in the service, not in a separate pod.
There is one mechanism for all of these — the .vast’s plugins: list
(next section) — and it is uniform across the CLI and the MCP/service paths:
On your host (``vast`` CLI), a plugin you have already installed is detected and used as-is.
config_plugins.ensure_workspace_pluginschecks whether each declared distribution is importable; if you ranpip install/make venvyourself, nothing is re-fetched. Only the declared specs that are missing are installed — into a virtual environment under the project’s.robovast_plugins/directory (never your active venv).For a service/cluster campaign, the service installs each declared plugin into the workspace’s ``.robovast_plugins/`` and imports it off ``sys.path`` when it composes (
config_plugins.ensure_workspace_plugins). The install runs once, on the credentialed service, so a private-repo clone needs credentials only there; the driver then uses the installed plugin directly — there is no staging round-trip and no separate pod.
Dependencies come from each package’s own metadata into that same environment, so
the one real constraint is that your plugin must declare its dependencies
correctly (e.g. scenario_mt’s shapely and its fpm @ git+…). There
is no separate host-venv “injection” step and no wheel-shipping; a plugin
installed only in your host venv but not declared in plugins: will not
reach the service.
Do not declare a dependency on robovast itself. Your plugin is loaded into
robovast’s process, so the host always provides it; the declaration is redundant,
and it makes your package unnecessarily expensive to resolve anywhere the host is
not already present. validate_project reports it once the plugin is installed.
Declaring a plugin in the ``.vast`` (CLI, MCP, and ``robovast-service``).
Declare the plugin inside the ``.vast`` with a top-level plugins: list of
pip requirement specs. This is the single portable mechanism: it works for the
CLI (where you may instead just install the plugin yourself — it is detected) and
for an MCP/service campaign, where the LLM only authors inputs into a workspace
and .vast / .osc are the only file types it can write inline:
plugins:
- scenario_mt @ git+https://github.com/secorolab/metamorphic_testing@main
- some_published_plugin==1.2.3
- ./plugins/my_plugin-1.0.0-py3-none-any.whl # a wheel you uploaded
These are installed into the workspace. Before variation types are resolved
from entry points, config_plugins.ensure_workspace_plugins (called once at
the top of generate_scenario_variations — the sole convergence for local and
service composition) installs the declared specs (with dependencies) into a
venv under <workspace>/.robovast_plugins/ and puts that venv’s
site-packages on sys.path so the entry points resolve. A .installed
marker (a hash of the specs and of the environment they were resolved against)
makes it idempotent.
Why a venv rather than pip install --target. A plugin is resolved against
the host, not in isolation from it: inside a venv pip treats what the host already
provides as satisfied, and declines to uninstall anything living outside the
environment it targets. --target cannot do either — pip forces
--ignore-installed whenever it is given, so every dependency is re-materialized,
and a plugin depending on robovast got a second robovast installed beside it.
Since importlib.metadata deduplicates distributions by name and keeps the first
on sys.path, that copy’s entry points became the only ones the process could see.
The same mechanism, for the same reason, backs the experiment image build
(image_build._VENV_SETUP).
The key property: .robovast_plugins/ lives in the workspace the driver
composes from, so the plugin is imported straight off sys.path with no
staging step. The install runs where the workspace lives:
the
robovast-serviceruns it increate_campaign(and for validate/preview) — its environment is what reaches the source and holds any git credentials;on your host (
vastCLI), a declared plugin already installed in the active venv is detected and used as-is — install it yourself and it is not re-fetched. Only missing specs are installed, and always into the workspace venv (never your site-packages), so a run is non-invasive.
Sources. An index pin needs no source access. A git URL works when the
install environment can reach it — for a private repo in the service, provide
a GitHub token at vast cluster setup (below). A workspace-relative
wheel you create_uploaded is the fully offline path (no git, no
credentials) for a private or unpublished plugin. On a failed install the error
is raised synchronously from the service call (create/validate/preview), so an
MCP user sees an actionable message instead of an Unknown variation class
surfacing later on the campaign’s status.
Because the robovast-service is one long-lived process serving many
workspaces and Python cannot unload a module, a plugin already loaded (from another
workspace) cannot be swapped; ensure_workspace_plugins logs a warning in
that case (the already-loaded version wins until the service restarts).
Private-repo credentials (``vast cluster setup``). When a git+https
plugin points at a private repo, provide a token via ROBOVAST_GIT_TOKEN (or
GITHUB_TOKEN / GH_TOKEN) — either exported in the environment or, more
conveniently, set in the project’s .env file (every vast command loads
./.env before it runs). deploy_service
stores it in a robovast-git-credentials Secret and mounts it read-only as a
file into the service pod (/var/run/secrets/robovast-git/token). The token is
handled so it is not accessible to any workspace or command:
it is never put in an environment variable (which every child process / command would inherit), never written to
~/.gitconfig, and never placed on a command line (ps-visible);it is supplied to
gitonly for the onepip installsubprocess, via a throwawayGIT_ASKPASShelper in an owner-only temp dir that is removed immediately after;it never enters a workspace or a campaign’s inputs (the driver imports the already-installed
.robovast_plugins/from the workspace and needs no credentials at composition time).
Composition moves to a subprocess when ``plugins:`` is declared. The plugin and its pinned dependencies are then imported only there, never in the long-lived service process, so a plugin needing a different version of something RoboVAST also uses cannot disturb it. An operator who needs hard isolation from untrusted plugin code should still prefer the uploaded-wheel source, which needs no credential at all.
That subprocess is also why a variation’s auxiliary container needs a hand. The
execution backend publishes its runner factory in a ContextVar, which does not
cross a process boundary — so a campaign declaring plugins: and a variation
with get_required_container (floorplan generation, say) once could not compose at
all on a cluster backend, having neither the backend’s factory nor a local docker
to fall back on.
robovast.common.container_runner_proxy closes that: the parent serves its own
live factory on a Unix socket beside the job file, and the worker installs a factory
whose runners forward run / close / expose back across it. The runner —
and with it the Kubernetes client, the storage client and the credentials both
authenticate with — stays in the parent; only the four calls of the
ContainerRunner contract cross,
plus workspace, which is a path both sides can already see. Command output is
streamed frame by frame, so a plugin’s progress still reaches the campaign log while
the command runs, and a failed command still raises
subprocess.CalledProcessError in the worker with its output intact.
Nothing about a backend is serialized, so this works for every backend without any
of them describing itself — including the local docker one and any added later.
When no factory is active (a plain CLI run) no socket is served and the worker uses
its own docker, exactly as before.
For a single dependency-free variation you can skip packaging entirely and use a
<path>.py:<Class> file reference resolved relative to the .vast directory
(see below).
Two pitfalls when exposing the package as a poetry extra of this repo (e.g.
nav = ["robovast-nav"]) rather than as an independent third-party plugin:
The extra’s package name must also be declared as an optional dependency in
[tool.poetry.dependencies](e.g.robovast-nav = {path = "src/robovast_nav", optional = true}for an in-repo sibling package) —poetry checkcatches the mismatch if not.For a cluster run, the built-in
robovast/robovast-navcode comes from the service image (vast cluster setupdeploys it), so dev changes to those sources reach a run by rebuilding/redeploying that image — publish your own set (make release-images PROJECT=docker.io/<you> PUSH=1) and pointROBOVAST_PROJECTat it to iterate. This applies only to the built-in sources; independent plugins are handled bydiscover_plugin_installsabove (installed into the workspace, no image rebuild needed).
If your plugin’s package pulls in a dependency that itself needs system
shared libraries (e.g. robovast-nav hard-depends on
pyside6-essentials, whose bundled Qt6 libs need libGL.so.1 and
friends to even import, regardless of whether any GUI is ever shown), the
controller image needs those apt packages too — see the
container/controller/Dockerfile apt-get block for the list verified
against robovast-nav. A missing system lib shows up the same way as a
missing extra: the plugin’s entry point fails to load and the variation
type is reported as unknown.
Add Command-line Plugin
To create a plugin for the vast CLI:
Create a Click group or command in your package
Register it in your pyproject.toml under [tool.poetry.plugins.”robovast.cli_plugins”]
The plugin will be automatically discovered and added to the vast command
Example plugin registration:
[tool.poetry.plugins."vast.plugins"]
variation = "variation_utils.cli:variation"
Add a config format for the system under test
The sut: channel addresses values inside the stack’s own configuration files.
yaml, json and xml ship built in; a stack configured by something else — a
bespoke .ini, a proprietary descriptor — needs a format, not a change to RoboVAST.
Subclass robovast.common.sut_formats.SutConfigFormat and register it under
[tool.poetry.plugins."robovast.sut_formats"]. The built-ins register through that same
entry point, so the route your package takes is the one this repository exercises on every
run.
class IniFormat(SutConfigFormat):
EXTENSIONS = (".ini", ".cfg")
def load(self, path): ...
def can_address(self, doc, path) -> bool: ...
def set(self, doc, path, value): ...
def remove(self, doc, path): ...
def dump(self, doc, path): ...
Four things decide whether a format behaves like the built-ins:
You own the path syntax. RoboVAST splits a destination once, on the first
., to find the source; the rest arrives at your format verbatim. Use dotted keys, XPath, JSON Pointer — whatever addresses your documents honestly.``can_address`` is the pre-check, and it is yours to answer. It asks whether a path is writable, not whether a value is already there: a factor may set a key the file leaves at its default, and refusing that would reject a correct campaign. What must exist is the parent. Raise
CannotAnswerwhen you genuinely cannot decide — that leaves the destination unchecked and reported, which is different from “no”, and keeping them distinct is what stops an unimplemented check from looking like a rejection.``set`` creates or replaces, and value may be a scalar, a mapping, a list or a fragment. A factor swapping a whole subtree must be the same call as one changing a number.
``remove`` is separate from ``set``, because a configuration block that is present and empty is not one that is absent, and stacks tell them apart. It is what
{$absent: true}reaches.
addresses is optional and may return None: it feeds error messages and editor
completion, never the check. A format whose address space is not enumerable — any XPath
expression is a potential address — loses only the listing.
Add Metadata Processing Plugin
Metadata processing plugins run after the generic and variation-plugin metadata
phases and can modify the metadata.yaml produced for each campaign. They are
configured in the .vast file under results_processing.metadata_processing:
results_processing:
metadata_processing:
- my_metadata_plugin
- my_metadata_plugin:
param1: value1
param2: value2
Each plugin must subclass robovast.common.metadata.MetadataProcessor and
implement the process_metadata method:
from pathlib import Path
from robovast.common.metadata import MetadataProcessor
class MyMetadataPlugin(MetadataProcessor):
def process_metadata(self, metadata: dict, campaign_dir: Path) -> dict:
# Modify metadata as needed
metadata["custom_field"] = "custom_value"
return metadata
Register the plugin in your package’s pyproject.toml:
[tool.poetry.plugins."robovast.metadata_processing"]
my_metadata_plugin = "my_package.metadata:MyMetadataPlugin"
Add Variation Plugin Metadata Hook
The Variation base class defines an overridable classmethod that returns an
empty dict by default. Subclasses implement it to attach domain-specific metadata
to each configuration entry in metadata.yaml:
from pathlib import Path
import yaml
from robovast.common.variation import Variation
class MyVariation(Variation):
@classmethod
def collect_config_metadata(cls, config_entry, config_dir: Path,
campaign_dir: Path) -> dict:
"""Load extra metadata from a YAML sidecar in _config/."""
data_file = config_dir / "_config" / "my_data.yaml"
if data_file.exists():
with open(data_file) as f:
return {"my_data": yaml.safe_load(f)}
return {}
collect_config_metadata is called once per configuration that used the
variation and returns a dictionary that is merged into the configuration’s
metadata entry.
Add PROV-O Provenance Hook to a Variation Plugin
Variation plugins can contribute domain-specific nodes to the campaign’s
PROV-O provenance graph by overriding collect_prov_metadata on the
Variation base class. The default implementation returns None
(no contribution). It follows the Contribution hooks: the shared contract contract, and its
failure policy is the “record the gap in the record” case: a hook that raises is
logged and named in the graph as a ProvenanceGap, so an incomplete
provenance record says that it is incomplete.
This hook is the right place for provenance that is tightly coupled to a specific variation — for example, a floorplan generation variation knows which map and mesh files it produced and can declare their lineage in the graph.
Return type: ProvContribution (or None to contribute nothing):
from robovast.common.variation import Variation, ProvContribution
class MyVariation(Variation):
@classmethod
def collect_prov_metadata(
cls,
config_entry: dict,
campaign_namespace, # rdflib.Namespace for the campaign
config_namespace, # rdflib.Namespace for this config
gen_activity_id: str, # IRI of the config-generation activity
vast_id: str, # IRI of the vast file that contains it
):
"""Contribute domain-specific PROV-O nodes."""
from rdflib import PROV, Namespace
_ID, _TYPE = "@id", "@type"
MY_NS = Namespace("https://example.org/metamodels/")
config_cfg = config_entry.get("config", {})
my_file = config_cfg.get("my_output_file", "")
if not my_file:
return None
file_iri = config_namespace[my_file]
return ProvContribution(
# Extra graph nodes (entities, activities) appended to @graph
graph_nodes=[{
_ID: file_iri,
_TYPE: PROV["Entity"],
"wasGeneratedBy": gen_activity_id,
MY_NS["someProperty"]: "value",
}],
# Properties merged onto the concrete scenario node
scenario_properties={MY_NS["outputCount"]: 1},
# IRIs that each run activity should declare as "used"
run_used_iris=[file_iri],
)
ProvContribution fields:
graph_nodesList of JSON-LD node dictionaries appended to the PROV
@graph. Userdflib.PROV,rdflib.DCTERMS, or your ownNamespaceobjects as keys/values.scenario_propertiesDict merged onto the concrete scenario entity node for this configuration. Useful for adding counts or classification properties (e.g. number of goals, number of obstacles).
run_used_irisList of IRIs that every run activity in this configuration will declare as
prov:used. Typically the IRIs of entities generated by this variation that are consumed at runtime (e.g. a map file, a mesh file).
Note
collect_prov_metadata receives rdflib.Namespace objects
(campaign_namespace, config_namespace) so you can construct
campaign-relative IRIs with campaign_namespace["some/path"].
rdflib is a required dependency of the core robovast package.
Add a Simulator Backend
A backend supplies what a campaign would otherwise restate for every simulator run: the
image, the packages, the environment, and how the simulator is started. Registered in the
robovast.simulators entry-point group; implementation:
robovast.common.simulators. The user-facing side, the two shapes, and a worked
example are in Simulators.
Two constraints that are not obvious from the base class:
It must import without its simulator installed. apply_backend runs in the
long-lived service process, which has no reason to carry a MuJoCo or an Isaac runtime.
A backend declares strings and container specs; anything genuinely needing the simulator
returns a ContainerSpec from input_files and runs inside the simulator’s image.
It can arrive through plugins:, but not via the driver’s plugin-install phase.
That phase (controller._install_plugins) is deliberately add_to_path=False and in
any case runs long after the image and environment hooks. What makes a packaged backend
work is a separate, earlier install: ServiceBase._build_specs_for resolves the
campaign’s plugins: before extracting the build specs, precisely because the
container plan depends on the backend. The ordinary answer is still an installed
distribution of the service environment, like robovast_nav, or a .vast-relative
file ref — a wheel per campaign is a real cost. See Simulators for the three
options in preference order.
Note
That two install sites decide this, and only one of them is documented where a backend author would look, is worth revisiting: whether a backend should be shippable per campaign is a design question, and today the answer is an accident of call order.
Where it runs in composition. apply_backend() is called once at the top of the
execution extraction in generate_scenario_variations(), so the container plan, the
image builds and the run environment all read one already-merged mapping instead of each
re-asking the backend and risking different answers. The campaign always wins: a backend
fills in keys the author left out and never overrides one they set.
Add Input Generator Plugin
Input generators produce a campaign’s derived inputs before it is composed — the mirror
image of postprocessing, at the other end of the campaign. They are declared as
execution.generate and registered in the
robovast.input_generators entry-point group. Implementation:
robovast.common.input_generation.
Where it runs in composition. run_input_generators() is called from
generate_scenario_variations() before collect_filtered_files(), and each entry’s
outputs are appended to run_files. That position is the whole design, and everything else
follows from it: the outputs reach hash_run_files() (so they enter the configuration
identity), prepare_campaign_configs() copies them into <campaign>/_config/, and the
run container bind-mounts them at /config/<path>. There is no second code path for
generated files anywhere downstream. A per-configuration artifact — one a variation
produces per cell — is staged at that same /config/<path> instead of the campaign’s copy,
so a generator and a variation put a file in the same place and the campaign-wide one simply
loses the path when a cell owns it. It also means generation is host-side, before
publication, so the cluster gets the artifacts with no extra work.
Generation vs. variation. Both can produce artifacts, and the test for which you want is:
does it produce configurations, or only files? A generator produces files, once per
campaign. Something that multiplies the configuration set — one config per floorplan, with the
generated paths bound to scenario parameters — is a variation
even when it also generates, because execution.generate has no way to emit configurations.
Return value: (success: bool, message: str), or None for success. Raising and
returning False are reported identically, so use whichever carries the better message.
Reserved key. out belongs to RoboVAST, not the plugin: it is validated, created as a
temporary directory, and swapped into place only on success. So outputs can be expanded into
run_files without importing the plugin (which the isolated compose subprocess relies on),
and a half-written artifact can never be mistaken for a finished one.
Staleness. Declare what was read by calling write_manifest(out_dir, paths), which
writes .generated.json into the output directory; the next composition hashes those paths
and skips the generator when nothing moved. Report the real set — for anything compiled from
a description that can reference others, that includes the transitive ones. A generator that
reports nothing is never cached, and a cached result is honored only while its outputs are
still on disk unchanged: staleness fails towards doing the work, never towards serving a stale
artifact.
Creating an Input Generator:
from robovast.common.input_generation import BaseInputGenerator, write_manifest
class MyGenerator(BaseInputGenerator):
"""Compile <thing> into a campaign input."""
#: Bump when the OUTPUT FORMAT changes, so an upgraded plugin regenerates
#: even though none of its inputs moved.
FORMAT_VERSION = 1
@classmethod
def get_required_container(cls, parameters):
"""Optional: an aux image, when the tool is not installed alongside RoboVAST.
Same contract as a variation's — a container in an aux pod (the campaign's
during a run; one held by the container-exec manager when a preview
composes), or an ephemeral ``docker run`` under ``vast configuration
generate`` on a development machine. Reached via
``self.container_runner``,
whose ``workspace`` is visible at the same path on both sides (use
``stage_for_container`` / ``collect_from_container``).
"""
return None
def __call__(self, vast_dir, out_dir, source=None, **params):
sources = compile_thing(os.path.join(vast_dir, source), out_dir)
write_manifest(out_dir, sources)
return True, f"compiled {source}"
Registration:
[tool.poetry.plugins."robovast.input_generators"]
my_generator = "your_package.generators:MyGenerator"
A project can also skip packaging entirely and reference a local file:
- ./tools/gen.py:MyGenerator: {out: files/thing}.
Usage in .vast config:
execution:
generate:
- my_generator:
out: files/thing
source: things/input.yaml
Note
The generator must be importable by the process that composes the campaign, which for
a running service is vast serve — not the shell the user typed in. The unresolved-name
error prints sys.prefix for exactly this reason. An aux container sidesteps the
question entirely, which is the main argument for declaring one.
Add Postprocessing Command Plugin
Postprocessing plugins are Python functions that process run result directories (e.g., convert rosbag data to CSV). They are registered as entry points and executed before analysis.
Return value: A plugin must return (success: bool, message: str). It may optionally return a third value, a list of provenance entries, so that each produced file is recorded (e.g. which CSV was created from which rosbag). Each entry is a dict with keys: output (path relative to results_dir), sources (list of paths), plugin (plugin name), params (optional dict). If returned, these entries are merged and written into postprocessing.yaml in each run folder (<campaign-name>-<timestamp>/<config>/<run-number>/).
Steps that work run by run: a plugin whose output for a run depends only on that run, the
scenario jobs it links, the other runs of those jobs and the campaign-level inputs
(_config, _execution, _transient) declares scope = "run". A cluster may then run
it on parts of the campaign in parallel (Splitting a campaign’s postprocessing): every part
is a set of whole scenario jobs with their runs, and the step must write only into those. The
default, scope = "campaign", runs the step once over the whole tree. Only the leading run of
run-scoped steps is split – a run-scoped step listed after a campaign-scoped one runs after
it, over the whole tree.
Steps that run in the campaign’s execution image: some work can only happen in the image the
runs used — deserializing a bag needs the message definitions of the custom types it recorded. Such
a plugin derives from
ExecutionImagePlugin and names the
command to run there instead of implementing __call__:
import os
from robovast.results_processing.postprocessing_plugins import ExecutionImagePlugin
class DecodeCamera(ExecutionImagePlugin):
def image_command(self, ctx, topic="/camera"):
# A file in the scripts directory, then its arguments. ctx says where the
# campaign is inside the container, and whether to force or be verbose.
argv = ["decode_camera.py", "--topic", topic]
if ctx.provenance_file:
argv += ["--provenance-file", ctx.provenance_file]
return argv + [ctx.campaign_dir]
def image_files(self):
# Shipped beside the conversion scripts, under their own names.
return [os.path.join(os.path.dirname(__file__), "decode_camera.py")]
The postprocessing Job runs exactly that command from /scripts through ros2_exec.sh
in its image container; vast results postprocess on a development machine runs it with
docker_exec.sh against the campaign’s execution image. A parameter the command does not
take fails the step. The script runs without
robovast — only the standard library, what the image provides, and the files beside it
(rosbags_common among them). It writes its outputs into the campaign tree; on the cluster, every
file it writes is delivered back, whether or not it records provenance. To record provenance it
appends {"output", "sources", "plugin", "params"} entries (paths relative to the campaign) to
--provenance-file with rosbags_common.write_provenance_entry.
Creating a Postprocessing Plugin:
from typing import Tuple, Optional, List
def my_postprocessing_command(
results_dir: str,
config_dir: str,
custom_param: Optional[str] = None,
provenance_file: Optional[str] = None,
) -> Tuple[bool, str]:
"""Convert custom data to CSV.
Args:
results_dir: Path to the <campaign-name>-<timestamp> run directory to process
config_dir: Config file directory (for resolving relative paths)
custom_param: Optional custom parameter
provenance_file: Optional path for provenance JSON (for container scripts)
Returns:
Tuple of (success, message) or (success, message, provenance_entries)
"""
import subprocess
import os
script = os.path.join(config_dir, "tools/script.sh")
cmd = [script, results_dir]
if custom_param:
cmd.extend(["--param", custom_param])
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
return False, f"Failed: {result.stderr}"
return True, "Success"
Register in pyproject.toml:
[tool.poetry.plugins."robovast.postprocessing_commands"]
my_postprocessing_command = "your_package.postprocessing_plugins:my_postprocessing_command"
Usage in .vast config:
analysis:
postprocessing:
- my_postprocessing_command:
custom_param: value
Add Publication Plugin
Publication plugins package or distribute the results directory after postprocessing. They are plain callables (functions or class instances) that operate on the full results directory.
Return value: A plugin must return (success: bool, message: str).
Creating a Publication Plugin:
from typing import Optional, Tuple
def my_publication_plugin(
results_dir: str,
config_dir: str,
destination: Optional[str] = None,
) -> Tuple[bool, str]:
"""Upload results to a remote storage location.
Args:
results_dir: Path to the results directory (parent of campaign directories).
config_dir: Directory containing the .vast config file; relative
paths should be resolved from here.
destination: Remote destination URL or path.
Returns:
Tuple of (success, message).
"""
import subprocess
dest = destination or "s3://my-bucket/results/"
result = subprocess.run(
["aws", "s3", "sync", results_dir, dest],
capture_output=True, text=True,
)
if result.returncode != 0:
return False, f"Upload failed: {result.stderr}"
return True, f"Uploaded results to {dest}"
Register in pyproject.toml:
[tool.poetry.plugins."robovast.publication_plugins"]
my_publication_plugin = "your_package.publication_plugins:my_publication_plugin"
Usage in .vast config:
results_processing:
publication:
- my_publication_plugin:
destination: s3://my-bucket/results/
Add Cluster Config Plugin
To add a new cluster configuration option for RoboVAST, create a class that inherits from robovast.execution.cluster_config.base.BaseConfig. Register your cluster config in your pyproject.toml under [tool.poetry.plugins.”robovast.cluster_configs”]. The key is the name used to select the configuration, and the value is the import path to your configuration class.
[tool.poetry.plugins."robovast.cluster_configs"]
"YourClusterConfig" = "robovast_<yourplugin>.your_cluster_config:YourClusterConfig"
To test your cluster configuration, call
prepare_setup_cluster()
with an output directory. It writes the files and the instructions for performing the setup
steps by hand, without applying anything. There is no CLI verb for it — vast cluster setup
is the applying path.
Staging for experiment-image builds. A build’s context (the project dir plus the
generated Dockerfile) is a scratch tree on the service’s own disk, under the results root’s
_staged/image-builds/<build-id>/ slot (cluster_image_build.context_slot, written by
stage_context). The build Job’s init container fetches it from the service’s data plane
with a token scoped to that slot (pod_access), so a cluster config contributes nothing
here: no storage backend and no credential reaches the Job, only a token that opens one
slot.
A staged context is deleted again when the build ends (ClusterService calls its own
discard_staged on the slot; cluster_image_build.staged_context_build_ids lists
what is still staged for the sweep of contexts whose Job is gone).
Pod DNS for unresolvable hosts. get_host_aliases() parses
ROBOVAST_EXTRA_HOST_ALIASES (<host>=<ip>, comma-separated, grouped by IP into the
shape of the k8s field) and is spliced into the build Job
(cluster_image_build.build_job_manifest) and campaign Jobs
(kubernetes_backend). Override it if a deployment knows its aliases from somewhere
other than the environment. A malformed entry raises rather than being skipped: a dropped
alias reappears as an unexplained no such host inside a pod, far from its cause.
The boundary is worth keeping in mind when adding pod specs: hostAliases writes
/etc/hosts in the pod, so it governs what the pod’s own processes resolve — the
BuildKit push, or a node inside the scenario. The image pull is performed by the
node’s container runtime before the pod exists, so no pod-level field can influence it;
that stays node configuration, exactly like registry TLS trust.
Add a MCP Plugin
Contribute data, not read tools. A domain package (a robot type, a stack) makes its results readable through core rather than through tools of its own. A per-domain read tool is a second reader of facts the index already holds: it drifts from the first, it exists only where its package is installed, and every domain that adds one spends the tool budget every client pays for. The seams that make a domain’s facts readable everywhere:
A table that follows a contract. A pose table that follows the pose contract appears in
pose_track_viewwith no registration. A measurement outside a contract is still a table, and its meaning goes into column notes.Scene markers. A variation’s
config_view_dataplaces neutral markers (a planned path, goals, obstacles). The config view draws them,get_config_contributionserves them,get_track_deviationmeasures against apathmarker, anddraw_configdraws them.A config panel’s
plot. A package that renders a contributed file in the browser draws it fordraw_configtoo (Web UI internals).
An MCP plugin is for what these cannot carry, such as an action a domain offers.
Create a class with a name property and a register(mcp) method:
# my_package/mcp_plugin.py
from fastmcp import FastMCP
class MyMCPPlugin:
@property
def name(self) -> str:
return "my_plugin"
def register(self, mcp: FastMCP) -> None:
@mcp.tool()
def my_tool() -> str:
"""A custom tool."""
return "hello"
Then register the class as an entry point in pyproject.toml:
[tool.poetry.plugins."robovast.mcp_plugins"]
my_plugin = "my_package.mcp_plugin:MyMCPPlugin"
The plugin is picked up automatically the next time the server starts.
A call the tool rejects — an unknown or missing argument — is answered with the arguments
the tool does take. Where a tool lacks an argument a caller will reach for on purpose,
declare why with robovast.mcp_server.lacks.lacks(), and the reason is added to that
answer instead of costing description tokens on every request:
from robovast.mcp_server.lacks import lacks
@lacks(timeout="the bound follows from what is run")
def my_tool(command: str = "") -> dict: ...
@lacks("there is only one, and it belongs to the service")
def stop_it() -> dict: ...
Where the plugin’s tools answer a question an agent would otherwise put to a core tool,
give the class an
instructions attribute: one short paragraph appended to the server’s MCP
instructions, the only text a client shows before any tool is chosen. Core text names no
plugin tool, so the routing exists exactly where the plugin is installed. The server
refuses to start when the instructions with every plugin’s paragraph exceed what a client
shows (the instructions limit); keep the paragraph to a sentence or two.
Add Search Strategy Plugin
A search strategy drives the closed-loop search (see Iterative Search): it proposes
parameter sets, is told their evaluations, and produces a final report. Strategies
are algorithm-agnostic and share one config schema; only the per-strategy
strategy_parameters block differs.
Subclass robovast.search.strategy.SearchStrategy and implement the four
abstract methods. Optionally set PARAMS_MODEL to a Pydantic model — the
framework validates search.strategy_parameters against it and passes the
parsed object as params:
from pydantic import BaseModel
from robovast.search.strategy import SearchStrategy
from robovast.search.types import ParamSet, SearchReport
class MyParams(BaseModel):
step: float = 0.1
class MyStrategy(SearchStrategy):
PARAMS_MODEL = MyParams # optional; omit (None) for no parameters
def ask(self, n: int) -> list[ParamSet]:
"""Propose n parameter sets (keys match search_space dims)."""
...
def tell(self, evaluations) -> None:
"""Ingest the evaluations of the batch just run."""
...
def is_done(self) -> bool:
"""True when the budget is exhausted / converged."""
...
def report(self) -> SearchReport:
"""Return the deliverable (ranked best, archive, Pareto front)."""
...
self.search_space, self.objectives and the validated self.params are
available on the instance. For single-objective strategies, self.objective_value(ev)
returns the sole objective sign-oriented so that higher is always better.
Register the class under robovast.search_strategies; the key is the
search.strategy name:
[tool.poetry.plugins."robovast.search_strategies"]
my_strategy = "your_package.strategies:MyStrategy"
A strategy can also be loaded from a local file relative to the .vast without
packaging, using strategy: ./search/my_strategy.py:MyStrategy (the same
load_ref mechanism used for extractors and search postprocessing).
Add Extractor Plugin
The extractor is the single, SUT-specific scoring step: it reads a parameter set’s per-config result directory and returns named objectives (optimized) and measures (quality-diversity behavior axes; ignored by non-QD strategies). This is the one place system-under-test logic lives.
Subclass robovast.search.extractor.Extractor. It is constructed with the
extract.params from the .vast (so thresholds / column names can be swept
without editing code), and aggregation over the config’s runs is its
responsibility:
from pathlib import Path
from robovast.search.extractor import (Extractor, ExtractResult,
completed_run_dirs)
class MyExtract(Extractor):
# __init__(self, **params) is inherited; params land on self.params
def extract(self, config_dir: Path) -> ExtractResult:
runs = completed_run_dirs(config_dir) # helper: finished runs
failures = sum(1 for r in runs if _failed(r))
return ExtractResult(
objectives={"failure_rate": failures / max(len(runs), 1)},
measures={}, # {} when unused
)
objectives and measures are named dicts, so single- and multi-objective
use the same shape. The framework records how many runs backed each result.
Register under robovast.extractors (referenced by search.extract.plugin),
or load from a local file with extract.plugin: ./search/extract.py:MyExtract:
[tool.poetry.plugins."robovast.extractors"]
my_extract = "your_package.extractors:MyExtract"
The extractor reads what a postprocessing plugin produced (e.g. per-run
metrics.csv) — it does not compute raw metrics itself. Pair it with a
postprocessing plugin (below) that writes metrics.csv from raw artifacts:
list that plugin in search.postprocessing (run before extract) and/or in
results_processing.postprocessing (analysis), so search and the analysis
notebooks read the same metrics. Postprocessing plugins load identically in both
lists — by entry-point name or a local ./path.py:Class file reference.
Campaign Store and Results Indexing
Every campaign — batch or search — is described by a single sqlite store,
campaign.db (robovast.common.store.STORE_FILENAME), written at the
root of the campaign directory. It is the single source of truth the results
GUI reads, and the seam an in-cluster controller or web UI can later read/stream.
A campaign runs one or more batches: a batch-mode campaign (no search:
block) has exactly one batch of the enumerated configs; a search campaign has one
batch per ask/tell round.
Schema
robovast.common.store.CampaignStore is a thin wrapper over five tables:
campaign (1) --< batch (1) --< unit (one per param set / config) (1) --< run (one per repetition)
campaign (1) --< job (one per execution job) ...............< run (via run.job_id)
campaign —
mode(batch/search),config_dir(base directory against whichvisualization.results.explorer.notebooksnotebooks resolve),config_json(the full config), an opaquestrategy_stateblob for resumable strategies, and the campaign’s execution provenance:robovast_version,execution_type(local/cluster),image,image_revision(therepo@sha256the runs actually used),execution_started_at,elapsed_s, plusexecution_jsonholding the rest of_execution/execution.yaml. Those are the fields one compares across campaigns, and the SQL interface can attach several campaigns at once — see What goes in the database, and what stays a file. They are written byrecord_executiononce the backend has producedexecution.yaml, so they are NULL for a campaign that died before execution began. Noteexecution.yaml’s ownexecution_timeis a start timestamp, not a duration, hence the column name. What the campaign was asked for is not here: it lives in_execution/launch.yaml, because the service can write that before the run starts while these columns can only be filled after it (see the launch record).job — one row per execution job, holding that job’s
sysinfo.yamlverbatim insysinfo_json. It is a table rather than a column onrunbecause sysinfo is written once per job: a packed multi-config job runs several(config, run)pairs which reach the same file through each run dir’sjobsymlink. A per-run copy would repeat the blob and destroy the fact that those runs shared a machine — which is what makes “did the slow runs land together?” answerable.job_diris campaign-relative (_jobs/batch-0/job-3), or the run’s own directory for an older layout that wrote sysinfo beside the run.batch — one ask/tell round (search), or the single batch (
idx=0) of a batch-mode campaign.unit — one evaluated parameter set (search) or one configuration (batch): the sampled
params,objectives/measures(JSON;{}for batch), and theresult_dir.n_samplesand the aggregatestatusare roll-ups of the unit’srunrows, kept for convenience.run — one repetition of a unit (schema v2+). Mirrors that run’s
test.xml:status(passed/failed/error/unknown),passed(0/1),errors/failures/tests,duration_s,start_timeandfailure_message.run_idis the numeric run index within the config dir — so it is not unique on its own;config_namelives onunit.job_idpoints at the job it ran in. A run whosetest.xmlis missing or unparseable is still recorded, asunknown— never dropped.
Two definitions of the schema, on purpose
_SCHEMA is the full current layout, applied in one step to a fresh database, so a
reader can see what a table looks like without replaying history. _MIGRATIONS is the
append-only ladder that upgrades an existing store; entry i takes user_version i
to i+1 and is never edited once shipped, because some database on disk has already
applied it. Note migration 0→1 is a frozen copy of the v1 layout rather than _SCHEMA
— reusing _SCHEMA there would jump an old store to today’s tables and the later
ALTER TABLE steps would then fail on columns that already exist.
Adding a column therefore means touching both, in the same position.
test_fresh_and_migrated_schemas_match builds a store each way (from fresh, and from
every older user_version) and compares the resulting sqlite_master column by
column, so the two cannot drift apart.
Why run exists — the data-model layering
run is the operational source of truth for per-run outcomes, and it is why
listing campaigns is cheap. test.xml (JUnit, one per run) is the runner’s
on-disk contract; the controller already parses it at record time, so capturing a
run row there is free. Pass/fail counts are then one GROUP BY status over
run — no filesystem walk — and are available live, before postprocessing.
The central index’s run_view is the analytics-wide view over these rows
(joining sysinfo and the unit’s params); the ingest mirrors campaign.db rather
than re-parsing every test.xml. Heavy per-run measurement data (metric
time-series) is streamed into the index — campaign.db remains the lightweight
live store.
test.xml runner artifact (per run) — the on-disk contract
-> captured live at record time (data already in hand)
campaign.db.run operational source of truth — queryable DURING the run
-> postprocessing joins sysinfo/params/metrics
index runs view analytics-ready wide view (param_* columns, metrics) — DERIVED
Note
Schema v1 stores (written before the run table) migrate forward on open to
an empty run table.
robovast.common.campaign_index.backfill_run_rows() fills them from disk
test.xml; the service does this lazily for finished campaigns, and the
summary path falls back to the test.xml walk
(get_vast_configuration_info()) until it is
backfilled, so counts are never under-reported.
Who writes it
The controller (
robovast.execution.controller.CampaignController) writes the store live for both modes as each batch is evaluated, so progress is queryable while a campaign runs. It owns the campaign id, the flat results layout (<campaign>/<config>/<run>/) and the batch loop; anExecutionBackend(KubernetesBackend) only dispatches one batch’s jobs.The post-hoc indexer
robovast.common.campaign_index.build_campaign_store(campaign_dir)reconstructs the same store by scanning a finished results tree (reusing thecampaign_datareaders). It is used for campaign dirs not produced by the controller — e.g. one unpacked from an archive somebody else made — and is idempotent (mtime-guarded;force=Trueto rebuild). Controller-written stores are left untouched. It has no CLI entry point, so a tree that has no store keeps none. The function itself is the register step ofvast campaign import: an archive somebody else produced usually has no store, and extraction alone leaves the campaign invisible to every store-driven view — which is what keeps the primitive alive, and out of a routine dead-code sweep.
Taking a campaign in
robovast.service.ingest owns both halves. import_archive()
unpacks an archive into a results root; ingest_campaign()
registers what came out and reports per stage, since a campaign archive carries three
version surfaces of its own (the .vast’s, campaign.db’s and the analysis DB’s) which
can independently be older, newer, absent or corrupt. Neither re-implements a migration –
the config ladder is applied in memory and the store migrates on open, so this module observes
and reports.
Three entry points, one implementation: vast campaign import (locally, or streamed to a
reachable service), POST /campaigns/import behind the web UI’s upload button, and the
import_campaign MCP tool. Only the first two ever move bytes; both do it through the
archive side channel documented in HTTP API, and the import itself always takes a path
on the host that will hold the campaign.
Two refusals in import_archive are load-bearing rather than defensive. The archive must
hold exactly one top-level entry, computed with a ./ root ignored – reading the first
path segment literally makes tar czf x.tar.gz -C <results> . look like one entry named
., which resolves to the results root, so a forced import would have deleted every
campaign to make room for itself. And that entry must satisfy
is_campaign_dir(), because the local listing and the delete
guard both filter on that shape: a differently-named directory would import, report every
stage ok, and then be invisible and undeletable.
Store-driven results views
The results GUI (RunResultsAnalyzer) discovers campaigns by scanning
<results_dir>/*/campaign.db — there is no filesystem-walk or depth-based
heuristic. It reads the campaign/batch/unit rows to build the tree
(campaign → batch, search only → config), resolves notebook workloads from
config_json against config_dir, and enumerates only the run-level leaves
from each unit’s result_dir.
Campaign state and control
A campaign is driven by a CampaignController
running in the driving process — the robovast-service, one worker thread per
campaign. There is no
separate controller pod and no in-pod HTTP control channel: monitoring and control
are ordinary RobovastInterface calls the
service answers directly, so the CLI, MCP and web UI all use one path.
Live state
ControllerState is the thread-safe
holder the controller writes and readers snapshot. The controller calls
update / set_phase at each batch boundary; a reader takes a consistent
snapshot (deep copy), which is exactly what get_status returns. The one
status model, Status, is reused
verbatim by the interface, the MCP tools and the TS client.
Because the service drives many campaigns as threads in one process rather than one
pod each, per-campaign state is isolated explicitly: the container-runner factory is
a ContextVar, controller.log is filtered to its worker thread, and each aux pod
/ result prefix is keyed by campaign id.
Status: phase and stage
phase is an open string advanced through a documented vocabulary, with
stage carrying finer markers so new states slot in without a schema change:
|
Meaning |
|---|---|
|
Accepted: registered, listed, and addressable by id, with the service’s pre-flight (project push, registry/base-image resolution) still to do. The first phase every campaign has, and the one that guarantees a caller can always find what it just started — see A started campaign is findable. |
|
Building the experiment image (only when the |
|
Image settled; handing off to the worker (campaign context, backend construction). |
|
Installing the declared |
|
Expanding the config variations (batch mode); its log is |
|
Batch loop executing. |
|
Search stop condition met (or a |
|
Streaming the raw pre-postprocessing archive to the configured share (only when
|
|
Chained analysis postprocessing (rosbag→CSV Job + index ingest) running. |
|
Done; every run’s output is on the service’s results volume. A post-run step may still
have failed: the runs are the deliverable, so a failed upload-to-share or
postprocessing keeps the phase |
|
The runs were aborted. |
|
Terminal: cooperatively stopped; the driver died without recording an outcome; or (after a restart) not reconstructable from disk. |
phase_since records when the current phase was entered (and only moves on an actual
change, so a defensive re-set does not restart the clock). It exists because a phase name
alone cannot separate slow from wedged: an image build in progress and one that will
never finish both read building. Readers render it as an age — the MCP status returns
phase_age_s, and the web Monitor shows it beside the phase dot while a pre-run phase is
in effect, where there is no progress bar to watch instead.
A started campaign is findable
Invariant: from the moment the service accepts a campaign, its id resolves on every
read path — get_status and list_campaigns (also the MCP’s running_only filter). This is what
makes the standing advice (“a timed-out start is not a failed start; check, never retry”)
actually true: retrying a start that in fact succeeded creates a second campaign, and the
only defense is that the first one can be found.
Two things back it. Registration happens before the slow work — create_campaign
records the campaign in the service’s registry and returns, and the driver builds the image —
so the campaign is live from t=0 rather than from whenever its results directory
appears (see A campaign waits for its image; the request does not). list_campaigns unions that registry into
its id set beside the disk scan, so a campaign is listed for the whole length of its
pre-flight — without that it would be registered and addressable by id, but undiscoverable
by anyone who did not already know the id, which is precisely the caller whose start
response was lost.
Discovering campaigns across a service restart
list_campaigns builds its id set from two sources: the results root on disk, and the
in-memory registry of what is being driven.
The first is what survives the process, and it is the whole picture — a campaign is a
directory under <results_root>/<campaign_id>/ on the service’s results volume, so a
service that restarts finds every campaign from every previous life by the same
iterdir. There is no
second home to enumerate and no index of campaigns to keep in step with the tree.
That is also what makes the records cheap to read. _summary_for, _started_at_for,
_description_for and _status_from_disk all resolve the campaign through one seam,
campaign_dir, and read campaign.db and _execution/outcome.json from it as
local files — no fetch, no per-row round trip on a listing, and nothing that can be
reachable for one reader and not another.
The order is computed over every id before the page is cut, so each sort key has to be
answerable without building a summary per campaign: the recency keys come from the memoized
_started_at_for / _finished_at_for, and sort="size" from _results_bytes_for,
which reads the same outcome.json figure the row shows and memoizes it — None
included — once a terminal record says it. Only the page’s own campaigns get a
_summary_for.
A campaign waits for its image; the request does not
create_campaign is fire-and-forget, so the image build runs on the campaign’s worker,
not on the caller’s thread. Awaiting it inline meant a cluster start for a project with a
build: section died on the HTTP client’s 30 s read timeout while the server kept
going and the campaign succeeded — a reported failure for work that worked — and, since
the campaign was created only after the build, none of that work was observable while it
ran.
What the caller gets instead: the campaign id immediately, and a campaign in phase
building whose stage reads waiting for image <tag>. list_campaigns(running_only=True)
needed no change for it to appear there — a building campaign is a campaign, and the
listing already unions the in-memory registry. (Teaching that tool to enumerate builds
would have needed a listing endpoint that does not exist, returned entries that are not
campaigns, and created a second in-flight registry to keep consistent with the first.)
Builds are shared — build_hash is content-addressed over the spec and context, so
two campaigns needing the same image both wait on one build. Two consequences:
The phase means waiting for its image, not performing the build; otherwise two campaigns each appear to be building the same one. The
stageand thebuild.logheader both say so.Stopping a building campaign detaches it.
_await_build_imageraisesCampaignStoppedand touches neither the build Job nor the local build thread: a sibling may be waiting on that build, and the image is a cache entry rather than this campaign’s property. Nothing cancels a build today — the cluster teardown is label-scoped tojobgroup=scenario-runsand cannot reach ajobgroup=image-buildsJob, and the localdocker rm -f robovastcannot reach abuildxthread — and it must stay that way.
One wait loop: _await_build_image drives get_image_build_status
and get_image_build_log, which are interface operations each transport implements, so
the in-cluster BuildKit Job is waited on by the base’s code rather than by a loop of
ClusterService’s own.
The loop also tees the build log into the campaign’s own _execution/build.log,
which INFRA_PHASES serves as a leading BUILD section (see the MCP build workflow). That
is the campaign’s only durable record of the image it ran on: the live source dies with
the build, since a build Job is reaped at ttlSecondsAfterFinished, and a failed build
is exactly when someone comes looking. It is also why
controller._record_controller_outcome uploads build.log alongside outcome.json
— a campaign that died waiting for its image never reaches finalize_campaign at all.
This changes an error path on purpose: a failed build is now an inspectable failed
campaign — reason in its status, output in its own log — rather than a 500 and no campaign.
Building is part of the campaign’s driven work, not a precondition of its existence.
Control operations
stop(client.stop) — sets a cooperative flag on the campaign’sControllerState(request_stop); in the service this is a direct in-process call. Every wait the campaign’s thread sits in ends on that flag (wait_for_stop), and every boundary between two steps gives up on it (raise_if_stopped), so the phase a campaign happens to be in does not decide how long the stop takes. Which work a stop lands on, and what each one leaves behind, is the per-phase table in Client–Server Architecture.get_campaign_logs— servescontroller.logfrom a byte offset, the same file on the service’s results volume while the campaign runs and after. The web UI polls it to stream the log;vast … monitorrenders live status fromget_status.upload_to_share(launch flag oncreate_campaign) — when set, the driver streams a raw, pre-postprocessing archive of the campaign to the configured share the moment the runs finish, before analysis postprocessing. A share failure never loses the campaign: it staysfinishedand the reason is recorded onshare_error(durable). The backend streams it to the share provider with no on-disk copy. The download counterpart is the/data/campaigns/{id}/archivestream (the campaign as the service holds it, tarred on the fly off the results volume).run_share(client.run_share) — re-triggers the upload-to-share on a finished campaign, from the stored campaign alone (works after a service restart, no live entry). The provider comes from the environment, so adjustingROBOVAST_SHARE_TYPEand re-triggering uploads to a different destination. Mirrorsrun_postprocessing.
API reference
Live campaign state holder (ControllerState).
Every campaign is driven by a CampaignController
that runs in the driving process — the robovast-service. The controller advances a shared
ControllerState; the service reads its snapshot()
directly to answer GET /campaigns/{id}/status (no separate control server, no
pod-IP hop — those existed only when the controller lived in its own pod).
The status contract it carries (Phase, Status and the phase-group
predicates) now lives in robovast.client.status, so foundational common
modules can depend on it downward instead of common reaching up into
execution. It is re-exported here verbatim, so from
robovast.execution.control_server import Status (and Phase etc.) keeps
working everywhere. fastapi/uvicorn are not needed here.
- class robovast.execution.control_server.ControllerState(**initial)
Thread-safe holder the controller writes and the service reads.
The controller calls
update()/set_phase()at each batch boundary (andupdate()for run-level progress within a batch); the reader takes a consistentsnapshot().request_stop()/stop_requestedback the cooperativestop(now an in-process call fromclient.stoprather than an HTTP command).- property postprocessing_stop_requested: bool
Whether postprocessing was asked to end (see
stop_checker()).
- property progress_suspended: bool
- property project_released: bool
Whether the campaign has stopped reading the project it was launched from.
- raise_if_stopped(reason: str, scope: str = 'runs') None
End the caller with
CampaignStoppedwhen scope was asked to stop.For the boundaries between steps, where the wait primitive has nothing to wait on: a campaign staging its project, installing plugins or composing its sweep is doing bounded work in sequence, and the cheapest honest place to give up is before the next step rather than in the middle of one. reason names the step, because it becomes the campaign’s recorded stop message.
- release_project() None
Record that the campaign no longer reads the project it was launched from.
Set once the campaign has staged its configurations, which for a batch campaign is the last thing it does with that project (see
CampaignController._on_configs_staged). Never set for a search campaign, which composes out of the project again for every generation. Until it is set, editing that project changes what the campaign is about to run.
- request_stop(scope: str = 'runs') None
Ask scope’s work to end cooperatively.
Defaulting to
STOP_RUNSrather than stopping everything: a caller that does not say what it is stopping means the campaign’s runs, which is what a stop meant when there was only one of them.
- resume_progress() None
- set_phase(phase: str, stage: str | None = None) None
Advance to phase, stamping when it started.
phase_sincemoves only on an actual change, so re-setting the current phase (some paths do, defensively) does not keep resetting the clock a reader uses to tell “slow” from “wedged”.Reaching a new phase is itself forward movement, so it also restarts
progress_since. Without that, a campaign that spent ten minutes invariationwould enterrunningalready carrying a ten-minute-old progress clock, and be reported as stalled before its first run had a chance to finish.A stage belongs to the phase that set it, so a phase change clears it unless the new call names one. Without that,
stageoutlived its phase for the rest of the campaign’s life: a campaign that had waited for an image reported “waiting for image(s) simulation, sut” while it ran, while it postprocessed, and after it finished – a sentence that was true once and then read as a live statement about a campaign that was already over. The one call site that cleared it by hand (stage=""on enteringstarting) is the workaround this replaces.Re-setting the same phase keeps the stage: some paths do that defensively, and clearing there would wipe a stage the same phase had just set.
Whether the share upload was asked to end (see
UploadProgress).
- property stop_requested: bool
Whether the campaign’s runs were asked to end.
Deliberately still the runs and nothing else. A stop of the analysis or of the upload must not read as “the campaign was stopped”: that is what made a stopped run also discard its analysis, because one bit answered both questions.
- stop_requested_for(scope: str) bool
Whether scope’s work was asked to end.
- suspend_progress() None
Pause the run-progress poller (see CampaignController._poll).
The poller lists the whole campaign prefix every few seconds. During a batch’s result download over the same (off-cluster) storage tunnel, those extra list calls compound tunnel contention; suspending the poller for the download keeps the single transfer uncontended.
- update(**fields) None
- update_runs(**fields) None
Merge fields into the
runssub-model, atomically.update()assigns whole attributes, soupdate(runs={...})would drop every counter the dict omits. The counters are written from different places (the progress poller, the batch-completion tally, the outcome tally) and must not clobber each other, and merging under the lock keeps that safe without each caller having to read-modify-write.
- wait_for_stop(timeout: float, scope: str = 'runs') bool
Wait up to timeout seconds, returning early and True if scope is stopped.
The replacement for
time.sleepin every poll loop the campaign’s own thread runs: a sleeping loop only notices a stop at the end of the interval it happened to be in, so the interval that makes a poll cheap is also the delay a stop pays. Waiting on the event instead makes the two independent – the poll stays as rare as it should be, and the stop lands the instant it is requested.Returns what
stop_requested_for()would answer, so a caller can wait and decide in one step.
- class robovast.execution.control_server.Status(*, phase: str = Phase.INITIALIZING, phase_since: float = <factory>, progress_since: float = <factory>, progress_deadline_s: int | None = None, waiting_for_capacity: bool = False, batch_since: float | None = None, search_since: float | None = None, stage: str | None = None, mode: str | None = None, campaign_id: str | None = None, batch: int = 0, batches_done: int = 0, budget: list[~robovast.client.status.BudgetItem] = <factory>, runs: ~robovast.client.status.RunProgress = <factory>, best_objective: float | None = None, stop: dict | None = None, error: str | None = None, postprocessed: bool = False, postprocessing_error: str | None = None, share_error: str | None = None, results_bytes: int | None = None, extra: dict = <factory>, health: list[~robovast.client.status.HealthFinding] = <factory>, health_skipped: list[str] = <factory>, nodes_skipped: dict = <factory>, updated_at: float = <factory>)
The controller’s live state, served by
GET /campaigns/{id}/status.phaseis an open string the controller advances through a documented vocabulary (initializing→building→starting→variation→running→finishing→importing→postprocessing→sharing→finished/failed);stageandextraexist so future markers (e.g."upload-to-share-done") slot in without a schema change.What is on the share is deliberately not here. It was, as
share_provider, and nothing ever wrote it – which is the shape of the mistake: the share is another system’s state, so a copy of it on a campaign goes stale the first time somebody deletes an archive out of band.list_share_archivesasks the share.Before adding a field: this is a hot fan-out payload. The web UI renders every campaign in the list as a card and each card polls
GET /campaigns/{id}/statusevery 1.5 seconds, so anything here is paid for once per campaign on screen, per poll, per open tab – and some of it is computed per read (health). What belongs here is therefore BOUNDED state and cheap cursors: a phase, a counter, a best-so-far, a batch index. A SERIES does not, however small it looks:batch_historywas one (a row per batch, growing all run long), nothing ever read it, and it was removed in favour ofGET /campaigns/{id}/search/history– a route fetched only while something is displaying it, keyed on thebatches_donecursor that lives here. The rule, and where a live run view lands, is indocs/http_api.rst.- batch: int
- batch_since: float | None
- batches_done: int
- best_objective: float | None
- budget: list[BudgetItem]
- campaign_id: str | None
- error: str | None
- extra: dict
- health: list[HealthFinding]
- health_skipped: list[str]
- mode: str | None
- model_config: ClassVar[ConfigDict] = {'validate_assignment': True}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- nodes_skipped: dict
- phase: str
- phase_since: float
- postprocessed: bool
- postprocessing_error: str | None
- progress_deadline_s: int | None
- progress_since: float
- results_bytes: int | None
- runs: RunProgress
- search_since: float | None
- stage: str | None
- stop: dict | None
- updated_at: float
- waiting_for_capacity: bool
Per-Cluster Resource Resolution
The resolution logic for per-cluster resources (the {context-name: value}
mappings documented under Per-Cluster Resource Limits in
Cluster Execution) lives in robovast.execution.cluster_execution.cluster_context:
Kubernetes context awareness and per-cluster resource resolution.
Resource values in the .vast config file may be given as a per-cluster
list keyed by the real Kubernetes context name instead of a scalar:
resources:
cpu:
- gke_my-project_us-central1_my-cluster: 4
- minikube: 8
memory:
- gke_my-project_us-central1_my-cluster: 10Gi
- minikube: 20Gi
Scalars always work and are the recommended default when a single cluster is
used. Pass the matching context name via --context/-x when running
commands against a specific cluster.
- robovast.execution.cluster_execution.cluster_context.get_active_kube_context() str | None
Return the name of the currently active Kubernetes context.
Reads the active context from the local kubeconfig (
~/.kube/configorKUBECONFIG). ReturnsNonewhen the context cannot be determined (e.g. kubeconfig is absent).
- robovast.execution.cluster_execution.cluster_context.get_config_context_names(config_path: str) set[str]
Extract all context names used in per-cluster resource lists.
Scans a
.vastYAML config for any field that uses the per-cluster list syntax ([{context-name: value}, …]) and returns the union of all keys.- Parameters:
config_path – Absolute path to a
.vastYAML config file.- Returns:
Set of context name strings. Empty when no per-cluster lists are found.
- robovast.execution.cluster_execution.cluster_context.list_all_contexts() list[tuple[str, str]]
List all available
(label, kube_context_name)pairs from the kubeconfig.- Returns:
List of
(label, kube_context_name)tuples sorted by name. Returns an empty list when no kubeconfig is available.
- robovast.execution.cluster_execution.cluster_context.require_context_for_multi_cluster(kube_context: str | None, config_path: str | None = None) None
Raise
ValueErrorwhen a multi-cluster config is used without--context.Scans config_path for per-cluster resource lists and raises an informative error when more than one context name is present and no kube_context was specified.
Which config to scan is the caller’s decision, never discovered here: this used to find a
.robovast_projectby walking up to the filesystem root, so a project far above the CWD could demand--contextfrom a command that was never told about it. There is no ambient project at all now – callers pass the config the operator named with--vast, orNone.This is a no-op when:
kube_context is already set (the user supplied
--context).config_path is
None— nothing named a config, so there is nothing to scan.The config uses only a single context name (or only plain scalars).
- Parameters:
kube_context – The Kubernetes context name (
Nonewhen the user did not pass--context).config_path – The
.vastto scan, orNonewhen none was named.
- Raises:
ValueError – When multiple context names are found and kube_context is
None.
- robovast.execution.cluster_execution.cluster_context.resolve_resource_value(value: Any, context: str | None) Any
Resolve a resource value for the active Kubernetes context.
Handles two forms:
Scalar (
int,float, orstr): returned as-is.Per-cluster list (
[{context-name: value}, …]): the entry whose key matches context is returned.
- Raises:
ValueError – When the value is a per-cluster list but context is
None, or when the context has no entry in the list.- Parameters:
value – Raw resource value (scalar or per-cluster list).
context – Active Kubernetes context name, or
None.
- Returns:
Resolved scalar value, or
Nonewhen value isNone.
- robovast.execution.cluster_execution.cluster_context.resolve_resources(resources: dict, context: str | None) dict
Resolve all resource fields in a resources dict for the active cluster.
Calls
resolve_resource_value()for every key in resources and returns a new dict with all per-cluster lists replaced by their resolved scalar values.- Raises:
ValueError – Propagated from
resolve_resource_value()when a per-cluster list has no entry for context.- Parameters:
resources – Raw resources dict (e.g.
{'cpu': 15}or{'cpu': [{'gke_my-project_…_cluster': 4}, {'minikube': 8}]}).context – Active Kubernetes context name, or
None.
- Returns:
New dict with resolved scalar values (
Noneentries removed).
Job admission internals
User-facing behaviour is in Cluster Execution; this is the machinery behind it.
RoboVAST admits its own jobs. A campaign’s Jobs are created by
AdmissionController,
one at a time, each only once the cluster has room for it — so nothing is ever
submitted to a queue and left to wait. runs_per_job defaults to 1, so a
typical campaign’s plan is upwards of a thousand Jobs, and none is created before
the cluster can hold it.
Why creation is the admission point. A Job’s spec.template is immutable:
patching it is rejected, so the request a Job is created with is permanent and
anything that wants to size a Job per node has to decide before it exists. Holding
the plan in RoboVAST’s own store and creating on admit makes that immutability
irrelevant rather than something to work around.
Ordering is a property of the queue, not of thread scheduling. One global
queue drains in (priority, campaign rank, campaign start) order, and any campaign thread that
wakes drains it — so an older campaign’s later batch still outranks a younger
campaign’s first, which submission order alone would get backwards. A search
submits its batches one after another, so ordering by submission time makes an
older campaign look younger with every batch and the two take turns instead of
the older one finishing.
Capacity, and the double-counting hazard. Free capacity is
allocatable − committed requests − headroom, per node — a pod runs on one
machine, so a cluster-wide figure cannot see fragmentation and admitting against it
is how jobs end up Unschedulable while the cluster reports room. A Job created
moments ago has no pod bound yet, so the next measurement cannot see its requests;
handing the same cores out twice is exactly the over-admission this has to avoid.
The controller keeps an in-flight ledger and stops subtracting a reservation the
instant the measurement starts counting the real pod, which needs no timer and
leaves no window where the same cores are counted twice or not at all.
A pin narrows, and only a probe claims its node. A WorkItem may carry a
pin (the one node id it may use) and an accepts_node gate (whether its owner’s
work may go to a node yet); may_use requires both, and a pinned item never takes a
node without an identity label. Two kinds of work are pinned. A calibration probe is
pinned to the node it measures and, when it does not fit, claims that node for the
rest of the drain pass, so smaller lower-ranked work cannot keep taking the room it is
waiting for. A campaign confined with execution.kubernetes.jobs.node is pinned to
its node with reserves=False: it always has another job queued, so a claim would
renew every pass for its whole life and shut every lower-ranked campaign out of that
node. It waits for room like any other work, and preflight judges it against its own
node’s capacity instead, since no claim stands behind it. The alias is resolved inside
the job node pool before a Job of the campaign exists, and _pin falls back to the
campaign’s node wherever the queue granted none. Postprocessing is not confined.
Waiting for capacity is not a failure, and is not a stall. A job that has not
been created yet is PLANNED in the controller — a fact it holds, not something
inferred from a pod. list_campaign_jobs reports it waiting; the no-progress
deadline is suppressed while a batch is entirely queued, because that deadline is a
per-run budget and no run of the campaign is running. Getting this wrong is what
made a healthy third campaign report as wedged behind two others.
Web UI internals
The web frontend (user guide: Web UI) is a plain Vite + React + TypeScript
app (MUI + TanStack Query) living at
frontend/ui/. It has its own package.json / node_modules and is
independent of the Python build; npm run build emits frontend/ui/dist.
It is deliberately not a plugin framework or a shared UI shell — most of what a bespoke shell would provide maps to stable, widely-used OSS (server-state polling → TanStack Query, charts → Plotly, forms → Monaco/rjsf), so the app is an ordinary React app that reuses those libraries directly.
Thin client of the interface. All service access goes through one seam,
frontend/ui/src/lib/robovastClient.ts, which mirrors
robovast.service.interface.RobovastInterface — the Routes paths and the
request/response models (including Status)
— 1:1, exactly as the Python HTTPTransport does. When the interface changes,
update this file. Pages call the client via TanStack Query (refetchInterval drives
the Monitor’s live polling; mutations drive create/stop). Current pages:
pages/Monitor.tsx (the launch bar merged into it — there is no separate Launcher),
pages/config/ConfigPage.tsx, pages/results/ResultsPage.tsx and
pages/admin/AdminPage.tsx, sharing components/StatusView.tsx for the live
Status render.
Two things on the Admin page are worth knowing before changing them.
NavTopic.footer renders a topic at the foot of the sidebar rather than in the main
list, and Admin is the only user of it: it is about the service, so it belongs beside the
usage meters reporting that same service, and listing it with Campaigns and Results would
rank it as their peer. It is a field on the one TOPICS declaration in App.tsx
rather than a second prop, so navigation still resolves from one source and hashNav
needs no change — a leaf topic already falls out of the existing grammar.
components/LogPanel.tsx is the one live-log renderer, for the campaign log, the job log
and the service log alike. It is its own module rather than part of StatusView.tsx:
reaching it through there would drag BatchObjectiveChart, DetailsBox and the ETA
maths into a lazily-loaded page that needs none of them. That the same component fits all
three is not a coincidence: every live log here is a fetch(offset) -> LogChunk behind
one SSE loop.
A log panel is a tail, and both ends of it are bounded. A campaign’s assembled
infrastructure log reaches tens of megabytes, while the pane it is read through is a few
hundred pixels tall — so the stream serves an over-long frame from its end
(_sse_log_frame_cap in service/app.py, which leaves a note naming vast campaign
log for the whole thing), and the panel keeps only a bounded tail of what it has received
(trimHead). The second bound is not about memory: the body is rebuilt from the whole
buffer on every delta, so an unbounded one re-renders a span per line of the entire log
twice a second. Dropping the head costs the offset protocol nothing — next_offset is
where the log continues, not how much was sent — so a resumed connection still resumes at
the right byte.
An open stream with nothing in it is ambiguous, and the panel must not resolve it by
guessing: these servers flush the response headers before their first pull, and that pull is
network I/O. So useLiveStream reports received — whether a frame has actually
arrived, a delta or the heartbeat that means “read it, there was nothing” — and only
then does the panel call the log empty rather than still loading (logFooter).
The usage recording and the service-log ring both live in the serving layer
(service/app.py and service/service_log.py), not on RobovastInterface. The
precedent is _sse_campaign_list and /healthz: they describe the process that is
serving rather than the campaigns it drives, and there is nothing a transport would
implement differently. The usage ring is per-app (on app.state) so two apps built in one
process record separately; the log ring is process-global, because a logging handler cannot
sensibly be otherwise.
The service serves the SPA. robovast.service.app.build_app() mounts
frontend/ui/dist at / (_mount_ui / _ui_dist) after registering the API
routes, so the interface routes win and the SPA is served from everything else. The
UI is therefore same-origin with the API (no CORS in production) and starts with the
service. The build location is frontend/ui/dist relative to the repo, overridable with
ROBOVAST_UI_DIST; package the built assets into the service/controller image so the
in-cluster service ships the UI. If no build is present the service runs API-only.
Dev loop. cd frontend/ui && npm run dev runs Vite (:5173) and proxies the API path
prefixes to a running vast serve (ROBOVAST_SERVICE_URL to retarget), keeping the
browser same-origin for hot-reload development.
Navigation is the URL hash, and nothing else holds it. frontend/ui/src/lib/hashNav.ts
is the whole grammar — pure, React-free, and the only place a hash is parsed or spelled:
#/<topic> a leaf topic (Config)
#/<topic>/<view>[/<campaign>] a topic with views (Results)
#/results/<view>/<campaign>/<config>[/<run>] ...down to one node of that campaign
#/results/<view>/<campaign>?batch=<i> ...a search campaign's round
#/results/explorer/<campaign>/…?tab=<name> ...and which of its notebook tabs
#/config/campaign/<campaign> the frozen ``_config/`` of one campaign
#/execution?import=<search> the campaign view, share-import dialog on <search>
navFromHash parses into Nav; hashFor spells one back; nextNav is the transition
behind a sidebar click. Three rules live there rather than in a page, and each is pinned by
hashNav.test.ts:
The last form is the odd one out and says so in its own comment: ?import= is a request
carried by a link, not an address. It is parsed for the execution topic only — every
page stays mounted (KeepAlive), so a #/config?import=x that resolved would have the
campaign view open a dialog over a page nobody is looking at, whereas a ?tab= a view
ignores is simply inert. The campaign view clears it once taken, so the address stops
claiming a dialog that has been closed. It is also the one URL form with a reader outside the
browser: vast share import parses the same fragment (share_cli.campaign_from_ui_link),
parse-only, so the copy button produces something that works in both places a campaign is
imported.
The node address is positional, matching the on-disk and
/resultsspelling of a run (One address space for files) rather than inventingcfg/runmarkers.ResultsSelis a discriminated union over the four tree levels, so a config name and a batch index cannot both be held — an impossible URL is not representable rather than forbidden in prose.A view’s hash carries only what that view acts on.
hashForgives the Explorer the whole selection plus its tab, the Run view a run-level selection only (it cannot replay a batch or a config), and the Data browser the campaign alone. A hash is a view’s address, not a store for state the view ignores.``configCampaignId`` is dropped by every transition. A campaign’s frozen config is reachable only from its card, and that guarantee is
nextNavdropping the field — a mistake there does not misplace a view, it exposes one.
App owns the Nav state and writes the hash two ways, and the difference is load-bearing:
select assigns location.hash (pushes a history entry, as do the cross-page jumps in
lib/nav.ts), while setResults uses replaceState — a selection made inside a view must
not cost a Back press. It is guarded on hashFor(next) === hashFor(nav), which is total by
construction and cannot go stale as Nav grows a field.
Turning a selection into the tree’s node id, and back, is resultsTree.ts’s
selectionNodeId / selectionOf, beside the id builders themselves — the URL is a third
builder of node ids, so it must not spell one itself. resolveSelection answers “does this
campaign have this node, and which round is it in” from the rows the tree already loaded; because
a finished campaign’s runs are fixed, it is a useMemo and not a watcher.
Saying something short-lived: useToasts. frontend/ui/src/components/ToastProvider.tsx
is the app-wide way to state a passing fact — mounted once in main.tsx, used as
const { notify } = useToasts(). It is the sibling of DialogProvider: that one asks a
question and blocks on the answer, this one states a fact and gets out of the way. Three rules
travel with it.
Failures do not go through it. The Severity union in lib/toasts.ts has no error
member, which is the rule expressed as a type: a refusal or an error carries backend text worth
reading twice and keeps its inline Alert with ErrorText. Only successes and dispatches
become transient.
It is not a notifier. The provider draws a rectangle and nothing else. A caller that also
wants an OS-level notification — the campaign lifecycle watcher in CampaignStreamProvider,
and the upgrade handover in AdminPage — calls lib/browserNotify.post itself, beside its
notify. That keeps the Notification API out of a React component, and out of a flag threaded
through the queue that only some callers would set. post enforces its own preconditions,
including that the tab is actually hidden, so a later caller cannot forget them.
Both callers have the same shape, which is what the sink is for: work that outlives the view
that started it. A campaign runs on the service; a service roll keeps polling from a
KeepAlive-mounted Admin page. In each case the thing that finishes has no reason to expect
the user to still be looking at the page that started it.
The opt-in is asked once, in context. components/NotificationAsk.tsx is a banner above
every view, raised the first time useCampaignStream shows a running campaign and
browserNotify.shouldAsk() is still true — a permission ask needs a reason on screen, and
arriving is not one. It is deliberately not a toast: a toast clears itself after ten seconds by
contract, and an ask nobody was looking at would be spent for nothing. Either button stores an
answer (setOptedIn), which is what makes the key in lib/browserNotify.ts three-state —
absent means never asked, and a decline that stored nothing would be re-asked on every
campaign. The standing switch lives in AdminPage’s This browser paper; it is kept out of
ServiceConfigPanel on purpose, since that panel reports one shared service environment and
this preference is per browser on an instance several people share.
The prompt itself can only be raised from a real click: Firefox and Safari resolve a
gesture-less requestPermission() to default without ever showing it. That constraint is
what shapes both surfaces — the banner exists to produce the click, and the Admin control is a
switch rather than a setting read at start-up.
It is not a route to ntfy. Campaign lifecycle already reaches a phone from the server
(robovast.execution.notify), once per campaign. Fanning out from the browser would send one
push per open tab and would put the ntfy token in the client.
The queue itself (lib/toasts.ts) and the phase diff behind the lifecycle notices
(lib/campaignEvents.ts) are pure and unit-tested; the provider around them is not. Note
that remote panels (frontend/panel-kit/) share only react/react-dom with the host,
so a federated panel cannot reach this context — one that needs it must be handed a builtin.
Frontend tests stay minimal. npm run test runs vitest over the pure modules in frontend/ui/src/lib/ — and nothing else.
The checks that matter most here are still npm run lint (tsc --noEmit) and
npm run build; a spec is the exception, not the habit.
A spec earns its place only where the logic is non-obvious and regression-prone and
checking it by hand means clicking through the UI: node-id and route stability, grouping,
rollups, formatting. src/lib/resultsTree.test.ts is the model — the Results tree’s node
ids are a contract between two components that build them independently (the tree renders
them, the Run view reconstructs one to highlight the current run), so a drift shows up only
as “the picker stopped opening on the selected run”, which no compiler catches.
What not to do, because each of these costs far more than it protects here:
no component or DOM tests, no
jsdom, no@testing-library, no snapshots of rendered output — the UI is a thin client ofRobovastInterface, so behaviour belongs in the Python tests, which is where it is enforced;no network mocking — if a test needs the service, it is testing the service;
nothing
tscalready proves (a required field being present, a union being exhaustive);no test framework stack beyond vitest’s defaults over the existing
vite.config.ts(which is also where the@alias comes from, so a spec needs no path config of its own).
Note that CI does not build frontend/ui at all, so these run locally and in agent
sessions rather than as a gate — another reason to keep them few and fast. They are npm run
test inside frontend/ui; the Python suite is make test at the repo root (or make
test-<subtree>, e.g. make test-service, to re-run only what you are editing). CI calls
make test too, so the command a developer runs and the one that gates a merge cannot
drift apart.
The Python suite needs a Postgres, because campaign results live in one, and it provides its
own rather than asking you for one: a container named robovast-test-pg (see
tests/pg_provision), started on the first run that finds it absent and left running
afterwards so later runs cost a CREATE DATABASE instead of a container start. Each
session works in its own database inside it, so two suites can run at once, and drops it on
the way out. export ROBOVAST_TEST_PG_DSN=... to use a server you already have instead,
and docker rm -f robovast-test-pg to take the suite’s back. Without a Docker daemon and
without that variable, the tests that need a database skip and say so in their reason.
One colour scheme. Every colour the UI paints that is not a one-off comes from
frontend/ui/src/colors.ts. A Style object holds them; buildTheme(style) in
src/theme.ts maps one onto MUI’s palette roles, so a component reaching for
'primary.main' or 'error.main' resolves there. The module keeps four groups apart
deliberately, because which group a colour is in decides whether it may move when the brand does:
surface — the ground, a neutral dark grey, and deliberately not the marketing site’s near-black green. This is a tool that stays open for hours in front of long log tails, 3D scenes and charts, and on a tinted ground every one of those picks up the tint; a colourless ground leaves the accent and the status colours as the only hues on screen, which is what makes them read as meaning something;
brand — the accent (mint), transcribed from the site’s CSS custom properties rather than imported (it is a static page with no build step and no package to depend on, and a shared palette package would couple a deployed service’s bundle to a website). The favicon,
frontend/ui/public/favicon.svg, is the same mark in the same mint;status — phase chips, capacity meters, scenario-tree nodes, log severities. Semantics, not identity: they must stay put when the brand moves, or a re-brand silently re-labels every campaign. Held to one lightness band so no single status shouts and all of them sit clearly below the accent, with the hues kept meaningful (green good, amber attention, rose bad, blue working). The site has no equivalent, having no notion of a run that failed;
data — the categorical series scale, shared by the Vega eval charts, the run-view time series and the variation previews so a series keeps its colour between them. Kept in a brighter register than status on purpose: status is painted as filled chips and bars where a deep tone reads well, while a series is a 1px line that has to survive being thin. An encoding, so its order is load-bearing — append rather than reorder.
One thing is pointedly not in the status group: src/components/runLog/ansi.tsx maps ANSI
SGR codes to colours, and those describe what the program printed. Harmonising them into the
scheme would misreport a tool’s own output, so that table keeps its own values.
A second style is a new object, not a refactor. ACTIVE names the one style there is
(midnightMint); implementing Style and passing it to buildTheme is the whole of adding
another. What is deliberately not built is runtime switching: the flat tokens are module
constants resolved at build time, which is what lets a canvas-painting panel read a colour without
a hook. Switching while the app runs needs those panels to read from useTheme() instead, and
every token a panel uses to live on the theme — that is the entire delta, and nothing in the
current shape blocks it.
Import a token instead of writing a hex. Before this module the surfaces were spread across the theme and two components, the panel-canvas colour was copied into six files, the series scale existed twice — with a comment on one copy asking the other to please stay in sync — and the scenario tree and the log view each had their own private green and red.
Extending. Add an operation by giving robovastClient.ts a method mirroring the
new interface op, then a page/tab that queries it.
Resource usage (``/usage``). resource_usage is a backend-agnostic interface op
returning ResourceUsage (CPU cores + memory bytes,
capacity vs. used, and a parallel_runs flag). How the service measures lives entirely in
its _compute_resource_usage: ClusterService sums node allocatable (capacity,
reusing kube_client.parse_resource) and the requests of the non-terminal pods bound
to those same nodes (used) — so callers (the top-bar chip, the resource_usage MCP
tool) never branch on backend. Summing both sides over one node set is what keeps used <=
capacity: a pod still queued for a node requests cores nothing has granted, and counting
it reported “29.7/24” on a 24-core cluster. Pending work is jobs_pending, not usage.
That scenario-run tally is the other half of the op, and it is counted from Jobs, not
pods: running = executing, pending = accepted but not executing. A Job waiting its
turn has no pod at all — the state every cluster batch starts in, because the admission
queue has not created it yet — so a pod-based count reports a freshly launched sweep as
0/0 with its whole queue waiting for capacity.
ClusterService._scenario_job_tally therefore delegates to
cluster_execution.list_jobs_with_phase (the single place Jobs + pods become a phase, so no
consumer re-derives it and drifts) and folds waiting/blocked into pending, namespace-
scoped because it answers “what is this service running” while CPU/memory must stay cluster-wide.
The UI’s sidebar meter reads running/(running+pending) and hides itself when both are zero.
The reading goes through one TTL-cached path on the base class
(ServiceBase.resource_usage memoizes for _USAGE_CACHE_TTL under a lock), so many
polling clients cost one sampling per window.
The cluster read needs cluster-scoped RBAC (nodes are not namespaced): setup grants the
service ServiceAccount a read-only ClusterRole over nodes/pods
(service_deploy._service_rbac_manifests), so upgrading an already-deployed service to
this version requires setting it up again (vast cluster cleanup then setup,
or setup --force) to add the grant.
Config editor (frontend/ui/src/pages/config/) is where a .vast is authored:
a Monaco editor (frontend/ui/src/lib/monaco.ts bundles the editor + YAML workers and, via
monaco-yaml, drives completion/inline-validation from get_config_schema()). It edits a
workspace’s .vast (workspace = the project, since a browser has no CWD), autosaves +
debounce-validates through validate_project, and previews resolved configurations through
preview_configurations. These four ops — validate_project / preview_configurations /
get_config_schema / list_variation_types — live on RobovastInterface, so
every client reaches them; ServiceBase wraps validate_project_file /
generate_scenario_variations / ConfigV1.model_json_schema() / the
robovast.variation_types entry points, and validate_project swallows unexpected errors into
a structured problem so the editor’s live validation never 500s on in-progress YAML.
Per-variation previews. preview_configurations returns, per configuration, a previews
list ({variation_type, params, remote}) read from the config’s declared variations. The
editor (frontend/ui/src/preview/) renders built-in variation types host-native — builtins.tsx
maps a variation-type name to a Plotly component (distribution curves for
ParameterVariationDistribution*, a value list for ParameterVariationList). An external
variation plugin may instead ship a web preview: it sets Variation.WEB_PREVIEW to a
package-relative dir holding a built Module Federation remote (remoteEntry.js exposing a
./preview React component); the service serves it at
GET /variation_types/{name}/assets/{path} and preview_configurations fills in the remote
descriptor, which RemotePreview.tsx loads at runtime (sharing the host’s React singletons).
Built-ins ship no assets; a type with neither shows just the resolved parameters.
Results viewer (frontend/ui/src/pages/results/): pick a
campaign, browse its results schema, run read-only SQL, and chart the result with
Vega-Lite (frontend/ui/src/preview/VegaLiteChart.tsx — rows bound in as data.values).
The two data-query ops — describe_campaign_data / query_campaign_data_sql — were
promoted onto RobovastInterface; the actual SQL lives in one shared, directory-based
helper, robovast.results_processing.data_query (mode=ro + a sqlite3 authorizer,
campaign.db attached as schema campaign). Both callers reuse it: the service methods
resolve the campaign dir the same way (campaign_dir under the results root), and the
MCP run_data plugin resolves it via
results_resolver or delegates to a configured service — so CLI, MCP, and the web UI query
results identically, local or cluster. User-declared plots (visualization.results.data_browser.plots in the .vast,
robovast.common.config.PlotSpec) are surfaced by list_campaign_plots and rendered by
the same Vega-Lite component.
Run view panel framework (frontend/ui/src/lib/dashboard/) — the Run view (user guide:
Web UI) is a small plugin framework with three deliberate seams, all designed so a
future live view (watching a running system instead of replaying a rosbag) slots in
without touching any panel:
Panels are plugins with two delivery mechanisms behind one contract. A
PanelPlugin= manifest (type name + layout defaults) + React component implementingPanelProps({spec, clock, data}). Core built-in panels self-register viaregisterPanel(registry.ts) by importingfrontend/ui/src/panels/run_view/index.ts. Remote panels — package-provided or user-authored — are loaded at runtime as Module-Federation remotes (see below) and never touch the static registry. The.vast’svisualization.results.run_view.panelsspecs are normalized byparsePanels.ts(single-key shorthand →{type, ...fields}; the same shorthand is accepted by the Pydantic schema, seePanelConfig._flatten_shorthand). Validtypevalues are the core built-ins (BUILTIN_PANEL_TYPES) ∪ installedrobovast.panel_typesentry-point names ∪custom— validated inPanelConfig._known_typeinrobovast.common.config(so a package panel is valid only when its plugin is installed). Adding a core built-in is still one file + oneBUILTIN_PANEL_TYPESentry.Two surfaces, one framework. The Config tab’s column (The config view) uses the same registry factory, the same spec parsing and the same panel frame; what differs is the layout grammar (one column, so a panel declares only a
height—ColumnHost.tsx, notPanelHost.tsx) and the props (ConfigPanelProps: a resolved configuration, and no clock). A config panel lives infrontend/ui/src/panels/config/, registers withregisterConfigPanel, and its type joinsBUILTIN_CONFIG_PANEL_TYPES. Package-provided config panels use the samerobovast.panel_typesentry-point group and the same asset route; the panel class’sSURFACE("run"by default,"config"otherwise) is what tells them apart, so naming a run panel invisualization.config.panelsis refused – naming the surface the panel does belong to and the block to declare it under, rather than reporting it as unknown. A config panel must setSURFACE: the default is"run", so a class that forgets it registers on the wrong surface and its own campaigns then read as “unknown config-view panel type”.A panel type may also declare what a
.vastmay say to it, asCONFIG_CLASS– the same attribute a variation type uses, soget_plugin_detailsdescribes a panel’s fields without knowing anything about panels, and the editor completes them from the served schema. Fields take aBinding: a literal,{param: ...}(a scenario parameter),{internal: ...}(a key a variation left on the configuration) or{role: ...}(a contributed file), resolved by one function inpanel-kit(resolveBinding) so no panel parses its own bindings. Declaring the model is opt-in: a type that declares none keeps the free-formextrarule, which is what lets the run view’svega_lite/layers/seriesbindings stay free-form. A type that declares one gets its keys checked, and a misspelled binding becomes an error naming the valid fields instead of a panel that silently draws nothing.A config panel that renders a contributed file may also draw it without a browser, for the
draw_configMCP tool: it declaresFILE_ROLE(thecontribution.filesrole it reads) andplot(ax, path, projection, read), which draws the file onto a matplotlibAxes.read(path)returns a campaign file’s bytes by its campaign-relative path, so a format that names sibling files (a map YAML and its image) fetches them the same way.plotreturnsFalsefor a projection it does not draw and raises when the file cannot be drawn; it imports matplotlib inside the function, so the panel module still imports without it. The package that renders a format in the browser is then the one that renders it here, and core never learns the format. Reference:Map2DPanelTypeinrobovast_nav/panels.py.Data comes only through
DataProvider(dataProvider.ts): rows by table+time, nearest-sample lookups, a generic run-scopedfetchRun(endpoint, params)(GET a campaign endpoint withconfig_name``+``run_idapplied — how a panel reaches a specialized endpoint without the generic seam knowing about it, e.g. the costmap panel’s nav grids), andrunFileUrlfor per-run artifact files — today implemented over the read-only data-query endpoints (dbDataProvider); a live implementation would wrap a rosbridge buffer.timeSeries.tswraps one table as a time-indexedTimeSeriesSource(at(t)/upTo(t)).Time comes only from the shared
PlaybackClock(clock.ts), an external store (not React state) so display-rate updates don’t re-render the tree; canvas panels subscribe imperatively viauseCanvasClock.
Package-provided & user-authored panels (Module Federation). A remote panel is loaded
at runtime by exactly the seam the variation-preview path uses (above) — the two now share
their machinery. Server side, _resolve_plugin_asset(group, name, rel, asset_attr) (in
service/app.py) and _plugin_remotes(group, asset_attr, url_builder, module_attr) (in
service/client.py) are the generalized forms of the old variation-only helpers;
_variation_remotes and _panel_remotes are thin wrappers. A package panel is a
class in the robovast.panel_types entry-point group declaring WEB_PANEL (its built
bundle dir, shipped as package data) + PANEL_MODULE; the service serves it at
GET /panel_types/{name}/assets/{path} and list_campaign_panels attaches its remote
descriptor. A user-authored custom panel names a bundle path relative to the .vast;
_collect_analysis_input_files stages the bundle into the campaign’s _config/ and it is
served at GET /campaigns/{id}/panel_assets/{path} (resolve_campaign_panel_asset,
path-confined). Both descriptors are {name, remote_entry_url, module}; on the client,
useRemoteComponent (frontend/ui/src/lib/remote.ts, factored out of RemotePreview.tsx) loads
the module — seeding the host’s React singletons — and PanelHost mounts it with the full
PanelProps. Reference example: the costmap panel, relocated out of the core UI into
robovast_nav — Python descriptor in robovast_nav/panels.py, build sources in
src/robovast_nav/web (a @module-federation/vite remote exposing ./costmap,
built into robovast_nav/web/dist). It is the first real remote, and validates the
variation-preview loading path too. Its data endpoint is likewise package-provided —
see Package-provided service data endpoints below — so both halves of the costmap panel live in
robovast_nav.
Write a remote against frontend/panel-kit (@robovast/panel-kit), never against a copy
of the host’s types. Sharing only react/react-dom at runtime is a runtime constraint, not
a source one: the contract (PanelProps, DataProvider, PlaybackClock) and the
clock-driven scaffolding (useCanvasClock, the time-index binary search, keyframes for
samples too large to preload) are in-tree source that both the host and each remote compile into
their own bundle. Resolve it with a tsconfig paths entry plus a matching vite
resolve.alias — see src/robovast_nav/web for the two lines. Do not add it to the MF
shared map: bundling a private copy is what keeps a version skew between an installed package
and a newer host UI an ordinary build rather than a remote-load failure. The earlier arrangement — a
hand-maintained contract.ts mirror plus a re-implementation of the host’s canvas/clock
scaffolding — is what let a fetch-staleness bug exist in the costmap panel and nowhere else.
A panel type may also declare an optional REMOTE_NAME — the Module-Federation container
name, defaulting to the entry-point name (one container per type). Panels that share a single
built bundle set the same REMOTE_NAME so _plugin_remotes emits that shared name while each
type keeps its own asset URL (/panel_types/<name>/assets/... all resolve to the one bundle).
robovast_nav uses this to host its panels in one robovast_nav container (see
robovast_nav/web/vite.config.ts), sharing React/vendor chunks.
Nav2 behavior tree (when a package should ship no panel at all). nav2’s internal behavior
tree is shown in the run view, and robovast_nav contributes none of the rendering. nav2’s
/behavior_tree_log is the only generic, always-on source of BT state, but it is
topology-free — a flat stream of status transitions keyed by node_name. The data half
splits across the two extension seams the same way costmap’s does: the core rosbag handler
nav2_bt_to_csv (rosbags_process.py; core because HANDLER_REGISTRY isn’t
plugin-extensible and it needs the in-container rosbags_process step) writes the raw
nav2_behavior_tree transitions table, and robovast_nav’s nav2_bt_tree postprocessing
plugin (a robovast.postprocessing_commands entry point) reconstructs structure from the BT
XML nav2 ran and joins it into a nav2_behaviors table.
That table uses the same schema as scenario_execution’s behaviors table, and that is the
whole point: the built-in scenario_tree panel renders any tree expressible in it. So
robovast_nav ships the nav2_behavior_tree type but no renderer — its panel module is a
handful of lines that render the built-in one with nav2’s table, title and empty-state hint.
Deriving a panel is what makes that possible. PanelProps.builtins carries the host’s own
panel components to a remote (PanelHost’s RemotePanel reads them from the registry at
mount), because a Module-Federation remote shares only react/react-dom and cannot import
the host’s modules. A derived panel inherits every later improvement to the built-in — child
ordering, node-kind glyphs, feedback, source lines, scrolling — instead of needing them
implemented twice.
The package did once ship a full port of the built-in panel in plain React; it was deleted for
exactly that reason. Deleting the type along with it was a mistake worth recording: a type name
is not a duplicate. Configs name it (frozen _config copies of past campaigns included), and
PanelConfig._known_type validates against installed entry points, so removing the entry point
turned every one of those configs into a validation error and an “Unknown panel type” box.
The rule, then: a package needs a renderer only when the host cannot draw its data at all.
costmap qualifies — binary grids need their own service endpoint. A package whose data is a
table in an existing schema still gets a type of its own, but derives the panel.
Package-provided service data endpoints (robovast.service_endpoints,
robovast.service.endpoint_plugin) — an installed package contributes a run-scoped data
endpoint served at GET /campaigns/{id}/<name>?config_name=…&run_id=…&… → JSON, with no core
edit and no frontend change (the run view already reaches any such endpoint via
data.fetchRun(name, params), frontend/ui/src/lib/dashboard/dataProvider.ts). This closes the last
core-coupling for a self-contained analysis package: it ships a postprocessing plugin (writes an
index table), a service endpoint (serves it), and a panel (renders it) — all via entry
points. The mechanism mirrors the MCP-plugin loader: a ServiceEndpoint Protocol
(name + handle(ctx)) and load_service_endpoints(); build_app registers one route per
plugin (before the SPA mount) and dispatches to handle with a ``RunDataContext`` facade —
ctx.open_db() (a read-only connection to the central index, from the public
data_query.open_data_db),
ctx.run_dir(config, run), ctx.params. Handlers raise KeyError/ValueError/
DataQueryError → 404/400 via the shared _guard. Cluster-transparent because dispatch
resolves the campaign dir through the public impl.campaign_dir(campaign_id) seam, which is
a directory under the results root — so a plugin endpoint works on every deployment
unchanged. Endpoint names should be package-namespaced (nav/foo) to avoid
collisions; core route names are reserved (RESERVED_CAMPAIGN_ENDPOINTS). Scope: run-scoped
GET→JSON only — binary/large per-run artifacts are already served by the file address space,
GET /results/<campaign>/<config>/<run>/<path> (DataProvider.runFileUrl), and
producing data is a postprocessing plugin’s job. Reference: robovast_nav’s
CostmapEndpoint (robovast_nav/service_endpoints.py), relocated verbatim from core’s old
read_costmap_frame — it reads the costmaps table via ctx.open_db() and returns the frame
dict the costmap panel decodes.
3D scene viewer core (frontend/ui/src/lib/scene3d/) — renders the browser scene
descriptor (scene.json/scene.bin), which a simulator backend’s exporter produces;
the format is RoboVAST’s, see Replay artifacts: the scene and the capture. sceneLoader.ts
builds a three.js Group and returns an imperative animation API (jointMap /
basePose); viewport.ts is a plain-three viewport (renderer/camera/lights/grid/orbit
controls + the Z-up wrapper). The wheel is the viewport’s own, not the orbit controller’s
(cursorDolly.ts): scaling the orbit radius toward a fixed pivot makes the travel a geometric
series converging on that pivot, so a notch moves millimeters once you are close and the pivot can
never be passed through — flying the eye and the pivot along the cursor ray keeps the radius, and
with it the step size, constant. The same change makes a fixed far plane visible, so viewport.ts
sizes the frustum each frame to enclose the world’s bounding sphere — measured from the scene, not
from the pivot, which the wheel now carries along and which is therefore constant by design.
Extractability rule: files in this directory import only
``three`` — never ``@/…`` (see its README) — it is shared-candidate code, so all
robovast-specific wiring lives in the consumer, frontend/ui/src/panels/Scene3DPanel.tsx, which
binds the vast spec, fetches the descriptor via DataProvider.runFileUrl
(GET /results/<campaign>/<config>/<run>/<path> — the address is the run’s real
directory, so the loader’s relative sibling fetches, scene.bin/textures, stay in
it), and drives basePose from the clock.
Deferred: a bundle code-split (Monaco + Plotly + Vega + Module Federation make the SPA large).
Analysis postprocessing: local vs cluster
The rows query_campaign_data_sql reads are produced by analysis postprocessing. It splits
along a natural seam: only the batched ``rosbags_*`` → CSV step needs ROS2; everything after it
(the index ingest, metadata) is plain Python.
What the index ingest reads. One table per data file, discovered per run directory:
*.csv, and *.jsonl for producers that need more than a flat table can express. A JSONL
file declares its layout in the format key of its first record, and _JSONL_READERS
maps that to the function turning its records into rows — dispatch on the file’s own declaration
rather than on its name, so adding a producer does not mean hardcoding a filename in the ingest.
Rows are then typed and inserted through exactly the same path as a CSV’s, which is why column
types are inferred per column and a JSONL file whose records have differing keys still gets a
column for each (the column list is the union over rows, not the first row’s keys).
The one such producer today is behaviors.jsonl, written by scenario_execution’s
--bt-log, on for every run. Because the runner writes the file itself — rather than the table
being converted from a recorded /scenario_execution/snapshots topic, which only works for ROS
runs and would leave mode: base campaigns with no behaviour-tree data at all — both kinds of
run produce the behaviors table by the same path, and the snapshots topic need not be in the
bag. Its first seven columns are the ones nav2_behaviors (see above) shares, so the same panel
renders both; to them it adds what the JSONL carries: child_index, type,
additional_detail, feedback_message, is_active, tip_id, osc_file,
osc_line, osc_column and removed. The numeric status column is re-derived in the
reader from the status name, because those 1–4 codes come from py_trees_ros_interfaces — plain
py_trees Status values are strings.
Both backends run the rosbag conversion in the campaign’s own execution image — the
system-under-test’s image, recorded in <campaign>/_execution/execution.yaml — because rosbags
carry the SUT’s custom ROS2 message types and only deserialize there. On the cluster backend the
image is pinned to the immutable digest the run pods used (captured from the pods and stored as
image_revision; campaign_execution_image prefers it), so a re-postprocess months later
deserializes against the exact image the runs recorded the bags with — not whatever a floating
:latest resolves to then.
Local. run_postprocessing reads that image: and passes it to
docker_exec.sh --image, which bind-mounts the campaign dir (-v $INPUT_DIR:/input) — so the
container writes CSVs in place. Nothing to sync. The scripts are bind-mounted from the
driver’s own package (-v $SCRIPT_DIR:/scripts:ro, $SCRIPT_DIR =
robovast.results_processing.data), so the script version always matches the driver.
Cluster. A pod cannot bind-mount the caller’s filesystem, so all of postprocessing runs
as one Job (robovast.execution.cluster_execution.postprocess_job), with one copy of the
campaign in it. A campaign emptyDir mounted at /campaign in every container carries
the tree, and Kubernetes alone orders the containers — initContainers run sequentially to
completion in declaration order:
stage(initContainer, sidecar image) fetches the campaign as one tar stream from the service’s data plane —curl | tarofGET /data/campaigns/<id>/archive, narrowed bystage_include()— and lands it on that mount. Nothing of ours sits between the socket and the disk, so its memory is the pipe. Calibration probes and the log this Job will write are never staged, and where no conversion is configured neither are the rosbags — nothing else in the pod opens one.convert(initContainer, the campaign’s execution image) runs every step that needs that image — therosbags_*→ CSV conversion and any otherExecutionImagePlugin— in order, each as its plugin’s ownimage_command, against the same mount. It touches a marker beside the campaign first; every file changed after it is this container’s output. It exists only where the campaign has such a step. It exits 0 whatever its steps returned and hands their status tohostin a file beside the marker: a failed initContainer starts nothing after it, so a step that refused a few bags would otherwise take every converted one down with the pod.hostthen runs, delivers, and fails the Job with its own exit code3, which the failure reason names as the conversion’s.host(container, controller image,postprocess_host) runs everything else — the other plugins, the index ingest and metadata — and is what delivers: one tar of what the Job derived,PUTto the data plane, which writes it into the campaign’s directory on the results volume. It is the only container given the campaign’s token.
The conversion container holds no credentials of any kind — not the campaign’s data-plane token, not the index DSN: it is an arbitrary user image and reads and writes the shared mount only. Further:
image = the campaign’s execution image (never a default — a missing
image:is an error, not a silent wrong-image conversion);the conversion scripts, and the files each image step ships, are delivered as a per-campaign ConfigMap built from the driver’s own
robovast.results_processing.dataand mounted read-only at/scripts— the K8s analog of the local-v $SCRIPT_DIR:/scripts:robind-mount. This is deliberate: sourcing the scripts from the driver rather than from a separately-versioned controller image guarantees the in-cluster scripts match the driver that generates the conversion command, so an off-cluster/dev driver running ahead of a published image cannot skew them (a skew surfaces as a spurious--output-rooterror). The scripts are self-contained (stdlib + ROS2 libs, norobovastimport) and small; nothing is ever baked into the user’s image;each image step runs over the shared campaign tree itself, so every output lands at its campaign-relative path and goes to its canonical key with no mapping step.
The host stage is pure Python and reuses the normal pipeline with the image steps skipped
(run_host_postprocessing), so there is no second copy of the postprocessing sequence — the
same function vast results postprocess calls, run beside the data instead of fetching it.
Two entry points share that one implementation (postprocess_campaign):
auto-chain — the campaign driver, when
create_campaign(postprocess=True)setsRunOptions.postprocess(an option, not a process-global env var, so concurrent campaigns in the one service process stay distinct). It runs after ``store.close()`` (campaign.dbmust be flushed — the index’srunstable is built from it) and before ``_finalize``, so the results ride the campaign’s existing upload rather than needing one of their own.explicit re-run —
run_postprocessing, the cluster’s body for the interface operation: the service has no ROS runtime, so it submits the Job, whose pod is what holds the campaign, and materialises only the few small status objects it edits and publishes back — before and again after the pod, since the pod is what wrote the provenance the reconstruction reads. This backs the web Retrigger postprocessing dialog, the MCPrun_postprocessingtool, and the CLI. It reads the campaign’s own_config/<name>.vast(which the edit dialog overwrites in place — the single source of truth resolved bycommon.results_utils.campaign_vast), refreshes the durable outcome (clearing/settingpostprocessing_error), and is dispatched in the background viaServiceBase._dispatch_background, which registers a tracked campaign entry set to thepostprocessingphase (busy-guarded against a second concurrent op) and returns at once, so the campaign view shows it live. A minutes-to-hours re-run therefore never blocks the caller.
The upload-to-share step mirrors this: a failure records share_error (durable) instead of
being swallowed, and run_share() re-triggers it
(web Retrigger upload-to-share, MCP run_share, POST /campaigns/{id}/share/run) — also via
_dispatch_background (sharing phase). Both re-triggers need no live in-memory campaign entry,
so they work after a service restart.
A post-run step failure is deliberately not a campaign failure: the phase stays finished and
the reason lives on postprocessing_error / share_error. After a restart the cluster service
reconstructs a campaign’s status from the _execution/outcome.json in its directory, falling
back to the on-disk run artifacts (reconstruct_status_from_disk) when no outcome was recorded — so a
finished campaign reads as finished, never a bare unknown.
Querying RoboVAST campaigns
Using [rdflib](https://rdflib.readthedocs.io/), you can query the generated metadata graph using [SPARQL](https://www.w3.org/TR/sparql11-query/).
Load the metadata graph
from rdflib import Graph
g = Graph()
g.parse("metadata.prov.json)
Loading SPARQL queries
To do so, you can load any of the queries below as text, and use the query method for any graph g:
with open("query-file.rq", "r") as f:
query_string = f.read()
qres = g.query(query_string)
for row in qres:
# Process your results
print(row)
Below are a few example queries demonstrating the PROV relationships in the metadata graph.
Scenario inputs
FloorPlan models:
SELECT ?floorplan ?creator ?date
WHERE {
?floorplan rdf:type env:FloorPlanModel .
OPTIONAL {?floorplan prov:wasAttributedTo ?creator .}
OPTIONAL {?floorplan dcterms:modified ?date .}
}
Vast file:
SELECT ?vast_file ?creator ?date ?abstract_scenario
WHERE {
?vast_file rdf:type robovast:VastConfiguration .
?vast_file dcterms:references ?abstract_scenario .
?abstract_scenario rdf:type scenarios:AbstractScenario .
OPTIONAL {?vast_file prov:wasAttributedTo ?creator .}
OPTIONAL {?vast_file dcterms:modified ?date .}
}
Generation
FloorPlan Model-to-Model Transformation:
SELECT ?floorplan ?activity ?jsonld_file ?agent
WHERE {
?floorplan rdf:type env:FloorPlanModel .
?activity rdf:type robovast:FloorPlanTransformation .
?activity prov:used ?floorplan .
?jsonld_file prov:wasGeneratedBy ?activity .
OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
}
FloorPlan Artefact Generation:
SELECT ?source_files ?activity ?gen_file ?agent
WHERE {
?activity rdf:type robovast:FloorPlanGeneration .
?activity prov:used ?source_files .
?gen_file prov:wasGeneratedBy ?activity .
OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
}
Generation of Concrete Scenario
SELECT ?vast_file ?ref_file ?activity ?agent ?gen_file
WHERE {
?vast_file rdf:type robovast:VastConfiguration .
?activity prov:used ?vast_file .
?vast_file dcterms:references ?ref_file .
?gen_file prov:wasGeneratedBy ?activity .
OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
Test Execution
Test results generated from a test run:
SELECT ?scenario ?config_files ?activity ?agent ?gen_file ?start_time ?end_time
WHERE {
?scenario rdf:type smm:ConcreteScenario .
?activity prov:used ?scenario .
?activity prov:used ?config_files .
OPTIONAL{?activity prov:startedAtTime ?start_time .}
OPTIONAL{ ?activity prov:endedAtTime ?end_time . }
?gen_file prov:wasGeneratedBy ?activity .
OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
Postprocessing
Postprocessing of a bagfile:
SELECT ?bag_file ?activity ?agent ?gen_file ?start_time ?end_time
WHERE {
?bag_file rdf:type robovast:ROSBag .
?activity prov:used ?bag_file .
?gen_file prov:wasGeneratedBy ?activity .
OPTIONAL {?activity prov:wasAssociatedWith ?agent .}
OPTIONAL{?activity prov:startedAtTime ?start_time .}
OPTIONAL{ ?activity prov:endedAtTime ?end_time . }
}
Metadata and Graph postprocessing:
SELECT ?metadata_file ?graph_file ?md_activity ?graph_activity ?agent ?start_time ?end_time
WHERE {
?md_activity rdf:type robovast:PostprocessingMetadata .
?graph_activity rdf:type robovast:PostprocessingGraph .
?metadata_file prov:wasGeneratedBy ?md_activity .
?graph_file prov:wasGeneratedBy ?graph_activity .
?graph_activity prov:used ?metadata_file
OPTIONAL {?md_activity prov:wasAssociatedWith ?agent .
?graph_activity prov:wasAssociatedWith ?agent .}
OPTIONAL{?md_activity prov:startedAtTime ?start_time .}
OPTIONAL{ ?md_activity prov:endedAtTime ?end_time . }
}
Analysis
Identifying which variation types were used on each config:
SELECT ?config ?variation_type
WHERE {
?config rdf:type smm:ConcreteScenario .
?config robovast:variations/rdf:rest*/rdf:first ?variation .
?variation rdf:type ?variation_type .
FILTER (?variation_type != prov:Activity)
FILTER (?variation_type != robovast:Variation)
}
Getting the failure rate by environment:
SELECT ?env_model (SUM(?failures)/COUNT (?activity) * 100 AS ?result) (COUNT (?activity) AS ?total)
WHERE {
?conf rdf:type smm:ConcreteScenario .
?activity prov:used ?conf .
?activity rdf:type robovast:TestExecution .
?activity robovast:success ?success .
BIND(IF(?success=true, 0, 1) AS ?failures) .
?conf dcterms:references ?env_model .
?env_model rdf:type env:FloorPlanModel .
} GROUP BY ?env_model