Client–Server Architecture
RoboVAST’s operations are exposed behind one interface, implemented once
server-side, so the vast CLI, the MCP server, and the web UI
are all thin clients of the same contract. This lets an LLM (or any client)
author, run, re-postprocess, and reason over large-scale simulation campaigns
through an interface that can live on a different host from the client.
One interface, one implementation, many clients
There is a single operation contract,
robovast.service.interface.RobovastInterface, mirrored 1:1 in three
bindings:
the service HTTP endpoints (
robovast.service.app, FastAPI, OpenAPI at/docs);the client
robovast.service.client.RobovastClient;the MCP tools and ``vast`` CLI commands, which wrap the client.
Campaign status reuses robovast.client.status.Status verbatim — the live state
the campaign driver publishes — so every client reads one status vocabulary regardless of
where the campaign runs.
Four distributions, layered by audience
The contract above is one thing; what you have to install to use it is another. RoboVAST ships as separate distributions so each audience installs what it needs and — the part that matters — can decline what it does not.
Distribution |
Contains |
Adds |
|---|---|---|
|
the |
|
|
service core, config and variation, results, the MCP server, the campaign controller |
no Kubernetes client |
|
the service implementation (Kubernetes), its cluster-config plugins, the deploy and operator commands |
|
|
navigation variation types and panels; the roqsim simulator backend |
per package |
The dependency direction is the design. robovast-client depends on nothing of
ours, robovast depends on it, and robovast-cluster depends on robovast.
robovast must never depend on robovast-cluster — that edge back is what would make
the graph cyclic, and it is why it cannot be offered as an extra. Anywhere that installs it
needs its own step.
They share one import namespace: robovast/, robovast/service/ and
robovast/execution/ carry no __init__.py in any distribution, so they are PEP 420
namespace packages and the trees merge at import time. No import path depends on which
distribution a module ships in. In a deployed pod the core is installed editable from
/opt/robovast/src while robovast-cluster’s files land in site-packages — one
namespace, two origins, no shadowing.
The practical consequence, and the reason it is worth the packaging: pip install
robovast-client is 13 packages and ~30 MB against the core’s 88 and ~290 MB, and a
vast assembled from it lists exactly the verbs it can run. See Using the client.
Where the driver runs
A campaign is driven by one robovast.execution.controller.CampaignController
(batch or search) that runs in the driving process: the robovast-service runs
the driver in a worker thread (one per campaign) over KubernetesBackend, which
creates the scenario Jobs.
There is no separate per-campaign controller pod. The service is already a
long-lived, in-cluster process with the cluster config, the Kubernetes API, the
campaigns’ own volume and the workspace (plugins installed) — so it hosts the driver
directly instead of staging the project out and launching a second process to do
it. stop and live status are therefore in-process operations the service
exposes over the interface, not a network hop to a controller. Two things a
campaign still runs as their own Kubernetes workloads, because each needs to be
one: the scenario/postprocessing Jobs, and — only for variations that declare
one — a per-campaign auxiliary-container pod the driver execs into.
A stop is read wherever a campaign waits. The flag
(ControllerState.request_stop) is set by the service, and the campaign’s own thread
reads it through two primitives and nothing else: wait_for_stop replaces the sleep in
every poll loop, so a wait ends the instant the flag is set rather than at the end of
whichever interval it was in, and raise_if_stopped ends the campaign at the boundary
between two steps. Below the execution layer — pip, the composition worker, the
conversion — the same thing is a should_stop predicate
(stop_checker()) and the helpers in
robovast.common.stop, which kill a running subprocess’s whole process group.
What a flag cannot end is work on another machine, which is why
ClusterService.stop also tears down that campaign’s in-flight Jobs — reusing the
campaign-scoped cleanup_cluster_campaign (the same cleanup
vast cluster jobs-cleanup performs). The running pods terminate now, and the batch
loop finds them gone (running_jobs treats a gone Job as finished) rather than
waiting out runs the campaign has given up on. The deletions are label-scoped to the one
campaign, so other queued/running campaigns are untouched, and to its Jobs: a campaign’s
aux pod belongs to the composition span that created it and is deleted when that span
ends, so a stop leaves it alone and only a reaper (jobs-cleanup, a campaign being
deleted) collects one. A batch that ends while stopped sweeps its own Jobs once more on
the way out, because the queue can create one between that teardown and the loop’s next
read of the flag, and nothing else would collect it.
Which work a stop lands on, and what it leaves:
Phase |
Scope |
What ends it |
What the campaign is left as |
|---|---|---|---|
|
runs |
the boundary between two steps; the composition worker’s process group, and the auxiliary container a variation is waiting on – the exec is given up and the span’s pod goes with it |
|
|
runs |
pip’s process group is terminated |
|
|
runs |
the wait is abandoned |
|
|
runs |
the service’s teardown (this campaign’s Jobs), which the batch loop then sees |
|
|
postprocessing |
the pipeline between steps; the conversion’s process group, or its Jobs |
|
|
share |
the upload’s own progress callbacks |
|
terminal |
— |
nothing is running |
the stop is refused rather than answered with a flag nothing will read |
Service shutdown is deliberately not the same thing. What exiting does to a
running campaign is a property of the implementation, answered in its own
_shutdown_running_campaigns: ClusterService leaves its Jobs running, because they
outlive any one service process and the next one adopts them
(Cluster Execution). Stopping a campaign is stop; exiting the service
is not, and it never was a good way to say it — the cooperative stop persists a
terminal outcome.json, and a campaign that has recorded an ending is one no
successor will pick up again.
A stopped campaign is reported as a clean terminal, not a failure. The batch wait
loop raises CampaignStopped the moment it sees the cooperative-stop flag, the
controller sets phase "stopped", and the builders’ finish tail
(_finish_campaign) is skipped — the per-run results the pods already delivered are
left as the campaign’s output. The "stopped" outcome is persisted like a failure
(_record_controller_outcome writes _execution/outcome.json), so
the phase survives a service restart instead of reconstructing as an ambiguous
"finished". Every terminal-phase filter (listings, the --wait-and-download
waiter, cleanup’s live-set) counts "stopped" as done.
A stop that arrives once the runs are over — during postprocessing — ends differently,
and both halves of the difference matter. It is not reached by the service’s teardown (which
is label-scoped to jobgroup=scenario-runs so that it cannot cancel a shared image
build, and the postprocessing Job is jobgroup=postprocessing), so the wait polls the
flag itself: await_job deletes the Job — which makes a Job this process only
re-attached to stoppable as well. And the campaign is not "stopped" — its runs all
finished and their results are complete, so it ends "finished" with the reason on
postprocessing_error and no derived data, exactly as a postprocessing failure does,
because in both cases what is missing is only the derived data and only a re-run supplies
it. stop says which of the two stops it performed in its reply, and the reason is
recorded as a cancellation rather than through the failure account: no failure log is
authored, because a stop is not a fault.
What a cancelled postprocess leaves is bounded by design rather than by luck. The rosbag
conversion — the long step, and the one a stop is usually aimed at — is written to
survive being killed: a bag records itself as converted only once its handlers have
finished, and every output is rewritten rather than appended, so an interrupted bag is
simply redone. The steps after it (the index ingest, the metadata, the provenance record)
run in the postprocessing pod, so a stop landing during the ingest interrupts it — and
that is survivable for the reason the ingest is written with autocommit: campaign_ingest.ingest_campaign()
clears a campaign’s rows before writing them, so a re-run replaces a partial load rather
than doubling it. An ingest handed a directory that holds no campaign at all — neither
campaign.db nor a single run directory — is refused before that clear, so a wrong
path cannot empty a campaign’s rows and then record the emptiness as its answer: the
registry’s entry is what separates “ingested and measured nothing” from “never ingested”,
and a query trusts it to make that distinction.
What is never left is a campaign that claims derived data it does not have. The provenance record is written last, after the ingest, precisely so that its presence means every step succeeded — so a cancelled campaign has none, reads as not postprocessed, and asks for the re-run that settles it.
Winding down is a race against uvicorn’s graceful-shutdown deadline, so the signal
handler raises a process-wide flag (robovast.common.shutdown) before the
clock starts, and the layers that would otherwise fight the teardown consult it. The
SSE streams do not wait for their
next pull either: a watchdog closes the stream the moment shutdown is announced and
abandons the worker thread, because a pull that returns after the deadline gets its
response task canceled and the cancellation logged as an “Exception in ASGI
application” traceback with the server already gone.
MCP tools ─┐
vast CLI cmds ──┤→ RobovastClient (one interface)
web UI ──────┘ │
HTTPTransport → robovast-service
(in-cluster Deployment, KubernetesBackend,
results volume)
The service core is backend-agnostic: it dispatches execution to an
ExecutionBackend, which ClusterService answers with KubernetesBackend. See
Deployment Modes and Access for how a client reaches the service.
One container plan, built once
A campaign declares its containers by name (execution.containers)
and robovast.common.containers.plan_containers() turns that into the containers a run
actually starts. The backend, the image builds, ``exec_in_container`` and the docs read that
one map. A second lookup anywhere would be free to disagree with what the pod started, and
the disagreement would be silent – a diagnostic entering one container while the campaign
ran another.
Two consequences worth stating, because they invert what every pre-v2 campaign assumed:
The scenario’s container is not necessarily the campaign’s “the image”. With a simulator or a system under test in its own container, “the main container’s image” is no longer a synonym for “what this campaign is testing”. Anything that treated the two as one – digest capture,
execution.yaml, the compat check,exec_in_container– resolves through the plan instead.A role is not a container count.
scenario/simulation/sutare always valid names to ask about; one campaign backs all three with a single container and another with three. Onlyscenariois guaranteed to exist.
Simulators are declared, not assembled: a backend
(robovast.common.simulators, Simulators) contributes container blocks, an
environment and a SimulationInterface ref before the plan is built, so nothing downstream
knows which simulator it is looking at.
One thing in that plan is per-configuration: the simulator’s own keys. A world belongs to
a configuration, so each one resolves its own sim block over the campaign’s default
(Varying the simulator). The plan is still built once and still holds one entry per container –
images, resources and package lists stay campaign-level, and every consumer of
plan_containers is untouched. What varies is the simulator’s command (one argv token:
the world) and the overrides document mounted beside the scenario channel’s parameter file.
Both are produced where the backend already renders a per-job manifest, by the same backend
hooks apply_backend calls – so a backend cannot answer one thing at composition and
another at dispatch.
That is also why the packer groups work items by their resolved block: containers start once per job and are not restarted between packed items, so a job that mixed two worlds would run the second configuration against the first one’s compiled model.
Asking the simulator a question
Some facts only the simulator has. Which components a world defines, what it is built from, and
which model values a run may change are all settled by resolving the world – which needs the
simulator installed, and RoboVAST deliberately does not have it (the backend package is strings
and container specs, so the long-lived service carries no MuJoCo). A backend therefore answers
by returning a question, ContainerQuery: a command and
the image to run it in, whose one line of JSON RoboVAST reads.
Four rules make those answers trustworthy, and each of them was a bug first:
In the image the campaign runs. Which world a ref even names depends on what is installed, so a query answered in a fixed base image describes a different world – or none, for an experiment whose world ships in its own built image.
simulator_image()is the one place that decides, becauseapply_backend()applies the same precedence to the run and a second copy would be free to disagree with it.An unanswerable question says so.
WorldQueryUnavailablenames which reason it is – no backend, an unbuiltbuild:<tag>, no container runner here, a command that failed. A pre-check that logged this at debug and carried on was indistinguishable from a check that passed, which is how a misspelled plugin key reached the container after the image pull.Half an answer beats none. A simulator that exits non-zero having still printed a payload answered what it could, and that payload is taken. A world whose model does not compile in the image (a
*_rosworld described where the colcon-packaged bridge does not resolve) can still say which plugin keys it has – that half needs no build – and discarding the reply cost the campaign a check it could have had. The rule is generic: nothing here knows which half was lost, because the simulator says so in the payload’s ownerrors, and each half that goes unchecked is warned about by name.A container runner is not implied. The query runs a container, and in-cluster a container runner exists only inside a composition, which creates the aux pod it asks for, so outside one the query is refused with that reason rather than quietly running
docker runon the serve host. A standalone aux pod for one-shot queries is the follow-up that would lift it.
validate_project’s world check reads one more thing off the same answer: whether the image’s
own plugins publish every config key the world sets (robovast.service.world_keys). It
compares two published CLI answers of that image – the components and config paths of roqsim
scenes describe, and python3 -m roqsim.introspection describe <plugin> – plus
roqsim.schema.INJECTED_KEYS, the set of keys roqsim accepts on any component, printed by the
image’s own interpreter. RoboVAST imports none of it. The catalog answers are cached per resolved
image identity in robovast.service.image_catalog, the cache the image_catalog MCP tools
read too, and all uncached plugins of one world are asked in one exec. Its findings are advice by
design: a docstring-published key list can be incomplete, so the finding states what the catalog
shows rather than a verdict the check cannot make.
And one more: the state a trial starts from. The describe the check asks for resets the world it
built, as a run does before each trial, and the answer carries warnings – what that state holds
that will not stop the world from loading but is likely to make a run misbehave, in the simulator’s
own {check, message, hint} (roqsim: two bodies placed inside one another, which the contact
solver flings apart on the first steps). Each becomes advice carrying the simulator’s words, because
the simulator does not refuse it either; nothing here knows which checks exist. A reset that raises
comes back as errors.reset and is an error, since no run of that world reaches its first step.
One function runs every such query –
robovast.common.config_generation.describe_world_payload() – because the two callers
(the sim-override pre-check, and
describe_world() for a caller writing an
override) share all three traps.
Workspaces vs. campaigns
Two independent identifiers:
``workspace_id`` — a server-side folder of editable project inputs (
.vast/.osc/ run files / binaries). Clients need no filesystem of their own (robovast.service.workspaces).``campaign_id`` — a self-contained campaign produced by running a workspace’s project. It snapshots its project into
_config/and is addressed bycampaign_idalone.
Workspaces are independent of campaigns. Editing or deleting a workspace
never affects an existing campaign, and there is no campaign→workspace link.
Results/query operations key on campaign_id only.
Both travel as an archive. A workspace leaves the service as a tar.gz of its
project files under one directory, and goes to the same share a campaign’s results do
(Sharing Results). Importing one always creates a new workspace: the archive
carries project files and no identity, so there is no id to restore and nothing to
replace — the opposite of a campaign import, which restores a campaign under its own id.
A batch campaign reads the workspace only while it prepares. It composes there, resolves what each configuration references, and stages those files; from then on it reads nothing but its own campaign directory, so the workspace can change underneath it without changing what it runs. It says so, and a bulk push waits only for that window — which closes before the first container starts.
A search campaign is the exception, and holds the workspace for its whole life: it composes again for every generation, so a file edited between two generations would make them two different experiments under one campaign id. Giving a search the same freedom means giving it its own copy of the project to compose from, which is not yet done.
One project binding. workspace_id is the only project binding the service
accepts, on every backend: a campaign always runs a workspace’s .vast, and
config_path selects among several .vast files in that workspace. There is no
“current project” anywhere — not on the server, and not in the CLI either: every command names
its own input, and vast workspace run takes the same workspace-and-path pair the API
does. Omitting workspace_id is refused rather than resolved from somewhere else,
because such a fallback ignores config_path and so could run a different .vast
than the caller named.
A project reaches the service by upload, with vast workspace init, and edits need a
re-push.
The service implementation is resolved, not imported
vast serve runs the one service implementation installed, and finds it through the
robovast.execution_backends entry-point group — the same mechanism as simulators,
variation types and panel types, through the same resolver
(robovast.common.plugin_ref.load_ref). The core registers none; robovast-cluster
registers cluster, a class with a build(), and the core never names
ClusterService. With none installed the service refuses to start, naming the
distribution that ships one.
Listing the entry points must not import them, so that refusal can be a sentence rather
than a ModuleNotFoundError from a module nobody asked for; and importing the base loads
no implementation. Both are pinned by tests/service/test_serve_backends.py.
As with simulators, an implementation must import without the thing it drives — it is
imported in a process that may have no kubeconfig. Reaching for one belongs inside
build().
That indirection is what lets the cluster implementation ship as its own distribution,
robovast-cluster (src/robovast_cluster/), rather than as part of the core. An
install without it carries no kubernetes, boto3 or google-cloud-storage at
all, and still validates, composes, stores workspaces and processes results. Declining
it is a supported configuration for everything but running a service.
An implementation refuses by name
An implementation is allowed to offer less than the interface describes. What it may not do is answer such a request with anything but a refusal that says so: an operation it accepts and quietly does nothing with looks, to every client, exactly like one that worked.
So there is one exception for it, UnsupportedOperation, and one sentence:
<operation> is not supported by the <implementation> implementation, with a hint where
there is somewhere else to go. Every RobovastInterface implementation declares its
IMPLEMENTATION (cluster, http, and the test suite’s null), and raises the
refusal in its own class, never as a default on the base. The exception is a
ServiceError: the app maps it to 501 with the sentence as detail and
unsupported_operation in the error header, the HTTP transport hands the caller the same
status, code and sentence, and in process it is the very same object, so the CLI, the MCP
tools, the web UI and a raw HTTP client all show one line (HTTP API, “Status codes”;
tests/service/test_unsupported_operation.py).
That is also the line between “not supported” and the refusals that keep their own status:
a launch refused for now is a 409, and a request the environment cannot honour is a
400. A bare NotImplementedError reaching a route is still a 500, because it
still means a bug.
One base, the implementation’s hooks
ServiceBase (robovast.service.service_base) is the abstract, in-process half of the
interface; ClusterService (robovast-cluster) is its one production implementation.
The base holds what is correct wherever the runs happen: hosting a campaign’s driver on a
worker thread, the registry of what is being driven, the workspace store, the caches and
the event log, every reader that resolves a campaign under the results root, and every
operation whose body reaches the cluster only through a hook. What depends on where the
runs happen is an @abstractmethod on the base — which backend runs a job, which exec
runner a diagnostic uses, what a job’s log and state are read from, what exiting does to a
running campaign — and a body in ClusterService. A subclass missing one cannot be
constructed, and the error names it.
tests/service/null_service.py is a second subclass, NullService, that runs nothing
and refuses every driver-needing operation by name, so the base’s own code — workspaces,
files, records, listings, the routes over them — is tested without a Kubernetes-shaped
fake. Two things follow. The base imports no Kubernetes and names no driver in a body
(tests/service/test_service_base_hooks.py pins that, and tests/service/test_serve_backends.py
pins that importing it loads no implementation). And a body on the base is one that is
correct for any implementation, or it is a hook.
robovast-cluster ships into the same import namespace as the core rather than under
a name of its own, so no import path changes: robovast/ and robovast/execution/
carry no __init__.py in either distribution, making them PEP 420 namespace packages
whose two source trees merge at import time. In a deployed pod the core is installed
editable from /opt/robovast/src while the cluster distribution’s files land in
site-packages — one namespace, two origins, no shadowing. Adding an __init__.py to
either directory would break the merge silently, so neither has one.
The direction of the dependency is load-bearing: robovast-cluster depends on
robovast, never the reverse. That is why it is not installable as an extra — a
robovast[cluster] extra is the edge back, and closes the cycle. Every place that
installs it does so as its own step: the controller Dockerfile, make venv and make
build each name it, and a test guards the Dockerfile, because an omission there cannot
fail before deployment.
Container exec: a diagnostic that cannot become a run
exec_in_container runs one command in an experiment image. Everything about its design
follows from one requirement: it must be structurally incapable of producing a result,
not merely discouraged from it. An exec that could write /out or leave a
campaign-shaped directory would become the path of least resistance, and its output would
look like a campaign’s while having no pinned provenance and no repetitions.
So: no campaign id, no campaign directory, /out never mounted, and OUTPUT_DIR
pointing at a container-local path that dies with the container. A staged /config is
read-only, and the workspace — mounted at its own /sources/<id> address so a path from
write_file is usable verbatim — is read-only too.
Two sources, one path. A workspace project (workspace_id + config_path) and an
existing campaign (campaign_id) both resolve to a project directory, because a
campaign’s _config/ is one. From there the staging is identical:
build_campaign_data → filter_configs_by_name → prepare_campaign_configs, and
the image resolves from that project’s .vast exactly as a run resolves it.
The entrypoint is rendered, never inherited. prepare_campaign_configs substitutes
the init and post-run blocks a pod needs into entrypoint.sh (the hand-off that tells
the pod’s uploader this container has finished writing), so a campaign’s staged
entrypoint carries what its backend asked for. render_entrypoint exists so the
bare-image case can have that script without expanding a config tree it does not need.
Why reuse the entrypoint at all, rather than building an environment: the environment a
run sees is the ROS overlay, /ws/install, the init block, execution.env and
PRE_COMMAND — and it will grow. A diagnostic that reconstructed it would drift and
answer a different question than the run. Reuse also has a visible consequence worth
knowing: the entrypoint redirects its own stdout when given no argv, so a started
scenario logs to a file inside the container and the result carries log_path instead
of that output.
A running campaign is never a target. There is no code path from this operation to a job’s container or pod. An earlier draft made exec’ing a live campaign “safe” by annotating its log; that was an admission the operation was wrong rather than a fix. To inspect a live stack, the caller starts that stack in its own exec container.
One container, so there is no state to manage. At most one exists at a time, under a
fixed name (robovast-exec). That removes
session ids, a listing operation, and the leak class: a stray container is found and reaped
by name at service start. Its cost is reported instead of hidden — ResourceUsage carries
exec_container, so a cluster with no room can be attributed to the caller’s own container.
Two clocks bound it, and keeping them apart is load-bearing: an idle reap that runs
only while nothing the tool started is alive (otherwise a running scenario would be killed
while it worked), and a hard deadline at max(idle cap, limit + grace) so a project
declaring timeout: 900 is not truncated at the idle cap. Limits are derived rather than
passed — ExecRequest has no timeout field — and the result reports which rule applied,
so a timed_out result names its own remedy.
The cluster-specific half is a small protocol (ExecRunner): KubeExecRunner runs an aux
pod plus pods/exec. Everything
else — validation, staging, limits, the lifetime state machine — is shared, as are the
pod primitives both in-cluster users need (wait_pod_ready, wait_pod_gone,
exec_stream in robovast.execution.cluster_execution.kube_client; they live in common because the execution
engine may not import robovast.service).
The diagnostic stages the way a run stages. In-cluster, /config is written as a
staged tree on the service’s disk and fetched from its data plane by an init container on
the shared sidecar image, with a token scoped to that one slot — the same transport a
campaign Job and an image-build context use. A ConfigMap would be simpler but caps the
tree at ~900 KiB and answers “too big” with “run it as a campaign instead”, the exact
expense this tool exists to save. The deeper reason is that a diagnostic with its own
staging path can pass on a config the run would fail to stage.
Staging fits an init container because it happens exactly once per pod: the manager
discards a redundant staging when it reuses a held pod, and replaces the pod outright when
the identity changes, so /config never needs refreshing under a live pod. A staged tree
is torn down with its pod; a tree left by a dead service process is reaped at the next
stop, since nothing else would.
A composition’s aux pod moves its workspace the same way, so there is one in-cluster
transport rather than three. Its contract is harder than the exec runner’s — a
bidirectional transfer around every run() on a kept-alive pod, which an init
container running once before the pod starts cannot serve — so the pod carries a
transfer container from the sidecar image that idles beside the aux container, sharing
its emptyDirs and holding the slot’s token, and the runner execs the fetch and the delivery
there. It has to be a container of ours because the aux image belongs to a plugin author
and is not ours to add tools to, and so no credential reaches that image either. The
runner’s workspace is its slot’s directory on the service’s disk, mounted in the pod at
the same absolute path, so nothing is copied on the service side in either direction.
Nothing but the commands themselves goes through pods/exec: the exec channel is a text
websocket the client cannot half-close, so a receiver waiting for EOF there waits forever.
The cost is that campaign composition needs the data plane reachable from the pod; it
fails loudly rather than falling back, since a second transport is the thing not wanted.
One address space for files
Every file the service can reach has a single address, and that address is the URL
that serves it — the string a caller passes to read_file is the string it can
GET:
Address |
Root |
Writable |
|---|---|---|
|
the campaign directory |
no |
|
the workspace’s project directory |
yes |
Three properties are load-bearing.
Content lives apart from control. /campaigns/{id}/ and /workspaces/{id}/ are
control namespaces — a fixed vocabulary of service-owned verbs (status, logs,
query, validate …, plus every name a service-endpoint plugin registers). A user’s
config or output file may be called anything, so putting content there means a route can
shadow a file name — today, or the next time a route is added. Separate content
namespaces make “no reserved words, ever” true by construction.
The namespace is the permission, dispatched once rather than checked per operation:
there is no PUT/POST/DELETE route under /results at all, so a write there
is a 405 from the router. Results are what a campaign produced, and the one writer of a
campaign directory is the campaign — a pod delivers its own output through the data
plane’s control route (the data plane), which is why that route is not
a write verb here. Each namespace is confined against its own root via
robovast.client.safe_path.safe_join; a results address must never resolve inside a
workspace, or the read-only tree would inherit the writable one’s permissions.
The path is the real on-disk path. <config_name>/<run_id>/<file> for a run
artifact; _config/, _execution/, _transient/, _jobs/ by their actual
names. (The predecessor route interposed a synthetic segment that matched no directory
and collided with the .vast run_files: inputs — two opposite meanings for one
name.)
A trailing slash means “directory”: GET on it returns a listing, non-recursive by
default with a total — a campaign holds one directory per configuration and one per
run, so a recursive listing of its root is thousands of entries. (A bare owner,
/results/<campaign>, also lists: it can only be a directory.) The slash is how a
client says which representation it wants, not part of the path: list_files and
read_file mean the same thing whichever way the address was spelled, because the
HTTP client normalizes before it builds the URL. Making the character load-bearing would
give one interface call two answers depending on which client made it. Listing entries are
plain names, directories suffixed /, relative to the echoed address, so the next
address is a concatenation. GET on a file returns its bytes (mime-typed, so a browser
resource fetching siblings by relative URL resolves within its own directory);
?as=text returns a paginated, binary-refusing text view. Paging happens server
side — reading 100 lines of a cluster log transfers 100 lines.
The service serves /results from that directory, with FileResponse: Range and
ETag come free, a byte range is a seek rather than a transfer, and a listing is a
scandir. That is what a campaign being on the service’s own volume
buys, and it is why there is no “which part of the campaign do you need?” seam anywhere in
the read path: every part of it is a file the service can open.
The data plane: tar streams, in their own process
A campaign’s readers hold a directory; its producers are pods on other machines. That is the one boundary bytes still have to cross, and everything crossing it is a tar stream over HTTP:
Route |
What crosses |
|---|---|
|
what a scenario Job extracts into |
|
a pod’s whole |
|
the campaign, whole or narrowed to what a postprocessing pod reads |
|
the scratch trees a build, exec or auxiliary pod is handed and hands back |
A tar, rather than a file API. One stream is one request, so a pod’s entire output tree costs one round trip instead of one per file; and a tar carries executable bits and symlinks natively, so nothing has to be restored on the other side — which is exactly what a per-object protocol keeps getting wrong, because the metadata travels beside the bytes rather than in them.
Compressed only where it leaves the cluster. A stream between a pod and the service is
a plain tar. Run output is mostly recordings that barely compress, so gzip there buys
almost no size and costs a core per stream: a single gzip holds a transfer near
70 MB/s where the plain tar moves at the disk’s speed, and on the pod side that core is
taken from the scenario whose output it is. A download for a person keeps gzip, compressed
on every core by pigz, because it may cross a link where the size is what matters.
One delivery per pod, not per container. A scenario pod’s containers share /out,
so each of them uploading would walk and send the same tree. Instead every container
writes a done marker on the shared /ipc volume when its own files are complete, and a
single uploader container waits for the markers and delivers once
(pod_upload). It is a regular container, so
the Job is complete only when the results are home and fails when they could not be
delivered — which is what lets the driver read a finished Job as delivered results rather
than polling for them.
Its own process. The routes are a separate app
(robovast.service.data_app) over a class that knows directories, tokens and tar and
nothing else — no transport, no campaign registry, liveness read from the campaign’s own
terminal record. In the cluster Deployment it runs as its own container with its own
limits, behind an nginx front that owns the pod’s single port and routes /data/ to it
unbuffered; a vast serve started by hand mounts the same routes into its one process,
because there the isolation buys nothing and a second port would cost every client. The point of the
split is that a dozen pods delivering gigabytes at once slow each other down and nothing
else: bulk bytes never share an event loop with the run view or the admission queue.
What a pod is given. An address (ROBOVAST_DATA_URL), the campaign it is working on,
and a token scoped to that campaign’s — or that slot’s — data routes and refused
everywhere else (HTTP API, “Who may call it”). No storage credential of any kind
exists in a pod, so reading a pod spec discloses nothing beyond which campaign it serves.
Token-efficient file transfer
Because inline tool content passes through an LLM’s context (costing tokens twice — once to generate, once per later turn), the file API is split:
Inline authoring is limited to ``.vast`` / ``.osc`` — the small text an LLM writes.
edit_filesends a diff, so the validate→fix loop stays cheap.Everything else uses the HTTP PUT side channel:
create_uploadreturns a one-time, TTL-scoped URL the clientcurls the bytes into (curl -X PUT --data-binary @file <url>), so run files, notebooks and binaries never enter the token stream. Executability is preserved (explicit flag or#!shebang auto-detect).
Data flow and result access
A campaign’s durable home is the service’s results volume:
<results_root>/<campaign_id>/, one directory, written by the campaign and read by
everything else. GET /data/campaigns/{id}/archive tars it on the fly into the
response (no scratch, nothing buffered in memory), which is what vast campaign
download and the web UI Download button use. Refusing there with a 409 — “the
results are already on this host’s filesystem” — asserts something true of a caller on
that host and false of everyone else, so a service reached over the network could not be
downloaded from at all.
The external tar.gz share is a separate system, with its own credentials and
its own lifetime, reached through vast share. It is opt-in at launch
(upload_to_share): when set, the driver streams the campaign to the share the
moment the runs finish, before postprocessing — so the shared copy is the minimal
raw snapshot while the campaign carries the derived data. Which variant an
archive is rides in its name (<campaign-id>.raw.tar.gz /
…postprocessed.tar.gz) and is read off the campaign rather than passed in, so the
campaign-end upload and a later vast share export cannot disagree.
The share is not a subset of what a service has. A campaign can be cleaned up here
while its archive stays up there — that overlap, not a hierarchy, is why
vast share import exists and why nothing caches “does this campaign have a share
copy” on the campaign. GET /share/archives asks the share, with the service’s own
credentials, for the callers (a browser) that can hold none. It answers newest campaign
first, keyed on the timestamp inside each campaign id: no provider reports a modification
time, and when the campaign ran is what a reader of that listing is after. A campaign present
as both variants is two entries sharing an id, ordered by object name between themselves so
that two calls cannot disagree — and an importer names the archive, not the campaign, when
it means one of them in particular.
Analysis postprocessing is editable and re-runnable: the raw rosbags are
always preserved, so results_processing.postprocessing entries can be changed
and re-run to compute different metrics later without re-executing the campaign.
Because it is config (not captured data), an edit overwrites that block in the
campaign’s own ``_config/<name>.vast`` in place — no override files, no revisions
(robovast.service.postprocessing_edit); the raw rosbags and the as-ran
configuration/execution are untouched. A re-run is dispatched in the
background and shows up in the campaign view as a live postprocessing phase (see
Re-running a finished campaign’s post-run steps).
What goes in the database, and what stays a file
A campaign writes a lot of artifacts, and each is reachable in exactly one of two ways: as rows in a database, or as a file through the one address space. The rule deciding which:
An artifact belongs in the database when it is a per-entity record you filter, aggregate, or join across runs — and stays a file when it is a whole document read once.
“One row” is not the test; aggregatable is. A campaign’s _execution/execution.yaml
is a single document, but its contents (which robovast, which image) are exactly what one
compares across campaigns — and the SQL interface can attach several campaigns at once —
so it is lifted onto the campaign row. Applied to what a campaign writes:
Artifact |
Home |
|---|---|
each run’s |
DB — |
each job’s |
DB — |
per-run metric CSVs |
DB — one index table per CSV stem, after postprocessing |
|
DB — provenance columns on |
|
DB — the index’s |
the |
Both — |
each configuration’s |
Both — the file is what each channel was given, a whole document a person reads when
replaying a cell by hand. Its values also reach the DB, as |
|
File — the campaign’s terminal status, read by |
|
File — what was done to a campaign’s runs other than running them, keyed by job
artifact dir, each entry tagged |
|
File — how the campaign was asked for; a whole document, read once by a
retrigger. Its |
|
File — already duplicated into |
|
DB — one row per log event, with |
|
DB — one row per container per process name per ~1 s tick, with |
|
File + the log tools — the raw bytes, and the live case is the point of reading
them: a campaign is not in the index while it runs. The tools reduce them on read
( |
Two consequences worth stating. A file is never also a table: putting
configurations.yaml in the DB would create a second source of truth for the resolved
configuration list. And a document in a cell is not a queryable artifact:
campaign.config_json holds the whole .vast, which exceeds the per-cell limit and
comes back truncated, so it is queried through the index’s JSON operators
(config_json::jsonb -> 'execution' ->> 'image'; the index is Postgres, which has no
SQLite json_extract) or the config_view rows — never SELECT config_json.
Querying results
Per-run metrics are consolidated into the central results index (one
table per CSV stem) plus a runs dimension table — per-run
status/duration_s and each varied parameter as a param_* column, on whichever
channel it was written (Which channel a factor was written on, and its column).
That runs table is the analytics-wide view over campaign.db’s run
table (the operational source of truth for per-run outcomes, written live from
each test.xml); see the store schema. The MCP
run_data plugin exposes read-only SQL
(query_campaign_data_sql + describe_campaign_data), with campaign.db
attached as schema campaign — so campaign.run is queryable for raw
pass/fail even before postprocessing ingests the campaign. Joining runs to any
metric table on (config_name, run_id) answers “how does <param> affect
<metric>” in one query. Analysis notebooks read the same tables through
robovast.common.analysis.db, which scopes a table to the notebook’s DATA_DIR (see
Reading results).
Not versioned, because it is derived. The metrics store carries no schema version and no
migration ladder — the deliberate difference from campaign.db, whose
SCHEMA_VERSION does carry one. The store is authored: written
as the campaign runs, and the only record of what happened, so an old one must be upgraded in
place. The metrics are derived: postprocessing rebuilds them from the run directories, which
keep their CSV/JSONL, so the upgrade path is to run postprocessing again — which re-executes no
trial, and needs neither ROS nor the campaign’s execution image, since only the rosbags_* →
CSV step does and everything after it is plain Python. A migration here would be code
maintained to reproduce what the ingest already does.
A version would not gate reads anyway, because it is too coarse to answer the question a reader actually has. What gates a query is whether the columns are there, which the reader checks directly.
Two flat views carry the joins, so a caller cannot omit one. run_view (one row per
run: config, status, duration, params, search round, host record) and config_view (the .vast as
one row per key) are objects in the index, queried unqualified. They exist because a
forgotten join does not raise — run_id is unique only within a configuration, so a
query filtering on run_id alone silently returns rows from every configuration and
averages across them. Making the join part of the schema removes that failure mode rather
than documenting it. Each is created WITH (security_invoker = true), so the row-level
security on the tables underneath applies to whoever queries the view rather than to its
owner — without it an unscoped FROM run_view answers with every campaign’s runs.
Every ingest rebuilds them (index_views.create_views()), because which views the index
can support depends on which tables it holds: the first campaign to record a probe is what
brings the view over it. Where an underlying table is missing, run_view keeps its column
set and reports NULL for the host and batch columns — one query shape whatever the index
holds, with “not recorded” reading as NULL rather than as a broken query. A view gaining a
column therefore needs no migration and no re-postprocessing of the campaigns it serves: the
next ingest of any campaign rebuilds the views, and every campaign already in the index has
the column.
One index is shared by every campaign, so its DDL is serialized. Two postprocessing runs
can be rebuilding the views, or creating the table a stem needs, at the same time, and none
of the spellings that look safe are: IF NOT EXISTS checks the catalog and then creates,
a DROP before a CREATE leaves a window between them, and CREATE AGGREGATE and
CREATE POLICY have neither. The writer that loses such a race is refused with a duplicate
key on pg_type — a message naming neither the relation nor the concurrency — so every
CREATE/DROP/ALTER against the index is issued under one session-level advisory
lock (index_schema.ddl_lock()), and the paths that hold it ask what is missing first,
so an ingest with nothing to change takes no lock at all. A table’s creation, its new columns
and its widenings (index_schema.ensure_table()) are decided from the column verdicts
read again under the lock, not from the read that found DDL necessary: decided from that
earlier read, a writer that queued behind another could retype a column the other had just
widened back to the narrower type it saw, or create over a table that now exists and record
its own verdicts over the ones it was built with. The first read is safe to act on only when
it says nothing needs to change, because a verdict only ever widens. The view rebuild is one
transaction as well as locked: a reader that arrives mid-rebuild waits for the swap instead
of being told run_view does not exist, which would read as a campaign with no runs.
The ingest session trades durability for speed, because the index is derived.
campaign_ingest.ingest_campaign() turns synchronous_commit off on its connection for
the length of the ingest and puts the connection’s own value back afterwards. Each of the
ingest’s autocommitted statements then returns without waiting for its WAL flush; a crash of
the database server can lose the last commits before it, but cannot corrupt the index or
apply a commit in part. Nothing but the ingest writes that way, and what it can lose is
rebuilt from the campaign’s files by ingesting them again. The ingest ends by running
ANALYZE on every table it cleared or wrote (index_schema.analyze_tables()), so the
first queries over a campaign’s fresh rows are planned with statistics that describe them
rather than with whatever autovacuum last recorded — or, for a table the ingest just created,
with none.
describe_campaign_data lists both views first and carries the canonical query for each
question a caller is likely to ask — the per-run lookup, a configuration’s parameters, how
a metric was produced. That is deliberate: the replacement for a tool has to be discoverable
from the tool output an assistant already reads, not only from this page.
Columns are typed from the data, not left as text. A CSV yields only strings, so an
untyped ingest makes every comparison lexicographic — ORDER BY timestamp puts
"10.022" before "9.5", shuffling a trajectory and producing a path length that is
wrong by a factor rather than an error. So ingest infers a type per column
(robovast.results_processing.csv_types): a column whose every non-empty value is a
number becomes INTEGER/REAL and is stored numerically, and everything else stays
TEXT verbatim. The rule is deliberately strict — one n/a demotes the column, and
"007" is text, because a zero-padded identifier must keep its text.
param_* columns are typed the same way from their resolved values.
describe_campaign_data reports each column as "name TYPE", which is what tells a
caller whether a column can be ordered directly or needs CAST(col AS REAL).
A non-finite value is a measurement, so it is stored as one. A range with no return, a
path length for a trial where no path came back, a ratio with no denominator: these reach
the ingest as CSV text (inf, nan) and as Python floats, from a .jsonl file or
from the campaign record’s own parameters. REAL is double precision, which holds
Infinity, -Infinity and NaN natively, so the column stays numeric and NULL
is left to mean that nothing was measured. A container is JSON-encoded with
allow_nan=False and each non-finite float written as the string "inf", "-inf" or
"nan": Python’s json otherwise writes Infinity, -Infinity and NaN,
which JSON has no tokens for and which Postgres refuses when the column is cast — failing
the whole query rather than the row that holds one. The *_json columns mirrored from
campaign.db are re-encoded the same way on their way into the index.
The declaration never outlives the evidence. A column is declared by the first run that
writes it, but the evidence is every run — a later one can turn an INTEGER column real,
or a numeric column textual. A stale declaration is not cosmetic: a schema claiming REAL
over a column holding one 'n/a' makes AVG() return a plausible wrong number (SQLite
reads the text as 0) and MAX() return the text itself — the original bug wearing a
different hat. So after the last run is ingested, any table whose verdict moved is rebuilt
with the corrected types and its values carried over (_retype_table), which also
normalizes a mixed column so every row in it is text. Only affected tables are touched, so
the usual case rebuilds nothing.
Because a type alone cannot say “numeric except in the runs that failed”, such a column is
also recorded in the index’s _column_notes and surfaced by describe_campaign_data
as column_notes on the owning table — postprocessing logs a warning too, but no SQL
caller ever reads that log. The note names the fix (exclude the text rows, e.g.
WHERE col GLOB '[0-9-]*') because CAST alone would silently read them as 0.
Run view (web panel framework)
The web Run view is a small panel framework, built so it can grow (more panels, a future 3D scene, and eventually a live/streaming data source) without reworking the core. It has four decoupled contracts, and panels only ever see the bottom three:
Declaration — the campaign’s
.vastcarries a top-levelvisualization.results.run_view.panelslist (robovast.common.config.VisualizationConfig), served to the UI bylist_campaign_panels(the same raw-load pattern aslist_campaign_plots) with the contributed panels merged in:config.ALWAYS_ON_PANELS(theplaybacktransport, which no run view can do without) and the configured simulator’sdefault_panels. Merged there rather than in the UI, so the served list and the view cannot disagree; a campaign that declares the type itself keeps its own entry. The frontend normalizes the result against each panel type’s registry defaults (frontend/ui/src/lib/panels/parsePanels.ts).Registry + host — panel plugins self-register (
frontend/ui/src/lib/dashboard/registry.ts);PanelHostresolves each spec’s anchor/size to CSS and mounts the component. Adding a panel is oneregisterPanelcall.Shared panel kit —
@robovast/panel-kit(frontend/panel-kit/) holds the panel contract (PanelProps/PanelSpec/DataProvider/PlaybackClock) and the clock-driven scaffolding (useCanvasClock, the time-index binary search,keyframes). It exists because the host and the package-provided panel remotes are separately-built npm packages that must agree: a remote shares onlyreact/react-domat runtime, so before this it kept a hand-maintained copy of the host’s types and its own copy of the scaffolding — and the drift is what let a staleness bug live in one panel and nowhere else. Both consumers resolve it by path (atsconfigpathsentry plus a matching viteresolve.alias); it has no build of its own and is deliberately not an MFsharedmodule, so each side bundles its own copy and a version skew is an ordinary build rather than a remote-load failure.Clock —
PlaybackClock(frontend/panel-kit/src/clock.ts) is the single shared time source (seconds on the rosbag timeline). The playback panel is the only writer; the rest subscribe. It is an external store, so the ~display-ratetupdates while playing don’t re-render the tree.Data seam —
DataProvider(declared infrontend/panel-kit/src/dataProvider.ts, implemented bydbDataProviderinfrontend/ui/src/lib/dashboard/dataProvider.ts) is how a panel gets rows/frames by table + time, decoupled from transport. Today it reads one run’s rows from the index through the existingquery/describeendpoints, plus the dedicatedcostmapendpoint for grids. The interface (nearest/series/timeRange/has/fetchRun) is shaped so a futureliveDataProviderover a live topic buffer drops in without touching any panel.
Camera delivery. A recorded video is the one panel input that never passes through the data
seam: the panel resolves a videos row to a URL (DataProvider.runFileUrl) and puts it in a
<video src>, and the browser does the fetching, ranged requests and all. Nothing streams
media through the service’s Python. The videos row exists because the encode re-times frames
onto a constant rate and drops the bag stamps, so t_start is the only thing that can place
the file on the clock; the get_camera_frame MCP tool reads the same row, so the two surfaces
cannot hold different opinions about when a recording began. The panel is a reader of the
clock like every other, which is why it shows no controls of its own — two things in charge of
“now” is a run view disagreeing with itself.
That path is also why the route asks the transport for a path outright rather than probing for
the method. FileResponse is what carries Range, and every transport has local_file
— the interface defines a refusal and the service subclasses ServiceBase — so a presence
check can only ever succeed, and treating it as meaningful is how a caller ends up believing the
service cannot serve a path when it can. It answers or it refuses by name; there is nothing to
probe. The service answers, because a campaign’s results are under its own results root.
Screenshots are the deliberate opposite of geometry. scene_cache builds a scene
descriptor per world, so one build serves every run that used it — worth a background thread,
a cache and a status to poll. robovast.service.screenshot renders one moment of one run from
a caller’s viewpoint, so its key would be a camera pose and a time and would never be hit twice:
no cache, no thread, one synchronous POST that returns the image or the reason. That second
part matters as much as the first — an asynchronous render would have to stash its failure
somewhere the caller could find later, which is exactly the in-memory dictionary that makes a
failed scene build visible to nothing but get_run_scene_status. A render is still kept,
under a fresh name the response’s Content-Location carries: not for reuse, but so a caller
can pass it on by address. The store is bounded by age and count rather than cached, and pruned
each time a render is kept.
Costmap delivery. A grid needs a table of its own. The generic topic flatten packs an
array field into one cell and the read path caps a cell at 2 KB, so a run view reading SQL
could never get a frame out whole, and the flattened table carries none of the geometry a
frame has to be drawn against. The rosbags_costmap_to_csv handler
(robovast.results_processing.data.rosbags_process.CostmapToCsvHandler) instead
decodes each grid once during postprocessing and re-encodes it compactly — int8 cells
zlib-compressed, base64 in a costmaps table row with the pose/geometry metadata. The
/campaigns/{id}/costmap endpoint (robovast_nav’s CostmapEndpoint, a
robovast.service_endpoints plugin — it is not a core interface operation) delivers the
frame nearest a time untruncated; the browser inflates it with the native
DecompressionStream. The costmaps table description in describe_data_db gives an
LLM the map’s size in meters, resolution, and layers for spatial reasoning without decoding
grids.
Alongside the frame it returns t_prev / t_next, the timestamps recorded either side of
it for that topic. This is the one panel that fetches per clock position rather than
preloading its series, so “nearest” alone is not an interpretable answer — the query always
returns something, however far away, and the panel could not tell a current frame from the
first or last one clamped to a cursor minutes off. From that pair
(frontend/panel-kit’s frameValidity) it derives the interval over which the frame stays the
nearest one — so it re-requests only when the answer could change, and a latched topic such as
/map is fetched once per session — and the local publish period, hence how far the cursor
may drift before the layer is reported as absent instead of drawn as current. A latched topic
has no neighbors, so it is exempt from staleness by construction rather than by name.
That staleness threshold is floored at the panel’s own fetch cadence, which is not a detail: nav2 publishes costmaps far faster than any viewer fetches them (50 Hz local costmaps are ordinary, against a ~140 ms round trip plus throttle). Deriving the threshold from the publish rate alone inverts there — every frame is judged stale within 40 ms, long before it could be replaced, and the layer is blanked essentially permanently. A frame older than the publish period because the viewer sampled coarsely is not stale data; the threshold exists to catch real gaps and off-the-end clamps, which are seconds.
The interface surface
The operation contract (Phase 0 + workspaces + postprocessing shown; data-query
lives in the run_data MCP plugin):
Workspaces —
create_workspace/list_workspaces/get_workspace/delete_workspace/create_upload. A workspace’s files are not separate operations: they are the writable half of the file address space below.create_workspacealso takesfrom_campaign, which seeds the new workspace from that campaign’s frozen_config/— the one bridge from the read-only results tree back into editable inputs, and a reconstruction rather than a copy (the scenario is placed where the.vastdeclares it, as a retrigger does).Files —
list_files/read_file/read_file_bytes/write_file(.vast/.osconly) /edit_file/delete_file/create_upload, each taking one address (see One address space for files). The upload grant isPOST /uploadsrather than a workspace sub-route: its request already names the workspace in its address, and a path segment that had to agree with it would be an argument the handler either ignores or has to re-check. Bulk directory sync is client-side glue over these primitives, not a new operation:sync_directory_to_workspace(inrobovast.service.project_push) walks a local directory and driveswrite_filefor.vast/.oscandcreate_uploadfor everything else, with an optionalprunethat deletes workspace files absent locally. It is the single implementation behind two callers — thevast workspace init/vast workspace updateCLI commands and the web UI’s drag-a-folder upload — so both stay transport-agnostic and can never drift.Campaigns —
create_campaign(backend implicit in the deployment;upload_to_shareis a per-campaign launch flag on the request, not a separate operation) /get_status/list_campaigns/list_jobs/get_job_log/stop/ the/archivestream.list_jobs+get_job_logare the live per-job view (the current batch’s execution units and a single running job’s log);ClusterServiceimplements them over the campaign’s Kubernetes Jobs +read_namespaced_pod_log, merging every container the job runs into one append-only stream through the sharedcommon.log_tail.MergedLogBuffer, since a job is not one container: the ROS shape gives the simulator and the system under test their own, and on the cluster those are native sidecars, which live inspec.initContainersand are found viacluster_execution.kube_client.pod_workload_containers. They report live state only; the persisted per-run logs remain part of the campaign result data, served byget_campaign_logsunchanged.GET /campaigns/eventsis a browser-only Server-Sent-Events transport over the samelist_campaignspull (the same server-side-loop idiom as the campaign log stream, so there is no second enumeration to drift): it pushes the full list on connect and on every change, which is how the webui shows a launched campaign — and itsbuilding/variation/running/ … phase — the instant it is registered, without polling.list_campaignsstays the authoritative pull for MCP and the CLI. The stream takes the pull’ssort/order, because the order is applied before the page is cut and so cannot be re-applied by a client; the web UI keeps its app-wide stream in the default order, which the start/end notices read, and opens a second one only while its list is shown in another. Both draw from one rule: a campaign tracked in the in-process registry reports its liveControllerState; an untracked one is reconstructed from its recorded facts (reconstruct_status_from_diskovercampaign_dir) — the same precedenceget_statususes. Which campaigns exist is the union of the results directory and the registries of what is being driven.Postprocessing —
get_postprocessing/update_postprocessing/run_postprocessing. The structured*_postprocessingpair is the programmatic API (MCP, CLI);get_postprocessing_source/update_postprocessing_sourceare the YAML-text twin the web UI re-run dialog edits in Monaco (mirroring the run-view*_panels_sourcevisualization editor). Both overwrite the edited block in the campaign’s own_config/<name>.vastin place.run_postprocessingdispatches the re-run in the background and returns at once (watch the campaign view for progress).Data query (MCP
run_data) —describe_campaign_data/query_campaign_data_sql.
New disk-consuming work is admitted against a free-space reserve. create_campaign,
retrigger_campaign, build_image, create_archive_upload, import_campaign and
run_postprocessing each call ServiceBase._admit_storage first (ClusterService’s
own build_image and run_postprocessing call it too). It reads
ResourceUsage.storage_refusal, which resource_usage computes once from the
disk and results readings it already takes (robovast.service.storage_reserve), so the
refusal and the meters are one measurement. With the reserve set to 0 nothing is read; a
reading that fails is logged and not judged, as an unmeasured meter is not a full disk, so a
launch never depends on the permissions the capacity reading needs. A refusal is
InsufficientStorageError, a 507 over HTTP — the status a write that already failed for lack
of space is also given.
Accepted work is never refused, but on a cluster it stops starting Jobs at the reserve.
Admission alone would not keep the node clear of its eviction threshold: a campaign accepted
above the reserve goes on creating Jobs, and every one writes its results into the same disk.
So the admission queue asks a space gate before each drain
(AdmissionController(space_gate=...)), which measures the filesystem the campaigns land
on with robovast.common.disk_reserve.disk_shortfall – on a node-directory deployment,
the node’s own disk. While it is short, nothing is created: every waiting campaign’s refusal
reads waiting for disk space: ... in its log and on its stage, the no-progress
deadline treats it as queued, and a postprocessing Job that times out waiting says so rather
than that the cluster was full. Jobs already running go on and deliver – the reserve is the
room their results land in – and admission resumes by itself once space is freed. Stop and
delete are never guarded: they are what free space.
The caches a clear may empty are copies of durable data, and nothing else.
service_cache / clear_service_cache sweep the scene cache; the
results directory is the durable home and is never offered. That the clear has nothing else to offer is the point: a campaign’s bytes exist
in exactly one place, so no copy of them can go stale, be swept mid-read, or need a lock
that a reader of them would have to respect.
Why images resolve the way they do
Container images says what the mechanism is. This is why it is that, because it is the part someone will otherwise re-litigate.
A family image ref glues three independent facts into one string, and authoring all three in one place is what makes image configuration a five-variable problem:
harbor.example/robovast / robovast-roqsim : latest
\_______ WHERE _______/ \____ WHAT ___/ \ WHICH /
deployment config never a choice version
WHAT is never authored. Which member a container needs follows from its role and the
campaign’s mode. Once that is granted, a .vast has no reason to mention a family image
at all, and the hard question — what should an override do to a digest-pinned ref in a
published dataset’s config? — does not arise rather than being answered.
``family:<member>`` mirrors ``build:<tag>``. Both are symbolic refs core resolves late
from context the author does not hold, and both fail loudly if one reaches a container spec
unresolved. The alternative considered was a rewrite pass that matched family
repositories in already-concrete refs and moved them to another project. It was
rejected: it needs a
whitelist of names to match, and anything a whitelist can match by mistake it can
redirect by mistake — a campaign’s own sut image silently pulled from somewhere the
author never named. A prefix marker cannot do that, because writing it is a request.
Resolution is per-campaign and explicitly threaded. image_project rides on
CreateCampaignRequest → RunOptions → build_campaign_data →
generate_scenario_variations, and is resolved in one pass right after apply_backend.
Two properties force the threading rather than an ambient module-level or context value:
a campaign is composed in a worker thread, and a plugins:-declaring config is composed
in a subprocess of that thread (which is why the value travels in the compose worker’s job
file, not its environment). The service drives several campaigns concurrently, so a process
-wide value would be whichever campaign set it last. The composition cache key includes the
project for the same reason: the composed data carries resolved refs, so an entry composed
against one project must not satisfy a request for another.
The image build is a second resolution point, threaded the same way. A build spec is not
read from the composed campaign data — it is re-derived from the raw config, because it has to
exist before composition (the images are what the campaign then runs in). So
extract_build_specs calls apply_backend itself, and therefore has to resolve what that
contributes itself: image_project rides CreateCampaignRequest → RunOptions →
_start_build_images / _resolve_built_images → extract_build_specs.
Missing this was asymmetric, which is what hid it: a container taking the default member was
resolved on the composition path, so sut and scenario built correctly while the one
container declaring a backend: carried family:robovast-roqsim into its Dockerfile’s
FROM. Docker read that as repository family, tag robovast-roqsim, and the campaign
died in BuildKit with a registry insufficient_scope — a credentials error three layers from
the cause. generate_dockerfile now refuses either prefix outright, so the promise that an
unresolved ref fails loudly holds for a build and not only for a pod spec.
Resolving into the campaign data — rather than at each point of use — is what makes
_execution/execution.yaml record a concrete image. Postprocessing reads that record to
choose the image it deserializes rosbags in, so a symbolic ref surviving there would be a
family: string handed to Kubernetes as an image name.
Where an image lives is the implementation’s answer, not a caller’s. Everything above is
about the recipe — which containers build and what their inputs hash to. Where the resulting
image ends up is the other half: ImageBuildStore, which
ClusterService answers with the deployment’s registry. An implementation overrides exactly
one factory (_images); every
question asked of a store — what is this image called here, is it actually here — is then
written once, and asked of the store rather than assumed. Two rules the ABC carries:
present()raises when it cannot tell. “I could not check” and “it is not there” are different answers; collapsing them is what turned a missing dependency into a missing artifact, three layers from the cause. The build path wants the opposite trade and says so explicitly (_registry_has_image): uncertainty means rebuild, because a redundant push is cheap and a wrong cache hit leaves pods inImagePullBackOff.only
ImageRef.identitymay cross the API boundary. The concrete ref is registry-qualified, and would carry registry knowledge to a client that must not have it — so there is exactly one field that is allowed out, rather than a rule to remember at each return.
One reader, several policies. Four things ask a campaign’s records which image it ran, and they want different answers: a re-run needs bytes it can obtain elsewhere, a cache needs anything that identifies bytes (a bare local id will do), postprocessing needs something it can run here and is right to accept a mutable tag, and the publication gate asks whether an outsider could obtain or rebuild them. Each of those is correct for its question.
They used to re-derive the record separately, with the precedence baked into each — and drifted,
visibly: the publication gate called a campaign opaque that the retrigger pre-flight called
pinnable, about one file on one disk. So campaign_image_record reads the record once and
returns facts with no verdict attached, and each policy stays with the caller that owns it. The
two predicates the policies differ on — image_is_pullable against image_identifies_bytes,
which differ by exactly the bare-local-id case — are named rather than spelled out at each site.
A diagnostic against a campaign does not use a store at all: the campaign recorded which
image each role ran on, and campaign_role_image returns those bytes. Re-deriving a hash
from the campaign’s frozen _config/ cannot work anyway — that snapshot holds the .vast
and the run files, not the build inputs — and the recording is the better answer regardless,
because a diagnostic about a run should run what the run ran.
A registry is not a node. The store answers where an image lives; whether a pod can
start it quickly is a different question. The build pushes to a registry and stops
there — the image exists and is on no node — so whoever
runs it first pays the whole multi-GB pull. That is usually exec_in_container, whose entire
justification is answering “is the package installed?” in seconds; the web UI’s 3D panel even
names the wait out loud (Fetching the simulation image onto the node).
So the service pulls the image onto a node as soon as it knows the image exists, with a
throwaway Job whose one container is that image running /bin/true
(image_warm). The pull is the work — nothing
reads the Job’s result, so a broken entrypoint or a nonzero exit warms the node just as well,
which is what makes it safe to point at an arbitrary experiment image. It fires where the
service learns an image is in the registry rather than where a caller asks for one: after a
build’s succeeded transition, at the two cache-hit returns, and — for the resolved base,
concurrent with the build itself — at submit. A campaign that builds inherits all of it, because
those fire points are on the build and not on the caller.
The family images are warmed from a different place and for a different reason:
vast cluster setup / upgrade, which is both the moment every node is cold for the
whole family — a tag bump or a moved project means the next campaign pays a full pull of
robovast-roqsim, the largest image there is — and the moment it is free, since the pod is
being restarted anyway so nothing is mid-campaign. It resolves from the caller’s own
ROBOVAST_PROJECT / ROBOVAST_PROJECT_TAG, which is correct precisely because those are
the values being baked into the service pod, so it warms the set the deployment is being
pointed at. robovast-controller is excluded: it is the Deployment, running
imagePullPolicy: Always, so the kubelet pulls it during the rollout already happening.
This also covers what the build fire points cannot — a campaign whose containers declare no
build: section runs family images that no build ever produced, so there is no
build-completion transition to hang a warm on.
They are also warmed with a different shape, and the asymmetry is the point. A Job schedules one pod and therefore warms one node, which is the right answer for an experiment image: the pod that wants it is the next one to run, and the only one that will. Any node may run any cell of a sweep, so for the family that leaves every other node to pay the pull — so the family gets a DaemonSet instead, one container per image, each asleep. An exited container’s image is collectable again, so init containers would warm a node and then let it go cold; a running container is what stops the kubelet reclaiming the bytes. The cost is a few millicores and tens of mebibytes per image per node, against nodes sized for a simulator.
Its containers run imagePullPolicy: Always while every campaign pod runs IfNotPresent,
which is deliberately backwards and is the whole reason the DaemonSet earns its place. A floating
tag is never re-pulled under IfNotPresent once a node holds bytes for it, and campaign pods
keep IfNotPresent precisely so that a sweep does not depend on the registry being reachable
at every pod start — which leaves this DaemonSet as the only place a re-pushed :latest
reaches a node ahead of the sweep that needs it (a pod whose ref is still a tag, such as a
composition’s auxiliary one, pulls it when it starts). It buys freshness, not reproducibility: a pinned ROBOVAST_PROJECT_TAG is
still what makes two runs comparable, and with a floating tag Always can just as well drift
new bytes into the middle of a sweep. The pod template carries the same restart annotation the
service Deployment does, because without it a re-pushed floating tag leaves every field
byte-identical, so the patch rolls nothing and Always never gets a container to start.
Being a DaemonSet, it gives up the self-collection below: it is meant to persist, so teardown removes it explicitly. Skipping that would leave a pod on every node holding multi-GB images for a deployment that no longer exists.
Three properties of the Job shape, and each replaces machinery rather than adding it:
Idempotent by name. The Job name is derived from the image ref, so a duplicate create is a 409 meaning “already warming”. No in-process record, and a service restart changes nothing — the same trick
build_id_foruses to make a resubmit idempotent.It terminates itself.
ttlSecondsAfterFinishedis not sufficient, and assuming it was would have reproduced a bug this codebase already paid for: TTL starts only once a Job is terminal, and withbackoffLimit: 0a pod wedged inImagePullBackOffleaves both counters at zero and the Jobactiveforever. That is precisely why the build path carries ablocked-phase probe. A prewarm has nobody watching it, so it getsactiveDeadlineSecondsinstead of a watcher.It never fails its caller. A failed prewarm leaves exactly the situation that held before the feature existed — a slow first pod — so raising would turn a missed optimization into a failed build. It warns instead, and the warning is load-bearing: nothing reads a prewarm back, so the log is the only place a permanently broken one can surface.
Deliberately outside admission, which is the same choice the build Job makes. A prewarm queued behind a full sweep would warm the node after the thing that needed it, which is worse than not warming; the cost is a few millicores, and admission counts its pods like any other once they are bound, so nothing is double-counted.
It is not wired to the 3D panel’s geometry compile, though that is the one place a human
watches this wait and the panel names it out loud. The compile runs in
campaign_role_image(..., SIMULATION_CONTAINER) — the exact image that campaign’s simulation
container ran — so at the moment a campaign finishes, the image is by construction already on
the node that ran it, and a prewarm there would find it present and exit in a second. The wait
the panel reports is the later one: a view that lands on a node which never ran that campaign,
or one whose kubelet has since garbage-collected the image. Neither is reachable from a
campaign-completion hook — the first is the placement limit below, and the second would evict a
prewarmed copy just as readily. Warming at view time is no better, since AuxPodSession
creates a pod with that image immediately anyway and a second pod would race the same pull for
no gain. So the honest answer is that this particular latency needs multi-node warming rather
than another fire point — which the family DaemonSet now provides for a campaign that ran a
family image directly, and still does not for one that built its own.
Two further things it deliberately does not do. It does not fire on the restart branch of
get_image_build_status, which holds only a build_id — and build_id_for does not
reverse into a ref, since it lowercases and folds _ to - where concrete_image_ref
does not, so a tag like my_sut would yield a ref no registry serves and a prewarm that
reports nothing while warming nothing. And exec_in_container does not wait on a prewarm in
flight: the kubelet serializes pulls per node, so a pod asking for the same ref queues behind the
prewarm rather than duplicating the transfer, and a wait loop would add latency without moving a
byte earlier.
The registry stays authoritative. Warmth is an optimization on top of it, never a source of
truth: present() still asks the registry, a node holding an image is not evidence it was
pushed, and nothing about digests or provenance changes. Placement is where the Job shape is
still best-effort — it carries no nodeSelector, and neither does the exec pod, so on a
single-node cluster they are necessarily the same node and on a larger one an experiment image
may be warm on a node the exec pod does not get. The family DaemonSet has no such limit by
construction, and carries no nodeSelector either for the opposite reason: it wants every
node. It does mirror the ResourceFlavor’s toleration, read from where that flavor is written, because a
warm pod that does not tolerate what campaign pods tolerate would skip exactly the nodes worth
warming — and report success doing it.
The pod env is the site default; the request overrides it. That ordering is the whole
reason a dev run needs no redeploy. Every per-image variable is carried into the service
pod, so vast cluster setup --force moves what it says it moves: one read in-pod and
set nowhere is a setting that appears to take and does not.
Reproducibility lives in the recorded digest. resolve_robovast_image resolves an
unpinned ref and warns, rather than refusing a mutable default outright so that the image a
campaign runs is always pinned by its author. That rule aims at the right thing from the
wrong layer: it makes every .vast carry a registry-specific string, which is how a
shipped example comes to pin a private registry only one site could pull. What makes a run
reproducible is the digest captured from it (pullable_digest /
_capture_image_digest, replayed by from_campaign) — a fact about what happened, not
an intention recorded beforehand.
The default tag is latest and not this installation’s version. Deriving it reads well
and is wrong: it assumes every version has a published tag, and CI publishes semver tags
only for v* pushes. Tried, it produced robovast:2.0 — a tag no workflow had ever
built — which is precisely the failure the whole change set out to remove.