How to run
vast workspace
Manage server-side workspaces (editable project inputs).
A workspace holds a self-contained project (.vast + scenario/run files) on
the service, independent of any CWD. These commands mirror the MCP workspace
tools and drive the same interface.
A workspace lives in the store of whichever service you talk to, so every
command here prints the target it resolved: a service on the conventional local
port if one answers, otherwise the one vast login stored.
vast workspace [OPTIONS] COMMAND [ARGS]...
delete
Delete a workspace and its inputs (existing campaigns are unaffected).
WORKSPACE may be a ws-… id or a workspace name (unique names resolve;
ambiguous ones must be deleted by id).
vast workspace delete [OPTIONS] WORKSPACE
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- WORKSPACE
Required argument
download
Fetch every file in WORKSPACE_ID into DIRECTORY.
The other direction of workspace init / update, so a project can be taken off a
remote service and worked on locally – and so a workspace somebody else authored can be
inspected without the web UI.
One archive, extracted into DIRECTORY: the same bytes vast share export --workspace
publishes, so a project taken off a service and one taken off the share are the same
tree.
vast workspace download [OPTIONS] WORKSPACE_ID DIRECTORY
Options
- --overwrite
Replace local files that already exist. Off by default: pulling over an edited copy of the same project would lose those edits irrecoverably.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- WORKSPACE_ID
Required argument
- DIRECTORY
Required argument
init
Create a workspace and upload every file from DIRECTORY into it.
.vast/.osc are written inline; all other files go through the upload
side channel (executability preserved). Hidden files/dirs (.cache etc.),
output dirs (results/, plus any --exclude) and any campaign results tree
— recognised by its contents, so a downloaded campaign under its own id is caught
too — are skipped, and every skip is reported. Prints the new workspace id.
vast workspace init [OPTIONS] DIRECTORY
Options
- --name <name>
Workspace name (default: the directory name).
- --exclude <NAME>
Directory name to skip (repeatable). Adds to the default [‘results’].
- --include-results
Upload campaign results trees too. Off by default: they are a campaign’s OUTPUT, so pushing them back as project input uploads every past campaign on disk. What was skipped is always reported.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- DIRECTORY
Required argument
list
List workspaces (newest first).
vast workspace list [OPTIONS]
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
preview
Show what the sweep expands to, without running any of it.
The answer to “how many configurations will this be” — asked before a launch rather than discovered from a campaign that turned out to be ten times the intended size.
vast workspace preview [OPTIONS] WORKSPACE [VAST]
Options
- --max-configs <max_configs>
Show at most this many configurations (0 = all).
- Default:
0
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- WORKSPACE
Required argument
- [VAST]
Optional argument
run
Run a .vast — the one way to start a campaign from a project.
WORKSPACE is a ws-… id or a workspace name. VAST is the path to the .vast
within that workspace; omit it when the workspace holds exactly one and the
service will resolve it, naming the candidates if there are several.
A workspace is a directory holding files and possibly several .vast files; a
project is one .vast and the files it references. So the pair (workspace, path)
names the project to run, which is exactly what the service accepts —
workspace_id is the only project binding it takes.
Fire-and-forget by default: it returns once the campaign is launched. Track it with
vast campaign wait <campaign-id> or the web UI. --wait-and-download instead blocks
until it finishes and pulls the archive down.
vast workspace run [OPTIONS] WORKSPACE [VAST]
Options
- --push <DIR>
Sync DIR into WORKSPACE before launching, creating the workspace if it does not exist. The two-step form (workspace init/update, then run) is the same thing spelled out.
- --filter <GLOB>
Run only configurations matching this name or glob (e.g. hall*).
- -r, --runs <runs>
Override execution.runs (default: the value in the .vast).
- --campaign-name <campaign_name>
Override the campaign name; the id becomes <name>-<timestamp>.
- --description <TEXT>
One line saying what this run is for. It is what tells two same-day <name>-<timestamp> campaigns apart in the web UI.
- --priority <N>
Which campaign the cluster queue admits first: higher goes first, 0 is normal, negative waits behind everything else. Ordering only — it never stops a run that has started. Refused by a service that has no queue.
Stream a raw (pre-postprocess) archive to the configured share when the campaign finishes.
- --allow-opaque-image
Launch even though a container names an image that declares no provenance:. Refused by default because nothing in the results could then say what ran; the exemption is recorded on the campaign.
- --image-project <PROJECT>
Registry/namespace to take the RoboVAST images from for this run, overriding ROBOVAST_PROJECT. Affects only images RoboVAST publishes — a container image your .vast names is run as written.
- --image-project-tag <TAG>
Tag to take those images at (default: ROBOVAST_PROJECT_TAG, else latest).
- --wait-and-download
Block until the campaign finishes and its results are uploaded, then download its archive into the current directory.
- --poll-interval <poll_interval>
Seconds between status polls when –wait-and-download is set.
- Default:
5.0
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- WORKSPACE
Required argument
- [VAST]
Optional argument
update
Re-sync DIRECTORY into an EXISTING workspace (id or name).
Uploads every file from DIRECTORY, overwriting in place — .vast/.osc
inline, everything else via the upload side channel. Hidden files/dirs,
output dirs (results/, plus any --exclude) and any campaign results tree
(recognised by its contents) are skipped, and every skip is reported. With
--prune it also removes workspace files that no longer exist locally, so
the workspace mirrors DIRECTORY exactly.
vast workspace update [OPTIONS] WORKSPACE DIRECTORY
Options
- --exclude <NAME>
Directory name to skip (repeatable). Adds to the default [‘results’].
- --prune
Also delete workspace files that are absent from DIRECTORY (full mirror). Off by default: update only adds/overwrites.
- --include-results
Upload campaign results trees too. Off by default: they are a campaign’s OUTPUT, so pushing them back as project input uploads every past campaign on disk. What was skipped is always reported.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- WORKSPACE
Required argument
- DIRECTORY
Required argument
validate
Check a project before spending any compute on it.
Reports every problem at once rather than the first, because they fail independently and fixing them one launch at a time is the expensive way to find out.
Computed service-side: the checks need the config schema, the scenario image’s own
parser and (unless --no-world-check) the simulator, none of which a client install
has. So this works the same whether the service is local or remote. The two container
checks are what cost more than a moment; each has a flag to skip it.
vast workspace validate [OPTIONS] WORKSPACE [VAST]
Options
- --no-world-check
Skip the world check, which builds the model in a container.
- --no-scenario-check
Skip parsing the scenario in the image that would run it. That is the only check that sees an import the image does not carry.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- WORKSPACE
Required argument
- [VAST]
Optional argument
world
Describe the world this campaign’s simulator will load.
The other half of authoring a sim: override: vast workspace world says what the
world offers — which components an override can address, and with --targets which model
values a run may change at all. Both are otherwise only refused inside the container, after
the image pull.
Answered by the simulator, in the image the campaign runs, so the reply names that image: which world a ref resolves to depends on what is installed there.
vast workspace world [OPTIONS] WORKSPACE
Options
- --path <VAST>
Which .vast in a multi-.vast workspace (default: the only one).
- --targets <GLOB>
Also report the objects matching GLOB whose model values a run may override, with their current values. Costs a model build.
- --entities
Also list the entities the world compiles, and what its start state warns about. Costs a model build and a reset.
- --json
Print the raw description as JSON.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- WORKSPACE
Required argument
vast campaign
Act on a campaign: list, watch, stop, read its log, re-run it.
Every verb drives the robovast-service, so these work the same whichever deployment runs the campaign.
To start a campaign from a project, use vast workspace run.
vast campaign [OPTIONS] COMMAND [ARGS]...
delete
Permanently delete one or more CAMPAIGNs wholesale.
Removes each campaign’s directory under the service’s results root, plus, on a cluster
service, any leftover Kubernetes Jobs. This is the full “forget this campaign” action;
vast share remove only touches the external share, which this command leaves
untouched.
Each campaign is deleted or refused on its own, and reported on its own line: the service refuses one that is still running (stop it first) and an id that is not a campaign id, and deletes the rest. The exit status is non-zero if any campaign was not fully deleted. An id that has nothing left here counts as deleted. This is irreversible.
vast campaign delete [OPTIONS] CAMPAIGN...
Options
- -y, --yes
Skip the confirmation prompt.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- CAMPAIGN...
Required argument(s)
download
Download campaign archives from the service, one .tar.gz each.
That is the whole command: it fetches <campaign-id>.tar.gz and stops. Nothing is
extracted, no results directory is written into, and no state is kept about what you
already have – the archive is yours, to keep, copy, unpack, or hand back with vast
campaign import.
The archive is the campaign as the service holds it, postprocessing and all. The
share’s raw, pre-postprocess snapshot is a different system with different
credentials: vast share download.
A campaign that is still running downloads too, and lands as
<campaign-id>.incomplete.tar.gz – runs that had not finished are simply not in it.
The service names that file, so the name cannot disagree with what is inside.
Writes into the current directory unless -o says otherwise – an archive is a
file, not a results tree, so a results directory is the wrong home for it.
One campaign that fails does not stop the others: each is reported on its own line and the exit summary counts what landed. A thin single-archive copy cannot do several, resume past a failure, or show progress on a multi-gigabyte transfer, so this is the one implementation.
vast campaign download [OPTIONS] CAMPAIGN...
Options
- -o, --output <output>
Directory to write the archives into [default: the current directory]
- -f, --force
Overwrite an archive of the same name that is already here
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- CAMPAIGN...
Required argument(s)
import
Take a campaign archive into the service, and postprocess it if it needs it.
ARCHIVE is a .tar.gz on this machine – one vast campaign download or vast
share download produced, or a colleague sent. It is uploaded to the service and
imported there, so the campaign lands where the web UI and every other client can see
it; the file itself is never deleted, it is yours.
Importing is more than extracting: listings and the web UI answer from campaign.db,
not from the results tree, so a campaign that is only unpacked is invisible. And when
the archive is a raw one – carrying no postprocessing record, which is what the
share holds – postprocessing is chained automatically, because a campaign without its metric tables
is not one you can ask anything.
Long-running, so it returns once the import is under way: the campaign appears
immediately at phase importing. Watch it with vast campaign wait <campaign-id>,
or in the campaign view.
It creates a campaign, which is why it is here and not in vast results: an upload
and one HTTP call, needing nothing of the local half of the tool. It sat in the group
that ships only with the full distribution, so the install most likely to be talking to
a remote service was the one that could not import into it.
vast campaign import [OPTIONS] ARCHIVE
Options
- --force
Replace a campaign of the same id already there.
- --rebuild-store
Reconstruct campaign.db from the results tree (the recovery for a corrupt one).
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- ARCHIVE
Required argument
list
List campaigns this service knows about: live ones first, then newest first.
The size column is what the results occupy, measured once when the campaign ended;
- means none is recorded (a campaign still running, or one never measured).
description is what tells two same-day <name>-<timestamp> ids apart, which is
why the launch verbs ask for one. A live campaign’s standing with the queue follows its
phase when it is not the default – prio +2, paused – because a held campaign
making no progress is otherwise indistinguishable here from a wedged one.
vast campaign list [OPTIONS]
Options
- --limit <limit>
How many to show, in the listed order.
- Default:
20
- --sort <sort_key>
Order by when a campaign ended (started, while live) or by its results size. Live campaigns lead either way.
- Default:
recent- Options:
recent | size
- --desc, --asc
Newest or largest first, or the reverse. A campaign with no size is listed last either way.
- Default:
True
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
log
Print a campaign’s unified infrastructure log.
The same divider-separated stream the web UI and MCP show — the variation
(config-generation), run (controller) and postprocessing phases in order, each
under a ===== PHASE ===== divider.
One reader, over HTTP, and no fallback to assembling the log from a campaign directory when no service answers – that needs the core installed, takes a path where every other verb takes a campaign id, and is a second implementation of “read the log” that a client-only install could not reach anyway.
vast campaign log [OPTIONS] [CAMPAIGN]
Options
- -f, --follow
Stream new output until the campaign finishes
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- [CAMPAIGN]
Optional argument
pause
Stop admitting new runs for a campaign; resume starts them again.
The runs it has already started finish normally and their results are kept: this holds back what is queued, so the campaign drains rather than stopping, and the cluster is free for something else within one run’s length. It keeps the priority it will resume at.
vast campaign pause [OPTIONS] [CAMPAIGN]
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- [CAMPAIGN]
Optional argument
postprocess
(Re)run analysis postprocessing for CAMPAIGN.
The rosbag->CSV step runs in-cluster and the campaign’s derived data is rebuilt.
Mirrors the web “Retrigger postprocessing” action and the MCP run_postprocessing
tool, so all three drive one implementation.
This was vast results reprocess, beside a vast results postprocess that did the
same job in-process against a results directory on this machine. Two postprocessing
paths is the same split vast exec local run was: one of them could only ever see a
local campaign, and it was the one the documentation reached for first.
Dispatched, not awaited. Postprocessing takes minutes to hours, so the campaign
re-enters its postprocessing phase and this returns – follow it with vast
campaign wait CAMPAIGN, exactly as after vast workspace run. A second run is
refused while one is in flight.
vast campaign postprocess [OPTIONS] [CAMPAIGN]
Options
- -f, --force
Bypass per-rosbag caches and reprocess all bags.
- --skip <PLUGIN>
Skip a postprocessing plugin (repeatable), e.g. –skip rosbags_to_webm.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- [CAMPAIGN]
Optional argument
priority
Set which campaign the queue admits first. Higher goes first; 0 is normal.
For getting a short campaign through while a long one is running: demote the long one (a negative VALUE) or promote the short one. Takes effect on the next admission pass, and applies to the batches the campaign has not submitted yet as well as the jobs queued now.
Ordering only – nothing already running stops. The campaign you demote keeps the runs it
has and gives up only the slots they release, so no partial run is produced and no results
are lost. To end a campaign instead, use vast campaign stop.
Needs a service that queues campaigns against each other; one with no queue refuses.
vast campaign priority [OPTIONS] VALUE [CAMPAIGN]
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- VALUE
Required argument
- [CAMPAIGN]
Optional argument
rerun
Launch a NEW campaign from what CAMPAIGN_ID recorded. The source is not modified.
Reuses the frozen config and the image the source recorded, so it runs the same code rather than today’s. A config older than the current version is migrated into the staging copy; the archived one is left exactly as its author wrote it.
The service runs the pre-flight itself and refuses on a blocking axis, naming what is
missing – so a re-run that could only fail in the backend is answered before the launch.
--force tells it to proceed anyway, which is worth having for an axis you have decided
you understand.
vast campaign rerun [OPTIONS] CAMPAIGN_ID
Options
- --check
Report whether this campaign can be re-run, and launch nothing. Costs nothing: it answers from what the campaign recorded. Exits non-zero when an axis is genuinely blocked.
- --force
Ask the service to launch even though the pre-flight reports a blocking axis. What is being overridden is printed before the launch.
- --to-workspace <NAME>
Do not launch. Materialise the campaign as a workspace with its config migrated as far as it could be and a marker at every decision left, to finish by hand. For a config no ladder step can carry forward.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- CAMPAIGN_ID
Required argument
resume
Admit runs again for a paused campaign, at the priority it was paused at.
vast campaign resume [OPTIONS] [CAMPAIGN]
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- [CAMPAIGN]
Optional argument
status
Print a campaign’s phase and progress once, and exit.
The single-read counterpart to vast campaign wait: use this for a campaign you are
not waiting on. Waiting is a separate verb because it can take days, and holding a
request open for that is a different thing from asking once.
vast campaign status [OPTIONS] [CAMPAIGN]
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- [CAMPAIGN]
Optional argument
stop
Ask a running campaign to stop.
Goes through the robovast-service, which drives the campaign in-process, so the stop reaches whatever it is doing and the campaign is published as usual. What it leaves behind depends on which work was running; the reply says. A no-op if nothing is running.
vast campaign stop [OPTIONS] [CAMPAIGN]
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- [CAMPAIGN]
Optional argument
stop-job
Kill ONE running job; the rest of the campaign keeps going.
For a job that is visibly wedged and will not exit on its own. This is not how a
campaign is ended – that is vast campaign stop, which stops all of it.
Only a running job can be stopped. The kill is permanent and recorded: the runs it
cut short report status='killed' in the campaign data, with the reason, and count
as neither passes nor failures.
vast campaign stop-job [OPTIONS] JOB_NAME [CAMPAIGN]
Options
- --reason <TEXT>
Why you stopped it — stored with the run and shown in the results
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- JOB_NAME
Required argument
- [CAMPAIGN]
Optional argument
wait
Block until CAMPAIGN is over; the exit code says how it ended, or why the wait did.
The service drives every campaign, so its phase is the campaign’s. Prints each phase change as it happens and exits when the campaign reaches a terminal one — past postprocessing, not merely past the last run.
Exists so a caller can wait without holding a request open, and is why the MCP
offers no campaign-wait tool: an agent harness can background this command and be
notified when it exits, hours or days later, where a blocking tool call would occupy
the conversation for as long as the campaign ran — and still not outlive the session.
The loop itself is wait_for_campaign_status(),
shared with every other surface that waits.
A stall ends the wait too (STALLED), because a stalled campaign never reaches a
terminal phase: it holds running for its whole life, so a waiter that stopped only
on terminality would never return and nobody would be told. The verdict is
stall_report()’s, not a second opinion computed here.
An ``error``-level health finding ends it too (HEALTH_FINDING), and earlier: a
stall is only visible once a run is past its declared budget, and needs one to have been
declared at all, while a simulator saying “sim time is not advancing” is true within a
minute and needs no budget. Whatever a finding means is the simulator’s business – this reads one
word, level, and passes the rest through.
Only a new stall or a new finding exits. A campaign already stalled when this
command starts is not news – whoever ran it has just been told – and exiting on the
first poll would make vast campaign wait unusable for exactly the state it reports: the message
says to re-run it after diagnosing, and a fresh waiter would exit immediately, forever. So
the first observation is recorded and only a rising edge stops the wait; for findings the
edge is per check, so one check firing repeatedly is one exit and not a stream.
vast campaign wait [OPTIONS] CAMPAIGN
Options
- --interval <interval>
Seconds between status polls.
- Default:
10.0
- --timeout <timeout>
Give up after this many seconds (default: wait indefinitely).
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- CAMPAIGN
Required argument
vast config
Manage run configuration.
vast config [OPTIONS] COMMAND [ARGS]...
generate
Generate VAST’s run configurations and output files into OUTPUT_DIR.
vast config generate [OPTIONS] VAST OUTPUT_DIR
Options
- --keep-transient
Keep and display temporary folders used during generation (e.g. by FloorplanGeneration).
- --no-cache
Skip cache lookup and force a fresh generation even if inputs are unchanged.
Arguments
- VAST
Required argument
- OUTPUT_DIR
Required argument
variation-points
List the variation points VAST’s scenarios offer.
Shows every scenario parameter that can be varied.
vast config variation-points [OPTIONS] VAST
Arguments
- VAST
Required argument
variation-types
List available variation types.
Shows all registered variation type entry points that can be used in the variations section of .vast configuration files.
vast config variation-types [OPTIONS]
list
List the configurations VAST expands to, without generating files.
Shows what would be generated, creating none of it.
vast config list [OPTIONS] VAST
Options
- --debug
Show internal values starting with _
Arguments
- VAST
Required argument
vast container
Run a command in an experiment container, and stop the held one.
For testing a container and its setup – an import, a ros2 pkg list, a file check
– before a campaign spends compute finding the same thing out.
vast container [OPTIONS] COMMAND [ARGS]...
exec
Test a container and its setup by running SHELL_COMMAND in the experiment image.
Produces no campaign data — nothing durable, no provenance, no repetitions. Use
vast workspace run to run the experiment itself.
Omit --config-name to check the bare image (python3 -c 'import x',
ros2 pkg list, file checks). Name one to stage that configuration; an empty
SHELL_COMMAND then starts its scenario, and its output goes to a log file inside the
container rather than to stdout (the path is printed).
The image is the one this host’s copy of the project builds, so a stale checkout
checks a stale image; the build:<tag>@<hash> that ran is printed with the exit line.
There is at most one such container at a time; vast container stop
ends it. No --timeout: the limit is derived from what is being run (the
project’s execution.timeout for a scenario, a fixed cap for a command) and
reported with the result.
vast container exec [OPTIONS] [SHELL_COMMAND]
Options
- --workspace <workspace_id>
Workspace whose project to use (with –config for the .vast).
- --config <config_path>
Which .vast in the workspace (workspace-relative).
- --campaign <campaign_id>
Use an existing campaign’s _config/ as the project instead.
- --config-name <config_name>
Stage this configuration. Omitted, the bare image is used.
- --keep-alive
Leave the container running so later calls can inspect it.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
Arguments
- SHELL_COMMAND
Optional argument
stop
Stop the held container-exec container, if there is one.
vast container stop [OPTIONS]
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
vast cluster
Set up and maintain the Kubernetes substrate campaigns run on.
These read what they need from the cluster itself, so they work from any directory.
To act on a campaign, use vast campaign; on the service, vast service.
Every verb needs a kubeconfig and arrives with robovast-cluster, so what this
lists depends on what is installed.
vast cluster [OPTIONS] COMMAND [ARGS]...
cleanup
Clean up the Kubernetes cluster setup.
Removes the robovast pod (registry and index), the robovast-service
Deployment, the build daemon and the cluster-wide pieces setup installed. The data
directories on the node are kept unless --delete-data asks otherwise.
This command can be run after completing all scenario executions to clean up cluster infrastructure resources (different from jobs-cleanup which only cleans up job pods).
If --cluster-config is not specified, it will automatically detect which
cluster configuration was used during setup by reading it back from the deployed
robovast-service — so this works from any host and needs no project.
When specifying --cluster-config explicitly, pass -n <namespace> if the
setup was done in a non-default namespace.
vast cluster cleanup [OPTIONS]
Options
- -c, --cluster-config <config_name>
Cluster configuration plugin to use (auto-detects if not specified)
- -n, --namespace <namespace>
Kubernetes namespace to clean up (required when using –cluster-config without prior setup)
- -o, --option <options>
Cluster-specific option in key=value format (can be used multiple times)
- -x, --context <kube_context>
Kubernetes context to use (default: active context in kubeconfig)
- --delete-data
Also empty this deployment’s data directories on the node: the workspaces, the results, the campaign index and the registry. Cleanup keeps them by default, because the results hold every finished campaign and a teardown is a very expensive way to discover that. Irreversible, and no archive is taken first – use vast share or vast campaign download for anything worth keeping.
- --forget-placement
Also remove the node labels recording where this deployment kept its data. Without this the labels stay, so a later setup lands on the same node with no flags – which is the point of them. The on-disk data is not removed either way.
- --vast <FILE>
A .vast to pre-flight against this cluster. Its only use here is to refuse when it declares per-cluster resource lists for several contexts and –context was not given – which would otherwise pick a cluster by accident. Optional, and read only for that check.
jobs-cleanup
Clean up jobs and pods from a cluster run.
Removes scenario execution Jobs and their pods directly (using your kubeconfig
— the -x context, -n namespace). By default removes all campaigns; use
--campaign for one.
The campaign’s results are untouched: they live with the service, and
vast campaign delete is what removes a campaign.
vast cluster jobs-cleanup [OPTIONS]
Options
- -i, --campaign <campaign>
Clean only jobs for this campaign (e.g. campaign-2025-02-27-123456). Without this, cleans all scenario-runs jobs.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
- --vast <FILE>
A .vast to pre-flight against this cluster. Its only use here is to refuse when it declares per-cluster resource lists for several contexts and –context was not given – which would otherwise pick a cluster by accident. Optional, and read only for that check.
monitor
Monitor scenario execution jobs on the cluster.
Displays progress per run: how many jobs have finished (completed or failed), how many are running, and how many are pending for each run.
By default, monitors only the contexts referenced in the .vast config file. Falls back to the active kubeconfig context when no per-cluster config is defined. Use –context to restrict monitoring to a single cluster. Only contexts with active or past jobs are shown.
This is intended for monitoring jobs created by a campaign launch.
vast cluster monitor [OPTIONS]
Options
- -i, --interval <interval>
Polling interval in seconds
- Default:
2.0
- --once
Print job status once and exit
- -x, --context <kube_context>
Kubernetes context to use (default: active context in kubeconfig)
- -n, --namespace <namespace>
Kubernetes namespace the scenario Jobs run in.
- Default:
default
- --vast <FILE>
Watch every context this .vast names, instead of only the active one. Ignored when –context is given, which names one directly.
setup
Set up the Kubernetes cluster for execution.
Deploys the robovast pod (the container registry and the campaign index), the
robovast-service Deployment that drives campaigns and keeps their results on its
results volume, the shared build daemon, and the cluster-wide pieces they need (RBAC,
the GPU device plugin, the placement labels).
This command should be run once before executing scenarios on the cluster for the first time.
If the cluster is already set up, this command will exit with an error.
Run ‘vast cluster cleanup’ first to clean up the existing setup,
or use --force to force re-setup.
Use --list to see available cluster configuration plugins.
Cluster-specific options can be passed using --option key=value.
Reads no project and no .vast: this deploys into a cluster and runs from any
directory. Which nodes the cluster’s pods may use is a property of the cluster. The
campaign job node pool comes from ROBOVAST_JOB_NODE_LABELS in the environment, a JSON
object of label -> value; setup and upgrade both write it into the service, so unset
means every node. --control-node-label is written on every run, so omitting it clears
what a previous setup configured.
The job node aliases a campaign may confine itself to (execution.kubernetes.jobs.node)
come from ROBOVAST_JOB_NODE_ALIASES in the environment, a JSON object of alias -> node
name. Setup and upgrade both reconcile the node labels to exactly what it states – unset,
every alias is removed – and refuse the whole set, before anything is changed, if an
alias is not legal or its node is not schedulable inside the job pool.
Share credentials (ROBOVAST_SHARE_TYPE and its provider variables — e.g.
ROBOVAST_GCS_BUCKET / ROBOVAST_GCS_KEY_FILE) are read from the host
environment / project .env at setup and handed to the in-cluster service
as a Secret, so --upload-to-share campaigns work from the cluster. A key
file is inlined into the Secret; nothing else is needed on the host.
vast cluster setup [OPTIONS] [CLUSTER_CONFIG]
Options
- --list
List available cluster configuration plugins
- -n, --namespace <namespace>
Kubernetes namespace for execution (used by cluster run)
- Default:
default
- -o, --option <options>
Cluster-specific option in key=value format (can be used multiple times)
- -f, --force
Force re-setup even if cluster is already set up
- --gpu-replicas <N>
Advertise N time-slicing replicas per physical GPU, so N pods can share one card. Without this flag a GPU that is present is used anyway (with a sensible default) and a cluster without one is left alone; passing it makes GPU support a requirement, so a cluster that cannot provide it becomes an error. N caps concurrency and does NOT partition VRAM: all N renderers allocate from the same card.
- --no-gpu
Set the cluster up without GPU scheduling, even if it has a GPU.
- -x, --context <kube_context>
Kubernetes context to use (default: active context in kubeconfig)
- --ingress-host <HOST>
Publish the service at this hostname, so users reach it in a browser without kubectl. Needs TLS (see –issuer/–tls-secret) and an access token; both are refused otherwise.
- --ingress-class <NAME>
IngressClass to use (e.g. nginx). Default: the cluster default.
- --issuer <NAME>
cert-manager ClusterIssuer to obtain the certificate from. tools/setup_ingress_tls.py creates one.
- --tls-secret <NAME>
Existing TLS Secret to serve, instead of a cert-manager issuer.
- --insecure-http
Publish over plain HTTP. The shared token then crosses the network in clear text; only for a trusted network.
- --rotate-token
Issue a new access token, logging everyone out. Without this an already-deployed token is preserved across re-runs.
- --data-root <PATH>
Put every node-local directory this deployment keeps under one path: the workspaces, the campaign results beside them, the registry and the build cache. The flag for the ordinary case – a machine with a disk mounted for RoboVAST – and any per-tenant flag below overrides it for its own tenant. Reads ROBOVAST_DATA_ROOT, so a .env can set it once per machine. It does NOT place a campaign job’s scratch: that is an emptyDir under the kubelet root, which is the node’s configuration rather than this deployment’s.
- --workspaces-path <PATH>
Host directory holding the service’s workspaces (default: /var/lib/robovast-workspaces). The campaign results root – where finished campaigns live – is placed beside it, and the campaign index beside that; neither needs a flag of its own.
- --workspaces-class <NAME>
Back the workspaces and the campaign results with PVCs from this StorageClass instead of hostPaths, which also unpins the service pod. The campaign index follows unless –index-class says otherwise. Stock RKE2 provisions nothing, which is why hostPath is the default.
- --results-size <SIZE>
Size of the campaign results PVC (default: 500Gi) – the volume every campaign lives on. Needs –workspaces-class, which is what backs it: without one the results are a directory on the data node, bounded by that disk, and there is no volume to size. The same flag on ‘vast service upgrade’ raises an existing claim.
- --index-class <NAME>
Back the campaign index with a PVC from this StorageClass. Without it the index takes the results’ backing: a PVC of the workspaces’ class where one is given, else a directory beside the results on the data node. Its own flag because Postgres is a different workload from the bulk of the results, and on a managed node pool the index is what a replaced node would otherwise take with it.
- --index-size <SIZE>
Size of the campaign index PVC (default: 20Gi). Needs a class, its own or the workspaces’: without one the index is a directory on the node and there is no volume to size.
- --registry-class <NAME>
Back the built-in container registry with a PVC from this StorageClass instead of a hostPath. Preferred where the cluster can provision volumes; stock RKE2 cannot, which is why hostPath is the default.
- --registry-path <PATH>
Host directory backing the registry when using hostPath (default: /var/lib/robovast-registry).
- --data-node <NODE>
Hold this deployment’s node-local data on this node: the workspaces, the results, the index, the registry and, unless –buildkit-node says otherwise, the build cache. Rarely needed – setup picks the node with the most free space the first time and records the choice as a node label, so later runs stay put without any flag. Naming a node moves the placement off whatever node holds it now and says so; the bytes are NOT migrated, so the new node starts empty – including the results, which is where finished campaigns live.
- --buildkit-class <NAME>
Back the shared build daemon’s cache with a PVC from this StorageClass instead of a hostPath. Prefer an SSD class: BuildKit’s snapshotter is small-file heavy and a slow disk becomes the bottleneck the cache was meant to remove.
- --buildkit-path <PATH>
Host directory backing the build cache when using hostPath (default: /data/robovast-buildkit). This is what makes a base image pulled once stay pulled.
- --buildkit-size <SIZE>
Size of the build cache PVC (default: 200Gi). Needs –buildkit-class: without one the cache is a directory on the node, bounded by that disk and the daemon’s GC ceiling, and there is no volume to size.
- --buildkit-node <NODE>
Hold the build cache on this node. Defaults to the data node, because auto-separating would put a 150 GB cache on whichever node was left over. Separate them where the disk is tight: these are the deployment’s two large on-disk tenants, and a full builder disk on the service’s node becomes DiskPressure evictions of the API.
- --performance-governor, --no-performance-governor
Set the nodes’ CPU governor to ‘performance’. ON by default, because a cluster whose clock moves with load produces numbers nothing downstream can detect or correct: a per-node figure is then taken against a clock that was not the same for every run. Needs a privileged pod with the host /sys writable; a cluster that refuses it (managed Kubernetes does) is WARNED about and setup continues. Naming the flag makes that refusal an error instead. –no-performance-governor skips it and leaves the hosts alone.
- --tailnet, --no-tailnet
Publish the service on a WireGuard tailnet instead of an Ingress: a node beside it dials OUT to a coordination server and answers under a stable name, so nothing listens on the internet and users need no kubeconfig. OFF by default. The coordination server and pre-auth key come from the environment (ROBOVAST_TAILNET_LOGIN_SERVER / _AUTHKEY), so a key does not land in shell history – but WHICH cluster is on a tailnet is decided here, because one .env and two contexts would otherwise publish whichever was current. Written on every setup: omitting it removes a node a previous setup deployed.
- --control-node-label <KEY=VALUE>
Run RoboVAST’s own infrastructure pods on nodes carrying this label; repeatable. Narrows rather than decides: these are ANDed with the node-local data placement setup chooses. Pass none to clear.
Arguments
- CLUSTER_CONFIG
Optional argument
Environment variables
- ROBOVAST_DATA_ROOT
Provide a default for
--data-root
- ROBOVAST_WORKSPACES_PATH
Provide a default for
--workspaces-path
- ROBOVAST_WORKSPACES_CLASS
Provide a default for
--workspaces-class
- ROBOVAST_RESULTS_SIZE
Provide a default for
--results-size
- ROBOVAST_INDEX_CLASS
Provide a default for
--index-class
- ROBOVAST_INDEX_SIZE
Provide a default for
--index-size
- ROBOVAST_REGISTRY_CLASS
Provide a default for
--registry-class
- ROBOVAST_REGISTRY_PATH
Provide a default for
--registry-path
- ROBOVAST_BUILDKIT_CLASS
Provide a default for
--buildkit-class
- ROBOVAST_BUILDKIT_PATH
Provide a default for
--buildkit-path
- ROBOVAST_BUILDKIT_SIZE
Provide a default for
--buildkit-size
vast service
Act on the robovast-service: read its log, roll it onto new bytes.
The verbs that need only a URL and a token ship with robovast-client; the ones
that reach a Kubernetes API server arrive with robovast-cluster.
vast service [OPTIONS] COMMAND [ARGS]...
cache
What the service’s rebuildable caches hold; with –clear, free what may go.
Only copies of durable data – a campaign’s files fetched from the object store, compiled 3D worlds – so clearing loses nothing but the time to rebuild what is next asked for. The thing to reach for when a launch is refused for free space. What a running campaign, an operation or a recent reader may still be using is kept, and listed with the reason.
vast service cache [OPTIONS]
Options
- --clear
Remove every entry nothing may still be using, and say what that freed.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
info
Which service is answering, which code it runs, and which backend it drives.
Call this first when something behaves unexpectedly. A service loads robovast once,
at startup, so after an edit a reachable service may still be running the old code;
comparing revision with your tree is how you find that out.
When revision is absent the check is unavailable and there is no substitute –
version is the package semver, so it stays the same across every edit and reading
it as a revision is a live trap.
robovast_version, the field the compatibility handshake compares, is not printed:
it resolves to a revision when there is one and to the semver otherwise, so it is
always already on one of these two lines. Printing it as version as well would show
the same SHA twice on every deployed service.
vast service info [OPTIONS]
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
log
Print what the robovast-service itself has been doing.
Not a campaign’s log – this is the service process: what it decided, what it refused,
and the reason behind a failure whose visible half was one terse line. Several failures
say so in as many words (“the real reason is only in the service log”), and until now
there was no way to read it short of kubectl logs.
The service keeps the last few hundred kilobytes in memory, so this covers what it has
been doing recently, not its whole life, and a restart clears it. A container that has
already died is only in kubectl logs -p deploy/robovast-service: a buffer inside a
process cannot outlive the process.
vast service log [OPTIONS]
Options
- -f, --follow
Keep printing new output until interrupted.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
mcp-stats
Which MCP tools agents actually call, and what happened when they did.
The same record the web UI’s Admin page shows, on a terminal: a ranking of tools by
call count with their failure share and durations, or the calls themselves with what
each was given and what it answered (--calls).
Deliberately a command and not an MCP tool. An agent asking this question would pay for the whole answer in context, and the question – “is this tool surface worth what it costs?” – is one asked about agents rather than by one.
The record lives in the central index, so it outlives a service restart but not the results store, and it covers a bounded window that the ranking prints.
vast service mcp-stats [OPTIONS]
Options
- --calls
Print the call log instead of the per-tool ranking.
- --tool <tool>
Only this tool.
- --failed
Only calls that failed.
- --limit <limit>
How many calls to print with –calls.
- Default:
50
- --csv
Write the call log as CSV to stdout, for a spreadsheet or a script.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
resources
Does the cluster have room, and is it reachable?
Ask before a sweep.
pending is work the backend has accepted but is not executing, which is why it is
counted apart from usage rather than folded into it – counting queued work as used
reported more cores in use than the cluster had.
vast service resources [OPTIONS]
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
restart
Roll the deployed service onto the newest image at its tag, and nothing else.
Asks the service to restart itself, so this needs only a URL and a token – which is the
point: vast service upgrade needs a kubeconfig, so somebody who reached the
deployment through vast login had the web UI’s button and no command at all.
This reconciles none of those. A version needing a permission the last one did not
will deploy and then fail at runtime with a 403, which reads as a bug rather than as a
missed migration. So this is the command for “new bytes are published and nothing else
changed”; for a version bump, a missed RBAC migration, a rotated Secret or a registry
move, use vast service upgrade. The Secrets in particular are rebuilt from the
operator’s environment, which the pod does not have, so they can never be done from in
there at any level of privilege.
Refuses while campaigns are live – their controller runs in the pod being replaced –
and names them. --yes skips the question.
vast service restart [OPTIONS]
Options
- -y, --yes
Do not ask before rolling over live campaigns. For scripts; without it a non-interactive run aborts rather than rolling silently.
- --wait
Block until the new pod is the one serving, instead of returning once the roll has been asked for.
- -n, --namespace <namespace>
Namespace the robovast-service runs in.
- Default:
default
- -x, --context <NAME>
Kubernetes context, when a command has to name one.
vast service upgrade
Move a running instance to a new RoboVAST version.
Rolls the Deployment onto the resolved image, reconciles RBAC, and waits for the new pod to be the one serving before reporting anything.
It fails, non-zero, if that pod does not take over: an image it cannot pull, a node it
cannot be scheduled on, or a container that crash-loops. The reason Kubernetes gave is
printed as soon as it appears, so a stuck upgrade names its cause in seconds instead of
looking like a hang – and “✓ upgraded and ready” now means it. Use --timeout for a
registry slow enough to need longer.
Always restarts the pod, even when nothing appears to have changed. That is the
point: an image ref that is a floating tag, or a change confined to the Secrets,
leaves the Deployment spec byte-identical, and Kubernetes then rolls nothing while
this command reports success. It also makes the restart the only way the env
Secrets are re-read – the pod loads them through envFrom at container start and
never again. The cost is a few seconds during which the API is unavailable.
RBAC reconciliation is not decoration: a version needing a permission the last one
did not — as /usage once needed a cluster-scoped ClusterRole — would otherwise
deploy and then fail at runtime with a 403, which reads as a bug rather than as a
missed migration.
--no-restart reconciles just that part — RBAC, the registry
ingress route, the optional tailnet node — and stops before the Deployment is touched. All three are picked up by
the running pod (the API server evaluates RBAC per request, and
workload, a route is the gateway’s own state), so a permission the running version is
missing can be granted without a version change and without the API blip. That is the
difference between fixing a missed migration and rolling a service: it is the only way
to do the former while a campaign is in flight, because the campaign controller lives in
the pod a roll would replace. It does not move the image and does not re-read the env
Secrets — for either of those, run the command without the flag.
The campaign job node pool and the job node aliases are applied, like setup does, from
ROBOVAST_JOB_NODE_LABELS and ROBOVAST_JOB_NODE_ALIASES in the environment – so an
upgrade from a shell without them clears the pool and removes the aliases, and says so –
after checking every alias against that pool, before anything changes. With
--no-restart the pool is not applied, since it lives in the pod’s environment.
--no-restart reconciles them too: they are node labels a campaign reads when it
starts, not the pod’s environment.
Before the roll it asks the service which campaigns are live and names them, for the
reason --no-restart exists: the pod being replaced is where their controller runs.
--yes skips the question. A service that cannot be reached is reported and the roll
goes ahead – a wedged service is a reason to upgrade, not a reason to refuse – and
that is said out loud, so a silent roll never has to be read as “nothing was running”.
Reads the environment like every vast command – ./.env then
~/.config/robovast/env – so ROBOVAST_PROJECT and ROBOVAST_PROJECT_TAG
are where the images come from, and this is the command that moves them. Both are
baked into the service pod as the site default; a campaign can still override them
per launch (vast workspace run --image-project), which needs no upgrade at all.
There is deliberately no --image. A flag would have been one-shot, because nothing
reads the deployed image back, so a later bare upgrade would silently revert it –
arriving through the command that looks safe. For a one-off, a real environment
variable already beats both files:
ROBOVAST_PROJECT=ghcr.io/cps-test-lab vast service upgrade
make image-digests PROJECT=... checks that a project holds a complete family
before you point a cluster at it.
This is the command for rolling out an image, not setup --force:
Campaign data lives on the service’s results volume and survives both.
vast service upgrade [OPTIONS]
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in
- Default:
default
- -x, --context <kube_context>
Kubernetes context to use (default: active context in kubeconfig)
- --timeout <SECONDS>
How long to wait for the new pod to take over before failing. Generous because a service that comes up owing campaigns restores them before it binds its port; a pod that is genuinely broken (ImagePullBackOff, a crash loop) still fails fast on its own, without spending this.
- Default:
1800.0
- --no-restart
Reconcile only what does not need the pod rolled – RBAC, the queues, the registry ingress route – then stop. For granting a permission the RUNNING version is missing without a version change or an API blip, e.g. while a campaign is in flight.
- --results-size <SIZE>
Raise the campaign results PVC to this size – the volume every campaign lives on. Expanded online by every StorageClass this deployment targets, so no campaign is interrupted. A claim cannot be shrunk, and a deployment whose results are a directory on the data node has no volume to size; both are refused rather than reported as done.
- -y, --yes
Do not ask before rolling over live campaigns. For scripts; without it a non-interactive run aborts rather than rolling silently.
Environment variables
- ROBOVAST_RESULTS_SIZE
Provide a default for
--results-size
vast service token
Show the access token, and what to hand users along with it.
Setup deliberately prints the token only once, so reading it back meant a
kubectl get secret ... | base64 -d incantation – which every operator then
keeps in their shell history, and which needs kubectl syntax to answer a RoboVAST
question.
The token is per cluster: an instance mints its own, and one instance’s token is simply wrong at another. That is the failure this command is most likely to prevent, since the mistake looks identical to a mistyped password.
vast service token [OPTIONS]
Options
- -n, --namespace <namespace>
Namespace the robovast-service runs in
- Default:
default
- -x, --context <kube_context>
Kubernetes context to use (default: active context in kubeconfig)
- -q, --quiet
Print only the token, for piping into something else.
vast results
Work on a results directory on THIS machine.
Every verb here names a path and none of them needs a login: this is the local half
of the tool, for someone holding a results tree – publishing it, generating its
metadata and provenance, merging campaigns. It ships with the full robovast
distribution, so a client-only install does not have this group at all, which is the
honest signal that none of it is a service operation.
What acts on a campaign is vast campaign – including postprocessing, which
lives there because the service owns the cluster the runs executed on.
vast results [OPTIONS] COMMAND [ARGS]...
merge-campaigns
Merge campaign directories with identical configs into one merged_campaign_dir.
Groups campaign-directory/config-directory by config_identifier from config.yaml. Run folders (0, 1, 2, …) from all campaigns are renumbered and copied. Original campaign directories are not modified.
vast results merge-campaigns [OPTIONS] MERGED_CAMPAIGN_DIR
Options
- -r, --results-dir <results_dir>
Source directory containing run-* directories (uses project results directory if not specified)
Arguments
- MERGED_CAMPAIGN_DIR
Required argument
postprocess-commands
List all available postprocessing command plugins.
Shows plugin names that can be used in the results_processing.postprocessing section
of the configuration file, along with their descriptions and parameters.
vast results postprocess-commands [OPTIONS]
Environment variables
ROBOVAST_INSECURE_SSLSet to
1to disable TLS certificate verification for remote fetches. This allows the CLI to continue when a remote host presents an invalid certificate. Use only with hosts you trust.