How to run

vast workspace

Manage server-side workspaces (editable project inputs).

A workspace holds a self-contained project (.vast + scenario/run files) on the service, independent of any CWD. These commands mirror the MCP workspace tools and drive the same interface.

A workspace lives in the store of whichever service you talk to, so every command here prints the target it resolved: a service on the conventional local port if one answers, otherwise the one vast login stored.

vast workspace [OPTIONS] COMMAND [ARGS]...

delete

Delete a workspace and its inputs (existing campaigns are unaffected).

WORKSPACE may be a ws-… id or a workspace name (unique names resolve; ambiguous ones must be deleted by id).

vast workspace delete [OPTIONS] WORKSPACE

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

WORKSPACE

Required argument

download

Fetch every file in WORKSPACE_ID into DIRECTORY.

The other direction of workspace init / update, so a project can be taken off a remote service and worked on locally – and so a workspace somebody else authored can be inspected without the web UI.

One archive, extracted into DIRECTORY: the same bytes vast share export --workspace publishes, so a project taken off a service and one taken off the share are the same tree.

vast workspace download growth-sim ./growth-sim
vast workspace download [OPTIONS] WORKSPACE_ID DIRECTORY

Options

--overwrite

Replace local files that already exist. Off by default: pulling over an edited copy of the same project would lose those edits irrecoverably.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

WORKSPACE_ID

Required argument

DIRECTORY

Required argument

init

Create a workspace and upload every file from DIRECTORY into it.

.vast/.osc are written inline; all other files go through the upload side channel (executability preserved). Hidden files/dirs (.cache etc.), output dirs (results/, plus any --exclude) and any campaign results tree — recognised by its contents, so a downloaded campaign under its own id is caught too — are skipped, and every skip is reported. Prints the new workspace id.

vast workspace init configs/examples/growth_sim # into the resolved service
vast workspace init [OPTIONS] DIRECTORY

Options

--name <name>

Workspace name (default: the directory name).

--exclude <NAME>

Directory name to skip (repeatable). Adds to the default [‘results’].

--include-results

Upload campaign results trees too. Off by default: they are a campaign’s OUTPUT, so pushing them back as project input uploads every past campaign on disk. What was skipped is always reported.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

DIRECTORY

Required argument

list

List workspaces (newest first).

vast workspace list [OPTIONS]

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

preview

Show what the sweep expands to, without running any of it.

The answer to “how many configurations will this be” — asked before a launch rather than discovered from a campaign that turned out to be ten times the intended size.

vast workspace preview [OPTIONS] WORKSPACE [VAST]

Options

--max-configs <max_configs>

Show at most this many configurations (0 = all).

Default:

0

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

WORKSPACE

Required argument

[VAST]

Optional argument

run

Run a .vast — the one way to start a campaign from a project.

WORKSPACE is a ws-… id or a workspace name. VAST is the path to the .vast within that workspace; omit it when the workspace holds exactly one and the service will resolve it, naming the candidates if there are several.

A workspace is a directory holding files and possibly several .vast files; a project is one .vast and the files it references. So the pair (workspace, path) names the project to run, which is exactly what the service accepts — workspace_id is the only project binding it takes.

Fire-and-forget by default: it returns once the campaign is launched. Track it with vast campaign wait <campaign-id> or the web UI. --wait-and-download instead blocks until it finishes and pulls the archive down.

vast workspace init . # push a workspace
vast workspace run my-experiment my.vast –description “pilot”
vast workspace run my-experiment –push . # push and launch in one step
vast workspace run [OPTIONS] WORKSPACE [VAST]

Options

--push <DIR>

Sync DIR into WORKSPACE before launching, creating the workspace if it does not exist. The two-step form (workspace init/update, then run) is the same thing spelled out.

--filter <GLOB>

Run only configurations matching this name or glob (e.g. hall*).

-r, --runs <runs>

Override execution.runs (default: the value in the .vast).

--campaign-name <campaign_name>

Override the campaign name; the id becomes <name>-<timestamp>.

--description <TEXT>

One line saying what this run is for. It is what tells two same-day <name>-<timestamp> campaigns apart in the web UI.

--priority <N>

Which campaign the cluster queue admits first: higher goes first, 0 is normal, negative waits behind everything else. Ordering only — it never stops a run that has started. Refused by a service that has no queue.

--upload-to-share

Stream a raw (pre-postprocess) archive to the configured share when the campaign finishes.

--allow-opaque-image

Launch even though a container names an image that declares no provenance:. Refused by default because nothing in the results could then say what ran; the exemption is recorded on the campaign.

--image-project <PROJECT>

Registry/namespace to take the RoboVAST images from for this run, overriding ROBOVAST_PROJECT. Affects only images RoboVAST publishes — a container image your .vast names is run as written.

--image-project-tag <TAG>

Tag to take those images at (default: ROBOVAST_PROJECT_TAG, else latest).

--wait-and-download

Block until the campaign finishes and its results are uploaded, then download its archive into the current directory.

--poll-interval <poll_interval>

Seconds between status polls when –wait-and-download is set.

Default:

5.0

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

WORKSPACE

Required argument

[VAST]

Optional argument

update

Re-sync DIRECTORY into an EXISTING workspace (id or name).

Uploads every file from DIRECTORY, overwriting in place — .vast/.osc inline, everything else via the upload side channel. Hidden files/dirs, output dirs (results/, plus any --exclude) and any campaign results tree (recognised by its contents) are skipped, and every skip is reported. With --prune it also removes workspace files that no longer exist locally, so the workspace mirrors DIRECTORY exactly.

vast workspace update ws-ab12 configs/examples/growth_sim
vast workspace update growth_sim configs/examples/growth_sim –prune
vast workspace update [OPTIONS] WORKSPACE DIRECTORY

Options

--exclude <NAME>

Directory name to skip (repeatable). Adds to the default [‘results’].

--prune

Also delete workspace files that are absent from DIRECTORY (full mirror). Off by default: update only adds/overwrites.

--include-results

Upload campaign results trees too. Off by default: they are a campaign’s OUTPUT, so pushing them back as project input uploads every past campaign on disk. What was skipped is always reported.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

WORKSPACE

Required argument

DIRECTORY

Required argument

validate

Check a project before spending any compute on it.

Reports every problem at once rather than the first, because they fail independently and fixing them one launch at a time is the expensive way to find out.

Computed service-side: the checks need the config schema, the scenario image’s own parser and (unless --no-world-check) the simulator, none of which a client install has. So this works the same whether the service is local or remote. The two container checks are what cost more than a moment; each has a flag to skip it.

vast workspace validate [OPTIONS] WORKSPACE [VAST]

Options

--no-world-check

Skip the world check, which builds the model in a container.

--no-scenario-check

Skip parsing the scenario in the image that would run it. That is the only check that sees an import the image does not carry.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

WORKSPACE

Required argument

[VAST]

Optional argument

world

Describe the world this campaign’s simulator will load.

The other half of authoring a sim: override: vast workspace world says what the world offers — which components an override can address, and with --targets which model values a run may change at all. Both are otherwise only refused inside the container, after the image pull.

Answered by the simulator, in the image the campaign runs, so the reply names that image: which world a ref resolves to depends on what is installed there.

vast workspace world tiago_pick
vast workspace world tiago_pick –targets ‘gripper_right*’ –json
vast workspace world [OPTIONS] WORKSPACE

Options

--path <VAST>

Which .vast in a multi-.vast workspace (default: the only one).

--targets <GLOB>

Also report the objects matching GLOB whose model values a run may override, with their current values. Costs a model build.

--entities

Also list the entities the world compiles, and what its start state warns about. Costs a model build and a reset.

--json

Print the raw description as JSON.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

WORKSPACE

Required argument

vast campaign

Act on a campaign: list, watch, stop, read its log, re-run it.

Every verb drives the robovast-service, so these work the same whichever deployment runs the campaign.

To start a campaign from a project, use vast workspace run.

vast campaign [OPTIONS] COMMAND [ARGS]...

delete

Permanently delete one or more CAMPAIGNs wholesale.

Removes each campaign’s directory under the service’s results root, plus, on a cluster service, any leftover Kubernetes Jobs. This is the full “forget this campaign” action; vast share remove only touches the external share, which this command leaves untouched.

Each campaign is deleted or refused on its own, and reported on its own line: the service refuses one that is still running (stop it first) and an id that is not a campaign id, and deletes the rest. The exit status is non-zero if any campaign was not fully deleted. An id that has nothing left here counts as deleted. This is irreversible.

vast campaign delete [OPTIONS] CAMPAIGN...

Options

-y, --yes

Skip the confirmation prompt.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

CAMPAIGN...

Required argument(s)

download

Download campaign archives from the service, one .tar.gz each.

That is the whole command: it fetches <campaign-id>.tar.gz and stops. Nothing is extracted, no results directory is written into, and no state is kept about what you already have – the archive is yours, to keep, copy, unpack, or hand back with vast campaign import.

The archive is the campaign as the service holds it, postprocessing and all. The share’s raw, pre-postprocess snapshot is a different system with different credentials: vast share download.

A campaign that is still running downloads too, and lands as <campaign-id>.incomplete.tar.gz – runs that had not finished are simply not in it. The service names that file, so the name cannot disagree with what is inside.

Writes into the current directory unless -o says otherwise – an archive is a file, not a results tree, so a results directory is the wrong home for it.

One campaign that fails does not stop the others: each is reported on its own line and the exit summary counts what landed. A thin single-archive copy cannot do several, resume past a failure, or show progress on a multi-gigabyte transfer, so this is the one implementation.

vast campaign download [OPTIONS] CAMPAIGN...

Options

-o, --output <output>

Directory to write the archives into [default: the current directory]

-f, --force

Overwrite an archive of the same name that is already here

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

CAMPAIGN...

Required argument(s)

import

Take a campaign archive into the service, and postprocess it if it needs it.

ARCHIVE is a .tar.gz on this machine – one vast campaign download or vast share download produced, or a colleague sent. It is uploaded to the service and imported there, so the campaign lands where the web UI and every other client can see it; the file itself is never deleted, it is yours.

Importing is more than extracting: listings and the web UI answer from campaign.db, not from the results tree, so a campaign that is only unpacked is invisible. And when the archive is a raw one – carrying no postprocessing record, which is what the share holds – postprocessing is chained automatically, because a campaign without its metric tables is not one you can ask anything.

Long-running, so it returns once the import is under way: the campaign appears immediately at phase importing. Watch it with vast campaign wait <campaign-id>, or in the campaign view.

It creates a campaign, which is why it is here and not in vast results: an upload and one HTTP call, needing nothing of the local half of the tool. It sat in the group that ships only with the full distribution, so the install most likely to be talking to a remote service was the one that could not import into it.

vast campaign import [OPTIONS] ARCHIVE

Options

--force

Replace a campaign of the same id already there.

--rebuild-store

Reconstruct campaign.db from the results tree (the recovery for a corrupt one).

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

ARCHIVE

Required argument

list

List campaigns this service knows about: live ones first, then newest first.

The size column is what the results occupy, measured once when the campaign ended; - means none is recorded (a campaign still running, or one never measured). description is what tells two same-day <name>-<timestamp> ids apart, which is why the launch verbs ask for one. A live campaign’s standing with the queue follows its phase when it is not the default – prio +2, paused – because a held campaign making no progress is otherwise indistinguishable here from a wedged one.

vast campaign list [OPTIONS]

Options

--limit <limit>

How many to show, in the listed order.

Default:

20

--sort <sort_key>

Order by when a campaign ended (started, while live) or by its results size. Live campaigns lead either way.

Default:

recent

Options:

recent | size

--desc, --asc

Newest or largest first, or the reverse. A campaign with no size is listed last either way.

Default:

True

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

log

Print a campaign’s unified infrastructure log.

The same divider-separated stream the web UI and MCP show — the variation (config-generation), run (controller) and postprocessing phases in order, each under a ===== PHASE ===== divider.

One reader, over HTTP, and no fallback to assembling the log from a campaign directory when no service answers – that needs the core installed, takes a path where every other verb takes a campaign id, and is a second implementation of “read the log” that a client-only install could not reach anyway.

vast campaign log [OPTIONS] [CAMPAIGN]

Options

-f, --follow

Stream new output until the campaign finishes

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

[CAMPAIGN]

Optional argument

pause

Stop admitting new runs for a campaign; resume starts them again.

The runs it has already started finish normally and their results are kept: this holds back what is queued, so the campaign drains rather than stopping, and the cluster is free for something else within one run’s length. It keeps the priority it will resume at.

vast campaign pause [OPTIONS] [CAMPAIGN]

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

[CAMPAIGN]

Optional argument

postprocess

(Re)run analysis postprocessing for CAMPAIGN.

The rosbag->CSV step runs in-cluster and the campaign’s derived data is rebuilt. Mirrors the web “Retrigger postprocessing” action and the MCP run_postprocessing tool, so all three drive one implementation.

This was vast results reprocess, beside a vast results postprocess that did the same job in-process against a results directory on this machine. Two postprocessing paths is the same split vast exec local run was: one of them could only ever see a local campaign, and it was the one the documentation reached for first.

Dispatched, not awaited. Postprocessing takes minutes to hours, so the campaign re-enters its postprocessing phase and this returns – follow it with vast campaign wait CAMPAIGN, exactly as after vast workspace run. A second run is refused while one is in flight.

vast campaign postprocess [OPTIONS] [CAMPAIGN]

Options

-f, --force

Bypass per-rosbag caches and reprocess all bags.

--skip <PLUGIN>

Skip a postprocessing plugin (repeatable), e.g. –skip rosbags_to_webm.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

[CAMPAIGN]

Optional argument

priority

Set which campaign the queue admits first. Higher goes first; 0 is normal.

For getting a short campaign through while a long one is running: demote the long one (a negative VALUE) or promote the short one. Takes effect on the next admission pass, and applies to the batches the campaign has not submitted yet as well as the jobs queued now.

Ordering only – nothing already running stops. The campaign you demote keeps the runs it has and gives up only the slots they release, so no partial run is produced and no results are lost. To end a campaign instead, use vast campaign stop.

Needs a service that queues campaigns against each other; one with no queue refuses.

vast campaign priority [OPTIONS] VALUE [CAMPAIGN]

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

VALUE

Required argument

[CAMPAIGN]

Optional argument

rerun

Launch a NEW campaign from what CAMPAIGN_ID recorded. The source is not modified.

Reuses the frozen config and the image the source recorded, so it runs the same code rather than today’s. A config older than the current version is migrated into the staging copy; the archived one is left exactly as its author wrote it.

The service runs the pre-flight itself and refuses on a blocking axis, naming what is missing – so a re-run that could only fail in the backend is answered before the launch. --force tells it to proceed anyway, which is worth having for an axis you have decided you understand.

vast campaign rerun [OPTIONS] CAMPAIGN_ID

Options

--check

Report whether this campaign can be re-run, and launch nothing. Costs nothing: it answers from what the campaign recorded. Exits non-zero when an axis is genuinely blocked.

--force

Ask the service to launch even though the pre-flight reports a blocking axis. What is being overridden is printed before the launch.

--to-workspace <NAME>

Do not launch. Materialise the campaign as a workspace with its config migrated as far as it could be and a marker at every decision left, to finish by hand. For a config no ladder step can carry forward.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

CAMPAIGN_ID

Required argument

resume

Admit runs again for a paused campaign, at the priority it was paused at.

vast campaign resume [OPTIONS] [CAMPAIGN]

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

[CAMPAIGN]

Optional argument

status

Print a campaign’s phase and progress once, and exit.

The single-read counterpart to vast campaign wait: use this for a campaign you are not waiting on. Waiting is a separate verb because it can take days, and holding a request open for that is a different thing from asking once.

vast campaign status [OPTIONS] [CAMPAIGN]

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

[CAMPAIGN]

Optional argument

stop

Ask a running campaign to stop.

Goes through the robovast-service, which drives the campaign in-process, so the stop reaches whatever it is doing and the campaign is published as usual. What it leaves behind depends on which work was running; the reply says. A no-op if nothing is running.

vast campaign stop [OPTIONS] [CAMPAIGN]

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

[CAMPAIGN]

Optional argument

stop-job

Kill ONE running job; the rest of the campaign keeps going.

For a job that is visibly wedged and will not exit on its own. This is not how a campaign is ended – that is vast campaign stop, which stops all of it.

Only a running job can be stopped. The kill is permanent and recorded: the runs it cut short report status='killed' in the campaign data, with the reason, and count as neither passes nor failures.

vast campaign stop-job [OPTIONS] JOB_NAME [CAMPAIGN]

Options

--reason <TEXT>

Why you stopped it — stored with the run and shown in the results

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

JOB_NAME

Required argument

[CAMPAIGN]

Optional argument

wait

Block until CAMPAIGN is over; the exit code says how it ended, or why the wait did.

Exit codes:
0 FINISHED Finished, past postprocessing.
1 FAILED Failed, or stopped.
2 STOPPED_WAITING Stopped waiting: –timeout elapsed, or the service
stopped answering. The campaign is unaffected and can
be waited on again.
3 NO_PHASE The service knows no phase for this id: a typo, or a
campaign that died before recording one.
4 STALLED Stalled: nothing has completed for longer than one run
may take. Still running, and nothing is waiting on it
now.
5 HEALTH_FINDING A running job’s simulator reported an error-level
health finding about itself. Still running, and
nothing is waiting on it now.

The service drives every campaign, so its phase is the campaign’s. Prints each phase change as it happens and exits when the campaign reaches a terminal one — past postprocessing, not merely past the last run.

Exists so a caller can wait without holding a request open, and is why the MCP offers no campaign-wait tool: an agent harness can background this command and be notified when it exits, hours or days later, where a blocking tool call would occupy the conversation for as long as the campaign ran — and still not outlive the session. The loop itself is wait_for_campaign_status(), shared with every other surface that waits.

A stall ends the wait too (STALLED), because a stalled campaign never reaches a terminal phase: it holds running for its whole life, so a waiter that stopped only on terminality would never return and nobody would be told. The verdict is stall_report()’s, not a second opinion computed here.

An ``error``-level health finding ends it too (HEALTH_FINDING), and earlier: a stall is only visible once a run is past its declared budget, and needs one to have been declared at all, while a simulator saying “sim time is not advancing” is true within a minute and needs no budget. Whatever a finding means is the simulator’s business – this reads one word, level, and passes the rest through.

Only a new stall or a new finding exits. A campaign already stalled when this command starts is not news – whoever ran it has just been told – and exiting on the first poll would make vast campaign wait unusable for exactly the state it reports: the message says to re-run it after diagnosing, and a fresh waiter would exit immediately, forever. So the first observation is recorded and only a rising edge stops the wait; for findings the edge is per check, so one check firing repeatedly is one exit and not a stream.

vast campaign wait [OPTIONS] CAMPAIGN

Options

--interval <interval>

Seconds between status polls.

Default:

10.0

--timeout <timeout>

Give up after this many seconds (default: wait indefinitely).

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

CAMPAIGN

Required argument

vast config

Manage run configuration.

vast config [OPTIONS] COMMAND [ARGS]...

generate

Generate VAST’s run configurations and output files into OUTPUT_DIR.

vast config generate [OPTIONS] VAST OUTPUT_DIR

Options

--keep-transient

Keep and display temporary folders used during generation (e.g. by FloorplanGeneration).

--no-cache

Skip cache lookup and force a fresh generation even if inputs are unchanged.

Arguments

VAST

Required argument

OUTPUT_DIR

Required argument

variation-points

List the variation points VAST’s scenarios offer.

Shows every scenario parameter that can be varied.

vast config variation-points [OPTIONS] VAST

Arguments

VAST

Required argument

variation-types

List available variation types.

Shows all registered variation type entry points that can be used in the variations section of .vast configuration files.

vast config variation-types [OPTIONS]

list

List the configurations VAST expands to, without generating files.

Shows what would be generated, creating none of it.

vast config list [OPTIONS] VAST

Options

--debug

Show internal values starting with _

Arguments

VAST

Required argument

vast container

Run a command in an experiment container, and stop the held one.

For testing a container and its setup – an import, a ros2 pkg list, a file check – before a campaign spends compute finding the same thing out.

vast container [OPTIONS] COMMAND [ARGS]...

exec

Test a container and its setup by running SHELL_COMMAND in the experiment image.

Produces no campaign data — nothing durable, no provenance, no repetitions. Use vast workspace run to run the experiment itself.

Omit --config-name to check the bare image (python3 -c 'import x', ros2 pkg list, file checks). Name one to stage that configuration; an empty SHELL_COMMAND then starts its scenario, and its output goes to a log file inside the container rather than to stdout (the path is printed).

The image is the one this host’s copy of the project builds, so a stale checkout checks a stale image; the build:<tag>@<hash> that ran is printed with the exit line.

There is at most one such container at a time; vast container stop ends it. No --timeout: the limit is derived from what is being run (the project’s execution.timeout for a scenario, a fixed cap for a command) and reported with the result.

vast container exec [OPTIONS] [SHELL_COMMAND]

Options

--workspace <workspace_id>

Workspace whose project to use (with –config for the .vast).

--config <config_path>

Which .vast in the workspace (workspace-relative).

--campaign <campaign_id>

Use an existing campaign’s _config/ as the project instead.

--config-name <config_name>

Stage this configuration. Omitted, the bare image is used.

--keep-alive

Leave the container running so later calls can inspect it.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

Arguments

SHELL_COMMAND

Optional argument

stop

Stop the held container-exec container, if there is one.

vast container stop [OPTIONS]

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

vast cluster

Set up and maintain the Kubernetes substrate campaigns run on.

These read what they need from the cluster itself, so they work from any directory. To act on a campaign, use vast campaign; on the service, vast service.

Every verb needs a kubeconfig and arrives with robovast-cluster, so what this lists depends on what is installed.

vast cluster [OPTIONS] COMMAND [ARGS]...

cleanup

Clean up the Kubernetes cluster setup.

Removes the robovast pod (registry and index), the robovast-service Deployment, the build daemon and the cluster-wide pieces setup installed. The data directories on the node are kept unless --delete-data asks otherwise.

This command can be run after completing all scenario executions to clean up cluster infrastructure resources (different from jobs-cleanup which only cleans up job pods).

If --cluster-config is not specified, it will automatically detect which cluster configuration was used during setup by reading it back from the deployed robovast-service — so this works from any host and needs no project. When specifying --cluster-config explicitly, pass -n <namespace> if the setup was done in a non-default namespace.

vast cluster cleanup [OPTIONS]

Options

-c, --cluster-config <config_name>

Cluster configuration plugin to use (auto-detects if not specified)

-n, --namespace <namespace>

Kubernetes namespace to clean up (required when using –cluster-config without prior setup)

-o, --option <options>

Cluster-specific option in key=value format (can be used multiple times)

-x, --context <kube_context>

Kubernetes context to use (default: active context in kubeconfig)

--delete-data

Also empty this deployment’s data directories on the node: the workspaces, the results, the campaign index and the registry. Cleanup keeps them by default, because the results hold every finished campaign and a teardown is a very expensive way to discover that. Irreversible, and no archive is taken first – use vast share or vast campaign download for anything worth keeping.

--forget-placement

Also remove the node labels recording where this deployment kept its data. Without this the labels stay, so a later setup lands on the same node with no flags – which is the point of them. The on-disk data is not removed either way.

--vast <FILE>

A .vast to pre-flight against this cluster. Its only use here is to refuse when it declares per-cluster resource lists for several contexts and –context was not given – which would otherwise pick a cluster by accident. Optional, and read only for that check.

jobs-cleanup

Clean up jobs and pods from a cluster run.

Removes scenario execution Jobs and their pods directly (using your kubeconfig — the -x context, -n namespace). By default removes all campaigns; use --campaign for one.

The campaign’s results are untouched: they live with the service, and vast campaign delete is what removes a campaign.

Usage: vast cluster jobs-cleanup
Usage: vast cluster jobs-cleanup –campaign campaign-2025-02-27-123456
vast cluster jobs-cleanup [OPTIONS]

Options

-i, --campaign <campaign>

Clean only jobs for this campaign (e.g. campaign-2025-02-27-123456). Without this, cleans all scenario-runs jobs.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

--vast <FILE>

A .vast to pre-flight against this cluster. Its only use here is to refuse when it declares per-cluster resource lists for several contexts and –context was not given – which would otherwise pick a cluster by accident. Optional, and read only for that check.

monitor

Monitor scenario execution jobs on the cluster.

Displays progress per run: how many jobs have finished (completed or failed), how many are running, and how many are pending for each run.

By default, monitors only the contexts referenced in the .vast config file. Falls back to the active kubeconfig context when no per-cluster config is defined. Use –context to restrict monitoring to a single cluster. Only contexts with active or past jobs are shown.

This is intended for monitoring jobs created by a campaign launch.

vast cluster monitor [OPTIONS]

Options

-i, --interval <interval>

Polling interval in seconds

Default:

2.0

--once

Print job status once and exit

-x, --context <kube_context>

Kubernetes context to use (default: active context in kubeconfig)

-n, --namespace <namespace>

Kubernetes namespace the scenario Jobs run in.

Default:

default

--vast <FILE>

Watch every context this .vast names, instead of only the active one. Ignored when –context is given, which names one directly.

setup

Set up the Kubernetes cluster for execution.

Deploys the robovast pod (the container registry and the campaign index), the robovast-service Deployment that drives campaigns and keeps their results on its results volume, the shared build daemon, and the cluster-wide pieces they need (RBAC, the GPU device plugin, the placement labels).

This command should be run once before executing scenarios on the cluster for the first time.

If the cluster is already set up, this command will exit with an error. Run ‘vast cluster cleanup’ first to clean up the existing setup, or use --force to force re-setup.

Use --list to see available cluster configuration plugins.

Cluster-specific options can be passed using --option key=value.

Reads no project and no .vast: this deploys into a cluster and runs from any directory. Which nodes the cluster’s pods may use is a property of the cluster. The campaign job node pool comes from ROBOVAST_JOB_NODE_LABELS in the environment, a JSON object of label -> value; setup and upgrade both write it into the service, so unset means every node. --control-node-label is written on every run, so omitting it clears what a previous setup configured.

The job node aliases a campaign may confine itself to (execution.kubernetes.jobs.node) come from ROBOVAST_JOB_NODE_ALIASES in the environment, a JSON object of alias -> node name. Setup and upgrade both reconcile the node labels to exactly what it states – unset, every alias is removed – and refuse the whole set, before anything is changed, if an alias is not legal or its node is not schedulable inside the job pool.

Share credentials (ROBOVAST_SHARE_TYPE and its provider variables — e.g. ROBOVAST_GCS_BUCKET / ROBOVAST_GCS_KEY_FILE) are read from the host environment / project .env at setup and handed to the in-cluster service as a Secret, so --upload-to-share campaigns work from the cluster. A key file is inlined into the Secret; nothing else is needed on the host.

vast cluster setup [OPTIONS] [CLUSTER_CONFIG]

Options

--list

List available cluster configuration plugins

-n, --namespace <namespace>

Kubernetes namespace for execution (used by cluster run)

Default:

default

-o, --option <options>

Cluster-specific option in key=value format (can be used multiple times)

-f, --force

Force re-setup even if cluster is already set up

--gpu-replicas <N>

Advertise N time-slicing replicas per physical GPU, so N pods can share one card. Without this flag a GPU that is present is used anyway (with a sensible default) and a cluster without one is left alone; passing it makes GPU support a requirement, so a cluster that cannot provide it becomes an error. N caps concurrency and does NOT partition VRAM: all N renderers allocate from the same card.

--no-gpu

Set the cluster up without GPU scheduling, even if it has a GPU.

-x, --context <kube_context>

Kubernetes context to use (default: active context in kubeconfig)

--ingress-host <HOST>

Publish the service at this hostname, so users reach it in a browser without kubectl. Needs TLS (see –issuer/–tls-secret) and an access token; both are refused otherwise.

--ingress-class <NAME>

IngressClass to use (e.g. nginx). Default: the cluster default.

--issuer <NAME>

cert-manager ClusterIssuer to obtain the certificate from. tools/setup_ingress_tls.py creates one.

--tls-secret <NAME>

Existing TLS Secret to serve, instead of a cert-manager issuer.

--insecure-http

Publish over plain HTTP. The shared token then crosses the network in clear text; only for a trusted network.

--rotate-token

Issue a new access token, logging everyone out. Without this an already-deployed token is preserved across re-runs.

--data-root <PATH>

Put every node-local directory this deployment keeps under one path: the workspaces, the campaign results beside them, the registry and the build cache. The flag for the ordinary case – a machine with a disk mounted for RoboVAST – and any per-tenant flag below overrides it for its own tenant. Reads ROBOVAST_DATA_ROOT, so a .env can set it once per machine. It does NOT place a campaign job’s scratch: that is an emptyDir under the kubelet root, which is the node’s configuration rather than this deployment’s.

--workspaces-path <PATH>

Host directory holding the service’s workspaces (default: /var/lib/robovast-workspaces). The campaign results root – where finished campaigns live – is placed beside it, and the campaign index beside that; neither needs a flag of its own.

--workspaces-class <NAME>

Back the workspaces and the campaign results with PVCs from this StorageClass instead of hostPaths, which also unpins the service pod. The campaign index follows unless –index-class says otherwise. Stock RKE2 provisions nothing, which is why hostPath is the default.

--results-size <SIZE>

Size of the campaign results PVC (default: 500Gi) – the volume every campaign lives on. Needs –workspaces-class, which is what backs it: without one the results are a directory on the data node, bounded by that disk, and there is no volume to size. The same flag on ‘vast service upgrade’ raises an existing claim.

--index-class <NAME>

Back the campaign index with a PVC from this StorageClass. Without it the index takes the results’ backing: a PVC of the workspaces’ class where one is given, else a directory beside the results on the data node. Its own flag because Postgres is a different workload from the bulk of the results, and on a managed node pool the index is what a replaced node would otherwise take with it.

--index-size <SIZE>

Size of the campaign index PVC (default: 20Gi). Needs a class, its own or the workspaces’: without one the index is a directory on the node and there is no volume to size.

--registry-class <NAME>

Back the built-in container registry with a PVC from this StorageClass instead of a hostPath. Preferred where the cluster can provision volumes; stock RKE2 cannot, which is why hostPath is the default.

--registry-path <PATH>

Host directory backing the registry when using hostPath (default: /var/lib/robovast-registry).

--data-node <NODE>

Hold this deployment’s node-local data on this node: the workspaces, the results, the index, the registry and, unless –buildkit-node says otherwise, the build cache. Rarely needed – setup picks the node with the most free space the first time and records the choice as a node label, so later runs stay put without any flag. Naming a node moves the placement off whatever node holds it now and says so; the bytes are NOT migrated, so the new node starts empty – including the results, which is where finished campaigns live.

--buildkit-class <NAME>

Back the shared build daemon’s cache with a PVC from this StorageClass instead of a hostPath. Prefer an SSD class: BuildKit’s snapshotter is small-file heavy and a slow disk becomes the bottleneck the cache was meant to remove.

--buildkit-path <PATH>

Host directory backing the build cache when using hostPath (default: /data/robovast-buildkit). This is what makes a base image pulled once stay pulled.

--buildkit-size <SIZE>

Size of the build cache PVC (default: 200Gi). Needs –buildkit-class: without one the cache is a directory on the node, bounded by that disk and the daemon’s GC ceiling, and there is no volume to size.

--buildkit-node <NODE>

Hold the build cache on this node. Defaults to the data node, because auto-separating would put a 150 GB cache on whichever node was left over. Separate them where the disk is tight: these are the deployment’s two large on-disk tenants, and a full builder disk on the service’s node becomes DiskPressure evictions of the API.

--performance-governor, --no-performance-governor

Set the nodes’ CPU governor to ‘performance’. ON by default, because a cluster whose clock moves with load produces numbers nothing downstream can detect or correct: a per-node figure is then taken against a clock that was not the same for every run. Needs a privileged pod with the host /sys writable; a cluster that refuses it (managed Kubernetes does) is WARNED about and setup continues. Naming the flag makes that refusal an error instead. –no-performance-governor skips it and leaves the hosts alone.

--tailnet, --no-tailnet

Publish the service on a WireGuard tailnet instead of an Ingress: a node beside it dials OUT to a coordination server and answers under a stable name, so nothing listens on the internet and users need no kubeconfig. OFF by default. The coordination server and pre-auth key come from the environment (ROBOVAST_TAILNET_LOGIN_SERVER / _AUTHKEY), so a key does not land in shell history – but WHICH cluster is on a tailnet is decided here, because one .env and two contexts would otherwise publish whichever was current. Written on every setup: omitting it removes a node a previous setup deployed.

--control-node-label <KEY=VALUE>

Run RoboVAST’s own infrastructure pods on nodes carrying this label; repeatable. Narrows rather than decides: these are ANDed with the node-local data placement setup chooses. Pass none to clear.

Arguments

CLUSTER_CONFIG

Optional argument

Environment variables

ROBOVAST_DATA_ROOT

Provide a default for --data-root

ROBOVAST_WORKSPACES_PATH

Provide a default for --workspaces-path

ROBOVAST_WORKSPACES_CLASS

Provide a default for --workspaces-class

ROBOVAST_RESULTS_SIZE

Provide a default for --results-size

ROBOVAST_INDEX_CLASS

Provide a default for --index-class

ROBOVAST_INDEX_SIZE

Provide a default for --index-size

ROBOVAST_REGISTRY_CLASS

Provide a default for --registry-class

ROBOVAST_REGISTRY_PATH

Provide a default for --registry-path

ROBOVAST_BUILDKIT_CLASS

Provide a default for --buildkit-class

ROBOVAST_BUILDKIT_PATH

Provide a default for --buildkit-path

ROBOVAST_BUILDKIT_SIZE

Provide a default for --buildkit-size

vast service

Act on the robovast-service: read its log, roll it onto new bytes.

The verbs that need only a URL and a token ship with robovast-client; the ones that reach a Kubernetes API server arrive with robovast-cluster.

vast service [OPTIONS] COMMAND [ARGS]...

cache

What the service’s rebuildable caches hold; with –clear, free what may go.

Only copies of durable data – a campaign’s files fetched from the object store, compiled 3D worlds – so clearing loses nothing but the time to rebuild what is next asked for. The thing to reach for when a launch is refused for free space. What a running campaign, an operation or a recent reader may still be using is kept, and listed with the reason.

vast service cache [OPTIONS]

Options

--clear

Remove every entry nothing may still be using, and say what that freed.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

info

Which service is answering, which code it runs, and which backend it drives.

Call this first when something behaves unexpectedly. A service loads robovast once, at startup, so after an edit a reachable service may still be running the old code; comparing revision with your tree is how you find that out.

When revision is absent the check is unavailable and there is no substitute – version is the package semver, so it stays the same across every edit and reading it as a revision is a live trap.

robovast_version, the field the compatibility handshake compares, is not printed: it resolves to a revision when there is one and to the semver otherwise, so it is always already on one of these two lines. Printing it as version as well would show the same SHA twice on every deployed service.

vast service info [OPTIONS]

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

log

Print what the robovast-service itself has been doing.

Not a campaign’s log – this is the service process: what it decided, what it refused, and the reason behind a failure whose visible half was one terse line. Several failures say so in as many words (“the real reason is only in the service log”), and until now there was no way to read it short of kubectl logs.

The service keeps the last few hundred kilobytes in memory, so this covers what it has been doing recently, not its whole life, and a restart clears it. A container that has already died is only in kubectl logs -p deploy/robovast-service: a buffer inside a process cannot outlive the process.

vast service log [OPTIONS]

Options

-f, --follow

Keep printing new output until interrupted.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

mcp-stats

Which MCP tools agents actually call, and what happened when they did.

The same record the web UI’s Admin page shows, on a terminal: a ranking of tools by call count with their failure share and durations, or the calls themselves with what each was given and what it answered (--calls).

Deliberately a command and not an MCP tool. An agent asking this question would pay for the whole answer in context, and the question – “is this tool surface worth what it costs?” – is one asked about agents rather than by one.

The record lives in the central index, so it outlives a service restart but not the results store, and it covers a bounded window that the ranking prints.

vast service mcp-stats [OPTIONS]

Options

--calls

Print the call log instead of the per-tool ranking.

--tool <tool>

Only this tool.

--failed

Only calls that failed.

--limit <limit>

How many calls to print with –calls.

Default:

50

--csv

Write the call log as CSV to stdout, for a spreadsheet or a script.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

resources

Does the cluster have room, and is it reachable?

Ask before a sweep.

pending is work the backend has accepted but is not executing, which is why it is counted apart from usage rather than folded into it – counting queued work as used reported more cores in use than the cluster had.

vast service resources [OPTIONS]

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

restart

Roll the deployed service onto the newest image at its tag, and nothing else.

Asks the service to restart itself, so this needs only a URL and a token – which is the point: vast service upgrade needs a kubeconfig, so somebody who reached the deployment through vast login had the web UI’s button and no command at all.

restart the image, by stamping the Deployment’s restart annotation. With
imagePullPolicy: Always that is what moves a floating tag onto new bytes.
upgrade that, plus RBAC, the registry ingress route, the credential
Secrets and the build daemon.

This reconciles none of those. A version needing a permission the last one did not will deploy and then fail at runtime with a 403, which reads as a bug rather than as a missed migration. So this is the command for “new bytes are published and nothing else changed”; for a version bump, a missed RBAC migration, a rotated Secret or a registry move, use vast service upgrade. The Secrets in particular are rebuilt from the operator’s environment, which the pod does not have, so they can never be done from in there at any level of privilege.

Refuses while campaigns are live – their controller runs in the pod being replaced – and names them. --yes skips the question.

vast service restart [OPTIONS]

Options

-y, --yes

Do not ask before rolling over live campaigns. For scripts; without it a non-interactive run aborts rather than rolling silently.

--wait

Block until the new pod is the one serving, instead of returning once the roll has been asked for.

-n, --namespace <namespace>

Namespace the robovast-service runs in.

Default:

default

-x, --context <NAME>

Kubernetes context, when a command has to name one.

vast service upgrade

Move a running instance to a new RoboVAST version.

Rolls the Deployment onto the resolved image, reconciles RBAC, and waits for the new pod to be the one serving before reporting anything.

It fails, non-zero, if that pod does not take over: an image it cannot pull, a node it cannot be scheduled on, or a container that crash-loops. The reason Kubernetes gave is printed as soon as it appears, so a stuck upgrade names its cause in seconds instead of looking like a hang – and “✓ upgraded and ready” now means it. Use --timeout for a registry slow enough to need longer.

Always restarts the pod, even when nothing appears to have changed. That is the point: an image ref that is a floating tag, or a change confined to the Secrets, leaves the Deployment spec byte-identical, and Kubernetes then rolls nothing while this command reports success. It also makes the restart the only way the env Secrets are re-read – the pod loads them through envFrom at container start and never again. The cost is a few seconds during which the API is unavailable.

RBAC reconciliation is not decoration: a version needing a permission the last one did not — as /usage once needed a cluster-scoped ClusterRole — would otherwise deploy and then fail at runtime with a 403, which reads as a bug rather than as a missed migration.

--no-restart reconciles just that part — RBAC, the registry ingress route, the optional tailnet node — and stops before the Deployment is touched. All three are picked up by the running pod (the API server evaluates RBAC per request, and workload, a route is the gateway’s own state), so a permission the running version is missing can be granted without a version change and without the API blip. That is the difference between fixing a missed migration and rolling a service: it is the only way to do the former while a campaign is in flight, because the campaign controller lives in the pod a roll would replace. It does not move the image and does not re-read the env Secrets — for either of those, run the command without the flag.

The campaign job node pool and the job node aliases are applied, like setup does, from ROBOVAST_JOB_NODE_LABELS and ROBOVAST_JOB_NODE_ALIASES in the environment – so an upgrade from a shell without them clears the pool and removes the aliases, and says so – after checking every alias against that pool, before anything changes. With --no-restart the pool is not applied, since it lives in the pod’s environment. --no-restart reconciles them too: they are node labels a campaign reads when it starts, not the pod’s environment.

Before the roll it asks the service which campaigns are live and names them, for the reason --no-restart exists: the pod being replaced is where their controller runs. --yes skips the question. A service that cannot be reached is reported and the roll goes ahead – a wedged service is a reason to upgrade, not a reason to refuse – and that is said out loud, so a silent roll never has to be read as “nothing was running”.

Reads the environment like every vast command – ./.env then ~/.config/robovast/env – so ROBOVAST_PROJECT and ROBOVAST_PROJECT_TAG are where the images come from, and this is the command that moves them. Both are baked into the service pod as the site default; a campaign can still override them per launch (vast workspace run --image-project), which needs no upgrade at all.

There is deliberately no --image. A flag would have been one-shot, because nothing reads the deployed image back, so a later bare upgrade would silently revert it – arriving through the command that looks safe. For a one-off, a real environment variable already beats both files:

ROBOVAST_PROJECT=ghcr.io/cps-test-lab vast service upgrade

make image-digests PROJECT=... checks that a project holds a complete family before you point a cluster at it.

This is the command for rolling out an image, not setup --force:

upgrade the image, RBAC, and the credential Secrets it can rebuild from
the environment (git, share, ntfy, registry). Recovers this
cluster’s config and ingress host from the cluster itself, so it
cannot lose them. The access token is preserved.
setup –force re-provisions: the registry/index pod, the storage placement. It
takes its options as arguments, so a re-run without the original
flags re-provisions with different ones. Also re-mints the access
token when asked (–rotate-token), logging everyone out.

Campaign data lives on the service’s results volume and survives both.

vast service upgrade [OPTIONS]

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in

Default:

default

-x, --context <kube_context>

Kubernetes context to use (default: active context in kubeconfig)

--timeout <SECONDS>

How long to wait for the new pod to take over before failing. Generous because a service that comes up owing campaigns restores them before it binds its port; a pod that is genuinely broken (ImagePullBackOff, a crash loop) still fails fast on its own, without spending this.

Default:

1800.0

--no-restart

Reconcile only what does not need the pod rolled – RBAC, the queues, the registry ingress route – then stop. For granting a permission the RUNNING version is missing without a version change or an API blip, e.g. while a campaign is in flight.

--results-size <SIZE>

Raise the campaign results PVC to this size – the volume every campaign lives on. Expanded online by every StorageClass this deployment targets, so no campaign is interrupted. A claim cannot be shrunk, and a deployment whose results are a directory on the data node has no volume to size; both are refused rather than reported as done.

-y, --yes

Do not ask before rolling over live campaigns. For scripts; without it a non-interactive run aborts rather than rolling silently.

Environment variables

ROBOVAST_RESULTS_SIZE

Provide a default for --results-size

vast service token

Show the access token, and what to hand users along with it.

Setup deliberately prints the token only once, so reading it back meant a kubectl get secret ... | base64 -d incantation – which every operator then keeps in their shell history, and which needs kubectl syntax to answer a RoboVAST question.

The token is per cluster: an instance mints its own, and one instance’s token is simply wrong at another. That is the failure this command is most likely to prevent, since the mistake looks identical to a mistyped password.

vast service token what to send a user
vast service token -q the token alone
vast service token -x prod a specific cluster
vast service token [OPTIONS]

Options

-n, --namespace <namespace>

Namespace the robovast-service runs in

Default:

default

-x, --context <kube_context>

Kubernetes context to use (default: active context in kubeconfig)

-q, --quiet

Print only the token, for piping into something else.

vast results

Work on a results directory on THIS machine.

Every verb here names a path and none of them needs a login: this is the local half of the tool, for someone holding a results tree – publishing it, generating its metadata and provenance, merging campaigns. It ships with the full robovast distribution, so a client-only install does not have this group at all, which is the honest signal that none of it is a service operation.

What acts on a campaign is vast campaign – including postprocessing, which lives there because the service owns the cluster the runs executed on.

vast results [OPTIONS] COMMAND [ARGS]...

merge-campaigns

Merge campaign directories with identical configs into one merged_campaign_dir.

Groups campaign-directory/config-directory by config_identifier from config.yaml. Run folders (0, 1, 2, …) from all campaigns are renumbered and copied. Original campaign directories are not modified.

vast results merge-campaigns [OPTIONS] MERGED_CAMPAIGN_DIR

Options

-r, --results-dir <results_dir>

Source directory containing run-* directories (uses project results directory if not specified)

Arguments

MERGED_CAMPAIGN_DIR

Required argument

postprocess-commands

List all available postprocessing command plugins.

Shows plugin names that can be used in the results_processing.postprocessing section of the configuration file, along with their descriptions and parameters.

vast results postprocess-commands [OPTIONS]

Environment variables

ROBOVAST_INSECURE_SSL

Set to 1 to disable TLS certificate verification for remote fetches. This allows the CLI to continue when a remote host presents an invalid certificate. Use only with hosts you trust.