Cluster Execution
RoboVAST can execute scenarios at scale on a Kubernetes cluster, running each run configuration as an independent Job and collecting its results into the campaign on the service’s own results volume. This section covers everything from cluster setup and job queueing to multi-context workflows and cloud-provider-specific configuration.
Overview
Every cluster run — batch and search — is driven by the
robovast-service, which runs the campaign in-process (one worker thread per
campaign) and creates the scenario Jobs itself. So cluster runs require a deployed
service (vast cluster setup installs it); vast workspace run finds it — a service answering on the conventional local port, or the deployed
one vast login recorded — pushes the local project into a server-side
workspace, and starts the campaign there. It is fire-and-forget
— it returns immediately with the campaign id, and the campaign continues in the
cluster.
A campaign runs a workspace’s project, named by the pair (workspace, path). Push the directory once, then launch by name:
vast workspace run growth_sim growth_sim.vast \
--description "pilot: new inflation radius"
--description is one line saying what the run is for. Set it: it is what tells
two same-day <name>-<timestamp> campaigns apart in the monitor and the web UI.
Repetitions come from the .vast’s execution.runs unless --runs overrides
them. Internally:
Launch — The client pushes the project to a workspace and calls
create_campaign. The service starts aCampaignControllerin a worker thread overKubernetesBackend. No per-campaign controller pod is created; the service is the driver.The workspace is named after the project’s directory and reused by later launches of the same project — it is overwritten (after asking; the default is yes, and off a terminal it proceeds and says so), and project files it holds that the local directory no longer has are removed, so it mirrors what is on disk. What the service generated inside it (
.cache/, staged plugins) is left alone, as isresults/— a campaign’s output is not project input. Reusing the workspace is what keeps one per project rather than one per launch; pass--workspace NAMEto push somewhere else.A workspace a campaign is still reading is refused, naming the campaign: a campaign composes and resolves files out of the workspace, so a push during that would change an experiment as it is being set up — and a half-pushed tree read mid-composition is worse than either version of it.
A batch campaign is finished reading once it has staged, which is before its first container starts, so the refusal usually lasts moments; afterwards the push goes through and says which running campaigns it will not reach. A search campaign re-composes every generation and so is refused for its whole life.
WorkspaceInfo.preparing_campaignsis what answers the refusal andrunning_campaignsthe note — live state held by the service driving the run, never a stored campaign→workspace binding, because a finished campaign is workspace-independent.Config staging + job creation — The driver composes each batch, writes the scenario configurations into the campaign on the results volume, and creates one Kubernetes
Jobper packed job. Each job runs afetch-inputsinit container that pulls the campaign’s inputs as one tar stream into/configand a mainrobovastcontainer that executes the scenario. (A composition that reaches for an auxiliary container — a variation, an input generator, the simulator’s own world query — gets a pod per container, created when it asks and deleted when the campaign ends; a composition that asks for none creates nothing. Composing outside a campaign —validate_project,preview_configurations, andexec_in_containerfor one configuration — gets the same pod held by the container-exec manager instead, so an authoring loop reuses one warm pod and idleness reaps it.)Queueing — a Job is created only when sufficient CPU/memory is available, so a campaign cannot oversubscribe the cluster. Step 2 therefore paces itself against this rather than creating the whole plan up front.
Result collection — Every scenario pod carries an
uploadercontainer that delivers the pod’s whole/outto the campaign as one tarPUT, so a Job is complete only when its results are in the campaign. The campaign directory on the service’s results volume is both the durable home and the delivery mechanism: the driver writescampaign.dband_executioninto it as the campaign runs, and the service streams downloads straight out of it (vast campaign download/--wait-and-download), so no external share is required. Pushing a copy to an externaltar.gzshare is opt-in at launch — enable Upload to share when done in the web UI launcher (or--upload-to-share/ the MCPupload_to_shareflag). When set, the driver streams a raw, pre-postprocessing archive to the configured share the moment the runs finish, before analysis postprocessing adds derived data — so the shared copy stays minimal and untouched. Track progress withvast cluster monitor;vast campaign deleteis what removes a campaign once its results have been handled.
Note
Live status, stop and monitor all go through the service — auto-detected
on the conventional local port, or the one vast login recorded — there is no
controller pod to kubectl port-forward into. The web UI additionally
streams each campaign’s controller.log live from the service, and offers a
Download button (the postprocessed tar.gz, tarred out of the campaign
directory as it is read) on finished campaigns.
Prerequisites
The following tools must be installed and available on PATH before using
cluster execution:
Tool |
Purpose |
Install |
|---|---|---|
|
Communicate with the Kubernetes cluster (apply manifests, port-forward, wait for pods) |
|
|
Install and upgrade the NVIDIA device plugin on a GPU cluster, via the Helm chart registry |
|
|
Terminal UI for monitoring pods, jobs, and logs in real time — not required but greatly simplifies observability during a run |
For GCP clusters the gcloud CLI is additionally required — see
GCP (Google Kubernetes Engine) below.
To render on a GPU, the node additionally needs the NVIDIA driver and the NVIDIA
container toolkit installed on the host. Nothing in RoboVAST installs those: they are the
node administrator’s, and the toolkit is what lets a container be handed the device at all.
RKE2 and k3s register a nvidia RuntimeClass when the toolkit is present, and that is the
signal RoboVAST detects — kubectl get runtimeclass tells you whether a cluster has it.
Everything above the host is provisioned by setup; see GPU rendering below.
Cluster Setup
Before the first run, deploy RoboVAST into the cluster:
vast cluster setup <cluster-config>
Available cluster configs (--list):
vast cluster setup --list
Setup acts on the cluster, not on a project: it reads nothing ambient and runs from any
directory. It reads no .vast either: which nodes the cluster’s pods may use is given as
flags (below).
The setup command:
Deploys the
robovastpod — the container registry and the Postgres campaign index (Where the registry and the index run) — therobovast-serviceDeployment that drives campaigns and holds their results, and the shared build daemon. Every cluster config deploys the same set; what differs is how their volumes are backed.Makes GPUs schedulable where the cluster has them, so a simulation container renders in hardware instead of in software (below). A cluster without GPUs is left exactly as it was.
Pinning pods to a node pool
Both node pools are set by the operator, never in a .vast. A .vast describes a
campaign; which machines a cluster’s pods may run on is a property of the cluster, and
carrying it in a campaign file put a deploy’s lasting, cluster-wide decisions somewhere that
travels with an experiment.
The campaign job pool is ROBOVAST_JOB_NODE_LABELS in the .env, a JSON object of label
to value; the control pool is a setup flag:
ROBOVAST_JOB_NODE_LABELS={"node-pool": "primary"}
vast cluster setup rke2 --control-node-label node-pool=extra
ROBOVAST_JOB_NODE_LABELS confines campaign jobs. It is recorded in the service’s environment,
because the admission controller is what enforces it: the controller counts free capacity
only on nodes inside the pool, and every job pod carries the labels as a nodeSelector
so kube-scheduler is bound by the same rule the accounting assumed. Neither half suffices
alone — filtering capacity would leave the scheduler free to place outside the pool, and
stamping alone would have admission reserving room on nodes the pods cannot reach.
--control-node-label places RoboVAST’s own infrastructure pods. It narrows rather than
decides: these are ANDed with the node-local data placement setup chooses (see below).
vast cluster setup and vast service upgrade both write the job pool from the
environment and print it, so a command run from a shell whose .env lacks the variable
clears the pool — run them from the shell that has the deployment’s .env. A value that is
not a JSON object of strings fails the command before it changes anything.
upgrade --no-restart does not apply it, because it lives in the pod’s environment.
--control-node-label is repeatable and written on every setup, so omitting it clears what
a previous setup configured. With neither set, pods schedule wherever Kubernetes puts them.
Naming a node a campaign may be held to
A campaign can confine all of its jobs to one machine with
execution.kubernetes.jobs.node: <alias> — for a measurement whose cells must not be
compared across different hardware. The .vast names an alias, never a node: the
operator states which machine each alias means in ROBOVAST_JOB_NODE_ALIASES, in the same
.env as the other standing settings of the deployment. The value is a JSON object of alias
to node name:
ROBOVAST_JOB_NODE_ALIASES={"bench": "node-a", "gpu": "node-b"}
An alias is lowercase letters, digits, - and _, starting and ending with a letter or
digit, at most 63 characters — the same rule the .vast key is held to.
vast cluster setup and vast service upgrade both apply it. Each reconciles the node label
robovast.io/job-node-alias=<alias> to exactly what the variable states — an alias it no
longer names is removed, and a command run from a shell whose .env lacks the variable
removes them all — and prints every alias it adds, moves or removes. upgrade --no-restart
applies it too: a campaign reads the labels when it starts, so no pod has to roll. The variable
itself is not copied into the service’s environment; the node names stay in the operator’s
.env and on the nodes. Like the placement labels, the registry survives cleanup
(including --forget-placement). kubectl get nodes -L robovast.io/job-node-alias shows
it.
An alias narrows the job node pool and cannot leave it. A node must be schedulable by a
campaign job and inside the job node pool — ROBOVAST_JOB_NODE_LABELS — and the whole set is
refused, before the command changes anything, if one alias fails or the variable is not valid
JSON. The same check runs again when a campaign that names the alias starts, because a node can
be cordoned, a pool changed or a second node labelled by hand in between. A campaign refused
there is told which alias failed and why — unregistered (with the aliases that are),
registered on several nodes, not schedulable, outside the pool, or on a node without its
identity label — but never the node.
vast doctor fails a pool that matches no schedulable node and every registered alias a
campaign could not use.
Pinning the clock
setup sets the nodes’ CPU governor to performance. On by default, and alone among
what setup does it reconfigures the host rather than reporting something about it — so it is
the one step to know about before pointing setup at a machine you share.
It is on by default because a cluster used for measurement whose clock moves with load produces numbers that are wrong in a way nothing downstream can detect or correct.
A node on a scaling governor changes clock speed with load, so a per-node figure a campaign records — CPU usage, realtime factor, run duration — is taken against a clock that was not the same for every run. Fixing the governor removes that variable; it does not claim to be the only one.
--no-performance-governor skips it and leaves the hosts alone. Naming
--performance-governor explicitly changes the failure policy rather than the outcome: a
cluster that refuses the DaemonSet is then an error instead of a warning, because someone who
asked for a fixed clock and silently did not get one would go on to trust measurements taken
on a scaling one.
A provider whose nodes are virtual machines moves the default rather than the flag. gcp
and azure schedule campaigns on cloud VMs, whose guest kernels expose no cpufreq policy at
all — the hypervisor owns the clock — so setup states that once and does not attempt it, rather
than spending a readiness wait every run to rediscover the same answer. Naming
--performance-governor is still obeyed there and still fails loudly: provider policy decides
what happens when nobody said, never what happens when somebody did.
What the governor buys does not go away with the knob. Comparability between runs is still worth
having on a cloud pool, and the levers there belong to the node pool rather than to a DaemonSet:
a machine type with a predictable clock, no shared-core and no preemptible instances in the
campaign pool, and node upgrades held for the duration of a campaign. The
cpu_governor_scaling warning per campaign is what keeps the remaining gap in the results.
It installs a privileged DaemonSet with the host’s /sys mounted writable, confined to
the job node pool when one is configured. Leaving it off is supported: RoboVAST reports a
cpu_governor_scaling warning per campaign, so the effect shows up in the results rather
than silently.
vast cluster cleanup removes that DaemonSet — the owner, not its pods, which a DaemonSet
would recreate at once — and says so. It does not put the nodes back. The pods do not
restore the previous governor when they stop, and nothing recorded what it was, so a node
stays on performance until something changes it; a reboot returns it to its own
configured default. On shared hardware that is a real lasting side effect, which is why
cleanup prints it rather than reporting a clean teardown.
Warning
On a cloud VM this cannot work, which is why gcp and azure do not attempt it.
Where it is attempted — an unlisted provider, or --performance-governor naming it —
the two ways it fails are different. A cluster that refuses the privileged pod, as GKE
Autopilot does, says so at create time. GKE Standard and EKS generally accept it, and
the DaemonSet then fails at runtime: a GCE or EC2 guest has no writable
/sys/devices/system/cpu/*/cpufreq, so the pod exits non-zero and
CrashLoopBackOffs on every node. Setup waits for a Ready pod and reports both cases
rather than either as applied. Node auto-repair would undo the setting anyway. Set the
governor through the node image where the node image is yours — and read the
cpu_governor_scaling advice, which is what tells you whether it took effect.
Where this deployment’s own data lives
The selectors above place job and control pods. This places the deployment’s own state,
which is node-local by default: a stock cluster ships no StorageClass, so hostPath is what
the workspaces, the campaign results, the campaign index, the registry and the build cache
all fall back to.
Where it goes is one flag, --data-root; which node it goes on needs no flag at
all. The two are separate questions and are answered separately below.
Setup decides the node once, and records the decision as a node label:
vast cluster setup rke2
robovast.io/data-node: not found on any node -- picking by free space
node-a 186.8 GB free (kubelet-nodefs) <- chosen
node-b 48.6 GB free (kubelet-nodefs)
node node-a labelled robovast.io/data-node=true
✓ Cluster setup completed successfully!
workspaces, results, index and registry on node-a (chosen automatically: most free disk)
build cache alongside it on node-a (--buildkit-node puts it on another disk)
recorded as a node label, so a later cleanup + setup returns here without any flag
Because a node label is cluster-scoped, it outlives vast cluster cleanup.
A later setup reads it back and lands on the same node with no flags at all:
robovast.io/data-node: node-a (existing label)
That is the point of the label. Without one, a cleanup + setup leaves no trace of the
previous placement, the scheduler is free to choose again, and on a heterogeneous cluster it
will: the service comes up on a different node with an empty registry – the blobs still on
the old node’s disk, intact and unreachable – while setup reports success. The Disk
meter changes at the same moment, because it reports the filesystem of the node carrying the
service pod (see Web UI), which is then a different machine.
The pods carry the label as a constant nodeSelector, never a hostname. That is what
makes the pin impossible to lose: there is no node value for a caller to forget, and so
none for an upgrade to drop.
How the node is chosen
In order, stopping at the first that applies:
--data-node NODE/--buildkit-node NODE, when given.The existing label – the ordinary path on every run after the first.
The eligible node with the most free space. Eligible means Ready, not cordoned, and carrying no taint the pod does not tolerate: ranking on size alone would pick a
NoSchedulecontrol-plane and leave the podPendingwhile setup reported the placement as chosen. Free space comes from the kubelet’sstats/summary, falling back toallocatable.ephemeral-storagewherenodes/proxyis not readable; the log names which was used, because they measure different things.
Two rules keep it sticky:
The build cache follows the data node, not “the other one”. Auto-separating would put a 150 GB cache on whichever node was left over, which is usually the smaller. So
--data-nodealone moves both labels – one name for the whole of this deployment’s on-disk state, because a cache left behind on the old node is exactly the stranded-bytes surprise the flag exists to make visible. Separate them with--buildkit-nodewhere the disk is tight: a full builder disk on the service’s node becomes DiskPressure evictions of the API, and once named, the cache stays where it was put.Naming a node is enough to move the placement. Typing a node name is already the deliberate act, so there is no second confirming flag. What the move costs is reported rather than guarded: setup names the node the label came off, and the bytes there are not migrated – the new node starts with an empty registry and rebuilds what it needs.
Nothing is pinned where nothing is on a node: pass --registry-class /
--workspaces-class and the pods schedule freely, because a nodeSelector on a
provisioned volume is noise at best and unschedulable at worst.
--control-node-label still applies and is ANDed with the
placement label rather than replaced by it: it narrows which nodes may be chosen, while the
label decides which one of them holds the data. A pool selector alone still lets the pod
float within the pool, which is this same problem at a smaller scale.
Which directories it uses
The node label above decides which machine. This decides where on it — normally with one flag, because the ordinary case is a machine with a disk mounted for RoboVAST:
vast cluster setup rke2 --data-root /media/data
That places every node-local directory this deployment keeps:
/media/data/workspaces the service's workspaces
/media/data/results the campaign results root, where campaigns live
/media/data/index the campaign index, placed beside them
/media/data/registry the built experiment images
/media/data/buildkit the build cache
Each tenant can also be named on its own, and overrides the root for itself:
Flag |
Environment |
Default |
|---|---|---|
|
|
(unset) |
|
|
|
|
|
(unset — a hostPath) |
|
|
|
|
|
(unset — a hostPath) |
|
|
|
|
|
|
|
|
(unset — a hostPath) |
|
|
|
|
|
(unset — a hostPath) |
|
|
|
Every one reads an environment variable, so a .env — or ~/.config/robovast/env, for
what is true of the machine rather than of a project — sets them once instead of on every
setup.
Two tenants take no path flag, and for the same reason: derived data must not be separated from its source. The campaign results sit beside the workspaces and share their backing, because the service pod carries both and mirrors a campaign between them. The campaign index sits beside the results and shares their backing, because every row in the index was ingested from a campaign on the results volume – an index that outlived its sources would answer questions about campaigns nobody can reproduce or check, confidently. A path flag for either could only ever agree with its parent or be refused.
How much room the results get is stated separately, because they are the larger half by
orders of magnitude: --results-size sizes the claim the workspaces’ class provisions,
and the same flag on vast service upgrade raises one that already exists. Without a
class there is no volume to size – the results are a directory on the data node, bounded
by that disk – and the flag is refused rather than accepted and ignored.
--index-class is the one thing a derived tenant may state for itself, and it does not
break the rule: it takes the index off its parent’s disk without moving it off its parent’s
node. Postgres is a different workload from the bulk of the results, and on a managed node
pool the index is what a replaced machine would take with it while the campaigns it indexed
survive on their claim. Without it the index simply follows the results — a PVC of the
workspaces’ class where one is given, else a directory beside them on the data node.
In order, the first that answers wins: what you stated (flag or environment), then
--data-root, then what the cluster is already doing, then the default. That third step
is why a re-run with no flags leaves a moved deployment where it is — and why --data-root
outranks it: recovery answers for a caller that said nothing, while a root is a caller saying
where this deployment’s data goes. Ranked the other way, a root would move the registry and
the build cache and leave the workspaces behind.
Two settings that cannot both apply are refused rather than ordered, before anything is applied:
--registry-path and --registry-class both place the registry, and they cannot both
apply: a class provisions a volume, a path names a directory on the node. Pass one.
A class and a path for one tenant is the case worth catching: the class wins, so the path is
accepted, reported, and ignored. --buildkit-size without --buildkit-class is the same
failure a step smaller — it sizes a claim nothing will create.
Note
--data-root places this deployment’s own state, not the scratch a run produces on
its way into the campaign. A campaign job’s working directory is an emptyDir, so it lives
under the kubelet’s root directory on whichever node the job ran — which is right: an
emptyDir is per-pod, isolated and reclaimed automatically, and a shared host directory
would put concurrent runs on one node in each other’s way. On a node whose root filesystem
is small, moving those means moving the kubelet’s root directory, which is the node’s
configuration rather than RoboVAST’s.
Moving it, and forgetting it
kubectl get nodes -L robovast.io/data-node -L robovast.io/build-node # where is it?
vast cluster setup rke2 --data-node node-b # move it
vast cluster cleanup --forget-placement # forget it
The move says what it abandoned, which is the only place that node is ever named again:
workspaces, results, index and registry on node-b (as requested)
build cache alongside it on node-b (--buildkit-node puts it on another disk)
moved here from node-a; the bytes written there are NOT migrated
so this deployment starts with an empty registry and rebuilds what it needs;
campaigns held on the old node do NOT come with it
None of these move the data. The workspaces and registry bytes stay on the old node’s
disk; a moved deployment starts with an empty registry and rebuilds what it needs. The
results volume does not come with it either, and that one is not cheap: it holds every
campaign this deployment has finished. Archive what matters (vast share) before moving
the placement.
One thing a re-setup cannot do by itself: applying a manifest over an object that already
exists keeps the existing one, so a running robovast pod — the registry and the index
— cannot be relocated that way. Setup refuses rather than reporting a placement it did not
apply, and says what recreating it costs: delete the pod, or run cleanup first.
GPU rendering
A simulation container renders its cameras offscreen with MuJoCo. Given a GPU it uses EGL; without one it falls back to software rendering, which is correct but roughly an order of magnitude slower and burns a dozen CPU cores doing it. On a sweep that is often the difference between a campaign that finishes and one that does not.
There is nothing to configure. Setup detects a GPU and makes it schedulable, and the
container that runs the simulator then requests one because the cluster advertises it — no
flag, and no change to the .vast. The same file renders in hardware on a GPU cluster and
in software on a CPU one.
vast cluster setup rke2 -x local # provisions GPUs if the cluster has them
What that does, when a GPU is found: installs the NVIDIA device plugin with time-slicing so several pods can share
one card, and puts runtimeClassName: nvidia
plus NVIDIA_DRIVER_CAPABILITIES=all on the pods that ask for one. That capability is the
load-bearing part: the container runtime’s default (compute,utility) hands over the
device without the GL half of the driver, so the container gets a GPU it cannot render on
— and nothing errors, the job is simply slow. roqsim refuses to start in that state rather
than quietly producing a slower result.
A cluster with no GPU is not a problem to report: nothing is installed, no manifest changes, and setup does not fail. The same is true if the plugin cannot be installed or never comes up — unless GPUs were asked for explicitly, which turns those into errors.
Concurrency and VRAM. --gpu-replicas N sets how many pods may share one physical GPU:
vast cluster setup rke2 -x local --force --gpu-replicas 24
vast cluster setup rke2 -x local --no-gpu # opt out entirely
N caps concurrency; it does not partition device memory. Nothing in Kubernetes, in
the plugin, or in the driver gives each pod a share of VRAM — all N renderers allocate
from the same card, first come first served — so N is an assertion that N
simultaneous trials fit in it. Exceed that and a trial’s simulator fails mid-run.
Measured on an RTX A2000 12GB, one 640×480 offscreen context per pod, against a 337 MiB baseline (the node’s desktop session):
Concurrent contexts |
GPU memory used |
Above baseline |
Marginal per context |
|---|---|---|---|
1 |
430 MiB |
93 MiB |
93 MiB |
4 |
711 MiB |
374 MiB |
93 MiB |
8 |
824 MiB |
487 MiB |
60 MiB |
16 |
1574 MiB |
1237 MiB |
77 MiB |
So the cost is sub-linear — the driver shares part of its allocation across contexts on one GPU — and 16 concurrent renderers used 13% of the card, with GPU utilization at 45% while each rendered at 10 Hz. On this hardware VRAM is nowhere near the binding constraint: it is CPU quota that limits campaign width, which is why the default sits above that ceiling.
Do not read those figures as a budget for your own worlds. A 640×480 framebuffer of a trivial
scene is the floor: the framebuffer scales with the requested frame size, each camera builds
its own renderer, and meshes and textures are extra. Re-measure with nvidia-smi on the
node during a real campaign, or from the per-run resource_usage table, before raising the
default for a heavier world.
The default of 16 is chosen to sit above the CPU ceiling, so GPU quota is not what limits a
campaign: a three-container scenario job asks for roughly ten cores, so a 96-core node admits
about nine concurrent jobs either way. Raising N therefore changes nothing until per-job
CPU drops — which GPU rendering itself makes possible, by freeing the cores software
rendering was using.
The value is not stored anywhere. The node’s advertised capacity is the record, because unlike a remembered number it cannot go stale:
kubectl --context local get node <node> -o jsonpath='{.status.capacity.nvidia\.com/gpu}'
A bare re-run of setup --force preserves whatever count is deployed, so it will not
quietly undo a deliberate --gpu-replicas 24.
Opting a campaign out. Set gpu: 0 on the simulation container to leave the GPU alone
— worth doing for a camera-less world, which never renders and would otherwise hold a
replica for nothing:
execution:
containers:
simulation:
resources:
gpu: 0
gpu also takes the per-cluster form, for one .vast across a GPU and a non-GPU
cluster: gpu: [{local: 1}, {gcp-c4: 0}].
Which GPU did a run actually use? sysinfo records it, so it is a query over a
finished campaign rather than something inferred from wall-clock:
SELECT config_name, run_id, sysinfo_json FROM run_view
giving a gpu block per run — render_node, nvidia_model, nvidia_driver — which
together say whether that trial could render in hardware and on what. A GPU present with no
render node is the graphics-capability mistake described above.
Which backend was bound is a property of the process that rendered, not of the node, so it
is asked there instead: roqsim.rendering.bound_gl_backend() and bound_gl_device(), and
roqsim logs the pair at INFO when the application configures logging. The device string is the
informative half — egl alone is what any machine with a working hardware GL stack reports,
and on a multi-GPU host it does not say which card the driver picked.
Note
Fix a campaign’s GPU concurrency and record it. Time-slicing inflates each trial’s rendering time, so cells that ran at different concurrency are not comparable with each other — a result that looks like an effect of the variable under study.
To tear everything down after use:
vast cluster cleanup
Cleanup removes the device plugin too, but never the nvidia RuntimeClass or the host’s
driver and toolkit: those belong to the cluster and its node administrator.
Cleanup deletes named objects rather than the namespace, so what it removes is a list rather
than a sweep: the campaign Jobs and their pods, the image-warm DaemonSet, buildkitd, the
robovast-service Deployment and Service, the robovast pod, the controller RBAC, the
CPU governor DaemonSet, the NVIDIA device plugin, and whatever the cluster config’s own
cleanup_cluster owns.
Deliberately kept: this deployment’s data directories — the results (where every finished
campaign lives), the workspaces, the index and the registry — buildkitd’s volume claim, the
node identity labels, the job node aliases (Naming a node a campaign may be held to), and the placement
labels — see Where this deployment’s own data lives, and --forget-placement for the last of
those. vast cluster cleanup --delete-data empties the directories instead, irreversibly
and with no archive taken first. The one thing cleanup cannot undo is the CPU
governor setting itself; see Pinning the clock.
Running Scenarios
# Run every config the workspace's .vast expands to
vast workspace run my-experiment my.vast
# Override the number of runs from the CLI
vast workspace run my-experiment my.vast --runs 5
# Run only the configs matching a name or glob (batch campaigns)
vast workspace run my-experiment my.vast --filter my-config
run is fire-and-forget: it starts the campaign on the service and returns
immediately, printing the campaign id. The campaign continues in the cluster —
watch it with vast cluster monitor. (It needs a reachable service:
auto-detected on the conventional local port, or the one vast login stored; run
vast cluster setup first if you have none.)
Monitoring and Results
Check the status of a running (or recently completed) run:
vast cluster monitor
A finished campaign is already whole on the service’s results volume — its Jobs delivered
their results there as they ran — and vast campaign download (or
run --wait-and-download) streams it from there, no external share needed:
vast campaign download campaign-2025-06-01-120000
# -> ./campaign-2025-06-01-120000.tar.gz
That is the whole command. It fetches the campaign as this service holds it —
postprocessing and all — writes one .tar.gz, and stops: nothing is extracted, no
results directory is written into, and no state is kept about what you already have.
The stream is end-to-end, so a ~1TB campaign is never buffered on the service or in
memory. What you do with the archive afterwards is yours; to put it back into a
service, vast campaign import <archive>.
The share’s raw, pre-postprocessing copy is a different system, reached through
vast share (see Sharing Results). To push a copy there, either enable it
at launch (Upload to share when done in the web UI, --upload-to-share on
vast workspace run, or the MCP upload_to_share flag) or export a
finished campaign with vast share export -i <campaign-id>.
Clean up only the job objects, leaving every campaign where it is:
vast cluster jobs-cleanup
vast cluster jobs-cleanup --campaign campaign-2025-06-01-120000
That removes the scenario Jobs and their pods with your kubeconfig (-x context,
-n namespace) and touches no results: they live with the service, and
vast campaign delete is what removes a campaign — through the service, so a delete
never catches a campaign that is still running.
Push notifications (ntfy)
Because a run is fire-and-forget, the campaign driver can push ntfy.sh notifications so you don’t have to poll monitor. Set a
topic in your .env and subscribe with the ntfy mobile/desktop app:
ROBOVAST_NTFY_TOPIC=robovast-alice-campaigns # enables notifications
ROBOVAST_NTFY_SERVER=https://ntfy.sh # optional, this is the default
ROBOVAST_NTFY_TOKEN=tk_xxx # optional, for protected topics
You then get a message when a campaign starts, when each batch finishes, once an hour with the current run progress, when it is uploaded to the share, and exactly one message when the campaign ends — whether that end is a finish, a stop, or (urgently) a failure.
Re-running a campaign also announces itself, on the topic of the campaign it was re-run from: the new campaign gets its own id, and someone following the old one otherwise has no way to learn which. It is not an ending — the source campaign is unmodified and still sends its own — and the re-run announces its own start as usual.
The ending message is worth reading rather than glancing at. It is sent when the campaign is genuinely over, after postprocessing rather than when the last run stops, and it carries what the campaign actually produced: the run tally, and any postprocessing or upload failure. A campaign whose trials all passed but whose postprocessing failed is reported as “finished WITH PROBLEMS” — it has no CSVs and nothing queryable, and reporting that as a clean finish made a campaign with no metrics look identical to a complete one on the one screen nobody re-reads.
Notifications outlive whatever started the campaign, which is what makes them the right thing to rely on for a long sweep: a CLI wait, a terminal session, and an agent’s attention span all end sooner.
Notifications are optional and best-effort: with no topic set the driver stays silent, and an unreachable ntfy server never affects the campaign. Pick a different topic per user so notifications don’t cross over; each message carries its campaign id so concurrent campaigns sharing a topic stay distinguishable.
For an in-cluster service the ntfy config is read from your .env at
setup time and injected into the service pod (as a Kubernetes Secret, exactly
like the share credentials), so changing the topic means re-running setup
--force to redeploy. A vast serve started by hand reads the .env live.
Experiment image builds (registry)
Agent-built experiment images (a project’s build section) are built in-cluster by a BuildKit Job and pushed to a
container registry the cluster can pull from. RoboVAST runs that registry itself, as
a container in the robovast pod vast cluster setup creates, so there is nothing to
configure and no site prerequisite: a deployment can always build. See
Where the registry and the index run for why it lives there rather than beside the service.
The registry is published on /v2 of the host the service already answers on, and the
image prefix is that host — robovast.example.org/<tag>:<hash>. That is not a
cosmetic choice. An image ref is one string resolved by two different things: BuildKit
pushes to it from inside a pod (pod network, CoreDNS) and the kubelet pulls it on the
node (node network, node resolver, node TLS trust). Nothing in a pod spec reaches the
second one — see When the cluster cannot resolve the registry — so a …svc name works for the push and
fails for the pull, and a plain-HTTP registry needs registries.yaml plus a runtime
restart on every node. The service’s own published hostname already has real DNS and
a real certificate, so both halves work with no node configuration at all.
The same URL works from a workstation: docker login robovast.example.org and then
docker pull robovast.example.org/<tag>:<hash> needs no CA import, which is how you
reproduce a campaign’s exact image locally.
A published registry authenticates, and does so in the registry rather than at the
Ingress. The credential is minted at setup, kept in the cluster, and never rotated by a
re-run — the deployment’s own push, the nodes’ pulls and the service’s “already pushed?”
probe all read it from one dockerconfigjson Secret, so nothing has to be configured for
builds to keep working. vast service token does not print it: it is not a user
credential, and no client needs it.
Ask the cluster when you want it for a docker login from a workstation:
kubectl get secret robovast-registry-push \
-o jsonpath='{.data.\.dockerconfigjson}' | base64 -d
It is enforced by the registry and not by an Ingress annotation deliberately. The only
annotations for this are ingress-nginx’s, and a cluster whose controller is something else —
GKE’s gce, for one — accepts them, ignores them, and serves an open registry while
reporting success. What the registry enforces itself holds on every controller, and on the
cluster-internal route as well.
Note
An unpublished deployment leaves it open, because there is then no route to it: the prefix campaigns build into is derived from the Ingress host, so a deployment with no Ingress cannot build at all. Publishing is the moment it becomes reachable, so publishing is what turns auth on — the same rule that already refuses an Ingress without an access token.
Warning
A published deployment that predates this gains auth on its next setup, and a
campaign already running across that moment is the one case to plan for. Job pods created
before it carry no imagePullSecrets; while their pods keep running this costs nothing,
but one that restarts afterwards re-pulls without a credential and fails with a 401 that
reads as a missing image rather than a missing login.
Let running campaigns finish before re-running setup, or expect to re-launch them.
No campaign’s results are affected, and campaigns launched after the change get the
credential like any other.
The registry keeps serving anonymously until the robovast pod is recreated.
Setup keeps an existing pod as it is — recreating it on every run would restart the
registry and the index for nothing — so a changed container spec does not reach it, and
the credential alone changes nothing. Setup says so rather than reporting the hole closed: vast cluster cleanup
then vast cluster setup is what applies it, built images are rebuilt on demand, and
the clients already hold the credential and start using it the moment the registry asks.
A service without a registry prefix cannot build. The prefix is the service’s own published host — with no Ingress there is no address a node could pull a built image back from — so a project whose container adds packages fails at submit naming that reason.
Two different states produce it, with two different fixes, and the service cannot tell them apart: it reads the prefix out of its environment and has no RBAC to look at its own Ingress.
Published, but the prefix is unset.
vast service upgradere-bakes the prefix from the live Ingress. A deployment whose prefix was dropped by an oldersetupre-run without--ingress-host— which left the Ingress alone, so nothing looked wrong until a campaign was submitted — needs exactly that upgrade.Never published. Re-run
setupwith--ingress-host.
vast doctor -n <namespace> says which of the two you are in.
The same host is also written in as the service’s declared origin, from the same
--ingress-host. It is what the service reports as web_base, so a client that cannot
be handed bytes – a 20k-line log, a truncated query, a rosbag – can be handed a link to
fetch them out of band instead. The two are separate values on purpose, even though today
they are the same host: a registry prefix may point somewhere else entirely, and the origin
must not silently follow it.
An ``upgrade`` re-bakes it too, and needs no arguments to do it. The origin carries a
scheme as well as a host, and an upgrade holds none of the TLS arguments the Ingress was
created with – so it reads the whole URL back from the live Ingress, where the TLS block
decides the scheme, and states that. Which means a single vast service upgrade is
enough to publish the origin of a deployment that predates it, and enough to clear it
again for one that has been unpublished: reading the Ingress is what tells those two apart
from “nobody told me”.
A setup re-run without --ingress-host reads it back from the live Ingress the same
way, so the origin is always what the Ingress publishes. Every deploy replaces the Deployment
whole, so a value set on it by hand lasts until the next setup or upgrade.
A service that was never published declares no origin at all, which is honest rather than degraded: routes and file addresses still work, and only the absolute URLs are absent.
Storage defaults to a hostPath on the node the pod is pinned to, because a stock RKE2
cluster ships no StorageClass and a PVC there stays Pending forever. emptyDir is
deliberately not offered: every upgrade restarts the pod, which would discard every
built image on each version bump while already-submitted Jobs went to
ImagePullBackOff.
Only two things remain configurable, both about other people’s registries:
# Credentials for a private registry a .vast names in its `image:` field. Purely a
# PULL credential for THAT registry -- the build registry's own push credential is
# minted automatically at `setup` (see above) and is never configured here. Applied
# to campaign pods, aux/exec pods and the service's own image.
ROBOVAST_REGISTRY_SERVER=harbor.example.org
ROBOVAST_REGISTRY_USERNAME=<user>
ROBOVAST_REGISTRY_PASSWORD=<token>
# Optional: default base image when build.base_image is omitted.
ROBOVAST_BASE_EXPERIMENT_IMAGE=ghcr.io/cps-test-lab/sim-suite-nav2-eval:latest
Removing one of those variables and re-running upgrade deletes the Secret it
created, rather than leaving a revoked credential deployed and still attached as an
imagePullSecret.
The pull Secret that setup creates from those values is found by the service
itself — it looks for the fixed name it would have created (robovast-registry-push)
and uses it when present, so nothing has to tell it it exists. This matters for a
vast serve started by hand (mirrord exec): setup stores its env in the service
pod, which that process never reads, so without the lookup it would conclude there were
no credentials at all. Set ROBOVAST_REGISTRY_PULL_SECRET only to point at a
differently named object; it is an override, not a requirement.
When the cluster cannot resolve the registry
A registry whose name lives only in /etc/hosts on your workstation is not resolvable
from inside the cluster, and the push fails after the whole image has been built:
failed to push harbor.example.org/robovast/my-tag:<hash>: ... dial tcp:
lookup harbor.example.org on 10.43.0.10:53: no such host
Declare the mapping instead of editing CoreDNS — RoboVAST puts it in the pod specs it
creates (the build Job and campaign Jobs) as hostAliases:
ROBOVAST_EXTRA_HOST_ALIASES=harbor.example.org=10.0.0.9
# several: comma-separated, names sharing an IP are grouped automatically
ROBOVAST_EXTRA_HOST_ALIASES=harbor.example.org=10.0.0.9,mirror.example.org=10.0.0.10
Warning
This fixes what a pod resolves — the push, and anything the scenario itself looks
up. It does not fix the image pull: that is performed by the container runtime
on the node, which reads neither pod specs nor CoreDNS. An unresolvable registry
therefore also needs the name in each node’s own resolver (/etc/hosts or
registries.yaml) — the same node-level scope as registry TLS trust.
A real DNS record is the one change that covers both, and is preferable wherever you can add one. Note also that naming a registry by IP is not a substitute when it sits behind an SNI-based proxy: a client dialing a bare IP sends no TLS SNI, so such a proxy serves no certificate at all and the handshake fails before any CA could apply.
RoboVAST’s own registry avoids this entirely by being published on the service’s Ingress host, which by construction is a name both the pod and the node resolve.
Where the build context is staged
The build needs somewhere to put its context (the project directory plus the generated
Dockerfile) for the BuildKit Job’s init container to fetch. There is nothing to
configure: the service stages it in a staged slot on its own disk,
image-builds/<build-id>, and the init container fetches that slot from the data plane
as one tar stream (Where a campaign lives, and how pods reach it). A build belongs to no campaign, so it gets a slot
of its own rather than a campaign’s route, and the token in its environment reaches that
slot and nothing else.
The init container stays even though BuildKit can read an HTTP context by itself: the data
plane wants an Authorization header, and buildctl --opt context=<url> has no way to
send one.
What is not copied
The context skips BUILD_CONTEXT_IGNORE (robovast.common.build_context) —
.git, results/, __pycache__, build/, .venv and the rest — by exact
name, pruned while the context is staged.
It also skips campaign output directories, and those are recognised by structure
rather than by name: a directory holding an _execution/ child. There is no name to
list, because a campaign directory is named after its campaign id. results/ was
already excluded, but workspace run --wait-and-download extracts a campaign
beside the sources rather than into results/ — so a project that had used it a few
times was staging several hundred megabytes of rosbags around a fifteen-megabyte tree,
uploaded and mirrored back down once per container on every build, with nothing reporting
it. Structure means the rule cannot go stale on a project nobody had in mind when it was
written.
This changes what is copied, never what is hashed: a campaign output is not a
python_packages entry, so build_hash never saw it and the context hash is
unchanged either way.
A staged context is scratch, and is cleaned up — it is a full copy of the project directory, so a build per experiment would otherwise pile up copies on the service’s disk indefinitely. Nothing reads it after the build: a rebuild re-stages, the layer cache lives in the registry, and a failure is diagnosed from the build log. So:
the context is deleted as soon as the build is seen to be terminal (either outcome) — that is any
vast image buildwithout--no-wait, and every campaign submit, since both poll the status to completion;on the next build, any context whose Job no longer exists is swept. Build Jobs self-destruct at
ttlSecondsAfterFinished(1 h), so an absent Job means the build ended at least that long ago — or died with a previous service instance. This is the backstop for a--no-waitsubmit nobody polled, and it runs on a cache hit too.
Builds the service still has in flight are held back from the sweep explicitly: their
context is staged before their Job is created, so “no Job” on its own does not mean
stale. The Job itself needs no cleanup of ours, and neither does the :buildcache
tag — one tag per build tag, overwritten in place.
Caching in-cluster
Builds are solved by one long-lived BuildKit daemon (robovast-buildkitd), not by a
BuildKit spawned inside each build pod. The build Job still exists and still stages its own
context – it is now a client that dials the daemon, which works because buildctl --local
resolves its paths on the client and streams them over the session.
That daemon keeps its store on a volume, and the store is the point. When every build got a
fresh BuildKit, two costs were paid every single time and neither looked like anything but a
slow build: the base image was pulled again (measured at 95-110 s per container, on builds
where every layer was a cache hit), and RUN --mount=type=cache was discarded, so a pip
layer that missed re-downloaded its wheels in full. No registry-side cache can help with
either – a cold builder has to materialise the base whatever the registry holds.
Two registry-backed mechanisms sit above the daemon’s own store:
Is it already built? The service asks the registry whether
<prefix>/<tag>:<hash>already has a manifest, and skips the build if so. This is deliberately not derived from the build Job’s status: that Job is deleted afterttlSecondsAfterFinished(1 h) and the in-process record is lost on a service restart, after which a bit-identical image would otherwise be rebuilt and re-pushed. The probe fails closed — if the registry cannot be reached or authenticated, the image counts as absent and is rebuilt, because a wrong cache hit would leave the campaign pods inImagePullBackOff.Layer reuse across hashes. The build imports from and exports to
<prefix>/<tag>-<scope>:buildcache(mode=max, so intermediate layers are kept too). This tag is not hash-qualified — that is the point: the build for a new hash reuses the layers of the previous one, so changing one latepython_packagesgroup does not rebuild the ones before it. A failing cache export never fails the build (the image is already pushed by then); a failing import just makes the build slower. Both refs inherit the deployment’sINSECURE/ CA settings, since they address the same registry as the push.
The <scope> segment is what keeps that reuse from reaching too far. <tag> is the
container’s name, so it is sut, simulation or scenario for very nearly every
project there is. Keyed on the name alone, every project in a deployment exported
mode=max to the same three tags, overwritten in place — so a campaign whose large,
stable install group had been built an hour ago found it gone, because an unrelated
campaign with a container of the same name had built in between. The group an author had
deliberately ordered first to protect was the one that kept being rebuilt.
cache_scope (robovast.service.image_build) therefore hashes what decides which
layers exist — the resolved base image, the apt set, and the install groups in order —
with the per-iteration churn removed: a local wheel counts as its distribution name,
not its filename, so bumping pkg-0.1.24 to pkg-0.1.25 keeps the namespace it is
meant to reuse. A resolved VCS commit is deliberately not an input: a moved branch
changes a layer’s content, not the chain’s shape, and retiring the namespace for it would
throw away the layers below the one that actually changed.
The registry therefore needs room for one extra tag per built container per chain —
a project retires its own old scope whenever it changes its base image or its package
set. Grouping the python_packages list by change frequency is what makes the layer
cache pay off — see containers — and the scope is what makes
that grouping survive other projects building alongside it.
The build daemon
robovast-buildkitd is applied by setup and converged by upgrade, one replica with
Recreate. Its storage is chosen the same way the registry’s is:
vast cluster setup rke2 --buildkit-path /data/robovast-buildkit
vast cluster setup gcp --buildkit-class premium-rwo --buildkit-size 200Gi
A hostPath by default, because a stock RKE2 cluster ships no StorageClass and a PVC there
stays Pending forever. That makes the cache node-local, so --buildkit-node pins the
daemon; keep it off the registry’s node, since those are the deployment’s two large on-disk
tenants and the service pod is pinned to that one. On a cluster that can provision volumes,
prefer an SSD class: BuildKit’s snapshotter is small-file heavy, and a slow disk becomes
the bottleneck the cache was meant to remove.
The daemon’s budget and size are set in the .env — no flags — and applied by both
vast cluster setup and vast service upgrade. Unset takes the default, deleting a line
resets it, and an upgrade prints every value it changes on the running daemon:
Environment |
What it bounds |
Default |
|---|---|---|
|
the cache’s size |
|
|
free space kept on its disk |
|
|
cache kept even when old |
|
|
memory one build may use |
|
|
CPU one build may use |
|
|
build steps run at once |
|
A cache size is an amount (150GB) or a share of the disk (70%); memory and CPU are
Kubernetes quantities. A value that is not one fails the command before anything is applied.
A deployment whose daemon was given these as setup flags keeps them only until its next
upgrade: write them into the .env first.
Size the cache for the disk it lands on. The load-bearing setting is the free space it
keeps: it is measured against the filesystem, so it prunes the cache long before an
oversized ceiling is reached, and stops a full builder disk from becoming DiskPressure
evictions on its node. Unset, it is the service’s free-space reserve where that is stated in
gigabytes (ROBOVAST_DISK_RESERVE_GB, see Deployment Modes and Access), and never below 50GB.
A build killed for memory is reported as a resource failure fixable by infra, not
as a missing dependency: the daemon’s ceilings are one build’s ceilings. Raise
ROBOVAST_BUILDKIT_MEMORY where the nodes have room, or lower
ROBOVAST_BUILDKIT_PARALLELISM where they do not — peak memory is roughly the parallel steps
times the heaviest compile. The daemon reserves far less than its ceilings, because admission
subtracts every request from what campaigns may run; a ceiling below that reservation lowers
the reservation with it.
Changing any of these replaces the daemon pod, so upgrade --no-restart leaves the daemon as
it is.
Three consequences worth knowing before they surprise you:
An upgrade kills builds in flight.
Recreatereplaces the daemon pod, and a campaign waiting on a build detaches rather than cancels — so a sibling campaign can be waiting on a build a restart destroys. Nothing drains.The endpoint is unauthenticated. BuildKit offers mTLS and nothing else, and this ships with neither mTLS nor a NetworkPolicy. It is a ClusterIP with no Ingress, but that is not a boundary: campaign pods run images a
.vastchose and can reach it. The pip download cache is shared across every build and now persists, so anything that can dial the daemon can leave something in it for the next build to install. The registry does not share this weakness — it authenticates wherever it is published — so this is the one unauthenticated endpoint the deployment runs, excused only by having no hostname at all.A wedged cache has no remote remedy. Changing a cache scope fixes a bad registry cache; a bad local store needs the daemon restarted or its volume cleared, on the node that holds it.
If the daemon is not ready, a campaign is refused at submit, naming it — before staging a context or creating a Job. A daemon that dies mid-build is classified as an infrastructure failure rather than a problem with the project’s packages, and the daemon’s own recent log is appended when the client produced none.
Note
Rootless BuildKit needs AppArmor and seccomp Unconfined, for rootlesskit’s mount
namespace. That is set on the daemon; the build Job creates no such namespace and
carries no exemption. On nodes whose container runtime cannot pull from the registry
without host trust (e.g. an in-cluster registry over plain HTTP on RKE2/k3s
containerd), the node must be configured to trust it (registries.yaml) for
campaign pods to pull the built image — an external registry with valid TLS
avoids this.
Where the registry and the index run
vast cluster setup creates one robovast pod per deployment, holding this
deployment’s two pieces of setup-lifetime infrastructure:
registrythe container registry experiment images are built into, published on
/v2of the service’s own Ingress host.indexthe Postgres holding every campaign’s rows — one index, so a query across a search arm is one
WHEREclause rather than a per-campaign database. The service reaches it atrobovast.<namespace>.svc:5432; the DSN is assembled from the Service name and the namespace and handed to the service asROBOVAST_INDEX_DSN.
Campaigns are not in this pod. They live on the service’s results volume (Where a campaign lives, and how pods reach it), and everything this pod holds is re-derivable from them or rebuilt on demand: images are rebuilt when a campaign asks for them, and the index is re-ingested from the campaigns. That is why it is one replica with no standby and no backup — losing it costs time and nothing else.
Why they are not in the service pod. robovast-service is a Deployment and every
vast service upgrade rolls it, so a container living there is restarted by each
upgrade — including one that only bumps the controller image — and its volume follows the
Deployment rather than the cluster. Neither of these is service-lifetime state: the
registry holds the images already-submitted campaigns will be pulled from, and the index
holds rows that took hours to ingest. Here they are created once at setup and torn down
deliberately by vast cluster cleanup.
The index sits beside the results and takes their backing unless --index-class
says otherwise (Where this deployment’s own data lives). Every row in it was ingested from a
campaign on the results volume, so it must never outlive its sources: an index that did
would answer questions about campaigns nobody can reproduce or check, and answer them
confidently.
Both ports are on the pod’s single ClusterIP Service. It already selects exactly this
pod, so a second Service would duplicate the selector and add a name that must agree with
the DSN and the Ingress rule, for no isolation — a ClusterIP is not a security boundary.
The registry is nonetheless addressed through the service’s published Ingress host,
because an image ref is resolved twice — by BuildKit inside a pod and by the kubelet on a
node — and only a published name works for both; what the Ingress’ /v2 rule points at
is this pod’s Service.
Warning
A live pod is kept as it is, because recreating it on every setup would restart the
registry and the index for nothing. A pod that does not match the manifest therefore
does not come to match it by re-running setup, and vast cluster setup and vast
service upgrade both refuse rather than deploying a service whose /v2 route and
index DSN point at containers that are not there. Two shapes are refused: a pod missing
the registry or the index, and a pod that carries an object-store container — nothing
reads such a store, so every campaign in it is one the service cannot see. The remedy
for both is vast cluster cleanup followed by vast cluster setup. Campaigns in an
object store are not migrated, and nothing after the cleanup reads them: archive what
matters to a share first, and import it from there (vast campaign import).
Where a campaign lives, and how pods reach it
A campaign is a directory on the service’s results volume,
<results_root>/<campaign_id>/, and the only copy: campaign.db, _execution/ and every run’s output are
written into that one tree, and downloads, re-postprocessing, the
index and vast share all read it there.
Pods move bytes as tar streams
Every pod the service launches exchanges whole trees with the service’s data plane —
the /data routes, listed in HTTP API under Addressing files: one GET of a
tar for what it needs, one PUT of a tar for what it made. Both stream (a fetch is
curl | tar onto a mount, a delivery is tar | curl off one), so nothing is written
twice on the pod and nothing is buffered on the service.
A scenario Job runs a
fetch-inputsinit container that fetches the campaign’s inputs into/config— the campaign’s_config/and_transient/, with this cell’s own files on top. Of the per-job documents in_transient/the pod asks for its own by tag, so what it downloads stays the same size however many jobs the campaign has — and anuploadercontainer that delivers the pod’s whole/outas one tarPUTonce every workload container has written its done marker on the shared/ipcvolume. The uploader is a regular container, and that is the point: a Job is complete only when its results are in the campaign, and a delivery that could not be made is a failed Job rather than a quiet one.A postprocessing Job fetches the campaign — narrowed by
stage,skip_bags,batch_jobsand, for one part of a split postprocess (Splitting a campaign’s postprocessing),partto what it will actually read — and delivers its derived files back the same way.Aux, exec and build pods work on a staged slot instead (
GET/PUT /data/staged/<slot>), which the service stages under<results_root>/_staged/<slot>/: scratch beside the campaigns, sharing their disk and their meter, and discarded with the work that used it.
What a pod is given
A campaign’s pods get three values, and nothing else that reaches the campaign — ROBOVAST_DATA_URL (the service’s in-cluster address plus /data,
assembled from the Service name and the namespace, so no pod holds a name that resolves in
one cluster only), ROBOVAST_CAMPAIGN_ID and ROBOVAST_TOKEN.
The token is scoped: an HMAC of the scope under the deployment’s shared secret, which reaches that one campaign’s data routes and nothing else — not another campaign, not a control route, not the UI. It comes from a per-campaign Secret the service creates before the campaign’s first Job and deletes with its last, because a Job is created many times over and a literal in the pod spec would be readable in every one of them. A pod working on a staged slot gets a slot-scoped token as a plain environment value instead: it lives as long as its slot, and the slot is a scratch tree nothing else can be reached through.
One port, three containers
The service pod runs robovast-service (the control plane: the API, the web UI, the
campaign driver) and robovast-data (the data plane, vast serve-data) on Unix
sockets in a shared emptyDir, with an nginx robovast-front owning the pod’s single
port and routing /data/ to the data plane and everything else to the control plane.
Bulk bytes must not share a process with the control plane: a dozen pods delivering
gigabytes at once should slow each other down, never the run view or the admission loop,
and a second process with its own limits makes that structural rather than a matter of
tuning. The front is what keeps it from costing an address — one port is one thing to
publish, one Ingress rule, one thing a client has to discover — and it is nginx rather
than the control plane proxying, because a proxy there would copy every uploaded byte
through the event loop it exists to protect. Its configuration is rendered into a
ConfigMap by vast cluster setup and vast service upgrade, so it is versioned with
the code that depends on it and never edited on the cluster.
What the results volume is, and is not
It is a directory on a node (rke2, minikube) or a provisioned ReadWriteOnce claim
(the cloud providers), placed beside the workspaces and backed by their class — see
Where this deployment’s own data lives. It survives the service pod being restarted or
upgraded, and vast cluster cleanup leaves it alone; vast cluster cleanup
--delete-data is what empties it. A claim that is filling up is raised with
vast service upgrade --results-size, online, with no campaign interrupted; a claim
cannot be shrunk, and a deployment whose results are a node directory has no volume to
size, so both are refused rather than reported as done.
It is not a backup. One disk is one disk, and RoboVAST manages no snapshot of it:
vast share is the archive, and keeping the disk itself is the operator’s snapshot
schedule. Anything that must outlive the machine belongs in an archive.
It is also the meter. ResourceUsage.results is read from the service pod’s own
volume stats, against used + available rather than the reported capacity — a volume
with no size limit reports the whole node filesystem as its capacity, which is headroom
that is not there. A hostPath deployment has no per-volume figure at all: the kubelet
reports none for one, and the Disk row is that same filesystem (see Web UI).
Job Queueing
Note
If your cluster has Kueue installed, remove it by hand, once. Nothing here installs or removes it, so a cluster that has it keeps a controller pod and its CRDs indefinitely — inert, since no job carries a queue label, but running.
helm uninstall kueue -n kueue-system
kubectl delete crd -l app.kubernetes.io/name=kueue
Delete RoboVAST’s own ClusterQueue and ResourceFlavor first if the CRD delete
hangs — an orphaned custom resource is what holds its CRD open. Check for a leftover
kueue mutating/validating webhook afterwards: that is the one leftover that would
actually break Job creation, rather than merely idling.
A cluster set up fresh needs none of this.
Cluster jobs are queued by RoboVAST itself. A job is created only once the cluster has room for it — CPU and memory, and GPUs on a cluster that has them — so a large campaign cannot oversubscribe the nodes, and several campaigns launched at once share the cluster instead of fighting over it. There is nothing to configure and nothing to install: the queue lives in the RoboVAST service and is sized from what the nodes advertise, each cycle rather than once at setup.
Creating jobs as capacity appears, rather than creating them all and letting them wait, also means a campaign never asks one kubelet for a thousand image pulls at once.
A request no node could ever satisfy — more CPU than the largest machine has, or a GPU on a cluster with none — is refused at launch, with the request and each node’s allocatable named. That is a different thing from a cluster that is merely full, which is not an error and is simply waited out.
So is a pod that declares no CPU at all: a request of nothing fits every node, so the
queue would create the whole plan in one pass and gate nothing. A campaign whose containers
declare no resources.cpu is refused at launch, naming them. If only some declare it
the campaign runs, but the queue paces on less than the pod really takes — that is a
warning naming the silent containers, not an error.
Older campaigns finish first, unless you say otherwise. When several campaigns run at once, the one that started earliest is admitted first: each slot that frees up goes to the oldest campaign that still has work queued, so a campaign is not overtaken by one launched after it. Nothing needs configuring — the order comes from the campaign id, which carries its start time.
That is the right default for campaigns run in sequence, and the wrong one when a long
campaign is in the way of a short one: the long campaign takes every freed slot across all of
its batches, so a five-run pilot waits for a multi-day search to finish. A campaign’s
priority overrides it. Higher is admitted first, 0 is what a campaign nobody asked
about runs at, and negative waits behind everything else:
vast workspace run my-ws long.vast --priority -1 # start it out of everyone's way
vast campaign priority -1 <campaign-id> # or move it aside once it is running
vast campaign pause <campaign-id> # or hold it entirely
vast campaign resume <campaign-id>
Pause holds back what is queued; it does not stop what is running. The runs already
started finish normally and their results are kept, so a paused campaign drains rather than
stopping — the cluster is free for something else within one run’s length, and nothing has
to be re-run when you resume. It keeps the priority it will resume at. To end a campaign
instead, that is vast campaign stop.
Both act on the queue in which campaigns compete for the cluster. The service says that it
has one before anyone asks: can_schedule in
its version answer, printed by vast service info as its queue line and reported by the
MCP get_service_info; the web UI offers the entries only where it is true.
A campaign that is admitting nothing says which of the two reasons it is: a queued campaign reports what capacity it is waiting for, and a paused one reports that it is paused.
vast campaign list carries a live campaign’s standing after its phase when it is not
the default – [prio -1], [paused], [prio +2, paused] – as do the web UI’s
campaign card and the MCP list_campaigns. A finished campaign has no standing with the
queue and carries neither.
Two properties are worth stating, because they are what make this safe to leave on:
It never stops work that is already running. Order — by age, by priority, or a pause — applies to the queue only. An overtaken campaign’s runs finish undisturbed; nothing is preempted, and no partial run data is produced. The campaign ahead takes the capacity as it is released, not by taking it away.
The cluster still stays full. A high-priority job that does not fit does not block smaller ones behind it, so a younger campaign keeps using capacity the older one cannot. Utilization is unchanged; only the order in which queued jobs are admitted changes.
The one thing this does not do is reserve capacity. A search campaign postprocesses between batches, and while it is composing its next batch it has nothing queued — so a younger campaign legitimately fills the cluster in that window, and the older one reclaims the slots as they free. That is the intended trade: the alternative is idling the cluster to hold capacity for a campaign that is not yet asking for it.
Jobs waiting is normal. A campaign whose jobs sit in the queue is healthy: it
is waiting for capacity, not stuck. vast cluster monitor, the web UI
and list_campaign_jobs report such jobs as waiting — a status of its own,
distinct from pending (a pod exists and is being scheduled) and from blocked
(the job cannot start and needs a human). The reason capacity was refused rides along
as the job’s detail.
A campaign that is entirely queued is also not reported as stalled. The no-progress
deadline measures how long a run has gone without completing, and while nothing of
the campaign is running it is timing a queue rather than a run — so the verdict is
withheld and stall_verdict says why, instead of accusing a healthy campaign that is
simply behind an older one.
A waiting job has not been created on the cluster yet, so it appears in the web UI
only as the waiting N counter, never as a row: the per-job list mirrors the jobs that
actually exist there (what k9s shows), and those are the ones with a pod and a log to
read. list_campaign_jobs still returns every one of them with its reason.
The listing spans the two jobgroups a campaign creates for its own work — its trials
(scenario-runs) and its postprocessing conversion (postprocessing) — in one
set-based selector, because it is polled while the campaign is live and a second listing
would double both the Job read and the pod read behind it. Each row carries the
JobKind that says which it is, and only run rows reach JobCounts: the counts are
read as facts about runs, and a conversion or a probe among them would enter the run meter
and the ETA’s divisor. The conversion is the one job a campaign in its postprocessing
phase has, so listing it is the difference between a busy campaign and an apparently idle
one. Its row carries no log of its own, and that is deliberate: the conversion runs in init
containers, which the per-job log does not report (it reports the containers that run for the
pod’s whole life), and every container’s output is already written to the campaign’s
POSTPROCESSING section while the Job runs (publish_live_log() writes
_execution/postprocessing.log) — a copy in the campaign, which is the one that is still
readable after ttlSecondsAfterFinished has taken the pod away.
An unreachable cluster (VPN down, cluster stopped, a kubeconfig context pointing at
an endpoint that no longer answers) is reported the same way: one line naming the API
server and the transport error, within about a minute — every API call has a 10-second
connect timeout (ROBOVAST_KUBE_CONNECT_TIMEOUT), so a connect that cannot succeed
is not left to the operating system’s multi-minute TCP timeout. Read timeouts stay
unlimited; a slow answer is still an answer.
When it happens after the runs — the postprocessing step that follows a finished
campaign — the campaign stays finished with the reason on
postprocessing_error; the results are already published, and
run_postprocessing re-runs the step once the cluster is back.
Note
Admission does not depend on anything being installed in the cluster. If the service cannot measure the cluster it refuses to submit rather than falling back to creating every job at once, which is the one way a campaign could still overload the nodes.
What the queue is not
This is a special-purpose queue for RoboVAST’s own work, not a general cluster scheduler, and two consequences are worth stating rather than discovering.
The service runs as a single replica, and that is load-bearing. The queue lives in
the service process’s memory, so a second replica would be a second queue spending the
same free capacity against the same cluster. Nothing would report an error – the symptom
would be over-admission and pods that cannot be placed. Scaling the Deployment past one
replica requires making the queue cluster-wide state first; the replicas: 1 in the
service Deployment says so at the line.
A service restart leaves the campaigns that were running alone. The queue is the part that does not survive it; the campaigns do.
Jobs already created keep running. They carry no owner reference, so Kubernetes does not collect them, and startup reaping covers aux pods only – they run to completion and their uploaders deliver their results into the campaign, which is where a campaign’s results live anyway. Every transfer a pod makes – its fetch as much as its delivery – retries for longer than the service’s own startup budget, re-running the whole stream each time, so a Job that starts or finishes while the service is being rolled waits for it rather than failing for it.
Jobs still queued were never created, so there is nothing to orphan. They are re-queued when the campaign is adopted.
The successor process does not over-admit against the surviving Jobs. Capacity is measured from the pods actually bound to nodes rather than bookkept, so their requests are visible to it exactly like any other tenant’s – which is also what lets it adopt them without any bookkeeping of its own.
This is what ClusterService._shutdown_running_campaigns says, and why the service has no
shutdown-time job teardown: stopping a campaign is vast campaign stop, and exiting the
service is not. The distinction matters more than it looks, because a cooperative stop
persists a terminal outcome.json – and a campaign that has recorded an ending is one
no successor will ever pick up again.
And the successor picks them back up. At startup, before it answers anything, the
service re-launches every campaign on its results volume that recorded no ending
(campaign_resume). Discovery has that one source – the results root, minus every
campaign with a terminal _execution/outcome.json – because listing live Jobs would be
a second source for the same set, and two sources of one truth is one more way for them to
disagree. There is no resume mode anywhere in the launch path, the batch loop or the
controller; a resumed campaign is a re-launch under its own id, and three properties –
each equally true of a campaign starting now – are what make that safe:
the campaign’s records (
_execution/launch.yaml,campaign.db) are written into the campaign as it runs rather than at the end, so an unfinished campaign is already a directory holding what its earlier life produced;the batch runner plans against that root, adopting every job whose runs already have a verdict instead of running them a second time;
create_campaignis idempotent by name, so the campaign re-opens its row.
A search is picked up too. Nothing about its strategy is serialized; the strategy is
re-driven through the exact ask/tell sequence its own unit rows recorded, which
reproduces the original search for a strategy that is a function of its seed and its
evaluations – every strategy shipped here. That is why campaign.db is written through
at each batch boundary: those rows are the checkpoint. Two conditions are checked before the
campaign is re-launched rather than discovered halfway through its second half:
search.seed is set (an unseeded strategy re-seeds from entropy, so the replay would
rebuild a different search) and the strategy does not declare RESUMABLE = False.
Campaigns are deliberately left alone when nothing says what to run (no launch record,
or no frozen _config/), when the frozen config cannot be read unchanged (resuming would
mean migrating it mid-campaign, making the second half a different experiment from the
first), or when a search fails either condition above. Each says which it is in the service
log, keeps the crashed phase reconstruct_status_from_disk gives it, and its data
stays recoverable with vast campaign import.
A running postprocess is re-attached to separately (postprocess_reattach, called
from startup right after the resume). Postprocessing is a Job that outlives the service
process too, but only the waiting process writes the campaign’s postprocessing verdict –
so a restart mid-postprocess leaves a Job that converts every bag, finishes, and a campaign
still carrying the previous attempt’s message: fully derived data marked as none. The resume
above cannot cover it, because a postprocess retriggered on a finished campaign runs against
a terminal outcome.json, which owed_work excludes precisely so that a campaign that
recorded an ending is never restarted. What is owed here is a verdict, not work.
The live Jobs are found with one labelled listing (jobgroup=postprocessing) and the
label’s campaign resolved against the campaign directories on the results volume, then confirmed against the
campaign-level Job name – a discriminated Job is a search’s per-batch conversion and is
owed to its batch’s driver, not to any campaign record. A part of a split postprocess is
neither: the split recorded its parts and options beside them
(_execution/postprocess_parts/plan.json), and a live part makes the service start that
postprocess again with those options – it waits for the parts still running, runs the
rest, and completes the campaign. The waiter creates and replaces
nothing (the Job already mounts the scripts it was created with), publishes the live log
while it waits, and records nothing at all when the Job cannot be read: a campaign whose
conversion succeeded must never be marked failed because the API server was unreadable.
How free capacity is measured, and why it is not the whole cluster
Admission may only hand out capacity the scheduler can actually place against, so free
capacity is allocatable minus the requests of every pod already bound to a node, minus
a small headroom. Everything sharing the nodes counts: the CNI and ingress DaemonSets, the
robovast pod, the RoboVAST service, the build daemon, and the campaign pods themselves.
Counting at 100 % of allocatable instead admits one job more than the nodes can hold as
soon as anything else runs — the extra pod is created, the scheduler has nowhere to put
it, and it sits Unschedulable with Insufficient cpu. That is the failure a single
campaign rarely reaches and several concurrent ones reach reliably, which is what makes it
look like a concurrency bug rather than an arithmetic one.
It is measured every cycle, not snapshotted at setup. This replaced a fixed quota sized once when the cluster was set up, which had to be re-sized by hand after anything long-lived was added to or removed from the nodes, and was silently wrong until someone did. Nothing needs re-running now.
Headroom protects the shared tenants no campaign owns: ROBOVAST_NODE_HEADROOM_CPU
(default 1) and ROBOVAST_NODE_HEADROOM_MEMORY (default 2Gi), read from the
operator’s .env by vast cluster setup and vast service upgrade and written into
the service Deployment, like the build daemon’s settings above. Deliberately not a .vast
setting — a per-campaign override would let one campaign shrink the margin every other one
depends on. 0 is a value: on a single node whose control plane already sits in the
requests admission subtracts, the default would reserve the same core twice, and a pod the
node can place waits in the queue for good.
Capacity is counted per node, and each job is pinned to the node it was counted against. A pod runs on one machine, so a cluster-wide figure cannot see fragmentation: it says there is room while no single node has enough, and the job is created and then cannot be placed. It happens whenever the free cores are spread across nodes and no single node holds the 4.75 a pod needed.
The pin is the label robovast.io/node-id, whose value is the same hash a run records as
runs.node_label — so the selector that placed a run and the record of the machine it ran
on are provably the same node, and no hostname appears in any pod spec. setup and
vast service upgrade both apply it — the latter beside the RBAC it reconciles, so an
operator never has to know the label exists — and an unlabelled node still takes work:
it simply cannot be named, so its jobs are created without a selector and the scheduler
places them. A node that joined since therefore behaves exactly as it did before per-node
placement existed rather than refusing to run anything.
An unlabelled node is still counted as its own node. Accounting keys on the node’s identity, which is the label’s value where the label is there and is computed where it is not (it is an unsalted digest of the node name, so it can be recomputed rather than looked up). Only the pin depends on the label being present. This matters most exactly where unlabelled nodes are routine: when every one of them shared a single “no identity” key, a pool of them collapsed into one slot in the budget and all but one node’s free capacity was never offered to any campaign — silently, and the more nodes the worse.
A pinned pod whose node is momentarily full is contention, not a fault. It waits for that node — the fifteen-minute window, not the sixty-second one — because it is waiting for capacity that is coming back, and the alternative is destroying a run for being patient.
RoboVAST’s own infrastructure is not evenly spread. The service pod — which carries the workspaces and the campaigns — the registry and the index are pinned to one node wherever their volumes are node directories (Where this deployment’s own data lives), so that machine has materially less left for campaign work than an even split implies. Per-node budgets see this correctly, because they measure what is committed on each node rather than dividing a cluster total.
Reserve for the node itself, too. Kubernetes hands out allocatable, which is
capacity minus what the kubelet was told to hold back for the OS, the kubelet and
the container runtime. Distributions differ: where nothing is reserved, allocatable
equals capacity and a full cluster leaves the machine’s own processes competing with
pods for the last core — setup warns when it sees a node like that, because nothing
it can do reaches the setting that fixes it. On RKE2/k3s, reserve it in
/etc/rancher/rke2/config.yaml (k3s: /etc/rancher/k3s/config.yaml) on each node:
kubelet-arg:
- "system-reserved=cpu=1,memory=2Gi"
- "kube-reserved=cpu=1,memory=2Gi"
Applying it restarts the kubelet, which restarts every pod on the node: do it in a
maintenance window, not while campaigns are running. The new allocatable is picked up
by the next measurement with nothing to re-run (kubectl get node <name> -o
jsonpath='{.status.allocatable.cpu}' shows it took effect). This is independent of the
subtraction above and complements it: the kubelet reservation protects the machine, the
subtraction protects the schedule.
Per-node container sizing
A campaign asks for it, per campaign, with execution.sizing: calibrated (see
declaring the sizing mode). Before it places work on a node, one calibration probe runs there;
what it measured becomes that node’s figures, and every run of that campaign on that node is
sized from them. A campaign that says nothing keeps fixed and its declared sizing.
Sizing and execution.kubernetes.jobs.node answer different questions and compose.
sizing: calibrated makes each node’s allocation right for that node; node holds the
machine fixed across all of a campaign’s cells, so no comparison between cells is also a
comparison between machines (Naming a node a campaign may be held to). A pinned, calibrated campaign
probes its one node once and sizes every run from that.
The cluster’s part is the bootstrap: what a container asks for before anything has been measured for it, which is what the probe itself runs at.
Set in your .env, like the git token and the ntfy topic — setup carries it into the
Deployment, so it is a property of the cluster rather than of any campaign:
ROBOVAST_BOOTSTRAP_CPU={"sut": 5, "simulation": 3, "scenario": 2}
ROBOVAST_BOOTSTRAP_MEMORY={"simulation": "4Gi"}
A role left out keeps its default rather than disappearing, so raising one cannot silently drop another. With neither set the defaults below apply, and the service log says which of the two it is using.
A campaign may override it per container, by declaring resources on that container:
under execution.sizing: calibrated those figures are what the probe and every
not-yet-measured node run at, and the ceiling a measured figure may not exceed. The
deployment default still applies to every container that declares none, and per field — a
.vast stating only cpu keeps this memory. See
the sizing mode.
How the measurement is turned into an allocation is settable the same way, in a
calibration block on the container, defaulted per role from one more .env entry:
ROBOVAST_CALIBRATION={"sut": {"size_on": 100, "limit": "request"},
"simulation": {"headroom": {"cpu": 1.25}}}
The same block a container may carry, keyed by role — so an option added to it later is settable here with no new variable.
Warning
The three roles sum to a pod that has to fit your smallest node. The probe is pinned to the node it measures, so a sum larger than that node’s allocatable (less the cluster headroom) leaves it unmeasurable – refused at launch, naming the figure and the node.
Do not set a figure below what the container actually wants. The bootstrap is also
what the probe runs at, so a container capped under its own demand throttles against that
cap; the probe is then refused as having measured its ceiling rather than the demand, no
node is calibrated, and every run of the campaign stays on the bootstrap — the outcome
calibration exists to avoid, reached by tightening the one figure that must not be tight.
The refusal says so per node in the campaign log, naming the container and its throttle
ratio. Size these from a finished campaign’s resource_usage peaks, with margin.
Per role, because the three want very different amounts and CPU and memory rank them differently: the system under test wants cores and little memory, the simulator is the opposite — it compiles a world once and then sustains very little CPU. One ranking for both would starve whichever resource the other role dominates. A container whose name is not a known role takes a small default; it is one probe away from a measured figure, while a generous default for every unnamed container is what makes a probe unplaceable.
It is per container, so a three-container pod reserves the sum until its node reports — which makes the bootstrap the floor on what calibration costs, and a reason not to set it generously on a small or busy cluster.
Why per node at all: the same trial costs about 1.6x more CPU on the slowest machine of a mixed cluster than on the fastest, and wall time does not show it — a realtime-paced simulator holds one simulated second per wall second, so every machine finishes at roughly the same time and the difference lands entirely in CPU consumed. One declared number is therefore wrong on every node but the one it was measured on.
It is a validity matter as much as a throughput one. At a uniform 3.0 cores for the system under test, one node was quota-bound in 100 % of its runs at 2.5 and below while three others were never quota-bound at any allocation down to 2.0. Equal cores are not equal compute, so an equal declaration produces unequal conditions — the thing a uniform number was meant to prevent.
How the figure is found:
One probe per node, and it is never a campaign run. It writes to
_calibration/<node-id>/— a reserved directory nothing walks looking for runs — so it cannot enter the results in the first place. Every container of the probe writes there: the scenario’s results, the job artifacts, and what a sidecar writes per run (a simulator’s recording and pose record), each of which is named by its own variable and every one of which the probe points at its own directory. A campaign of 50 runs still delivers 50.A probe is listed, marked, and counted apart. It holds real capacity on a real node, so it carries the campaign’s labels and appears in the job listing — as
kind: calibration, named for the node it measures, and outside every figure inJobCounts, which a reader takes as facts about runs. It reports its own status like any other job, because a probe that failed is one worth looking at, but it cannot be stopped individually: there is no run to record as killed. In the web UI’s campaign view it is the row with thecalibrationchip beside its status.A node with an outstanding probe takes no campaign work. Otherwise the runs placed while it is measuring are the odd ones out on a node whose later runs are calibrated, reintroducing the inconsistency the probe exists to remove.
How a measurement becomes an allocation depends on the container’s role, and what decides that is whether anything would report that squeezing it cost something. The system under test is read at its maximum, as request and limit, so it never throttles: a run clipped mid-plan fails in a way that looks like the stack’s fault rather than the allocation’s. The simulator is read at the 95th percentile and keeps its ceiling — its peak-to-mean ratio is about 18, so reserving the maximum would cost more than not calibrating, and the realtime factor reports if the squeeze cost anything. The scenario runner is read the same way, but nothing grades how well it ran, so its ceiling is what must not be tight; on a probe its own tick rate fills that gap (see below).
Every one of those is a default, and a container may state its own under the sizing mode. Memory is different in kind and takes one rule for every role: the measured maximum, with headroom. Exceeding a CPU reservation slows a container; exceeding a memory one kills it, so no role is sized on a figure most of its samples sat below.
Memory is measured wherever the node can report it. Both cgroup layouts are read into the same columns, so a mixed cluster stays comparable. Where a node reports neither — an older runtime, a kernel without the counter — the container keeps its declared or bootstrap memory rather than being sized from nothing, and the same absence disables the OOM check for that node rather than passing it.
The scenario runner reports on itself, on probes only. A probe runs with
--tick-log, sotick_timing.csvrecords how closely the behaviour tree held its configured period. A probe whose scenario could not keep up measured a starved container, and is refused rather than believed. Campaign runs do not carry the flag: it is per-tick instrumentation on the trial’s hot path, and only the probe’s file is ever read.A calibrated figure never exceeds what the ``.vast`` declared. Calibration sizes a node’s jobs down to what they need; it does not raise a ceiling the author set.
Frozen once set, and dropped when the campaign ends. Continuing to adapt would mean run 5 and run 40 on the same node ran in different environments. The figures are deliberately not reused by the next campaign — they were measured under this one’s contention, for this one’s containers.
Pilots calibrate nothing. With no more jobs than the cluster has nodes, no node runs a second one, so the probe would cost as much as the work it was meant to improve. The mechanism is skipped and the campaign behaves as it did before any of this existed.
Whether it is worth turning on is a question about your cluster, not about RoboVAST.
The gain is the spread between your fastest and slowest node; on a homogeneous cluster there
is nothing to recover, and on an unlike one it is bounded by how much of a pod is the system
under test, whose ceiling calibration does not lower. A matched pair of campaigns — one with
execution.sizing: calibrated, one with fixed, everything else equal — measures it
directly, and run_health says whether the tighter ceilings cost the stack anything –
provided the campaigns declared a check under results_processing.health_checks. Nothing
runs undeclared, so an empty run_health for the pair means not graded, never not
degraded; that is the same rule as everywhere else in the table. Both campaigns’ grades sit
in the one index, so compare them in one query – naming both ids in the query call’s
campaigns argument – rather than expecting a per-campaign database. A query that names
only one campaign sees only that one, whatever its WHERE clause says.
Check the realtime factor when you do. clock_map_sim_span_s / clock_map_wall_span_s
is what says whether a tighter allocation cost the simulator its pacing, and a campaign whose
factor drops is no longer comparable with one sized any other way — worse results rather than
merely slower ones.
A peak measured on an idle probe is an unvalidated basis for a hard limit on a loaded machine. That is why this is switchable, and why the probe is one run rather than a guarantee. A workload with heavier planning spikes than the one a cluster was measured on has not been tested against its own calibration.
The evidence behind the design, kept here because it is what rules out the cheaper
alternatives. Two campaigns of one configuration times twenty runs, so the machine was the only
variable; forty trials, all passed. Per-container CPU comes from resource_usage, summed per
tick before averaging (a row is one process name, not a container), then divided by the run’s
realtime factor – cpu_percent is per wall second, so a node that meets fewer step
deadlines otherwise reads as cheaper than it is:
Measured on one four-node cluster, the same pod cost 1.6x more CPU per simulated second on
the slowest node than on the fastest — and the ordering did not follow clock speed. Your own
figures come from get_campaign_summary; the point is that the spread exists and is not
guessable from the hardware.
The ranking tracks microarchitecture rather than clock: the two Skylake-derived parts sit together at ~2.2 despite a 1.5x clock difference, Zen 3 at 1.71, Raptor Lake at 1.37.
A cached per-node factor is refuted, and that is why this is measured per campaign. Every
design that stores a number and reuses it – a scalar per node, a robovast.io/cpu-factor
label written at setup, a reference campaign run once – assumes a figure that transfers between
campaigns. Measured against two unlike campaigns on the same cluster, it does not. Container
rankings invert between nodes: one machine was the cheapest for the system under test and
among the dearest for the simulator, while another was the reverse, so no single per-node scalar
can be both greater and less than one at once – the shape is wrong, not the calibration. Even
per (node, container) it does not transfer: between two campaigns one node’s simulation cost
moved +42 % while another’s moved +1 %, flipping their order.
Why a hard limit is sized on the peak and not on p95. Sizing a limit at p95 x 1.25 looks
safe and is not: a container clipped at its limit does not lose the clipped work, it queues it,
so it stays pegged working the backlog off and the next spike arrives into a full budget. A
configuration whose static clip rate was 0.5 % produced 44 % saturation and lost 22 % of
the runs. That is why the system under test takes the peak as request and limit, while
everything else splits the two.
What remains true, and is why a campaign chooses. A peak measured on an idle probe is
still an unvalidated basis for a hard limit on a loaded machine; a workload with heavier
planning spikes than the cluster was measured on has not been tested against its own
calibration. execution.sizing: fixed — the default — honours the declared sizing exactly,
which is what a campaign wants when the allocation is itself the variable under study.
A probe that measured its own ceiling is refused, and the node stays on its current sizing rather than being sized from a limit: throttled past what its own statistic can absorb, or OOM-killed at all — a memory ceiling that binds kills rather than slows, so one is enough. Both counters come from the same file the sizing is read from.
A probe that loses a workload container stops the campaign, naming that container. Those
containers are native sidecars, so one that dies is restarted rather than ending the job.
The instance that comes back starts nothing — a workload brought back mid-trial would run a
fresh world under a stack still on the old one, so the sidecar entrypoint refuses a second
start in the same pod — and the probe would go on sampling a stack with a hole in it, hold
its node while it did, and what was read out of it at the end would measure the gap rather
than a trial, surfacing as whichever statistic that fragment fails with the container’s own
error nowhere in it. A probe runs one of the campaign’s own configurations, so this is not a
flaky trial to re-sample: every run would meet the same fault. The campaign therefore ends
on the crash itself, and what the container printed before it died is captured in
_execution/container_failures.json, beside the probe’s own output under
_calibration/<node-id>/.
How much throttling a container may survive depends on which statistic its figure comes from, because clipping removes the top of the distribution. A container sized on its peak is spoiled by the first clipped tick, so it keeps a strict allowance covering bring-up only. One sized on its sustained figure — a percentile that already discards a tail — is unaffected while the clipped ticks stay inside that tail, and is judged against exactly what the percentile throws away. A single strict allowance for both refuses probes whose sustained figure is perfectly good and leaves those nodes unmeasured, which costs more than the distortion it was guarding against.
Where calibration does not apply, the bootstrap stands and is checked. A campaign with no more jobs than the cluster has nodes, or a cluster that can grow, never probes. Those runs use the bootstrap, and one that is OOM-killed or throttled hard against it stops the campaign with an error naming the container. Unlike a declared or measured allocation — where such a run is recorded and kept — nobody chose the bootstrap for that workload, so a run that dies against it reports that the default does not fit rather than anything about the stack.
Note
Not applied on a cluster that can grow. There a job that fits no current node is created unpinned and the scheduler places it — possibly on a node that is already calibrated, at the declared size, which is the mixed sizing this exists to prevent and is invisible after the fact. Declared sizing everywhere is the honest behaviour there.
The same caveat applies on any cluster whose nodes are replaced while campaigns run
(managed node pools autoscaling, auto-upgrading or reclaiming spot capacity): a node that
joined since the last setup carries no robovast.io/node-id, so it cannot be probed
or pinned and its jobs run at the declared sizing beside calibrated ones. See
What does not hold on managed Kubernetes.
Which image bytes a run uses
Before the first batch creates a Job, every image ref the campaign’s pods will run is
resolved against the registry to the digest it names right now (one HEAD per distinct
ref, cached for the campaign), and each container carries an explicit
imagePullPolicy: IfNotPresent for a digest ref, Always for anything still a
tag.
Both halves matter, and the second is the one that bites. Kubernetes defaults the policy
to IfNotPresent — except for a :latest tag, where it silently becomes
Always. A campaign image is a floating tag in the ordinary case, so every container of
every scenario pod re-contacted the registry on every start for an image the node already
had. A batch of 35 pods is then ~140 registry round trips in one instant, against a
kubelet whose image-pull limiter is five per second (registryPullQPS, burst ten): the
pods past the burst come back ErrImagePull: pull QPS exceeded, and that is arithmetic
rather than bad luck. Pinning removes the round trips; writing the policy out removes the
dependence on how the tag happens to read.
It is also what makes a campaign one experiment: a floating tag can be re-pushed between
batch 1 and batch 50, and a system under test that changes underneath a sweep invalidates
the comparison the sweep exists to make. _execution/execution.yaml then records what
ran rather than what was asked for.
The pin covers the pods a batch creates. A composition’s auxiliary pod is created
before there is a batch to pin, so its image is the ref the spec names – resolved if it
is a family: member – and it carries the policy that ref implies, Always for a
tag. It therefore runs the bytes the tag names when the composition asks for them, which
is what a campaign’s own pods pin moments later; a tag re-pushed inside that window is the
one case where the container that described the world and the ones that run it differ.
Pin ROBOVAST_PROJECT_TAG for a campaign that must not depend on that.
Resolution is fail-soft. An unreachable registry, a ref this deployment holds no
credential for, or a registry that omits the digest header leaves the ref exactly as it
was — which is what would have run anyway — and it keeps Always, correct for a name
that may move. A campaign never fails to start because this could not be applied; the log
says which refs stayed unpinned.
Waiting, blocked, and merely busy
Three things a job that has not started can be doing, and the run loop treats them differently:
waitingPlanned, but not created yet: the cluster has no room for it. Normal, and unbounded — see above.
- busy
Created, but waiting its turn — for a node or for a pull. Either the resource exists on a node and something else is holding it, usually another campaign’s run; or the image exists and the credential works, and the pull is rate-limited behind the other pulls a whole batch of jobs asked for at the same instant (the kubelet’s own
registryPullQPSlimiter, or the registry’s account limit). Both start by themselves, so the batch waits 15 minutes (CONTENDED_GRACE_SECONDS) before giving up, and repeats the reason in the log each minute meanwhile. Failing either on the short timer threw campaigns away for the very conditions that resolve themselves.blockedIt will not start on its own: an image reference that names nothing, missing pull credentials, or a reservation no node can satisfy — one larger than the biggest machine, or a GPU that no node advertises because its device plugin is down. These look the same in ten minutes as in one, so the batch fails after 60 seconds (
BLOCKED_GRACE_SECONDS) with Kubernetes’ own message.
The job listing (vast cluster monitor, the web UI, list_campaign_jobs)
reports all three, and a busy job appears there as pending carrying Kubernetes’
message as its detail. pending is the literal truth about such a job — its pod exists
and has not started — and the reason it has not started is worth reading without being
worth acting on.
Reporting the two alike, as blocked, would put a red row and a Blocked: count in
front of anyone running two campaigns at once, for a job that starts by itself as soon as a
neighbour finishes; blocked is defined just above as the state that will not clear, so
the count that exists to say “someone must do something” would be saying it about nothing.
It is the same mistake the waiting phase exists to avoid, and it has the same cost: a
reader who learns to ignore the alarming word stops reading it when it is real. The run loop
separates them only by how long to wait — blocked fails the batch in a minute, busy gets
fifteen — and the reason is repeated in its log each minute meanwhile.
A busy job is therefore invisible in the monitor’s aggregate counts, which have no
per-job detail: it is one of Pending. The run loop’s log is where a CLI reader learns
which pending job is waiting on what.
The split is made per job, and on each side it is an allowlist of what is known to clear by itself — so a reason nobody anticipated is blocked, and fails fast, rather than sitting for the long grace on a guess.
Scheduling: the scheduler’s message plus what the nodes advertise. Insufficient
<resource> counts as busy only when the pod’s own requests fit inside some node’s
allocatable; any other stated cause — an untolerated taint, an unmatched node
selector — is blocked, as is every cause when the node list cannot be read.
Pulling: the kubelet’s message alone, since no node fact can make a throttled pull
permanent. Only a pull-attempt reason (ErrImagePull, ImagePullBackOff) whose
message names a rate limit counts as busy; a manifest that is not there, a registry that
refuses the credential, a host that does not resolve, and every non-pull reason are
blocked.
If a batch reaches the 15-minute limit, the cluster is oversubscribed rather than busy: check what else is running and whether the queue admits more than the nodes can hold — or, for a pull reason, run fewer jobs at once than the registry will serve.
Selecting a Cluster Context
RoboVAST uses kubeconfig contexts to address different clusters. Pass
the --context flag to any cluster sub-command to select a specific context
(as listed by kubectl config get-contexts):
# Use the currently active context (default)
vast workspace run my-experiment
# Explicitly target a context
vast workspace run my-experiment --context gcp-c4
The --context flag is available on workspace run, cluster setup,
cluster monitor, cluster jobs-cleanup, and cluster cleanup.
Contexts can be renamed to shorter, human-friendly identifiers:
kubectl config rename-context <old-name> <new-name>
Per-Cluster Resource Limits
When the same .vast file is used on multiple clusters that have
different hardware, resource fields (cpu, memory, gpu) can be expressed as
a list of {context-name: value} mappings instead of a plain scalar.
execution:
resources:
cpu:
- gcp-c4: 4
- local: 8
memory:
- gcp-c4: 10Gi
- local: 20Gi
secondary_containers:
- nav:
resources:
cpu:
- gcp-c4: 2
- local: 4
- simulation:
resources:
cpu:
- gcp-c4: 2
- local: 4
memory:
- gcp-c4: 8Gi
- local: 16Gi
Rules:
Scalars take precedence — a plain integer/string is used unchanged on every cluster.
For per-cluster lists the entry whose key matches the active context is used. If no entry matches, RoboVAST raises a
ValueError.Fields can be mixed:
cpuas a scalar andmemoryas a per-cluster list is valid.If a per-cluster list is present and no
--contextis supplied, RoboVAST will ask you to provide one.
Running the same config on two clusters:
vast workspace run my-experiment --context gcp-c4
vast workspace run my-experiment --context local
Cloud Provider Configurations
Four cluster configurations are shipped out of the box (vast cluster setup --list).
Select the one matching your environment. Read What does not hold on managed Kubernetes first: several parts of the
scheduler assume a static, hand-managed cluster, and on a managed one they degrade quietly
rather than loudly.
What does not hold on managed Kubernetes
The admission arithmetic itself is provider-agnostic: allocatable minus bound requests minus headroom, per node, measured every cycle. What is built around it was designed against a static bare-metal cluster, and these follow from that. None of them is a crash; each is a silent degradation, which is why they are written down.
A cluster whose growth ceiling nothing states is held at its current size. Admission never creates a job that no current node can hold, which is correct on a static cluster and self-defeating on an elastic one — a pod the scheduler cannot place is exactly what makes an autoscaler add a node. Given a ceiling, admission creates such work unpinned and lets the autoscaler respond, and a pool scaled to zero is a batch that waits rather than one that is refused. A cluster that has nodes is still sized against them: an autoscaler adds machines of its pool’s shape, so “no node is that large” stays a permanent refusal however many arrive.
The ceiling is recorded, not queried: setup and upgrade ask the provider — on
GKE that is gcloud, summing each node pool’s autoscaler maximum — and write the answer
into the service’s environment as ROBOVAST_CLUSTER_MAX_CPU / ROBOVAST_CLUSTER_MAX_MEMORY.
That is where admission reads it, because admission runs in the service pod, whose image
ships no cloud CLI. Setting those two variables before setup states the ceiling directly,
which is what a provider with no query of its own — an unlisted one, or one whose CLI is not
installed here — needs.
Recorded rather than live means the figure ages: resizing a node pool does not reach a
running deployment. Re-run vast service upgrade after such a change — the same lifecycle
the node identity labels below already have. With neither a provider answer nor the two
variables, the cluster is treated as static, exactly as before.
Node identity labels are applied at ``setup`` and ``upgrade``, not continuously.
robovast.io/node-id is what pins a job to the node its capacity was reserved on, and what
a calibration probe measures against. A managed node pool replaces nodes constantly —
autoscaling, auto-upgrade, auto-repair, spot reclaim — and every replacement arrives
unlabelled. Such a node still takes work (refusing it would turn adding capacity into an
outage) and still counts as its own node for capacity, but it cannot be pinned to and cannot
be probed, so its jobs run at the declared sizing beside calibrated ones. Re-run
vast cluster setup after the pool changes to bring new nodes back under
Per-node container sizing.
The CPU governor cannot be set on a cloud VM, so the cloud providers do not attempt one and say so instead. The comparability it buys is then the node pool’s to provide — machine type, no shared-core or preemptible instances, upgrades held for a campaign — rather than a DaemonSet’s. See Pinning the clock.
GCP (Google Kubernetes Engine)
Config name: gcp
The same deployment as everywhere — the robovast pod for the registry and the campaign
index, campaigns on the service’s results volume. What is provider-specific here is how
the cluster answers for itself (which GKE cluster a context names, how far its node pools may autoscale, how a
node reports its machine type) and how the volumes are backed.
Prerequisites:
Install and authenticate the
gcloudCLI.Install the GKE auth plugin required by
kubectlto authenticate against GKE clusters:sudo apt-get install google-cloud-cli-gke-gcloud-auth-plugin
Fetch the cluster credentials into your kubeconfig:
gcloud container clusters get-credentials <cluster-name> --region <region>
Optionally rename the context for brevity:
kubectl config rename-context \ gke_<project>_<region>_<cluster-name> gcp-c4
Pass
--ingress-class gcewhen publishing with--ingress-hoston GKE’s built-in controller. Unlike ingress-nginx it cannot route to a plain ClusterIP, so both Services the Ingress fronts — the UI on/and the registry on/v2— are annotated for container-native load balancing only when the class is named. A backend without it never becomes healthy, and the reason is visible in the load balancer rather than in anything RoboVAST prints.
Setup:
A GKE node pool replaces machines constantly — autoscaling, auto-upgrade, auto-repair,
spot reclaim — and a hostPath goes with the machine. So back every volume with a
StorageClass rather than the node’s disk:
vast cluster setup gcp \
--workspaces-class standard-rwo --index-class standard-rwo \
--registry-class standard-rwo --buildkit-class premium-rwo --buildkit-size 200Gi
--workspaces-class backs the campaign results too, which is where the campaigns are,
and --results-size says how large that claim is (Where this deployment’s own data lives).
Left on hostPaths, a replaced node takes the
campaigns, the index, the built images and the workspaces with it — and setup reports
success on the empty replacement. These are zonal disks, so the pods that mount them are
bound to one zone; that is the cost, and it is the intended one. Keeping the disks
themselves is the operator’s snapshot schedule, which RoboVAST does not manage.
Azure (AKS)
Config name: azure
The same deployment again; what is provider-specific is how a node reports its VM size.
An AKS node pool replaces machines rather than repairing them, so back every volume with a
StorageClass — managed-csi is the stock one:
vast cluster setup azure \
--workspaces-class managed-csi --index-class managed-csi \
--registry-class managed-csi --buildkit-class managed-csi
--workspaces-class backs the campaign results with it, which is where the campaigns
are. Keeping those disks is the operator’s snapshot schedule, which RoboVAST does not
manage. Like gcp, this provider does not attempt the CPU governor
(Pinning the clock).
Reaching it over a tailnet instead of publishing it
An Ingress asks for a public DNS record, a certificate for it, and an address that survives the cluster being rebuilt. A deployment that has none of those, or wants nothing listening on the internet at all, can be reached over a WireGuard tailnet instead: a Tailscale node runs beside the service, dials out to a coordination server, and answers on the tailnet under a stable name. No firewall rule is opened and no address changes when the cluster does.
Coordination is whatever you point it at — Tailscale’s own service, or a self-hosted
Headscale so that no third party is trusted with the tailnet.
RoboVAST passes --login-server through and lets the node register.
Off by default, and asked for per cluster. The credential lives in the environment, so
a pre-auth key does not land in shell history; which cluster is on a tailnet is decided on
the setup command, because one .env and two contexts would otherwise publish whichever
happened to be current — a deployment nobody meant to expose.
# .env — the credential
ROBOVAST_TAILNET_LOGIN_SERVER=https://headscale.example.org
ROBOVAST_TAILNET_AUTHKEY=<a pre-auth key from that server>
ROBOVAST_TAILNET_HOSTNAME=robovast # optional; the name users type
# the decision — this cluster, and no other
vast cluster setup <flavor> --tailnet
Users then reach the web UI at http://robovast on the tailnet with the access token from
vast service token. Nothing but a Tailscale client is needed — no kubeconfig, no
kubectl, no port-forward.
Written on every setup: omitting --tailnet removes a node a previous setup deployed,
rather than leaving one nobody remembers configuring still answering. --tailnet with no
credential in the environment is an argument error, since it would deploy a node that can
never register.
vast service upgrade reconciles a node that already exists, so a rotated key reaches
a running deployment without a re-setup — and creates none, so upgrading two clusters from
one shell cannot publish the second by accident. It sits beside the RBAC and the registry
route, so --no-restart picks it up without rolling the service pod.
Note
Plain HTTP on the tailnet, deliberately. The transport is already WireGuard, so a
certificate would encrypt what is encrypted. It works because the session cookie’s
Secure flag follows the request scheme — the reason an Ingress refuses plain HTTP is
that the token would cross an untrusted network, which is exactly what a tailnet is not.
Warning
This does not make in-cluster builds possible. The prefix campaigns push to has to be
pullable by the kubelet, and nodes are not on the tailnet — they resolve no tailnet name
and hold no key. So this publishes the service to people, never to the cluster’s own
container runtime, and can_build_images stays false. An Ingress, or an external
registry, is what changes that.
The node itself is unprivileged (userspace networking), so it deploys on a managed
cluster where the alternative — NET_ADMIN plus /dev/net/tun — is refused.
RKE2
Config name: rke2
Targets on-premise clusters managed by
Rancher RKE2. Stock RKE2 provisions no volumes, so every
directory this deployment keeps is a hostPath on the data node unless a class is
passed — including the results volume, which is where finished campaigns live.
Prerequisites:
Ensure the kubeconfig for the RKE2 cluster is available (typically provided by the cluster administrator as
/etc/rancher/rke2/rke2.yaml).
Setup:
vast cluster setup rke2
Notes:
The results survive the service pod being restarted or upgraded and a
vast cluster cleanup, but they are one directory on one node and no more: archive anything that must outlive the machine withvast share, or launch with Upload to share when done.vast cluster cleanup --delete-datais what empties this deployment’s directories, and nothing else does.The directories draw from the node filesystem and declare no bound, so watch the web UI’s Disk meter: a
hostPathcarries no per-volume stats of its own, and the disk it shares is the one that fills.
Minikube
Config name: minikube
Targets a local minikube cluster. The same deployment
as RKE2 — node directories for everything, the robovast pod for the registry and the
index, campaigns on the service’s results volume — on a machine that is usually the
developer’s own. Intended for development and local integration tests.
Prerequisites:
Start a minikube cluster:
minikube start
Setup:
vast cluster setup minikube
Notes:
Suitable for development and short-lived runs: archive anything that must outlive the machine with
vast share, and remove a campaign you are done with usingvast campaign delete.The results outlive the service pod;
vast cluster cleanup --delete-dataempties this deployment’s directories.