Deployment Modes and Access

The robovast-service (see Client–Server Architecture) is one FastAPI app, deployed into a Kubernetes cluster: a one-node cluster on your own machine, or one a team shares. This page covers the deployment, how a client reaches it, and the v1 security boundary.

One port owns reachability: a service that answers on the conventional port ``127.0.0.1:8800`` is found by every client — the web UI, the CLI, and the MCP server all auto-detect it there, so none of them needs configuring.** vast ui is just a shortcut that opens a browser at that port; it starts nothing.

vast serve also exposes its MCP tools at /mcp on that same port by default, so a single tunnel to 8800 reaches the web UI, the REST API, and MCP together — pass --no-mcp to serve the API without them (see MCP Server).

  • A one-node service — vast cluster setup minikube deploys it into a minikube or kind cluster on your own machine, and kubectl port-forward svc/robovast-service 8800:8800 puts it on the conventional port, where every client finds it (On this machine). vast ui opens it, and vast workspace run launches through it exactly as it would through a published one.

  • A published service — vast cluster setup --ingress-host … deploys and publishes it; users then reach it in a browser, or with vast login <url> for the CLI and MCP. No kubeconfig, no kubectl, nothing to hold open. In-pod, the Deployment runs vast serve. The service runs only inside the cluster — a campaign’s pods deliver their outputs to it over the cluster network, which cannot reach a process on your host — so debugging the driver against a real cluster means running it in that network with the Service’s traffic steered to it: mirrord exec --target deployment/robovast-service --steal -- vast serve.

  • Your own tunnel to either — an ssh -N -L 8800:127.0.0.1:8800 <host> or the kubectl port-forward above puts a service on the conventional port, and every client finds it there. That is the break-glass route when the Ingress itself is broken; it is kubectl’s feature, not a mode RoboVAST wraps.

Note

Every request needs the shared token. There is no unauthenticated mode: vast cluster setup generates one and preserves it across re-runs (--rotate-token issues a new one, logging everyone out), and vast service token prints it.

Browsers authenticate with a cookie obtained at /login; the CLI and MCP send Authorization: Bearer. The cookie is not a preference — EventSource cannot set headers, so it is what keeps the live streams in the web UI working.

Publishing the service insists on TLS, and on a token being configured. Both refusals are deliberate: a campaign names its own container image, so an open Ingress lets anyone who finds the URL run containers in the cluster.

The deployment

vast cluster setup deploys robovast-service as a Deployment + ClusterIP Service. It drives each campaign in-process (one worker thread per campaign) over the Kubernetes backend, creating the scenario Jobs itself, and keeps each campaign as a directory on the service’s results volume. There is no per-campaign controller pod. The CLI, the MCP server and the web UI share one workspace/campaign state, and campaigns survive client exit.

The one-node and the published deployment differ in the cluster configuration and in whether an Ingress is created, not in what the service does: minikube is a hostPath configuration for a single node, the others (rke2, gcp, …) the shapes teams deploy on (Cluster Execution).

There is no serviceless mode — no in-process one where the CLI calls the interface directly, with no service, no workspace and no CampaignOrigin. A campaign runs a workspace’s project through a service, and which service answers is the only thing that differs between a one-node deployment and a published one.

Access matrix

Deployment

Transport

How the client reaches it

One node, on this machine

HTTPTransport

kubectl port-forward svc/robovast-service 8800:8800, then http://127.0.0.1:8800 — the conventional port, which every client probes.

Published

HTTPTransport

https://robovast.<domain> directly, over the Ingress that vast cluster setup --ingress-host created. A browser logs in at /login; the CLI and MCP use vast login <url>. Nothing is held open and no kubeconfig is involved.

Either, through a tunnel

HTTPTransport

SSH tunnel — ssh -L 8800:127.0.0.1:8800 <host> — then talk to 127.0.0.1:8800. SSH provides authentication and encryption.

The published row is hardened once for the whole surface: one shared secret, presented as a cookie by browsers and a bearer header by everything else, in front of an Ingress that refuses to exist without TLS.

Walkthrough — the in-cluster service

The operator, once, with a kubeconfig:

python tools/setup_ingress_tls.py                # hostname + certificate
vast cluster setup rke2 \
    --ingress-host robovast.example.org \
    --ingress-class nginx --issuer robovast-ca   # deploys and publishes it

Setup prints the access token once, and waits for the pod to be Ready before reporting success — so an image that cannot be pulled is reported here, with the pod’s own reason, rather than as a connection failure from the next command.

To read it back later, vast service token prints the URL, the token and the three ways to connect (-q for the token alone). Each cluster mints its own token, so the URL it names is the only one that token opens.

Setup also stamps the timezone of the host it ran on into the service pod’s TZ, read from /etc/localtime’s symlink target (or /etc/timezone) and validated against that host’s tz database first. Campaign ids are minted from wall-clock time in the process that names the campaign — the service pod, for a cluster campaign — so without this every campaign directory is named in UTC while the people reading those names are not. An upgrade re-stamps it from the machine running the upgrade; a host whose zone cannot be determined logs a warning and leaves the pod in UTC, which is the pre-existing behaviour. Only this pod is affected: campaign Jobs get an env list built explicitly for them and inherit nothing from it, so their logs stay UTC, as do recorded timestamps everywhere (store keeps epoch seconds). A vast serve started by hand already runs in its host’s zone and needs none of this.

Everybody else, from a machine with no kubeconfig:

# a browser: open https://robovast.example.org and log in — nothing to install

pip install robovast-client                      # the CLI, 13 packages, ~30 MB
vast login https://robovast.example.org

claude mcp add --transport http robovast \
    https://robovast.example.org/mcp \
    --header "Authorization: Bearer <token>" \
    --header "X-Robovast-User: <name>"           # an agent: MCP over HTTP

vast login prints that last command filled in with the token and the name it just stored, so nobody has to assemble it by hand — and campaigns an agent starts carry the same name as the ones its owner starts.

robovast-client, not robovast: a user of a deployed service drives it over HTTP and needs no simulator, no Docker and no Kubernetes client. The full distribution is 88 packages against the client’s 13. See Using the client.

vast ui opens whichever service this machine talks to. When the Ingress itself is broken, kubectl port-forward svc/robovast-service 8800:8800 puts one back on the conventional port and every client finds it there.

What runs in the service pod

Three containers, one port. robovast-service is the control plane – the API, the web UI, the campaign driver – and robovast-data the data plane, the tar streams pods exchange with the service (HTTP API, “Addressing files”). Both are the same image, both listen on Unix sockets in a shared in-memory volume, and robovast-front, an nginx, owns port 8800 and routes /data/ to the one and everything else to the other. The Service and the Ingress see one port, exactly as before.

The split exists so that bulk bytes never share a process with the control plane: a dozen pods delivering gigabytes of run output at once slow each other down and nothing else, because the data container has its own CPU and memory limits and the front streams through it with request and response buffering off. Proxying through the control plane instead would copy every uploaded byte through the event loop the split is meant to protect; a second published port would be a second thing to reach, publish and secure. The data container is given the results volume and the access-token Secret and nothing else – no workspaces, no cluster configuration, no other credential – and verifies the scoped tokens the control plane mints for pods with that shared secret alone, so the two processes share nothing but the disk and a restart of either changes nothing for the other.

A vast serve started by hand has no front: it mounts the same data routes into its one process on its one port. There the isolation buys nothing and a second process would cost every client a port. The front’s configuration is rendered by setup and upgrade into a ConfigMap and never edited on the cluster.

Keeping the service up to date

Controllers are launched per campaign, so execution always tracks the configured controller image. The persistent service Deployment does not, so it has to be updated deliberately.

vast service upgrade

That rolls the Deployment onto the resolved image, reconciles RBAC, and waits for the pod to be Ready before saying anything. The access token is preserved, so nobody is logged out by a version bump.

It reads the environment like every vast command — ./.env from the current directory, then ~/.config/robovast/env — and that is also how the image is chosen: ROBOVAST_PROJECT and ROBOVAST_PROJECT_TAG (Container images). There is no --image flag, because a one-shot override is just ROBOVAST_PROJECT=ghcr.io/cps-test-lab vast service upgrade (a real environment variable beats both files), and a setting that should last belongs in a file.

This is the command that moves a cluster’s images, not setup --force. Both values are baked into the service pod as the site default; a single campaign can still override them per launch with vast workspace run --image-project, which needs no upgrade.

It always restarts the pod, even when nothing looks different, and that is deliberate. An image ref that is a floating tag, or a change confined to the Secrets, leaves the Deployment spec byte-identical; Kubernetes then creates no new ReplicaSet, the readiness wait passes immediately against the old pod, and the command reports success while nothing rolled. imagePullPolicy: Always does not rescue that — it governs a container that is starting, and none was. The restart is also the only thing that makes a changed Secret take effect: the pod reads them through envFrom once, at container start. The cost is a few seconds of API downtime, during which open MCP connections and log streams are dropped; campaigns run as their own Jobs and keep going.

The old pod goes before the new one starts. The Deployment’s strategy is Recreate, not the default rolling overlap, because the campaigns are on a ReadWriteOnce volume that one node may mount at a time: a replacement scheduled on another node while the old pod still held it would never start, and the upgrade would wait out its timeout on a Multi-Attach that the Deployment reports as merely “progressing”. A pod finishing a run during those seconds finds nothing listening and retries its delivery for long enough to outlast the replacement coming up — including the time it spends resuming campaigns before it binds its port.

RBAC reconciliation is not decoration. The /usage endpoint (cluster CPU/memory, shown in the web UI top bar and by the resource_usage MCP tool) once needed a new cluster-scoped ClusterRole over nodes/pods; a service deployed before that returned a permissions error until it was set up again. An upgrade that skipped RBAC would reintroduce exactly that, as a runtime 403 that reads like a bug rather than a missed migration.

The same endpoint’s measured CPU/memory (the filled series on the Admin page’s chart, cpu_measured on the MCP tool) needs two further things, and both fail soft: a metrics.k8s.io/nodes grant in that ClusterRole, and metrics-server on the cluster — which RKE2 ships by default. A service whose RBAC predates the grant, or a cluster with no metrics-server, keeps reporting capacity, reservations, disk and job counts, and states which of the two is missing instead of a 403 or a zero. So this one is a reconcile when convenient: run vast service upgrade and the fill appears.

The per-job cpu/memory on a campaign’s Jobs tab (and usage on list_campaign_jobs) needs the sibling grant, metrics.k8s.io/pods, and the same metrics-server. It is a separate rule because the two are granted independently, and it fails soft the same way and on its own: a deployment carrying only the older nodes grant keeps its Admin chart and simply shows no per-job meters, with metrics_unavailable naming pods and the command that reconciles it. Also reconcile when convenient. get and list are the whole of what metrics.k8s.io serves — it has no watch — so the numbers are polled, once per sample window for the whole service rather than per campaign or per job.

Before rolling, it asks the service which live campaigns the replacement could not pick up again, and names each with its reason. A live campaign is no longer reason enough on its own: its Jobs are not children of the pod being replaced, and the new pod re-attaches to them (Cluster Execution). What still blocks a roll is a campaign nothing could re-launch — one with no records, or a search with no search.seed. --yes skips the question; without it a non-interactive run aborts rather than rolling silently. A service that cannot be reached is reported and the roll proceeds, since a wedged service is a reason to upgrade rather than a reason to refuse, but it says so — a silent roll must never be read as “nothing was running”.

There is a smaller verb for the common case, and it needs no kubeconfig:

vast service restart

That asks the service to roll itself — the Deployment’s restart annotation, and nothing else. It is the same thing the web UI’s Admin page button does (The Admin page), and it exists because upgrade needs cluster access, so somebody who reached the deployment through vast login had a button in the browser and no command at all. It carries the same live-campaign guard and the same --yes.

It reconciles nothing. RBAC, the registry ingress route, the credential Secrets and the build daemon are all untouched, so a version needing a new permission will deploy and then 403 at runtime. Use it for “new bytes are published and nothing else changed”; use upgrade for a version bump, a missed RBAC migration, a rotated Secret, or a registry move. The Secrets cannot be done any other way: they are rebuilt from the operator’s environment, which the pod does not have.

The lifecycle verbs are deliberately distinct:

Command

What it changes

vast service upgrade

The image, RBAC, and the credential Secrets it can rebuild from .env (git, share, ntfy, registry) — which is how a registry move or a rotated password reaches the cluster — plus the settings it carries from .env, the free-space reserve and the build daemon’s among them. The access token is preserved, so nobody is logged out.

vast cluster setup --force

The same, plus it will re-mint the access token when asked (--rotate-token) — which does log everyone out, so it is not what you want for a version bump.

vast service restart

The image only, through the service’s own API — no kubeconfig needed. Reconciles nothing else, so it is for “new bytes are published” and not for a migration.

vast cluster cleanup

Removes the deployment entirely.

Campaign data lives on the results volume and survives all three — vast cluster cleanup keeps the data directories unless --delete-data asks otherwise. Plain setup over a live service is refused (Cluster is already set up).

That volume is a directory on the service pod’s node where the cluster provisions nothing (rke2, minikube) and a provisioned ReadWriteOnce claim where it can (Where this deployment’s own data lives). Either way it is one disk and not a backup: anything that must survive the machine belongs in an archive (vast share, or vast campaign download), and keeping the disk itself is the operator’s snapshot schedule, which RoboVAST neither takes nor manages.

Where a node setting comes from decides what each command does with it:

Setting

vast cluster setup

vast service upgrade

Campaign job node pool (ROBOVAST_JOB_NODE_LABELS in .env, Pinning pods to a node pool)

applied from the environment; unset, every node

applied from the environment, not with --no-restart; unset, every node

Job node aliases (ROBOVAST_JOB_NODE_ALIASES in .env, Naming a node a campaign may be held to)

applied from the environment; unset, none

applied from the environment, --no-restart included; unset, every alias is removed

A .env entry is the standing statement, so both commands apply it whole — run them from the shell that has the deployment’s .env.

Splitting a campaign’s postprocessing

On a cluster, a campaign’s postprocessing can run in several Jobs at once instead of one. The steps that work run by run — the rosbag conversion, run_log, resource_usage and every other step whose plugin declares it (Add Postprocessing Command Plugin) — run in one Job per part of the campaign’s runs, each staging only its own runs; the admission queue places them wherever the cluster has room. One more Job then runs the remaining steps and the index ingest over the whole campaign.

A part that fails still delivers what it derived, and that last Job still runs once every part has ended: the campaign is queryable less what the failed parts could not derive, and its postprocessing is reported as failed, naming the parts. Only a part that never ran – one that could not start, or a stop – leaves the campaign out of the index.

Nothing has to be set for this. By default a campaign is cut into as many parts as the cluster could run at once: its recorded size (ROBOVAST_CLUSTER_MAX_CPU, written by setup) divided by the CPU one conversion reserves, which is the campaign’s own results_processing.resources. A cluster that cannot say how large it is gets eight. How many of those Jobs run at the same time is, as for a campaign’s runs, the admission queue’s decision.

Set a cap when you want one:

# .env on the machine you run setup/upgrade from
ROBOVAST_POSTPROCESS_MAX_PARALLEL=8

It is then the most Jobs one campaign’s postprocessing is split into; 1 keeps the whole postprocessing in one Job. A campaign with fewer scenario jobs gets fewer parts either way. vast cluster setup and vast service upgrade carry the value into the Deployment and refuse one that is not a whole number of at least 1.

Keeping free space

The service keeps a free-space reserve, so that nothing it writes takes its disk past the point where the node evicts it. It honours the reserve two ways:

  • While its disk or its results volume has less free space than the reserve, it refuses to start a campaign, a re-run, an image build, an import or a postprocessing run, with a 507 naming the meter and the amounts.

  • On a cluster, campaigns already running start no new Jobs while the results volume is below the reserve. Each says so in its log and on its status (stage: waiting for disk space: ...); Jobs already running finish and deliver their results into the reserve, and admission resumes by itself once space is freed.

Stopping or deleting campaigns is never refused. The web UI’s sidebar and get_resource_usage (storage_refusal) show the same verdict. If clearing the service’s caches would help, the refusal says so: vast service cache --clear, or Service cache on the Admin page.

Unset, the reserve is 15% of the disk being written to: above the kubelet’s default hard eviction threshold (nodefs.available<10%), which evicts every pod on the node – the service included – with room for what running Jobs still deliver once admission stops. State an amount instead when you know your disk:

# .env on the machine you run setup/upgrade from
ROBOVAST_DISK_RESERVE_GB=150

It is an absolute amount in gigabytes (109 bytes); 0 keeps no reserve. An invalid value is an error, not a fallback.

On a cluster, keep it above the kubelet’s hard eviction threshold on the data node. That threshold is usually a fraction of the disk (nodefs.available), so convert it for your disk. Read it from the node:

kubectl get --raw "/api/v1/nodes/<node>/proxy/configz" \
  | jq '.kubeletconfig | {evictionHard, evictionMinimumReclaim}'

Keep it above the threshold plus evictionMinimumReclaim. Once a node crosses its threshold, the kubelet evicts and holds the disk-pressure taint until it has reclaimed that much more, so nothing pinned to the data node – the service among it – schedules there again until free space is back above the sum. With nodefs.available<5% and a minimum reclaim of 10%, that is 15% of the disk: the default reserve.

vast cluster setup and vast service upgrade apply it, and deleting the line resets it; vast service restart does not. The build cache keeps a reserve stated here free too, unless ROBOVAST_BUILDKIT_CACHE_MIN_FREE says otherwise (see the build daemon’s settings).

Checking a deployment

vast doctor              # prerequisites, capacity, permissions
vast doctor --flavor gcp # also what the gcp flavor needs
vast doctor -x local     # a specific kubeconfig context
vast doctor -n robovast  # a deployment in another namespace

Reads only, so it is safe at any time — which makes it usable both as the first step of an install and as the first step of debugging one. Every failure names its remedy. It checks the Python version, kubectl/helm/gcloud, that the kubeconfig resolves and the API server answers, that the caller may create ClusterRoles (setup does), and that the nodes report allocatable capacity at all. It reports the largest node rather than judging against a threshold: a campaign’s pod is whatever its .vast asks for, and a request no node can hold is refused when the campaign launches, naming both the request and each node’s allocatable.

It also checks whether the deployment can build experiment images, which is the one prerequisite that would otherwise surface only when a campaign is submitted and refused:

Check

Says

build registry

Whether a push target is configured. A published service with no prefix wants vast service upgrade; one that was never published wants setup with --ingress-host. Reading the Ingress is what lets it name which — the in-pod service cannot tell those apart and has to offer both.

registry route

Whether that target is actually reachable: the /v2 rule and the upload-size annotation. Both matter and fail differently — without the annotation every layer push dies on nginx’s 1 MiB default with a 413, while GET /v2/ answers 200.

Both are advisory: a deployment that cannot build is not a broken one. The namespace matters, so pass -n when the service is not in default.

From a machine with no kubeconfig, vast doctor still reports an image builds line — read from the service’s own handshake rather than the cluster. It is absent when the service is older than that field: absent is “did not say”, not “no”.