Results Processing
Every RoboVAST execution produces a results directory with a well-defined layout.
This page documents the output structure and how to postprocess and merge results
using the vast results command group.
Output Structure
The results directory is named with --results-dir: on the serve command the
deployment runs for where campaigns land, and on each vast results verb for which
tree to read.
Top-Level Layout
<results-dir>/
└── <campaign-name>-<timestamp>/ # One per execution (e.g. dynamic_obstacle-2026-03-04-152130)
├── metadata.yaml # Campaign metadata (auto-generated)
├── _config/ # Campaign-level configuration snapshot
├── _execution/ # Execution metadata
├── _transient/ # Intermediate/preprocessed data
├── _jobs/ # Per-job artifacts (sysinfo, resource usage, logs)
└── <config-name-1>/ # One directory per configuration variant
└── <config-name-2>/
Campaign-Level Directories
_config/ — Configuration Snapshot
A copy of all input files used during execution — and the source a retrigger reconstructs the
campaign from. To run a campaign again exactly as it ran, use Retrigger campaign in the web UI’s
campaign actions menu, start_campaign(from_campaign=<id>) over MCP, or
POST /campaigns/<id>/retrigger: all three read this snapshot together with the image recorded in
_execution/ and start a new campaign, leaving this one untouched.
Doing it by hand instead pushes the snapshot as a workspace and runs that:
vast workspace init <campaign-dir>/_config --name replay
vast workspace run replay <config-name>.vast
That path rebuilds the image rather than reusing the one the campaign recorded, so it needs the
sources the build: section names — which are not archived here. It is the escape hatch for when
the recorded image is gone; otherwise prefer the retrigger, which reuses the exact bytes.
All three launches are gated by the same pre-flight over these records; see
Re-running a campaign: the pre-flight.
The structure inside is domain-specific, but typically includes:
_config/
├── <name>.vast # The .vast configuration used
├── scenario.osc # OpenSCENARIO scenario file
├── analysis/ # Jupyter notebooks for analysis
│ ├── analysis_run.ipynb
│ ├── analysis_config.ipynb
│ └── analysis_campaign.ipynb
└── <run-files defined within vast-config> # e.g. launch files, models, scripts, parameters
_execution/ — Execution Metadata
_execution/
├── launch.yaml # How the campaign was ASKED FOR (see below)
├── execution.yaml
├── plugin_install.log # ``plugin install`` phase (pip output; only when plugins are declared)
├── variation.log # ``variation`` phase (config-variation expansion)
├── controller.log # ``run`` phase — campaign controller log
├── postprocessing.log # ``postprocessing`` phase (rosbag→CSV + index ingest)
├── share.log # ``share`` phase (export to share, when re-run)
├── import.log # ``importing`` phase (only on an imported campaign)
└── import.json # per-stage ingest report (only on an imported campaign)
Each pre-/post-run phase writes its own log file here; the service concatenates
them in phase order into the single live campaign log the web UI streams. The
plugin install phase (present only when the .vast declares plugins:) runs
first and captures the pip install output live, exactly like building,
variation and postprocessing.
A re-triggered step appends to its own phase file, so the campaign log shows what the
step you just asked for actually did rather than only what the original run did.
postprocessing.log grows on each run_postprocessing; a re-triggered upload-to-share
writes share.log, which is a phase of its own because an upload is not postprocessing and
a divider naming the wrong step is worse than no divider. (The share that runs inside a
campaign is part of the controller’s own narrative and stays under run.)
import.log is the same idea for the one phase that can precede everything else:
a campaign taken in from an archive or the share writes it while the bytes are still
arriving — which is why the campaign’s _execution/ directory is created before the
extraction rather than by it. An import whose account of itself only began after the
download would have no account of the download, which is the slowest and least
inspectable part of it.
execution.yaml contains:
execution_time: ISO timestamp of when the execution startedrobovast_version: Git commit hash of the robovast version usedruns: Number of runs per configurationexecution_type:clusterorlocalimage: the configured execution image reference (may be a floating tag such as…:latest)image_revision: the immutable digest the run pods actually used (repo@sha256:…), captured at run time on the cluster backend. Re-postprocessing reuses this exact image, so a later re-run deserializes the recorded bags against the same image the runs used even if the tag has since moved.cluster_info: Node count, labels, CPU manager policies (cluster only)
launch.yaml records the request, where execution.yaml records what happened:
config_filter, campaign_name, runs (as requested), postprocess,
upload_to_share and backend. It exists because the request was otherwise
unrecoverable — config_filter in particular was consumed during config expansion and kept
nowhere — so “was this the full sweep or a one-config pilot?” could not be answered about a finished
campaign, by a person or by a retrigger.
Read the two together for runs: launch.yaml’s runs: 0 means “take the .vast’s
execution.runs”, so 0 beside execution.yaml’s runs: 3 says the .vast asked for 3,
while 1 beside 1 on a .vast declaring 3 says someone piloted it. metadata.yaml nests
this under execution.launch so a published campaign is one document. It is written by the service
before the run starts (which is why it is a separate file: execution.yaml is written by the run,
and a campaign that fails before its first batch would otherwise have no record of what it was asked
to do). Campaigns from before this file existed simply have none.
controller.log captures the campaign controller’s own log for the whole run —
batch/search progress, backend job dispatch, postprocessing and stopping
decisions. For cluster runs the robovast-service writes this as it drives the
campaign (and the web UI streams it live); it is preserved in the campaign so a
downloaded or shared campaign is self-documenting without a live cluster.
_transient/ — Intermediate Data
_transient/
├── configurations.yaml # Fully resolved configuration parameters
├── entrypoint.sh # Generated container entrypoint script
├── secondary_entrypoint.sh # Generated secondary container entrypoint script
└── collect_sysinfo.py # System info collection script
configurations.yaml contains the fully resolved parameter values for every
configuration variant, including internal computed fields like navigation path waypoints
(_path), raster points (_raster_points), resolved file paths, and
_variations (list of applied variation plugins with name, start time, duration,
and any plugin-specific fields).
Configuration Directory
Each configuration variant gets its own directory:
<config-name>/
├── _config/
│ ├── config.yaml # Configuration identifier hashes
│ ├── scenario.config # Resolved scenario parameters (YAML)
│ ├── sim.config # Resolved sim block [if the config has one]
│ ├── sut.config # Resolved sut block [if the config has one]
│ ├── maps/ # [navigation only]
│ │ ├── <name>.pgm # 2D occupancy grid image
│ │ └── <name>.yaml # Map metadata (resolution, origin, thresholds)
│ └── 3d-mesh/ # [navigation only]
│ ├── <name>.stl # 3D environment mesh
│ └── <name>.stl.yaml # Mesh metadata
├── _transient/ # Per-config intermediate files
└── <run-number>/ # 0, 1, 2, ... (one per run)
scenario.config contains the actual scenario parameter values used for this
configuration, wrapped in a single key matching the scenario name:
test_scenario:
growth_rate: 0.5
initial_population: 50
One file per variation channel, and each holds what that channel resolved to for this
configuration: scenario.config the parameters the scenario file declares, sim.config
the simulator’s whole resolved block, sut.config the flat <source>.<path>: value
block the system under test was configured with. config.yaml holds none of them — it carries
only the identifier hashes that group configurations across campaigns.
The last two are records, not inputs: what a run reads is the mounted overrides file
(sim) and the rewritten config copies (sut). They sit beside the configuration so
that what each channel was given is readable without diffing two copies of a stack’s
configuration. The campaign-level _transient/configurations.yaml carries the same values
for every configuration in one document.
All three are also read into the index, so a factor can be filtered and grouped by whichever channel it was written on — see Which channel a factor was written on, and its column.
Run Directory
Each run directory holds the scenario output for one configuration at one run
number, plus a job symlink to that run’s job-level artifacts:
<run-number>/
├── test.xml # JUnit test result (pass/fail, duration)
├── job -> ../../_jobs/job-N # symlink to this run's job artifacts (see below)
└── <test-specific files> # Domain-specific scenario output, e.g. out.csv,
# or a scenario-recorded rosbag2/ bag directory
Anything the scenario itself produces (test.xml, scenario-recorded
rosbag2/, domain output) stays in the run directory. Infrastructure and
monitoring artifacts (sysinfo.yaml, resource_usage_*.csv, the system
log, and the entrypoint’s /rosout + /clock recording) belong to the job
and live under _jobs/job-N/ — reachable via the job link, e.g.
<run>/job/sysinfo.yaml (see Job Directory). The links are made when a batch
ends, so the runs of a batch that was stopped have none. Which job each run belongs to is
recorded in _transient/job_links.yaml from before the first job starts, and that is
what RoboVAST resolves a run’s job through.
Postprocessing adds two derived files to the run directory: run_log.csv, the job’s
container logs joined with /rosout and sliced to this run (see
run_log — everything the run said), and resource_usage.csv, the job’s resource-monitor samples
sliced the same way (see resource_usage — what the run cost). They live here, rather than beside
the job artifacts they were built from, because a run is what they describe — and because
the ingest that turns data files into tables globs run directories.
Every file a rosbags_* step derives from this run’s bag lands here too, for the same
reason — poses.csv, one action_<name>_feedback.csv and action_<name>_status.csv
per converted action, nav2_behavior_tree.csv, the costmap and per-topic CSVs. They are
the same for every campaign, so a downloaded campaign carries them. Each is named, with what it was derived from, in
_transient/postprocessing.yaml.
The conversion reads each bag once, and reads from it only the topics some configured handler converts: the bag reader is filtered to those topics, so a recorded topic no step asks for (a camera or laser-scan stream, say) is skipped by the storage plugin rather than handed to the converter and deserialized.
A common example of test-specific output is a scenario-recorded rosbag2/
directory (standard ROS 2 bag in MCAP storage, with a metadata.yaml listing
recorded topics and message counts). It is present only when the scenario
records a bag, and is distinct from the separate, job-level /rosout
recording under _jobs/job-N/logs/.
One clock per run
Every table a postprocessing step derives from a bag timestamps its rows with the bag’s
receive time. Under bag_record(use_sim_time: true) that is the simulator’s clock, and it
is the right one to key on: it is what the simulator actually stepped, so it is identical across
simulators and independent of how fast the machine ran — the same trial can take 520 s of wall
time on one backend and 100 s on another and still span the same sim seconds.
That shared clock is what lets the poses, the costmaps and the behaviour trees be read against
each other at all, and it is the timeline the web Run view scrubs. So a topic
that carries a clock of its own must be converted, with its original stamp kept in a
separately-named column rather than used as timestamp. There is a real instance:
/behavior_tree_log is stamped by nav2 from a wall clock even under use_sim_time, roughly
1.8e9 s away from sim time. Keying nav2_behavior_tree on it made the table unjoinable with
every other one and put each transition outside the run’s timeline entirely, so the behaviour-tree
panel had nothing to show at any playback time; it is now the receive time, with nav2’s stamp in
event_timestamp.
Recording /clock in the scenario’s bag_record(...) is worth the negligible space: it makes
the sim↔wall mapping recoverable, so a foreign-clock topic can be related to sim time afterwards
instead of guessed at.
The pose contract
A pose table answers one question — where was this thing, and when — and more than one producer
can answer it. poses comes from /tf in a rosbag; sim_poses is written by the simulator
itself, during the run, and is the only pose data a stepped (non-ROS) run has, since there is
no bag to derive anything from. A stack on some other middleware, a motion-capture ingest, or a
real-robot log joins them by emitting the same columns; nothing has to be registered, because the
ingest globs *.csv in each run directory and names the table after the file.
One table per producer, sharing the schema. That keeps postprocessing_steps provenance 1:1 and
stops a panel filtering frame: base_link from silently plotting two interleaved series; a query
that wants both writes one UNION ALL.
Column |
Meaning |
|---|---|
|
The named entity, in the producer’s own vocabulary: a TF child frame, a MuJoCo body, a motion-capture rigid body. |
|
The join key described above, and never re-keyed. What it measures depends on the producer: arrival time for a table converted from a transport, which no derivative may be taken from; the exact simulated time for one the simulator wrote itself. |
|
Measurement time: when the pose was true, from the producer itself. NULL when it cannot
state one (a latched |
|
Unix epoch seconds for the same sample, where the producer can state one. Not a pose clock:
it exists to join this table to what is stamped in wall time ( |
|
Meters. |
|
Quaternion, and the only attitude a producer emits. |
|
Derived at ingest; see below. |
|
World-frame velocity, empty when the producer cannot know it (TF carries none). |
World coordinates, as an invariant rather than a column. Every row is in the run’s single
global frame, so there is no reference_frame to read or to get wrong. This holds today rather
than being aspirational: the TF handler resolves every frame against map and fails loudly when
a required frame yields no map-relative pose, the ROS launches make map identical to the
simulator’s world by an identity edge, and MuJoCo’s xpos/xquat are world-frame by
construction. A producer that cannot express world coordinates does not satisfy the contract.
Difference ``stamp``, join ``timestamp``. Arrival time is only as fine as the /clock grid
the recorder’s own clock advances on, and is jittered by delivery on top — neither of which is the
interval the robot moved over. The failure is not subtle and does not look like noise. Measured on
one campaign: a ground-truth pose published every 18 ms onto a 10 ms grid arrived
20/20/20/20/10 ms, so a robot driving at a constant 0.238 m/s read as an alternating
0.214 / 0.428 m/s — the displacement between samples was identical in every bucket, and only the
denominator was wrong. Making the grid divide the period removes that systematic alias but not the
delivery jitter; only stamp removes both. calculate_speeds_from_poses picks the base for
you and reports which it used in time_base, so a comparison can assert both sides took their
derivative from the same column. Read that column together with the table it came from: time_base: timestamp
is the exact base on a simulator-written table and the degraded one on a transport-derived
table that carries no stamp, and the two are not comparable despite the identical label.
Every table that follows the contract gets a track summary, ``pose_track_view``. One row per
track, meaning one frame of one run as one table recorded it: points, length_m,
duration_s, avg_speed_m_s, max_speed_m_s, max_step_m, start and end pose, and the
bounding box.
It is computed over every recorded pose, on each table’s measurement clock (stamp where the
table has one, else timestamp), with position.z in the length where the table carries it.
source names the table, since the same entity recorded by two producers is two tracks. A
sample with no measurement time, such as a latched /tf_static transform, is not a point on a
track. A reposition counts as travel: a body spawned at the origin and then placed at its start
pose adds that jump to length_m, and max_step_m (the largest single step) is where it
shows. The view is built over whichever pose tables the index holds, so a new producer’s table
appears in it on the next ingest without being registered.
SELECT source, frame, points, length_m, duration_s, max_speed_m_s
FROM pose_track_view
WHERE campaign_id = '<id>' AND config_name = '<config>' AND run_id = 0;
Quaternion in, yaw out. Producers emit a quaternion and nothing else: roll/pitch/yaw is lossy
the moment a body pitches or rolls, which rules out a drone, a tilting arm, or a robot on a ramp.
The ingest then derives orientation.yaw for any table that has the quaternion columns and no
yaw, because the 2D consumers — the costmap panel’s heading marker, the nav MCP tools, the
notebooks — all want a heading and none of them should reimplement quaternion math in SQL,
JavaScript and pandas separately. It is a projection, and a _column_notes entry says so.
Wall → sim: the clock map
Everything a run logs is stamped in wall time — rosout’s receive time, and whatever each
container printed. Everything it is analyzed on is sim seconds. Relating the two is a per-run
clock map: a list of (wall_ts, sim_ts) samples, interpolated piecewise-linearly.
A single offset is wrong, which is why this is a sampled map and not a number. Measured on a
recorded run, dt/dw = 1.369 — the simulator ran 1.37× faster than the wall clock, so an offset
taken at the start is ~8 s out by the end of a 29 s run. Sim time can also pause.
Two producers, one format:
ROS — the entrypoint’s own recorder (
/rosoutand/clock) writes a bag in wall time for the whole container’s life, so each/clockmessage is an exact (wall receive, sim content) pair. Deliberately not the scenario’sbag_record: that one is sim-time on both axes, so it cannot carry the mapping, and it starts mid-run where the interesting failures are already over.roqsim (non-ROS) —
roqsim.capturestreams<recording>.clock_map.csvbeside the.npz, flushed per sample so a run killed by a timeout still has one.
The samples are decimated: one is dropped only when linear interpolation reproduces it within
5 ms, so a steady stretch costs two rows while a pause or a change of real-time factor keeps
exactly the samples that describe it (12000 /clock messages → 26 rows, measured).
Outside the sampled range there is no answer, and none is invented. The samples begin when the
simulator started publishing its clock, typically well after the container did, so a line logged
during image boot has no sim time — a different statement from “we could not compute it”.
runs.clock_map_source (ros_clock_bag / roqsim_run_npz / none) says which producer
answered, so a reader can tell a missing map from a quiet one.
run_log — everything the run said
One row per log event, across every container, on the run’s own playback clock. Written by
the auto-injected run_log postprocessing plugin as <run>/run_log.csv, which the usual
one-file-one-table ingest turns into the run_log table.
It is a join, not a concatenation, because a run’s output arrives twice: a launch container
forwards its nodes’ output to stdout and those nodes publish /rosout. On a measured
three-container campaign 473 of 521 rosout rows are the same event as a system*.log line, so
appending both would report most of the run twice. Matching is exact on (node, wall_ns, first
line of message), keyed on the producer’s stamp — keying rosout on the bag’s receive time
instead matched 0 of 521, because the transport delay puts every pair on a different nanosecond.
The join also supplies what neither source has alone: /rosout names the node but never the
container it ran in, and that is what a reader filters by. It comes from the file the stdout
twin was found in.
A packed job’s log is split between its runs, not shared with all of them. The artifacts are
written per job, and with execution.runs_per_job > 1 one job runs several configurations in
sequence into one log. Each line is claimed by exactly one run, so a run’s rows are its own —
another configuration’s trial is a different experiment, not context for this one. A run whose
job artifacts cannot be located, or which shares a job and never wrote test.xml, gets no rows
and is named in the plugin’s message; get_job_log is the whole-container view.
Columns: sim_time, wall_ts, time_source, in_window, container, node,
source, level, severity, message, file, function, line.
source—rosoutorstdout.WHERE source = 'rosout'is the rosout slice; there is no separaterosouttable, which would be a strict subset of this one.time_source— how the row got its wall time:stamp(the producer’s own, ns precision),inherited(a continuation line, e.g. a traceback frame, taking the time of the event it belongs to), ornone(nothing stamped anywhere before it).Producers stamp their own lines, which is why
stampis the normal case rather than a ROS-only luxury: rclpy writes the stamp, and so do the entrypoints’loghelper and scenario-execution’s logger. There is no capture step anywhere — a helper mounted in every container to timestamp each line as it is read means a second copy of every log line on disk (119 KB against a 108 KBsystem.log, measured) to describe output that is RoboVAST’s own and can simply say when it spoke.Third-party output that stamps nothing (a gz warning, a vanilla sidecar) is never dropped: it gets a row per line, inheriting from a neighboring event where one exists and reporting
nonewhere it does not. An untimed row is honest about being untimed; it is deliberately not backfilled from the next stamp, which would render exactly like a real time and claim the container booted at whatever second the first node came up.in_window— 0 for a line outside this run’s own wall window: its bring-up, its verdict, its teardown. Real output, kept rather than dropped, and flagged so a query can tell “during the trial” from “getting ready for it” and “cleaning up after it”.It is not the boundary of the trial, and must not be used as one. A run’s
test.xmlduration closes when its scenario stops, but the verdict line is logged after that — measured at ~1 ms late for a failing run and ~0.1 ms early for a passing one. Filteringin_window = 1therefore drops the verdict of every failing run. Where the trial ended is scenario_timestamps, and a line’s owner is its run’s log claim.severity— fromcommon.log_summary.severity_of, the same definition the status verdict and the MCP log tools use.
Read it from the web Run view’s log panel and the Explorer’s Log tab, or
from an agent with search_run_logs. What it makes possible that a log stream cannot:
-- which runs logged this, and did they fail?
SELECT r.config_name, r.run_id, r.passed, count(*) AS hits, min(l.sim_time) AS first_at
FROM run_log l JOIN runs r USING (config_name, run_id)
WHERE l.message LIKE '%CRITICAL FAILURE%'
GROUP BY 1, 2 ORDER BY hits DESC;
The raw streams stay files and stay the record of what was printed, and get_campaign_log /
get_job_log remain the way to read a campaign that is still running, since it is not in the
index yet.
Those files now carry a prefix on every line, including the infrastructure ones: Running as UID:
1000 reads [INFO] [1786264427.117714] [entrypoint]: Running as UID: 1000. That is the cost of
each line being placeable in time and attributable, and it is the shape most of the file already
had — 563 of 570 lines in a measured ROS run were stamped by their producer. Anything matching
^Running as UID needs updating; the live log panel is unaffected mechanically (it pages bytes)
but shows the prefixes too.
resource_usage — what the run cost
One row per container per process name per ~1 s sample, on the run’s own playback clock.
Written by the auto-injected resource_usage plugin as <run>/resource_usage.csv from
the job’s resource_usage_<container>.csv files, and ingested as the resource_usage
table like any other per-run CSV.
Why it is a table and not just those files: the cluster gives a job a fixed number of cores, so a simulator that starves the stack changes what the stack does. That is a competing explanation for any behavioral result, and it can only be ruled out in the same query as the behaviour.
-- did anything run out of CPU during the trial? (cpu_percent is PER-CORE:
-- one saturated core is 100, so the ceiling is 100 * available_cpus)
SELECT u.container, MAX(u.cpu) AS peak, 100.0 * r.available_cpus AS saturation
FROM (SELECT config_name, run_id, container, wall_ts, SUM(cpu_percent) AS cpu
FROM resource_usage WHERE in_window = 1 GROUP BY 1, 2, 3, 4) u
JOIN runs r USING (config_name, run_id)
GROUP BY 1, 3;
Three things differ from run_log, each deliberate:
The gap between two runs falls the other way. Both tables partition a packed job’s timeline — each tick and each line is claimed by exactly one run, so
SUMover a job’s runs is what that job consumed, and no run reads another configuration’s trial as its own. What differs is where inside the gap between two runs the boundary sits. A sample taken while the simulator is being reset is the cost of the run starting up, so the boundary is the earlier run’send_epoch. The gap’s lines, though, are the earlier run’s verdict and teardown followed by the later run’sExecuting scenario, and only that marker separates them — so for a log the marker is the boundary.Neither end of the trial window can stand in for it, and both fail in the awkward direction: a failing run’s verdict is stamped ~1 ms after
end_epoch, and the marker ~35 µs before the next run’sstart_epoch(it is logged, then the start is recorded). Cutting atend_epochfiles a failing run’s own verdict under its successor; cutting atstart_epochfiles every run’s own scenario-start line under its predecessor. When the markers cannot be matched one-to-one with the runs — rosout is only recorded once subscribed, so one can be missing — thestart_epochboundaries are used instead, which costs those microseconds rather than shifting every run by one.Rows are keyed by process name, not pid. Pids churn — a respawned node is a new pid and the same program — and no pid is comparable across runs.
num_pidsrecords how many shared a name in that tick.A run of a packed job with no ``test.xml`` claims nothing and gets an empty table rather than the whole job’s samples. It cannot be placed on the wall clock, and a table saying “no data” is honest where one stating another run’s numbers is not. A single-run job in that state still gets its whole trace — there is no other run to confuse it with, and a run killed mid-flight is the one whose trace matters most.
cpu_percent and memory_rss_bytes are sums over the processes sharing a name, and both
carry a _column_notes entry that describe_campaign_data shows: CPU is per-core, and
summed RSS double-counts pages shared with forks.
postprocess_system_usage.csv — what postprocessing itself cost
One row per postprocessing step – stage, convert, host – written to
_execution/postprocess_system_usage.csv, beside the campaign it
describes. The columns are the sampler’s own (memory_peak, memory_max,
cpu_usage_usec, nr_periods, nr_throttled, oom_kills, …), so a step’s figures
read the same way as a run’s system_usage rows and can be
compared with them. The same row is logged as one line per step in the POSTPROCESSING section,
so the usual question – did a step come near its limit? – is answered without downloading
anything.
``system_usage``, not ``resource_usage``. The distinction is the one that table pair
already makes: resource_usage is per process and every reader aggregates it that way,
while these figures are the cgroup’s accounting for a whole step and have no pid. Filing them
as processes would mean summing them as processes.
Read once from the kernel, not sampled. These steps are containers rather than runs, and
each is short, so the daemon’s per-second rows are the wrong shape twice over: a poll interval
longer than the step reports a comfortable figure for one that may have come close to its
ceiling, and memory.peak is a high-water mark the kernel already keeps that no sampler can
beat. monitor_resources.py --once writes exactly one row of the container-level probes;
the conversion step – which runs the campaign’s own image and can import nothing of RoboVAST
– invokes it from the /scripts mount the pod already has.
This is what makes results_processing.resources checkable: a campaign that raised it can
see whether it needed to, and the shipped default can be argued about with a number instead of
the absence of a failure.
Unlike resource_usage it stays a file rather than becoming an index table, because it
describes how a campaign was processed rather than what its runs did – a row under a run
would be summed as something that run consumed.
scenario_timestamps — where the trial ended
One row per run: config_name, run_id, timestamp (sim seconds), wall_ts,
status (succeeded / failed) and the message itself. Built while ingesting
run_log, from the first line scenario-execution’s own logger wrote announcing a verdict —
Scenario '<name>' succeeded., or the failure line add_result logs. The recognition lives
in one module, robovast.common.scenario_markers, and runs here and nowhere else: every
later reader queries this table instead of matching the log text again, which is what keeps the
web UI, search_run_logs and the playback clock from disagreeing about where a run ended.
“The first verdict in the log” is only the right answer because run_log is partitioned
per run, so a run’s rows are its own. When a packed job’s lines were instead shared with all of
its runs, the first verdict in every run’s log was the first scenario’s — so every run of the
job recorded that verdict, and a run whose own trial passed reported failed while
run_view.status said it passed. Sharing the lines is what made the simple rule wrong; the
rule itself was never the problem.
Both clocks, because they answer different questions. timestamp is what the playback
timeline is measured in. wall_ts is what run_log is ordered by, and it is the one the
log is cut on — the clock map does not extrapolate, so a run whose /clock stopped during
shutdown has NULL sim_time on every line after the verdict, sometimes including the verdict
itself. A sim-time comparison would keep exactly the lines a reader wanted rid of.
Everything after wall_ts is shutdown, not the trial: nodes being killed, lifecycle
transitions failing because their peer is already gone, TF errors from a publisher that has
stopped. That is what the run view’s shutdown toggle and the log tools’
hide_shutdown cut, both on by default.
status is the scenario’s verdict and can legitimately disagree with the run’s
test.xml verdict in run_view.status; comparing the two finds a scenario that reported
success while the harness failed, or the reverse. A NULL row is a run that reached no verdict —
killed by its deadline, say — and is left untrimmed rather than trimmed to a guess.
rosbag_attempts — the run recorded more than once
A recorder that restarts mid-trial writes a second bag beside the first, because
ros2 bag record’s default name carries a timestamp. The last attempt is the run:
everything else in the directory exists once — one run log, one verdict, one set of videos,
all of it the last attempt’s — so converting an earlier bag would put a different attempt’s
trajectory under this run’s outcome, and no table would say so. Only one attempt can be
converted at all, because every output name is derived from the run directory.
So the last attempt is converted, the earlier ones are not, and this table is the record:
one row per attempt, with role converted or superseded, and the start time each
was dated by. It exists only for a run that recorded more than once, which is why the
question is a query rather than a search through a postprocessing log:
SELECT config_name, run_id, bag FROM rosbag_attempts WHERE role <> 'converted'
The attempts are ordered by the start time in each bag’s own sidecar, and by the timestamp
in its name only when a sidecar is missing — never by a mix of the two, since one is epoch
and the other the recorder’s local clock. An attempt with neither — a bag whose name carries
no timestamp and whose sidecar was never written — leaves them unordered:
nothing is converted from that directory, every row says unordered, and the step’s log
names the directory to clear. Picking one anyway would be data that looks right.
One run’s ambiguity is never the campaign’s: every other run converts, and a bag the last attempt never finalized is reported as unreadable in the usual way (see its rosbag is unreadable).
test.xml — JUnit Test Result
Standard JUnit XML format with scenario execution results:
<testsuite errors="0" failures="0" name="scenario_execution" tests="1" time="49.03">
<testcase classname="tests.scenario" name="test_scenario" time="49.03">
<properties>
<property name="start_time" value="1772634122.583653"/>
</properties>
</testcase>
</testsuite>
Each run’s test.xml is the runner’s contract for that run’s outcome. The
controller mirrors it into campaign.db’s run table at record time (status,
pass/fail, errors/failures, duration, start time), so per-run outcomes are
queryable live and the postprocessed runs view is built from those
rows rather than by re-parsing every test.xml — see
the campaign store schema.
A run somebody stopped: killed
run_view.status is passed, failed, error, unknown — or killed,
which means an operator ended that run’s job by hand while the campaign was running (the
web UI’s per-job Stop, the stop_job MCP tool, or vast campaign stop-job).
A killed run is not a trial failure. Nothing was learned from it about the system
under test, so it is a missing measurement: RoboVAST counts it apart from the failures
everywhere it reports them — num_killed beside num_failed in the campaign counts
and get_campaign_summary, its own tally in the web UI’s Details panel, and
runs.killed on the live status. Treat it the same way in your own analysis:
SELECT config_name, COUNT(*) FILTER (WHERE status = 'passed') AS passed,
COUNT(*) FILTER (WHERE status IN ('failed','error')) AS failed
FROM run_view WHERE status <> 'killed' GROUP BY config_name
failure_message on a killed run names the surface that stopped it and the reason its
operator gave (manually stopped via webui: stuck in nav recovery), which is the only
record of why — so it is worth giving one.
Its rosbag is unreadable, and that is not a failure
Stopping a job kills its process mid-write, so the rosbag it was recording is never
finalized and can never be opened. The same is true of a job that ends abruptly for reasons
nobody chose — an evicted pod, an OOM-kill — and the postprocessing verdict does not depend
on which it was. A bag that cannot be opened is missing input, not a failed conversion:
the recorder never wrote the sidecar naming the bag’s storage plugin, so there is nothing to
convert and no retry will produce one. Postprocessing names such bags, skips them, and
succeeds — reported as N unreadable in the summary plus a NOTE listing the paths.
Failing the step over them would cost the metrics of every bag that was readable, which on
a large campaign is thousands for the sake of a handful.
--tolerate-under is narrower than that rule and survives alongside it: postprocessing is
still told which directories belong to jobs an operator stopped, so the NOTE can say how
many of the skipped bags were expected rather than merely observed.
A real conversion error still fails postprocessing — a bag that opened and whose handlers then refused is the case where the data existed and we did not produce it. And every bag is still attempted, because a job cut off between bags leaves readable ones whose data is worth having.
This is the pipeline’s general rule rather than a rosbag special case: a step that cannot
read something describes the gap and succeeds — run_log and resource_usage already
report theirs (“no job artifacts for 1 run(s): …”) and a run with no test.xml simply has
no trial window, so it is left out of the per-run metrics instead of breaking them. Only a
genuine conversion error fails a step.
killed replaces unknown and only unknown: a run of a killed job that had
already written a valid test.xml keeps its real verdict. That matters when a job packs
several runs (runs_per_job > 1), where the earlier ones routinely finish before anyone
stops the job — their results are measurement and are never overwritten.
A configuration that produced nothing: missing
Every status above belongs to a run. missing belongs to a configuration: the campaign
was composed with it and its results never reached the tree – never dispatched, or lost
between the cluster and the results root. It appears in run_view as one row with
run_id NULL, exactly as composition_failed does for a search draw that could not be
built, because a join through run would otherwise drop it.
This is what makes a shortfall visible at all. The results tree states what came back;
only the composition record (_transient/configurations.yaml) states what was asked for,
and without comparing the two a sweep that lost a seventh of its cells is indistinguishable
from a smaller sweep that ran perfectly. get_campaign_summary therefore counts
num_configs over the declared set and reports num_missing_configs,
missing_configs and a note when they differ; the aggregates beside them are over a
partial design, which is the one thing a summary must not leave unsaid.
SELECT config_name FROM run_view WHERE status = 'missing'
A campaign whose archive has no _transient cannot be checked this way, and says so in
the log rather than reporting a complete design it cannot vouch for.
A trial the runner threw away: invalid
invalid means a container the trial ran against crashed and was restarted under it.
The simulator (or the system under test) took the run’s state with it; the container the
kubelet starts in its place runs no workload, and the runner ends the job as soon as it reads
the restart off the pod. Whatever verdict the scenario reached in between describes a trial
that had already lost its process.
It is the one status that overrides a written verdict, and that inverts the rule stated
for killed just above. The inversion is the whole reason it is a separate kind rather
than another kill:
a killed run’s
test.xmlwas written before the intervention landed, so it is real measurement and overwriting it would destroy data;an invalid run’s
test.xmlwas written by the trial the restart broke — it is the confidently wrong result the detection exists to prevent, and it is at its most dangerous when it sayspassed, because nothing else about the run looks wrong.
Discarding is not destroying. The test.xml stays on disk, and the verdict being
overridden is recorded in the ledger entry, so the override is auditable and reversible.
Like killed, it is not a trial failure — it is an infrastructure fault, not evidence
about the system under test — so it is counted apart (num_invalid) and excluded from
pass rates:
SELECT config_name, COUNT(*) FILTER (WHERE status = 'passed') AS passed,
COUNT(*) FILTER (WHERE status IN ('failed','error')) AS failed
FROM run_view WHERE status NOT IN ('killed','invalid') GROUP BY config_name
A cell that loses every run this way scores no_sample rather than a fabricated 0.0,
and the search carries on. That is the point of the whole mechanism: before it, one crashed
sidecar in one job of one batch raised out of the batch wait loop and ended the campaign,
taking every completed batch with it.
What killed it: container_failure_view
The post-mortem is captured at the moment of the restart — while the pod still exists, because it is about to be deleted — and lands in two places, deliberately:
_execution/container_failures.jsonthe record: every field of the container’s termination state, plus the last 400 lines of the dead instance’s own log (
kubectl logs --previous, which nothing read before).campaign.db→container_failure_viewthe index into it, so the question is one query. It is in
campaign.dband not the results index on purpose: the index is loaded by postprocessing, and a campaign that dies mid-batch never postprocesses — which is exactly the campaign that needs explaining.
SELECT run_key, node_label, container, role, exit_code, signal_name, reason, memory_limit
FROM container_failure_view ORDER BY run_key
signal_name is usually the answer: exit_code 135 is 128 + 7, i.e. SIGBUS, and
137 is SIGKILL (an OOM kill). A NULL memory_limit means no limit was declared at
all, which is itself a finding — such a container is told by the downward API that it has
the whole node, and the pod’s shared /dev/shm is sized the same way. Join back onto
run_view on config_name || '/' || run_id = run_key.
For a SIGBUS the sizing half of the answer is in runs: shm_peak_bytes is what the
run’s shared-memory pool held at its fullest and shm_limit_bytes is the size that was in
force, so “it ran out” and “it was never that big” are one query apart. Both are NULL for a
campaign recorded before the monitor sampled the pool, which is unmeasured rather than unused.
Per-tick values are in resource_usage (shm_used_bytes / shm_total_bytes) for seeing
when it grew — one pool for the whole run, repeated on the tick’s rows, so MAX is the only
aggregate over them that means anything. Sizing it: Configuration under shm_size.
An invalidated job’s rosbag is unreadable for the same reason a stopped job’s is (it is
deleted at grace_period_seconds=0), and postprocessing tolerates both identically.
The kills themselves are recorded in _execution/interventions.json, which exists only for a
campaign somebody intervened in. One ledger holds every kind of intervention, each entry
carrying a kind – killed for a job stopped by hand, probed for a run somebody read
into while it was going – because “what was done to this run?” is one question and answering it
should not mean knowing to ask twice. What follows differs by kind and that is why the readers
do: a kill becomes a run status, while probed is a separate runs column in the index and
never touches the verdict. Putting an intervention into the measured outcome is the same mistake
that keeping killed out of num_failed avoids.
A probe’s granularity follows the job: with runs_per_job > 1 the whole packed job is marked,
because which of its runs was in flight cannot be recovered afterwards. That over-excludes rather
than admitting a perturbed run, which is the safe direction.
Warning
Two unrelated things are called a probe. runs.probed above is a campaign run
somebody looked inside while it was going, which is why it is recorded and excluded. A
calibration probe (Per-node container sizing) is a different object entirely: an
extra run that measures one node before the campaign places work there, which writes to the
reserved _calibration/ directory and so never becomes a run at all. Nothing in runs
or in interventions.json ever refers to one.
Note
metadata.yaml is not how a caller reads a campaign’s results. It is written
only by postprocessing, so nine MCP tools that parsed it answered “run postprocessing
first” for campaigns whose outcomes were already recorded in campaign.db; they were
replaced by read-only SQL over run_view (see Reading results: SQL, not a tool per scope). The file remains
as the campaign’s self-contained metadata document — what the FAIR/PROV-O export and
the publication zip read.
Job Directory
_jobs/job-N/ holds the artifacts of one job — the unit of dispatch (one
Kubernetes Job, or one local docker compose run). With the default
runs_per_job: 1 there is one job per run; with runs_per_job > 1
several runs share a job (and therefore share these artifacts). Each
run links to its job via <run>/job (e.g. <run>/job/sysinfo.yaml).
_jobs/job-N/
├── sysinfo.yaml # Hardware info (platform, CPU, memory) — stable
├── resource_usage_main.csv # Main container CPU/memory over the job
├── resource_usage_<secondary>.csv # Per secondary container [if multi-container]
├── system_usage_main.csv # Main container cgroup counters over the job
├── system_usage_<secondary>.csv # Per secondary container [if multi-container]
└── logs/
├── system.log # Main container system log
├── system_<secondary>.log # Secondary container log [if multi-container]
└── rosout_bag/ # /rosout recording [ROS mode]
resource_usage_*.csv files have columns timestamp (wall epoch seconds), pid,
name, cpu_percent, memory_rss_bytes, shm_used_bytes and shm_total_bytes,
one row per process per ~1 s, one file per container. For a packed job these span the whole
job; the resource_usage post-processing step slices them to each run (see
resource_usage — what the run cost). Both files span every instance of the container as well: a
container the kubelet restarts runs the sampler again against the same file, which continues
the record under its one header rather than replacing it, so the samples of the instance that
died — the high-water mark climbing towards its limit — are kept beside the ones of the
instance that came after. The seam shows as the cumulative counters going backwards: the
calibration reader drops that tick as a cgroup replaced, and a run whose container crashed is
invalid in the intervention ledger (Client–Server Architecture), so its per-run aggregates are never
compared with a run that kept one instance throughout.
system_usage_*.csv is the sibling for figures belonging to the container as a whole
rather than to a process — one row per ~1 s, no pid. It is separate because
resource_usage is per-process by contract: every reader aggregates it that way, so a
container-level figure written there would be summed as though it were one process among
many. Its columns are whatever the node could answer, so a column may be absent entirely
rather than empty — a runtime that cannot report and a container that never stalled are
different facts, and a zero would make the first indistinguishable from the second:
Columns |
What they answer |
|---|---|
|
Did the kernel STOP this container because it hit its own CPU limit? Throttling does not fail a run, it just makes it slower, so nothing else records it. |
|
Memory as the KERNEL accounts it, which the per-process rows cannot give: RSS counts a
shared page once per process, so summing it over a stack of ROS nodes sharing
libraries and a DDS shared-memory segment over-reports badly. |
|
What that memory is MADE OF, which decides how much of it must be reserved:
|
|
CPU time the kernel billed the cgroup — exact, where summing |
|
PSI: how long tasks were runnable but not running — the container crowded out,
which the throttle counters cannot show. |
|
The whole machine’s pressure. A node fact repeated on every row of every container, so never sum it across a pod — it is what separates “this pod asked for too little” from “this node is oversubscribed”. |
|
Allocations the kernel refused, and processes it killed for them. Memory is sized on the peak because exceeding it is a kill rather than a slowdown; this is the only place a mid-trial death names its cause. |
Everything above except the three memory figures is a monotonic counter, so read a delta
within the trial window, never a SUM and never a bare MAX (memory_current is a
gauge, memory_peak a high-water mark, and the memory.stat breakdown gauges; those
are read as they come). Prefer run_validity_view, which takes that delta and
applies the thresholds for you. It derives two flags, and they are opposite diagnoses with
opposite remedies: quota_bound means the container exhausted the quota its own
limits.cpu buys — raise the limit; contended means it was runnable and got no CPU
without having hit that ceiling — other work took cores it had not reserved, so raise its
resources.cpu request, or put fewer jobs on the node. A container can be both, and the
ceiling is reported first because that remedy is a line in the campaign’s own .vast.
Which channel a factor was written on, and its column
A campaign varies its factors on three channels (the destination reference), and the runs table gives each one a param_* column
so that a question about a factor is the same query whichever channel it came from.
The name differs, because the destinations do:
Channel |
Destination |
Column |
|---|---|---|
|
|
|
|
|
|
|
|
|
A scenario parameter keeps its own name, which is what an analysis has always read. A sim:
or sut: destination is a path, and its whole path makes no column anybody can type — the
sut: example does not fit in an identifier at all, and an XPath destination is not
identifier-shaped anywhere but its end. So the column is named from the end of the
destination, prefixed by its channel, and grows leftwards only as far as it must to stay
unambiguous: two friction keys under different components become param_sim_floor_friction
and param_sim_wall_friction rather than quietly sharing one column.
Uniqueness is decided over the whole campaign, not one configuration, because the table is one shape for every row in it.
Note
Do not guess a column name — read it from the table. The suffix rule depends on what
else the campaign varies, so the same destination is param_sut_inflation_radius in one
campaign and param_sut_inflation_layer_inflation_radius in another that also varies the
global costmap’s. describe_campaign_data lists what a campaign actually has.
Every destination’s value is in run_view.channels_json regardless, under the channel name
the .vast writes it on — including one whose name would collide with a scenario parameter,
or would not fit an identifier even at full length. Those get no column and are reported at
warning level during ingest, naming the destination: a column silently missing is the same
wrong answer as a column silently shared.
A measurement with no finite value
A metric can come out infinite or undefined and still be right: a range sensor reports no
return, a trial produced no trajectory so its path length is infinite, a ratio had no
denominator. Such a value is stored as a number — double precision’s Infinity,
-Infinity or NaN — in a column that stays numeric, whether the CSV wrote it inf,
-inf or nan. A value nobody measured is NULL, so the two never have to be told
apart by guesswork:
SELECT AVG(path_length) FROM nav_metrics; -- Infinity if any trial was censored
SELECT * FROM nav_metrics WHERE path_length = 'Infinity'; -- the censored trials
SELECT * FROM nav_metrics WHERE path_length = 'NaN'; -- the undefined ones
SELECT * FROM nav_metrics WHERE path_length IS NULL; -- and the unmeasured ones
In Postgres, NaN equals itself and sorts above every number, infinity included, so an
ORDER BY or a MAX over such a column puts it last.
Inside a JSON-encoded value — a container-valued param_* column, channels_json —
the same three appear as the JSON strings "inf", "-inf" and "nan", so
param_gaps::jsonb ->> 1 is nan rather than a token that would make the cast fail,
and (param_gaps::jsonb ->> 1)::double precision is the number.
Reading these files
Every path in the tree above has one address, and it is the same string
whether you type it at the CLI, pass it to an MCP tool, or GET it from the
service:
/results/<campaign_id>/<path>
The path after the campaign id is exactly the path in the tree — _execution/
outcome.json, <config-name>/<run>/test.xml — so what a listing shows is what
you can read. Campaign results are read-only: they are the record of a run, and a
rewritten record is one nobody can check the figures drawn over it against. Workspace
inputs live in the writable half of the same address space,
/sources/<workspace_id>/<path> (see Config editor).
vast files ls /results/nav-2026-03-04-152130/ # _config/ _execution/ …
vast files ls /results/nav-2026-03-04-152130/ -r # whole tree, files only
vast files cat /results/nav-2026-03-04-152130/_execution/outcome.json
vast files cat /results/nav-.../hospital-1-42/0/logs/system.log --lines 50 --offset 200
vast files get /results/nav-.../hospital-1-42/0/rosbag2/bag.mcap ./bag.mcap
ls lists one level at a time — a campaign holds a directory per configuration
and per run, so a recursive listing of the root is thousands of entries; it
reports total when it truncates. cat pages text and refuses binary;
get writes raw bytes, which is how you fetch one artifact without downloading
the whole campaign archive (that is vast campaign download <campaign-id>, which
writes one .tar.gz and does nothing else with it).
The same addresses work over HTTP (curl <service>/results/<campaign>/<path>)
and from an LLM through the read_file / list_files MCP tools — see
Files. Reading a campaign on this machine needs no running service; against a
cluster service the read serves that one file off the service’s results volume, not the
campaign.
If the service runs on your own machine, get_service_info also reports a
results_root you can open directly with your own tools; it is absent whenever
that would be a path you cannot actually read.
metadata.yaml — Campaign Metadata
Every campaign directory contains a metadata.yaml file that is
automatically generated after postprocessing completes. It aggregates
structural and domain-specific metadata about the entire campaign into a
single file.
The file is produced by a four-phase pipeline:
Generic metadata — collected by
MetadataGenerator(robovast.results_processing.metadata). This includes configurations, test results (pass/fail, timing, output files, sysinfo), execution metadata, run files, and the scenario file reference.Variation-plugin metadata — each variation plugin used during configuration generation can contribute additional metadata by overriding the
collect_config_metadataclassmethod defined on theVariationbase class. For example,FloorplanGenerationoverridescollect_config_metadatato load map and mesh YAML metadata from_config/. Thevariationsfield in each configuration entry lists all variation plugins that were applied, together with their execution timing (name,started_atas ISO timestamp,durationin seconds).User-defined metadata processors — custom plugins registered under the
robovast.metadata_processingentry-point group and configured in the.vastfile (see below).Derivation — each configuration entry gets a
derived_fromfield naming the.vastconfiguration it was expanded from, which is also theprov:wasDerivedFromedge from the cell to that configuration inmetadata.prov.json. A configuration that records no parent carries no such field.
A campaign whose recorded outcome says it ended early — stopped, failed or
crashed — is described with the runs it has: execution.ended_early names the phase,
a configuration may have fewer run directories than execution.runs planned, and the
campaign may have no run with a verdict, since the ending cuts runs short before they reach
one. Any other campaign with fewer runs than it planned, or none with a verdict, is refused
as a broken input — a shortfall on its own is also what an accidental gap looks like. More
runs than planned is refused however the campaign ended.
Example structure of metadata.yaml:
configurations:
- name: config-1
config:
growth_rate: 0.5
initial_population: 100
config_files: []
created_at: '2026-03-04T16:15:03.212496'
derived_from: config
variations:
- name: FloorplanGeneration
started_at: '2026-03-04T16:14:55.123456+00:00'
duration: 3.217
- name: PathVariationRandom
started_at: '2026-03-04T16:14:58.340789+00:00'
duration: 1.842
test_results:
- dir: config-1/0
success: 'true'
start_time: '2026-03-04T16:16:00+00:00'
end_time: '2026-03-04T16:16:49'
output_files:
- config-1/0/sysinfo.yaml
- config-1/0/logs/system.log
sysinfo: { ... }
postprocessing: {}
metadata: {}
run_files:
- _config/files/growth_sim.py
scenario_file: scenario.osc
execution:
execution_time: '2026-03-04T16:15:02'
robovast_version: abc123
runs: 2
execution_type: cluster
image: ghcr.io/example:latest
The metadata: block of the .vast file is passed through verbatim into
metadata.yaml and is used to configure PROV-O generation (see below).
See Add Metadata Processing Plugin and Add Variation Plugin Metadata Hook for how to add custom metadata processing plugins and variation metadata hooks.
metadata.prov.json — PROV-O Provenance Graph
After metadata.yaml is written, RoboVAST automatically generates a
W3C PROV-O provenance graph as
<campaign-dir>/metadata.prov.json (JSON-LD format) and an optional
metadata.pdf visualization (requires Graphviz dot).
The graph captures the full execution lineage of the campaign as a cyber-physical system test:
Software agents — RoboVAST and Scenery Builder with version info
Campaign activity — execution type, start time, number of runs
Scenario entities — abstract (
.osc) and concrete per-configuration scenariosConfig-generation activity — links the
.vastfile to the generated configsPer-run activities — success/failure, timing, sysinfo, output files
Domain-specific nodes — contributed by variation plugins (e.g. map/mesh entities for navigation, goal counts, obstacle counts); see Add PROV-O Provenance Hook to a Variation Plugin
Configuring the provenance graph
The metadata: section of the .vast file controls campaign-level
provenance properties:
metadata:
dataset_iri: https://purl.org/robovast/datasets/my-dataset/
# Optional: list of CPS agents (robots, manipulators, etc.) involved
# in the campaign. Omit entirely for agent-free campaigns.
agents:
- id: turtlebot4
type: robot
# Where this agent came from. One IRI, or several, or a mapping when
# the source carries a version.
derived_from: https://github.com/turtlebot/turtlebot4/tree/jazzy
# Files that configure this agent, relative to the configuration root.
configuration_files:
- files/nav2_params.yaml
- id: ur5
type: manipulator
derived_from:
- source: https://github.com/UniversalRobots/Universal_Robots_ROS2_Description
version: 2.1.0
dataset_iriBase IRI for the dataset namespace used in the provenance graph. All campaign, config, and run IRIs are constructed relative to this prefix. Defaults to
https://purl.org/robovast/datasets/default/.agentsList of PROV Agent nodes representing the physical systems under test (robots, manipulators, sensors, etc.). Each entry must have an
id(used as the IRI fragment;nameis accepted as a legacy spelling) and may carry arbitrary additional properties, which become properties of the agent node. If omitted, no agent nodes are added to the graph — suitable for software-only or simulation-only campaigns.Two keys are interpreted rather than copied through:
derived_fromWhat the agent was derived from, as
prov:wasDerivedFrom. Write a single IRI, a list of them, or — when the source has a version worth recording — a mapping withsourceand optionalversion. An entry that names no source is skipped with a warning rather than dropping the campaign’s provenance graph.configuration_filesPaths, relative to the configuration root, of the files that configure this agent. Each is matched against the campaign’s recorded
run_filesand aggregated into one plan entity per agent. A path with no match warns and is skipped, since a plan pointing at a file the campaign never carried would misdescribe the run.
Domain-specific provenance nodes (e.g. navigation map/mesh entities) are
contributed automatically by variation plugins that implement
collect_prov_metadata; no manual configuration is required.
Re-running a campaign: the pre-flight
Every re-run — Retrigger campaign in the web UI, vast campaign rerun <id>,
start_campaign(from_campaign=<id>) over MCP, POST /campaigns/<id>/retrigger — is answered
by the service walking the campaign’s records first, and refusing one that cannot work as
recorded: a launch that could only fail in the backend is refused before it starts. Five axes,
which fail independently and are all reported together:
configthe frozen
.vastis readable, and at a version the migration ladder can carry forward.hostthis robovast still speaks the recorded image’s container protocol.
imagesa new run can start from the images the campaign recorded. A container whose image the campaign built cannot be replaced, since the build context is not archived; one it merely declared is resolved again at launch.
pluginsthird-party
plugins:resolved to something re-installable.providerswhich asset-provider distributions supplied the campaign.
Only blocked refuses. unknown does not: a campaign recorded before a given field existed is
exactly what a re-run of an old campaign is, and refusing it for a record nobody wrote would defeat
the purpose. Every blocking verdict names the artifact and how to obtain it.
Read the report without launching anything — it stages nothing and starts no container — with
vast campaign rerun --check <id>, get_campaign_summary’s retrigger key, or
GET /campaigns/<id>/retrigger/check. Override it, for an axis you have decided you understand,
with vast campaign rerun <id> --force, start_campaign(from_campaign=<id>, force=True),
Re-run anyway in the web UI’s dialog, or force on the POST body.
Postprocessing
Postprocessing transforms raw run output (e.g. ROS bags, custom binary files) into
analysis-friendly formats (e.g. CSV). Commands are defined in the
results_processing.postprocessing section of the .vast file and executed by plugins
(see Add Postprocessing Command Plugin for how to write your own).
It runs automatically when a campaign’s runs finish. To run it again — after editing a postprocessing script, or on a campaign imported raw — name the campaign:
vast campaign postprocess CAMPAIGN
Options
- -f, --force
Bypass the per-rosbag caches and reprocess all bags, for example after updating a postprocessing script.
- --skip PLUGIN
Skip a postprocessing plugin (repeatable), e.g.
--skip rosbags_to_webm.
The campaign is the address, so the rosbag→CSV step runs in-cluster, where that campaign’s
runs ran, and the campaign’s derived data is rebuilt. It is dispatched, not awaited: postprocessing can take minutes to hours,
so the campaign re-enters its postprocessing phase and the command returns. Follow it
exactly as after a launch:
vast campaign postprocess my-campaign-2026-03-20-153630
vast campaign wait my-campaign-2026-03-20-153630
Postprocessing is cached by a hash of the results directory; when nothing changed the
step is skipped automatically, which is what --force bypasses.
Its last step loads the derived files into the central index. The index holds nothing that is not rebuilt from the campaign’s files, so the load commits without waiting for each write to reach the database’s disk, and it ends by refreshing the query planner’s statistics on the tables it wrote. A crash of the index’s database server during or just after a load can therefore lose the last rows written; postprocessing the campaign again loads them again.
Note
This was vast results reprocess, beside a vast results postprocess that did the
same work in-process against a local results directory. The second could only ever see
a local campaign, so there is now one postprocessing path, reached one way.
Re-running a finished campaign’s post-run steps
The two steps that run after a campaign’s scenarios finish — analysis
postprocessing and the upload-to-share — can each be re-triggered on a
finished campaign, and each works from the stored campaign alone: no live
campaign process is required, so a re-trigger is available even after the
robovast-service (vast serve) was restarted. Under the web UI’s Monitor
each finished campaign’s actions menu offers Retrigger postprocessing and Export
to share; the same operations are vast share export -i <campaign-id> on the
command line and the MCP tools run_postprocessing and run_share.
A third operation shares their shape without being a re-run: importing a
campaign this deployment never ran, from an archive
(vast campaign import <archive>) or from the share (vast share import
<campaign-id>). It is dispatched the same way, enters the importing phase, and
— when what arrived is a raw archive, carrying no postprocessing record — rolls straight
on into postprocessing, because a campaign without its metric tables is not one
anybody can query. Its per-stage verdicts land in _execution/import.json and its
narrative in _execution/import.log. A degraded import is usable-but-incomplete
rather than a failure.
A genuine failure is kept, as a failed campaign, and the refusal names what was
missing rather than which check noticed. Deleting the half-imported tree was tried and
was strictly worse: registering the campaign is what makes it visible while it arrives,
so the entry outlives the failure either way and removing the directory only took away
the import.log and import.json that explained it. Remove it with
vast campaign delete, or import again with --force.
The mirror of that check runs on the way out: an export refuses a campaign with no
frozen _config/ instead of writing an archive whose only possible future is an ingest
refusal on somebody else’s service, after a full transfer, with the source out of reach.
A re-trigger through the service is dispatched in the background and returns
immediately — postprocessing can take minutes to hours, so the campaign simply
re-enters the postprocessing (or sharing) phase and you follow its progress and
log in the campaign view, exactly like the original run; it returns to finished when
done. The web Retrigger postprocessing dialog therefore closes as soon as you click
Run. A second re-trigger is refused while one is already running. vast campaign
postprocess is the same dispatch from the CLI, so all three surfaces behave alike –
there is no longer a local, synchronous path that behaves differently from the rest.
Because it re-enters that phase as a tracked campaign, a re-trigger can also be stopped
like one: stop cancels it — deleting its Job — the campaign returns
to finished, and postprocessing_error says it was cancelled rather than that it
failed. Nothing that was already derived is lost, and nothing claims to be derived that is
not: an interrupted bag is redone next time, a partial index load is replaced rather than
doubled, and the provenance record that says a campaign carries derived data is written
last, so a cancelled campaign simply reads as not postprocessed (see Client–Server Architecture).
Ask for it again whenever the derived data is wanted.
Because a post-run step is separate from the runs themselves, a failure of one of
these steps does not fail the campaign. The campaign stays finished (its runs
are the deliverable and remain downloadable) and the failure is recorded on its own
field — postprocessing_error or share_error — which is durable (it survives a
service restart) and is surfaced as a warning badge in the UI. Re-running the step
successfully clears it. This is distinct from a run failure, which reports the
campaign phase as failed.
Editing the postprocessing parameters before re-running overwrites the
``results_processing.postprocessing`` block of the campaign’s own ``_config/<name>.vast``
in place — it is config, not captured data, so there are no override files or
revisions, and the raw rosbags and the as-ran configuration/execution are left
untouched. The edited config applies on both the local and the cluster backend. For the
upload-to-share, the target provider is taken from the service environment
(ROBOVAST_SHARE_TYPE and its credentials); adjust it and export again to upload the
same campaign to a different provider. The archive is named for what it now is —
a campaign exported after postprocessing goes up as .postprocessed.tar.gz, one
exported before it as .raw.tar.gz — and nobody passes that in: it is read off
postprocessing’s own provenance record (_transient/postprocessing.yaml), so the
campaign-end upload and a later export agree by
construction.
Custom postprocessing plugins that need third-party Python packages — an
entry-point postprocessing command, or the dependencies a local
./file.py:Class plugin imports — declare those packages in the .vast’s
top-level plugins: list (see Configuration). They are installed into the
campaign’s .robovast_plugins/ and put on sys.path before postprocessing
runs, including on a re-run in a fresh process.
Publishing Results
Publication packages or distributes the results directory using plugins defined
in the results_processing.publication section of the .vast file. Unlike
postprocessing (which operates per campaign run folder), publication plugins
receive the full results directory as input and are intended for tasks like
creating zip archives for upload or hand-off.
vast results publish [OPTIONS]
Options
- -r, --results-dir PATH
Directory containing the run results (parent of campaign directories). Required: there is no project file to take it from.
- -o, --override VAST_FILE
Use the given
.vastfile instead of the one stored in<campaign-name>-<timestamp>/_config/.
- -f, --force
Overwrite existing output files (e.g. zip archives) without prompting. Equivalent to setting
overwrite: trueon every publication plugin. Without this flag, plugins that find an existing output file will ask the user interactively (default answer: yes / overwrite).
Example:
# Publish using the project-configured results directory
vast results publish
# Publish and overwrite any existing archives without prompting
vast results publish --force
# Publish a specific results directory with an override config
vast results publish --results-dir /path/to/results --override my_project.vast
Listing Publication Plugins
vast results publish-commands
Lists all available publication plugins, their descriptions, and parameters.
Useful for discovering which plugins can be used in the
results_processing.publication section of the .vast file.
Merging Results
vast results merge-campaigns MERGED_CAMPAIGN_DIR [OPTIONS]
Merges campaign-directories with identical configs into one merged_campaign_dir.
Groups campaign-directory/config-directory by config_identifier from config.yaml.
Run folders (0, 1, 2, …) from all campaigns are renumbered and copied.
Original campaign-directories are not modified.
Arguments
MERGED_CAMPAIGN_DIRTarget directory where the merged campaign will be written.
Options
- -r, --results-dir PATH
Source directory containing campaign directories. When omitted the value required: there is no project file to take it from.
Listing Postprocessing Plugins
vast results postprocess-commands
Lists all available postprocessing command plugins, their descriptions, and
parameters. Useful for discovering which commands can be used in the
results_processing.postprocessing section of the .vast file.
Using --override to Supply a Local .vast File
By default vast results publish reads the .vast configuration from the
campaign snapshot stored in
<results-dir>/<campaign-name>-<timestamp>/_config/<name>.vast. This snapshot is copied
at execution time and may be out of date.
--override (short form -o) lets you point to any .vast file on disk,
for example your current working copy:
# Use a local/updated .vast file
vast results publish --override my_project.vast
When to use ``–override``
You want to apply updated postprocessing scripts to existing results without triggering a new execution campaign.
The results were produced in a different directory and the campaign snapshot points to stale paths.
You want to bypass the snapshot and always use the latest
.vastduring iterative postprocessing development.
Note
When --override is supplied, the same .vast file is used for
every campaign folder found under the results directory. The
config directory of the override file (its parent folder) is used to
resolve relative paths.