Replay artifacts: the scene and the capture
The run view’s 3D panel replays a run from two artifacts, and from nothing else — no ROS, no rosbag, no postprocessing, no results tables. Both are formats RoboVAST defines and a simulator may produce; this page is the contract a producer implements against.
artifact |
what it carries |
when |
who can produce it |
|---|---|---|---|
scene |
geometry: a body tree with rest transforms, named joints, geoms, materials, textures, skins, an initial camera |
static, per world |
any tool that can read the world — it need not be the simulator that ran |
capture |
motion: a time base plus named joint-value and body-pose tracks |
per run |
only the simulator that ran |
That split is what lets a second simulator be admitted later. Geometry is world-authored, so it can be compiled offline from an SDF, a USD, an MJCF or a floorplan by whatever tool reads that format; only the motion is a property of the execution.
Producing them is an optional capability. A simulator that emits both gets a replay; one that does
not simply has no scene3d panel, exactly as a Gazebo campaign has none today. The dependency is
declared per campaign and fails by absence — which is visible — rather than through a fallback that
renders something misleading.
roqsim is the first producer of both: roqsim export web for the scene, roqsim export capture (or its
scenario adapter, at shutdown) for the capture. The reference readers are
frontend/ui/src/lib/scene3d/sceneLoader.ts and frontend/ui/src/lib/scene3d/runCapture.ts.
When each is produced. A capture is written by the run, because only the simulator that ran can
write it. A scene descriptor is not written by the run: it is a function of the world, so the service
compiles it on first view — inside the campaign’s own pinned image — and caches it by world identity
(image digest + world + overrides), shared across every run and every campaign that used that
world. That is why the capture’s world identity is a contract and not decoration: it is the input to
obtaining geometry. See its delivery section.
The scene descriptor
scene.json + scene.bin + one tex_<i>.png per image texture, in one directory: the loader
fetches the binary and the textures as relative siblings of scene.json, which is why the file
address space preserves path segments.
The format is defined by its producer, roqsim/export_web.py, and its full field list lives there. Two
notes matter to a second producer:
It is a plain scene graph — bodies with rest transforms and a parent index, named joints carrying
type/axis/pos, geoms as primitives or indexed meshes — and nothing in it is MuJoCo-specific in substance.joints[].qposadris optional and legacy. It names an index into MuJoCo’s state vector, and the reference loader has never read it: animation is addressed entirely by joint name. A new producer should omit it.
The run capture
Two files in one directory, the same sibling convention:
<run>/capture/capture.json manifest: time base + track index + provenance
<run>/capture/capture.bin float32/float64 tracks, addressed by byte offset
A capture is per run — it describes one execution — so it is always addressed in the run’s file space, never the campaign’s.
The manifest
{
"format": "robovast.run_capture",
"version": 2,
"complete": true,
"frame": "world",
"time": {"base": "sim", "t0": 0.002, "t1": 29.324,
"off": 0, "dtype": "f8", "samples": 734, "width": 1},
"producer": "roqsim",
"producer_version": "<version>",
"world": "tiago_pick:tiago_pick",
"overrides": {},
"packages": {"roqsim": "<version>", "mujoco": "<version>", "numpy": "<version>"},
"seed": 1869948900,
"tracks": [
{"kind": "joint", "name": "arm_1_joint", "unit": "rad",
"width": 1, "samples": 734, "dtype": "f4", "off": 5872},
{"kind": "pose", "name": "base_footprint",
"width": 7, "samples": 734, "dtype": "f4", "off": 114576}
]
}
format and version are required; a reader refuses an unknown format, and a newer version by
name rather than guessing at it. Guessing would render something plausible and wrong, which is the
failure this whole design exists to avoid. What the versions are, and what a consumer owes them, is
below.
frame is the frame pose tracks are expressed in. It must match the scene’s geometry — for roqsim that is
the simulator’s world frame. This is not bookkeeping: a nav stack’s base_link lives in a map
frame that can be meters from the world origin, and no reader can tell the two apart from the numbers.
In configs/examples/basic_nav the offset is 8 m, so getting it wrong draws the robot in the wrong
room.
time.base is sim (simulated seconds from the run’s start) or wall. Declaring it is what lets a
capture-driven panel and a rosbag-driven panel be told whether they share a clock.
world + overrides are the world identity, not decoration: together they name the world this
motion belongs to, precisely enough for a consumer to obtain the matching geometry — which is how the
run view gets a scene descriptor without the campaign declaring one. A producer must therefore write the
overrides it actually built with.
overrides distinguishes three states, and conflating two of them renders confidently wrong geometry:
value |
meaning |
|---|---|
|
these overrides were applied; compile the world with them |
|
none were applied; compile the bare world |
key absent |
unknown — this producer did not record them. A consumer must not read this as |
producer, producer_version, packages and seed are provenance. A reader shows world and
producer when track names fail to resolve, because “this capture was recorded against a different
world” is the diagnosis in nearly every such case. packages carries the producer’s own library
versions (for roqsim: roqsim/mujoco/numpy), so a format or geometry mismatch across a version bump
is legible rather than mysterious. Each is whatever the image that recorded the run held, which is why
the sample above writes them <version> where every other field carries a real value; version
at the top is not one of them – that is the format’s own number and a reader refuses an unknown one.
seed may be null when the producer had none.
Format versions
version |
what changed |
|---|---|
1 |
The original. |
2 |
|
That last point is the one to carry into any future version, because it is what makes a version more than a label. Two rules follow from it:
Refuse a version you do not implement, by name — and say what to do about it. The fix is always the same one and it is never the capture’s: the reader has to learn the version. A message that suggests re-exporting with an older producer asks for something nobody can do.
Key any cache on the version, wherever you key it on a field the version redefines. The same
overrides document names a different world under v1 than under v2, so a cache keyed on the document
alone serves geometry compiled for something else — and it looks perfectly fine. RoboVAST does this in
scene_cache.cache_key; the rule exists because the failure is invisible, not because the cache is
special.
Adding a version
Add a row above, bump FORMAT_VERSION in robovast/common/run_capture.py and CAPTURE_VERSION
in frontend/ui/src/lib/scene3d/runCapture.ts — the second only if the change touches something
that reader reads, since the two consumers read different halves of the manifest and may legitimately
sit at different versions (the lower one refuses, which is degraded but honest). Then ask the question
that decides whether anything else is needed: does the new version change the meaning of a field a
consumer keys an identity on? If so it must reach that key, as v2’s does.
A purely additive change needs no version at all. A new track kind is skipped by a reader that
does not know it (see below), so bumping for one would refuse captures an
older viewer replays perfectly well.
There is deliberately no migration machinery. Nothing can rewrite a v1 override document into a v2 one from the document alone: telling a config key from a child component requires resolving against the world, which lives in the producer’s image. Nor is it needed — geometry is compiled in the campaign’s own pinned image, by the producer that wrote the capture, so an old document is always resolved by the code whose convention it used.
Tracks
Two kinds, which are exactly the two things a scene graph can be driven by:
kind |
width |
payload |
|---|---|---|
|
1 |
one scalar per sample, in the joint’s own unit ( |
|
7 |
one world-frame pose per sample — |
Names are the entire addressing scheme: no indices cross the interface. A producer resolves names
from its own model, which is why qpos and qposadr cannot appear here — no other simulator has a
MuJoCo state vector. The test to apply to any future field is “could this be written from
/joint_states and /tf?”
Rules a producer must hold
Emit a joint track for every joint whose value moves, and a pose track only for a body a reader could not otherwise place. A body is placeable when its parent is and its own joints are all named revolute/prismatic ones and it is not a mocap body — recursively, from the world body down. So a link welded to a moving parent, or hanging off it through joints, needs no pose track: the joint tracks and the geometry’s rest transforms already determine it. Emitting one anyway is not merely wasteful, it is two writers for one transform, and which wins is then an accident of ordering. (roqsim’s exporter got this wrong first time by testing only whether a body owns a joint, which produced 36 pose tracks instead of 2 on a real recording.)
Order pose tracks parents-first. A reader turns a world pose into a transform relative to the parent, so the parent must already be seated within the same frame.
Store each track sample-major — all of sample 0’s values, then sample 1’s. This is what makes a time window a contiguous byte range, so a reader can range-read the part it is showing rather than the whole file. Nothing needs that yet (a 30 s capture of a mobile manipulator is ~150 KiB), but it is the property that keeps windowing an implementation change rather than a format change.
Align every offset to its dtype. A browser’s new Float64Array(buffer, off, n) throws unless
off is a multiple of 8. A producer that aligned everything to 4 would work for most captures and fail
for some, which is the worst of both.
Use float64 for the time track (dtype: "f8") and float32 for values. A float32 second degrades
with magnitude, and timestamps are exactly where that shows.
Samples are in strictly increasing time order, and a track sharing the manifest’s time base has one value per time sample.
Reading it
A sample is addressed by time, never by index: nearest sample, ties to the earlier one, and never interpolated — blending two states produces a pose the simulation never had. Values are pushed into a sink rather than returned, so a 60 Hz redraw of a few dozen tracks allocates nothing.
A reader reports track names that resolve to no joint or body in the scene, with the manifest’s world
and producer, rather than replaying a partly-static world in silence.
Shaped for a live source
The capture is defined as a motion source — a time base plus named tracks — of which these two files
are one serialization. A live source is another, so following a running simulation lands as a second
implementation of MotionSource (frontend/ui/src/lib/scene3d/motionSource.ts) with no change to the panel
and none to this format. Five rules exist for that, each cheap now and expensive to retrofit:
t1may be absent andcompletemay befalsewhen the upper bound can still move. A consumer re-reads the range instead of caching it. A file is complete by construction.The track set may grow. A live source gains a track when a robot spawns or a pedestrian appears. A file’s never does — but a consumer that assumed otherwise would have to be rewritten rather than extended.
Samples are addressed by time, as above. For a live source “nearest” is simply “latest”.
The timebase is declared, so a live source can be told whether it shares a clock with other panels.
Reads are windowed (
fetch(t0, t1)), because a live source is inherently windowed. The file reader satisfies any window from the buffer it already has.
Every update reaches a consumer through subscribe, and the file reader fires it once for data that is
already loaded. An interface method nothing calls is a guess; one the only shipping implementation uses
is a contract.
Deliberately not specified until something produces and consumes them, because inventing a format
for an absent consumer is how formats rot: array tracks (width > 1, for a lidar scan) and a media
index (for camera video). Both are additive — a reader skips a track kind it does not know, so an
older viewer keeps replaying the motion it does understand.
Producing a capture with roqsim
Two ways, one implementation, so they cannot drift.
During a campaign run. The scenario adapter records while the run proceeds and derives the capture at shutdown, against the still-live model — no world rebuild, and no GL backend in the run container:
execution:
simulation: roqsim.scenario_adapter:MujocoSim
env:
- ROQSIM_WORLD: "/config/files/depot.yaml"
- ROQSIM_RECORD: "run.npz" # the recording (the run's ground truth)
- ROQSIM_CAPTURE_EXPORT_DIR: "capture" # -> <run dir>/capture/capture.json
# - ROQSIM_CAPTURE_FPS: "25" # optional; samples per *simulated* second
visualization:
results:
run_view:
panels:
- scene3d:
position: { fill: true }
scene: { path: _config/files/scene/scene.json, scope: campaign }
capture: { path: capture/capture.json }
vast config validate refuses a scene3d panel whose campaign never asks its runs to record: the
campaign would otherwise run, pass, and show a world that never moves.
After the fact, from a recording that already exists — which is how a different capture is produced without re-running anything:
roqsim export capture --state run.npz --out capture/
run.npz keeps its own role either way: it is roqsim’s ground truth, feeding roqsim render --state and
roqsim state --state, and everything else is re-derivable from it — the same relationship a ROS
campaign has with its rosbags.
Note
Both artifacts are written on a clean stop — the scenario ending, the adapter’s shutdown, a
closed viewer, one Ctrl+C, or a SIGTERM. A producer is expected to treat SIGTERM as a clean stop,
because under a campaign it is the usual exit: a container teardown, an eviction and a per-run
timeout all send it, and its default action would end the process with no flush at all. Only
SIGKILL leaves neither artifact, and the panel says so instead of showing an empty scene.
Adding a producer
For a simulator that is not roqsim:
Geometry. Either emit the scene descriptor directly, or compile it offline from the world the simulator ran — for an SDF world,
roqsim scenes sdf-to-scene→scene-to-mjcf→roqsim export webalready does this, and a campaign can run it as anexecution.generatestep so the descriptor is a frozen campaign input with a freshness manifest (see its delivery section).Motion. Write
capture.json+capture.binper run, holding to the rules above, and declare the currentversion(Format versions) — including addressingoverridesby path. From ROS data the tracks are/joint_statesfor the joint tracks and/tffor the pose tracks — with the caveat that the poses must be in the scene’s frame, not a map frame.Declare it. Add a
scene3dpanel withscene:andcapture:paths. Nothing else in the campaign changes, and no panel code is involved.