Analysis Notebooks

An analysis notebook is a plain Jupyter notebook a campaign declares under visualization.results.explorer.notebooks. The web UI Results Explorer executes one per selected tree node — campaign, batch, configuration or run — server-side, and renders it as HTML.

This page is about writing those notebooks and about the analysis library they read the campaign’s data with. Where they appear is the Explorer; see Web UI.

Writing Evaluation Notebooks

Notebooks are plain Jupyter .ipynb files referenced from the visualization.results.explorer.notebooks section of the .vast file:

visualization:
  results:
    explorer:
      notebooks:
        - MyAnalysis:
            run: analysis/analysis_run.ipynb
            config: analysis/analysis_config.ipynb
            campaign: analysis/analysis_campaign.ipynb

There are four reserved scopes, and a notebook declared under any other key is refused – the renderer keeps only the scopes it can address, so a misspelled one would otherwise leave the notebook staged and never rendered, with no tab and nothing said:

  • run – executed once per individual run directory (<campaign-name>-<timestamp>/<config>/<run-number>/).

  • config – executed once per configuration directory (<campaign-name>-<timestamp>/<config>/).

  • batch – executed once per proposing round of a search campaign. A batch has no directory of its own, so the notebook receives the campaign root as DATA_DIR plus an injected BATCH index; the tab appears only for a search campaign.

  • campaign – executed once per campaign directory (<campaign-name>-<timestamp>/).

Every declared path is relative to the .vast and must exist in the project that is pushed. vast config validate reports one that does not: staging skips it with a warning, and the campaign then runs to completion with a tab that cannot render.

The workload name (MyAnalysis above) becomes the label of its tab in the Explorer, and the way a link addresses that tab (?tab=MyAnalysis). One name is taken: a workload may not be called log, in any casing. The Explorer already appends a built-in Log tab to every run, so a second tab of that name would read the same and say nothing about which is which; the .vast is rejected rather than the tab bar growing an ambiguity.

The only hard requirement is that every notebook contains the line:

DATA_DIR = ''

When the GUI executes a notebook it replaces this line with the actual path for the currently selected item. The output is cached so subsequent views are instant.

Self-Contained Evaluation Notebooks

The self-contained pattern extends the basic requirement above: the notebook is written so it can be opened and executed directly in VS Code or JupyterLab (i.e. without the GUI) by setting DATA_DIR to a real path, while still remaining fully compatible with the GUI.

The approach

Set DATA_DIR to a real results directory in the very first code cell:

# Self-contained: set DATA_DIR to a real path during development.
# The RoboVAST GUI replaces this line at runtime.
DATA_DIR = '/path/to/results/dynamic_obstacle-2026-03-04-132444/my-config-1/'

When the GUI runs the notebook it replaces the entire DATA_DIR = ... line, so the hardcoded path is never used in production.

Reading results

read_table reads one of the campaign’s results-index tables and restricts it to what ``DATA_DIR`` selects — a run directory gives that run’s rows, a configuration directory that configuration’s, the campaign root everything. The same cell therefore serves all three notebook scopes, and no notebook names a file.

read_table(DATA_DIR, "behaviors")            # the scenario's behaviour tree
read_table(DATA_DIR, "poses", columns=["timestamp", "position.x", "position.y"])
read_table(DATA_DIR, "poses", where="frame = ?", params=("base_link",))
read_runs(DATA_DIR)                          # per-run outcome + each param_* column
list_tables(DATA_DIR)                        # what this campaign actually has

Tables are keyed (config_name, run_id); runs carries the same key, so joining it to a metric table relates what varied to what happened. read_sql(DATA_DIR, ...) is the escape hatch for joins and for the run_view / config_view views — it is deliberately not scoped.

What a metric varied with is usually the question, so with_params=True attaches each parameter of the owning run as a column, with the param_ prefix dropped (attach_params() does the same to a frame you already have). A parameter whose name collides with a column of the table raises rather than shadowing it:

df = read_table(DATA_DIR, "poses", with_params=True)
df["map_file"]        # the parameter, beside the measurements it belongs to

A parameter that names a file — map_file, mesh_file — holds a path relative to the campaign’s _config/. Resolve it with config_file rather than joining it onto DATA_DIR, which is only the campaign root at campaign scope:

config_file(DATA_DIR, df["map_file"].iloc[0])            # raises if it is not there
config_file(DATA_DIR, rel, config_name, must_exist=False)  # to test existence yourself

Which tables exist depends on the campaign. runs, behaviors, run_log, resource_usage and scenario_timestamps are produced whatever the simulator and whether or not the run used ROS. poses, costmaps, action_* and the nav2_* tables come from a rosbag, so a mode: base campaign has none of them. Ask list_tables or table_info rather than assuming, and name the columns an analysis cannot do without:

# Raises, naming the missing column and what the table does have, rather than
# returning a frame that is quietly missing it.
read_table(DATA_DIR, "behaviors", require=["status_name", "tip_id"])

Reading a search’s own record

read_table and friends read measurements, which postprocessing builds. A search campaign also keeps a record of what it proposed: one row per evaluated configuration with its objectives, its quality-diversity measures and the parameters it was drawn with, grouped into the batches that proposed them. That record is written as the search runs, so open_campaign_store() reads it without any postprocessed data — a batch or archive view therefore works on a campaign still in progress, and on one that was never postprocessed.

from robovast.common.analysis import open_campaign_store

store = open_campaign_store(DATA_DIR)          # campaign.db alone, read-only
store.row_factory = sqlite3.Row
store.execute("SELECT idx FROM batch ORDER BY idx")
store.execute("SELECT config_name, objective, measures_json FROM unit WHERE batch_id = ?", ...)

Tables are unqualified here (batch, unit, run, campaign, job); the same tables appear under a campaign. prefix when open_campaign_db() attaches them beside the metrics. objectives_json and measures_json exist only on unit — runs.objective lifts just the single scalar — so a multi-objective or quality-diversity campaign can only be read this way.

Three unit statuses are not results and must not be averaged over: composition_failed is a draw that could not be built into a configuration at all and never ran, missing is a configuration the campaign was composed with whose results never reached the tree, and no_sample is one that ran and lost every run to infrastructure. All three are coverage that was not obtained; count them rather than dropping them, or the campaign reads as having explored more than it did – and the first two carry no runs at all, so a query that joins through run drops them without saying so.

Reading requires postprocessing to have run — the campaign is not in the index before it, and a campaign whose postprocessing failed still reports finished. That case raises with the remedy in the message rather than falling back to the per-run files, which would answer a different question with less data. read_run_statuses() reads test.xml directly and so still works when there is no database at all.

The per-run file readers (robovast.common.analysis.files) remain available for what is genuinely not in the database.

Handling missing columns defensively

When developing against a specific dataset, guard against unexpected DataFrame schemas so the notebook fails clearly rather than with a cryptic KeyError:

required_cols = {'run', 'config', 'timestamp', 'frame'}
missing = required_cols - set(df.columns)
if missing:
    raise ValueError(f"DataFrame is missing expected columns: {missing}. "
                     f"Available: {list(df.columns)}")

Scoping DATA_DIR per notebook type

Use paths appropriate to the scope of the notebook:

Scope

Example DATA_DIR

What read_table returns

run

/<campaign-name>-<timestamp>/<config>/<run-number>/

that run’s rows

config

/<campaign-name>-<timestamp>/<config>/

every run of that configuration

campaign

/<campaign-name>-<timestamp>/

every run of every configuration

Note

The scope changes which rows come back, not which columns: every metric table carries config_name and run_id at all three levels, so a cell written for one scope runs unchanged at another. At run scope both columns hold a single value — grouping by them is redundant there but not an error, which is what lets the same cell serve every scope.

Benefits of the self-contained pattern

  • Interactive development: run all cells with Run All in VS Code without launching the GUI.

  • No context switching: tweak a visualization, re-run, inspect – all in one editor window.

  • GUI-compatible: the notebook works unchanged in the GUI; the hardcoded path is simply overwritten at runtime.

  • Reproducible: the path embedded in DATA_DIR documents which dataset the notebook was last developed against.

Typical development workflow

  1. Run an execution campaign to produce results.

  2. Open the relevant .ipynb file in VS Code.

  3. Set DATA_DIR to the actual campaign/config/run directory.

  4. Develop and iterate with Run All (or cell-by-cell).

  5. Once satisfied, commit the notebook. The GUI will use it via the visualization.results.explorer.notebooks section of the .vast file; DATA_DIR will be replaced automatically.

  6. To share the notebook with colleagues working on the same dataset, leave the real DATA_DIR value in place – they only need to update the path.