RoboVAST
Variation Automation and Scalable Testing for Robotic Systems
RoboVAST is an open-source framework for automated, large-scale integration testing of robotic software in simulated environments. Built upon proven foundations including the Floorplan-DSL for parameterizable indoor environment generation, scenario-execution for single run execution, and Kubernetes for orchestration, RoboVAST enables developers to systematically validate their systems across thousands of varied run scenarios.
Mobile robot navigation serves as our primary use case, focusing on indoor environments where robots must navigate with varying layouts, tasks, obstacles, navigation parameters and environmental conditions. This foundational application demonstrates RoboVAST’s core capabilities while providing immediate value to the mobile robotics community.
RoboVAST provides a comprehensive dataset designed to test multiple aspects of mobile robot software like Nav2, including localization, path planning, obstacle avoidance, and dynamic re-planning. This reference dataset can be used out-of-the-box or adapted to specific user requirements, significantly lowering the barrier to robust robotics testing.
Framework Architecture
Variation
RoboVAST combines multiple variation dimensions to generate comprehensive run suites. Environment generation uses the Floorplan-DSL to describe and generate diverse 3D indoor environments with parametric variation of room dimensions and connectivity. For systematic variation, it combines parameters specific to the use case, such as start/end poses, obstacle configurations, and sensor noise for mobile robot navigation. The modular architecture supports extensible addition of new variation dimensions based on specific application requirements.
Execution
RoboVAST orchestrates run execution by creating Kubernetes jobs that run individual robot simulations using scenario-execution, including screen capturing and ROS bag data collection. The platform handles parallel deployment across available cluster nodes along with required input and output data management. This architecture enables execution of multiple runs in parallel depending on cluster size, significantly reducing validation time while ensuring reproducible execution across distributed computational environments.
Result Processing
To support processing of campaign results, the framework provides tools with automated postprocessing, trajectory visualization, and log analysis of each campaign. These tools enable fundamental data extraction and organization for downstream analysis.
Evaluation
For evaluation of results, users can implement custom evaluation workflows tailored to their specific requirements. The framework includes examples for common evaluation patterns, enabling domain-specific evaluation using preferred analysis tools and methodologies.
Mobile Robot Reference Dataset
The RoboVAST dataset for mobile robots comprises thousands of mobile robot navigation tests conducted in Gazebo with ROS2 across diverse indoor environments and conditions.
The dataset serves as a comprehensive validation tool for navigation stacks such as Nav2, enabling developers to identify fundamental issues such as incorrect parameterization, setup problems, or software bugs. By testing their Nav2 configuration against the reference dataset, users can quickly assess system correctness and evaluate overall performance characteristics.
Contents:
- Quickstart
- Using the client
- Setup
- How to run
- Configuration
- Container images
- Simulators
- Example
- Variation Points
- Cluster Execution
- Overview
- Prerequisites
- Cluster Setup
- Running Scenarios
- Monitoring and Results
- Experiment image builds (registry)
- Where the registry and the index run
- Where a campaign lives, and how pods reach it
- Job Queueing
- How free capacity is measured, and why it is not the whole cluster
- Which image bytes a run uses
- Waiting, blocked, and merely busy
- Selecting a Cluster Context
- Per-Cluster Resource Limits
- Cloud Provider Configurations
- Reaching it over a tailnet instead of publishing it
- Sharing Results
- Scheduling reference
- Results Processing
- Output Structure
- Which channel a factor was written on, and its column
- A measurement with no finite value
- Reading these files
metadata.yaml— Campaign Metadatametadata.prov.json— PROV-O Provenance Graph- Re-running a campaign: the pre-flight
- Postprocessing
- Publishing Results
- Listing Publication Plugins
- Merging Results
- Listing Postprocessing Plugins
- Using
--overrideto Supply a Local.vastFile
- Search Page
- Analysis Notebooks
- MCP Server
- Overview
- Claude Code plugin
- Tool Taxonomy
- What each check can and cannot settle
- What the documentation corpus covers
- Using it: search an identifier, not a word
- One tool per question, not per shape of answer
- Reading results: SQL, not a tool per scope
- Files
- Where a link comes from
- Campaign control
- Is it working, or is it wedged?
- Building experiment images
- Asking what a world offers an override
- Testing a container and its setup
- What was called, and what it answered
- Available Tools
- Client–Server Architecture
- One interface, one implementation, many clients
- Four distributions, layered by audience
- Where the driver runs
- One container plan, built once
- Asking the simulator a question
- Workspaces vs. campaigns
- The service implementation is resolved, not imported
- An implementation refuses by name
- One base, the implementation’s hooks
- Container exec: a diagnostic that cannot become a run
- One address space for files
- The data plane: tar streams, in their own process
- Token-efficient file transfer
- Data flow and result access
- What goes in the database, and what stays a file
- Querying results
- Run view (web panel framework)
- The interface surface
- Why images resolve the way they do
- HTTP API
- Deployment Modes and Access
- Web UI
- The Admin page
- Staying up to date
- Being told what happened
- Bringing a campaign in
- Importing from the share
- Where a campaign came from
- The Details panel
- What a campaign declares, and where it lands
- Config editor
- Starting it
- Accessing it —
vast ui - Results viewer
- Reading the configuration a campaign ran
- Run view
- Development
- Replay artifacts: the scene and the capture
- Developer Guide
- Test your Robotic Software with RoboVAST
- Working across the distributions
- Releasing to PyPI
- Container Image Compatibility Version
- Extending RoboVAST
- Campaign Store and Results Indexing
- Campaign state and control
- Per-Cluster Resource Resolution
- Job admission internals
- Web UI internals
- Analysis postprocessing: local vs cluster
- Querying RoboVAST campaigns
- Future Work
- Visual artifact egress over MCP (video and rasters)
- A freshness contract for locally-derived campaign inputs
- Server-rendered figures (optional)
- Refocusing the MCP interface for a cluster-first service
- Scheduling: what admission still does not do
- Per-job GPU usage, and why device-wide sampling is not it