Skip to main content

🏗️ Core System Design

The system is composed of 11 ROS 2 packages and 2 standalone Python packages that provide configuration tooling and the visualization dashboard. Together, these components form a modular pipeline for LiDAR evaluation, where data flows through clearly separated stages of ingestion, processing, spatial filtering, visualization, and reporting.

The architecture diagram below illustrates the overall system design and how data moves across each submodule, from raw sensor input through to final evaluation outputs.

🧠 System Architecture Diagram​

1. Configuration & Setup

Configuration Files (one pair per run)

📄environment_configs/*.yaml
(zones, obstacles, world offsets, bags)
📄lidar_configs/*.yaml
(TF frame, topic, mount pose, sweeps)

Standalone Tool (not a node)

🖥️polysetup
(CLI, run via just recipes)
generates the URDF through lidar_zones · every path derived from --src-dir

Generated Artifacts

📦bench URDF
⚙️per-node parameter files + bringup launch
just setup-ws <environment> <lidar> → resolve config pair → generate → colcon build
generated configs▼

2. System Bringup

🚀lidar_test_bench_bringup
(ROS 2 launch entry point — just launch-bench)
launch file is regenerated per run; the recording chain is included or commented out to match the current mode
nodes running▼

3. Run Mode — just start-run

Mode A — Record new bags

just enable-bag-recording · hardware attached

🔁lidar_automation_manager
entry: /lidar_test_bench_initiate (Trigger)
  1. Cycle the vendor driver, apply each parameter under test
  2. Command the mount angle, let the scene settle
  3. Record one bag per case, straight into its case folder
  4. When every case is captured → /start_evaluation
🔌arduino_eth_bridge
(servo / angle commands over TCP)
🛰️LiDAR + vendor driver
⚙️Servo mount

Mode B — Replay existing bags

just disable-bag-recording · no hardware needed

🛠️Recording chain left out of the launch
  1. No driver, no automation manager, no actuators
  2. start-run calls /start_evaluation directly
  3. Evaluates bags recorded on an earlier run
the fast path for iterating on zones or metrics
both modes converge on the same entry point
evaluation started▼

4. Runtime Pipeline — ROS 2 Core Architecture

① Establish ground truth

📡lidar_bench_tf_broadcaster
(lidar_transforms)
publishes the bench frames and the robot description
🔻zones_orchestrator_node
(lidar_zones)
resolves each zone against those frames into baseline profiles

the bench frames come from the generated URDF, so the expected answer is known before any data arrives

expected geometry▼

The evaluation loop — clockwise, once per scan, then once per case

🧭② eval_framework_manager_node
(lidar_eval_orchestrator)
walks the case tree, replays each bag or takes live scans, and drives the loop
▶one raw scan
✂️③ pointcloud_filter_node
(lidar_pointcloud_filter)
cuts the scan two ways per zone — inside its volume, and along the rays aimed at it
▼points per zone
📐④ LidarMetricsEngine
(lidar_metrics_library — pure Python, not a node)
accumulates into each metric plugin, then reduces once the bag ends
◀when the bag ends
📈⑤ metrics_reporting_node
(lidar_reporting)
collects the case report, its visualization data and its recorded bag
▲next case in the sweep

↻ inner loop

② → ③ → ④ repeats for every scan in the bag, each pass measured against the same baseline profiles

Alongside the loop — live inspection, not part of the cycle

🪟visualizer_node
(lidar_reporting)
draws fitted planes as they are computed
zone & plane overlays → RVizmount angle commands → servolive scan inspection → Foxglove

Interfaces (lidar_test_bench_interfaces)

  • FilterCloud (srv)
  • GetProfiles (srv)
  • Visualization (srv + msg)
  • ExpectedZone (msg)
  • Plane3D (msg)
  • NumericalPointCloud (msg)
  • Point4D (msg)
finished run▼

5. Outputs & Reporting

🗃️lidar_eval_backends
(swappable persistence layer)
one interface, read + write · backend chosen by registry · pluggable credentials
📑Google Sheets + Drive
(the backend shipping today)
metrics tree · per-case visualization · recorded bags
📊polyview_app
(Streamlit dashboard — standalone)
reads results back through the same interface
adding another store = implement the interface + enable it in the registry; neither the reporting node nor the dashboard changes

Legend

Configuration input Standalone tool (not a node) ROS 2 runtime node Generated / internal Output / storage Physical hardware
Data flow Service interface Optional / external

📦 Repository Package Structure

The framework is organized into modular ROS 2 packages, each responsible for a specific stage of the LiDAR evaluation pipeline. This separation allows configuration management, sensor processing, evaluation, reporting, and hardware integration to evolve independently while maintaining a consistent data flow across the system.

ROS 2 Core Packages​

lidar_test_bench_bringup​

The primary entry point for launching the LiDAR evaluation bench. This package initializes the required ROS 2 nodes, loads global runtime parameters, and handles the launch coordination for the complete evaluation pipeline. Its launch file is regenerated by polysetup for each run.

lidar_test_bench_interfaces​

Defines the custom ROS 2 interfaces used throughout the framework. This package contains the custom topics, services, and action definitions required for communication between the orchestrator, filtering nodes, metrics backend, and reporting components.

lidar_eval_orchestrator​

The central orchestration package responsible for managing evaluation runs. It acts as the state machine for the bench, coordinating the lifecycle of data playback, point cloud processing, metric collection, and reporting workflows.

lidar_metrics_library​

A modular, plugin-based math and analysis library that computes sensor performance metrics (such as range error and depth accuracy) from filtered point cloud populations. It exposes an extensible API, allowing developers to implement custom metrics tailored to specific evaluation criteria.

lidar_zones​

Owns everything about zone geometry. It consumes YAML zone definitions, resolves them into geometric bounds against the TF tree, and serves the resulting baseline profiles — the ground truth every metric is scored against — to the rest of the pipeline, along with zone overlays for RViz.

Each zone geometry is a plugin selected by registry, so a new shape is added by writing one plugin rather than by editing the engine or its consumers. The package also generates the bench URDF consumed by lidar_transforms; that generation is invoked by polysetup at configuration time, not at runtime.

lidar_transforms​

Publishes the static structure of the bench. It holds the generated lidar_bench.urdf, brings up robot_state_publisher to publish it as the robot description, and runs a TF broadcaster that supplies the bench frames — cart, mount, sensor, and zone links — that zone resolution and metric evaluation are computed against.

lidar_automation_manager​

Provides automation utilities for high-level evaluation workflows, including automated rosbag recording, hardware rigging controls, and batch experiment management to ensure repeatable testing scenarios.

lidar_reporting​

Provides the reporting pipeline for aggregating and publishing evaluation results. It reads the per-case reports produced during a run, assembles the full metrics tree together with per-case visualization data and the recorded rosbags, and hands the finished result to a storage backend through lidar_eval_backends.

lidar_eval_backends​

The swappable persistence layer. A single interface defines both halves of the contract — writing a completed run and reading it back for analysis — so the reporting pipeline and the dashboard use opposite halves of the same backend.

Backends are chosen by registry rather than by import; a Google backend (Sheets + Drive) ships today, and adding another means implementing the interface and enabling it. Credential providers are pluggable on the same principle, so how credentials are obtained can change without touching the storage backend.

lidar_pointcloud_filter​

The primary data-path processing node responsible for isolating sensor data. It consumes incoming raw point clouds (sensor_msgs/msg/PointCloud2) and uses the geometric definitions from the zones generator to apply high-frequency spatial, projective, and boundary-box filtering.

📁 Configuration Folders

environment_configs​

Contains environment definitions used for LiDAR evaluation. Each configuration describes the geometric structure of an evaluation environment, including zones, obstacles, reference surfaces, and sensor test scenarios.

lidar_configs​

Contains LiDAR-specific configuration files describing sensor properties, mounting parameters, TF relationships, topics, and evaluation settings required to integrate a sensor into the framework.

🐍 Python Support Modules

polysetup​

A standalone CLI tool that configures the bench for one specific run. It is the first thing you run — before any launch — and it takes exactly two inputs: one lidar config from lidar_configs/ and one environment config from environment_configs/. From that pair it generates every artifact the runtime needs, so launching the bench never requires hand-editing a node's parameter file.

What it writes:

  • The bench URDF (lidar_transforms/urdf/lidar_bench.urdf) — cart geometry, mount pose, and the zone link tree, constructed through lidar_zones.
  • Per-node parameter files — the orchestrator's eval_framework_manager.yaml (input topic, lidar and environment identity, results location), zones_orchestrator.yaml and roi.yaml (zone frames and ROI regions), pointcloud_filter.yaml (lidar frame and per-zone projective padding), lidar_reporting/config.yaml (reporting identity and resolutions), and automation_manager.yaml (driver command, bag recording, sweep parameters, servo angles).
  • The bringup launch file — regenerated with the base node set, and with the recording chain included or commented out according to the current bag-recording setting.

It also computes values you would otherwise maintain by hand. The most significant are the automation manager's servo pan angles, solved geometrically from the sensor's horizontal FOV and the zones' combined edge-to-edge boundary rather than written into the lidar config — see the Developer Guide for that derivation and for the sweep model it feeds.

Every generated path is derived from the --src-dir argument at runtime, so polysetup behaves identically no matter which directory it is invoked from. It is not a ROS 2 node, but it imports the built lidar_zones package, so the workspace must be sourced before it will run. Configuration problems — a missing or malformed YAML file, or a zone region too wide for the chosen sensor's field of view — are surfaced as a single [ERROR] … line rather than a Python traceback.

It installs two console scripts. polysetup-ws-sync performs the configuration itself, naming the workspace root and both config files explicitly:

polysetup-ws-sync --src-dir /lidar_test_bench \
    --lidar-file /lidar_test_bench/lidar_configs/robosense.yaml \
    --env-file /lidar_test_bench/environment_configs/rocinante.yaml

An optional --backend-file accepts a backend config YAML for setups that need one.

polysetup-bag-recording --src-dir <ws> --bag-recording-status <true|false> records whether the run should capture fresh bags. It persists the choice so the next configuration pass includes or excludes the recording chain accordingly.

In day-to-day use you reach both through the just recipes described below rather than typing these paths.

polyview_app​

A visualization dashboard for inspecting LiDAR evaluation results. It provides interactive visualization of point clouds, evaluation zones, metric outputs, and sensor comparison results.

⚙️ Task Runner (just)

Workspace tasks are wrapped as recipes in the Justfile at the repository root, run with just. The recipes exist so routine operations don't depend on remembering absolute paths or flag spellings. Run just --list to see what the Justfile currently offers; a full run uses these:

RecipePurpose
just setup-ws <environment> <lidar>Configure the workspace for one environment/LiDAR pair, then build
just launch-benchBring up the evaluation pipeline
just start-runBegin a run
just stop-runStop an in-progress evaluation
just enable-bag-recordingRecord fresh bags on the next run
just disable-bag-recordingEvaluate already-recorded bags instead

just setup-ws <environment> <lidar>​

The standard way to configure a run. This is a thin, convenience-focused wrapper around polysetup: it resolves two friendly names into config file paths, fills in the workspace root, and hands off.

just setup-ws rocinante robosense

Note the argument order — environment first, lidar second — which is the reverse of how the underlying polysetup flags are usually written.

Each name is resolved against its own config tree: the first argument is searched under environment_configs/, the second under lidar_configs/. Resolution is deliberately forgiving about how you type the name and deliberately strict about what it accepts:

  • The .yaml / .yml extension is optional — rocinante, rocinante.yaml, and rocinante.yml all work.
  • Matching is case-insensitive and searches the config tree recursively, so configs may live in subdirectories.
  • If no config matches, the recipe fails and prints every available config under that tree, so you can see the valid names.
  • If more than one config matches, the recipe refuses to guess and lists the candidates for you to disambiguate.

Both failure modes exit before polysetup is invoked, so an unresolvable name never partially configures the workspace. On success the recipe echoes the two resolved paths, then calls polysetup with --src-dir set to the repository root automatically. Because that root comes from the Justfile's own location rather than the current directory, just setup-ws behaves identically from anywhere in the repository.

Once configuration succeeds the recipe builds the workspace, then reminds you to source install/setup.bash — sourcing inside a recipe would only affect that recipe's own subshell.

The recipe inherits polysetup's prerequisite: the ROS 2 workspace must be sourced first, since polysetup imports the built lidar_zones package. On a clean checkout this is a chicken-and-egg situation, so build once by hand before the first setup-ws.

Recording mode: just start-run​

The bench runs in one of two modes, and start-run behaves differently in each. The mode is set by just enable-bag-recording / just disable-bag-recording, which persist the choice so it survives across configuration passes.

Recording enabled — the automation manager owns the run end to end. start-run triggers it once, and it then drives the whole sweep itself: cycling the sensor driver, commanding mount angles, applying each parameter under test, capturing a bag per case, and starting evaluation when every case has been recorded.

Recording disabled — the recording chain is left out of the launch file entirely, and start-run begins evaluation directly against bags recorded earlier. This is the fast path for iterating on zones or metrics, since it needs no sensor and no hardware attached.

🔄 How Everything Works Together​

1. Configuration Loading​

The system begins by loading structured YAML configuration files defining:

  • LiDAR sensor setup
  • Environment zones
  • Evaluation parameters

These are parsed into a unified runtime configuration model.

2. System Initialization​

polysetup builds the evaluation environment before anything launches:

  • URDF construction of the test bench, generated through lidar_zones
  • TF tree definition for sensor and zone alignment
  • Regeneration of the bringup launch file and every node's parameter file

At launch, lidar_transforms publishes that URDF and broadcasts the bench frames, and lidar_zones resolves each zone against them into the baseline profiles the run is scored against.

3. Processing Loop​

Once running, the system executes a continuous evaluation pipeline:

  • Raw point clouds are ingested (live sensor or rosbag playback)
  • lidar_pointcloud_filter isolates each zone, producing both a 3D-region view and an angular-frustum view
  • lidar_eval_orchestrator feeds those per-zone clouds into the metrics library, scan by scan
  • Metrics accumulate state across the run rather than emitting a result per frame

4. Evaluation & Reporting​

When a case completes, each metric reduces its accumulated state to a result and the orchestrator writes a structured per-case report. lidar_reporting then aggregates those reports with per-case visualization data and the recorded bags, and publishes the result through a storage backend for the dashboard and any external analytics to read.

🧩 Core Concepts​

📍 Zone-Based Evaluation Model​

The environment is divided into configurable spatial zones defined via YAML. These zones enable:

  • Region-specific evaluation
  • Noise filtering outside areas of interest
  • Consistent benchmarking across sensors

🔌 Plugin-Based Metrics System​

Metrics are not hardcoded. Instead, they are:

  • Implemented as plugins
  • Dynamically loaded at runtime
  • Applied per-zone to filtered point clouds

This allows domain-specific evaluation without modifying core infrastructure.

🔁 Reproducible Evaluation Pipeline​

Every evaluation run is fully reproducible through:

  • YAML-based configuration
  • Fixed TF tree definitions
  • Deterministic processing pipeline

📊 Multi-Layer Output System​

Evaluation results are surfaced through multiple channels:

  • Structured report files (YAML/JSON)
  • Visualization dashboards
  • External analytics tools (Grafana, Sheets, etc.)

🧭 Summary​

This architecture enables a fully modular LiDAR evaluation pipeline where:

  • Configuration is externalized (YAML-driven)
  • Zones are extensible for different objects and shapes as long as the boundaries are defined correctly
  • Processing is deterministic and reproducible
  • Metrics are extensible via plugins
  • Visualization and reporting are decoupled from core computation
  • database backend are configurable via plugins

Together, these components form a scalable benchmarking framework for evaluating LiDAR performance across diverse robotics environments.