frank/ruvnet--RuView
1
0
Fork 0
mirror of https://github.com/ruvnet/RuView synced 2026-08-07 20:01:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: QEMU ESP32-S3 testing platform + swarm configurator (ADR-061/062) (#260)

Browse Source 
9-layer QEMU testing platform (ADR-061) and YAML-driven swarm
configurator (ADR-062) for ESP32-S3 firmware testing without hardware.

12 commits, 56 files, +9,500 lines. Tested on Windows with
Espressif QEMU 9.0.0 — firmware boots, mock CSI generates frames,
14/16 validation checks pass. 39 bugs found and fixed across
2 deep code reviews.

Closes #259

Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
rUv
2026-03-14 13:39:51 -04:00
committed by GitHub
parent a467dfed9f
commit 523be943b0
57 changed files with 9532 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/ADR-061-qemu-esp32s3-firmware-testing.md
+171 -3
View File
@@ -2,8 +2,8 @@
| Field | Value |
|-------------|------------------------------------------------|
| **Status** | Proposed |
| **Date** | 2026-03-13 |
| **Status** | Accepted |
| **Date** | 2026-03-13 (updated 2026-03-14) |
| **Authors** | RuView Team |
| **Relates** | ADR-018 (binary frame), ADR-039 (edge intel), ADR-040 (WASM), ADR-057 (build guard), ADR-060 (channel/MAC filter) |
@@ -32,6 +32,98 @@ Currently, **every code change requires flashing to physical hardware** on COM7.
Espressif maintains an official QEMU fork (`github.com/espressif/qemu`) with ESP32-S3 machine support, including dual-core Xtensa LX7, flash mapping, UART, GPIO, timers, and FreeRTOS.
## Glossary
| Term | Definition |
|------|-----------|
| CSI | Channel State Information — per-subcarrier amplitude/phase from WiFi |
| NVS | Non-Volatile Storage — ESP-IDF key-value flash partition |
| TDM | Time-Division Multiplexing — nodes transmit in assigned time slots |
| UART | Universal Asynchronous Receiver-Transmitter — serial console output |
| SLIRP | User-mode TCP/IP stack — enables networking without root/TAP |
| QEMU | Quick Emulator — runs ESP32-S3 firmware without physical hardware |
| QMP | QEMU Machine Protocol — JSON-based control interface |
| LFSR | Linear Feedback Shift Register — deterministic pseudo-random generator |
| SPSC | Single Producer Single Consumer — lock-free ring buffer pattern |
| FreeRTOS | Real-time OS used by ESP-IDF for task scheduling |
| gcov/lcov | GCC code coverage tools for line/branch analysis |
| libFuzzer | LLVM coverage-guided fuzzer for finding crashes |
| ASAN | AddressSanitizer — detects buffer overflows and use-after-free |
| UBSAN | UndefinedBehaviorSanitizer — detects undefined C behavior |
## Quick Start
### Prerequisites
Install required tools:
```bash
# QEMU (Espressif fork with ESP32-S3 support)
git clone https://github.com/espressif/qemu.git
cd qemu && ./configure --target-list=xtensa-softmmu && make -j$(nproc)
export QEMU_PATH=/path/to/qemu/build/qemu-system-xtensa
# ESP-IDF (for building firmware)
# See https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/get-started/
# Python tools
pip install esptool esp-idf-nvs-partition-gen
# Coverage tools (optional, Layer 5)
sudo apt install lcov # Debian/Ubuntu
brew install lcov # macOS
# Fuzz testing (optional, Layer 6)
sudo apt install clang # Debian/Ubuntu
# Mesh testing (optional, Layer 3 — requires root)
sudo apt install socat bridge-utils iproute2
```
### Run the Full Test Suite
```bash
# Layer 2: Single-node test (build + run + validate)
bash scripts/qemu-esp32s3-test.sh
# Layer 3: Multi-node mesh (3 nodes, requires root)
sudo bash scripts/qemu-mesh-test.sh 3
# Layer 6: Fuzz testing (60 seconds per target)
cd firmware/esp32-csi-node/test && make all CC=clang
make run_serialize FUZZ_DURATION=60
# Layer 7: Generate NVS test matrix
python3 scripts/generate_nvs_matrix.py --output-dir build/nvs_matrix
# Layer 8: Snapshot regression tests
bash scripts/qemu-snapshot-test.sh --create
bash scripts/qemu-snapshot-test.sh --restore csi-streaming
# Layer 9: Chaos/fault injection
bash scripts/qemu-chaos-test.sh --faults all --duration 120
```
### Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `QEMU_PATH` | `qemu-system-xtensa` | Path to Espressif QEMU binary |
| `QEMU_TIMEOUT` | `60` (single) / `45` (mesh) / `120` (chaos) | Test timeout in seconds |
| `SKIP_BUILD` | unset | Set to `1` to skip firmware build step |
| `NVS_BIN` | unset | Path to pre-built NVS partition binary |
| `QEMU_NET` | `1` | Set to `0` to disable SLIRP networking |
| `CHAOS_SEED` | current time | Seed for reproducible chaos testing |
### Exit Codes (all scripts)
| Code | Meaning | Action |
|------|---------|--------|
| 0 | PASS | All checks passed |
| 1 | WARN | Non-critical issues; review output |
| 2 | FAIL | Critical checks failed; fix and re-run |
| 3 | FATAL | Build error, crash, or missing tool; check prerequisites |
## Decision
Introduce a **comprehensive QEMU testing platform** for the ESP32-S3 CSI node firmware with nine capability layers:
@@ -145,7 +237,7 @@ This model exercises:
| 5 | Channel sweep | 5s | Frames on channels 1, 6, 11 in sequence |
| 6 | MAC filter test | 5s | Frames with wrong MAC are dropped (counter check) |
| 7 | Ring buffer overflow | 3s | 1000 frames in 100ms burst, graceful drop |
| 8 | Boundary RSSI | 5s | RSSI sweeps -127 to 0, no crash |
| 8 | Boundary RSSI | 5s | RSSI sweeps -90 to -10 dBm, no crash |
| 9 | Zero-length frame | 2s | `iq_len=0` frames, serialize returns 0 |
---
@@ -456,6 +548,53 @@ xtensa-esp-elf-gdb build/esp32-csi-node.elf \
-ex "continue"
```
### Debugging Walkthrough
**1. Start QEMU with GDB stub (paused at reset vector):**
```bash
qemu-system-xtensa \
-machine esp32s3 \
-nographic \
-drive file=build/qemu_flash.bin,if=mtd,format=raw \
-serial mon:stdio \
-s -S
# -s opens GDB server on localhost:1234
# -S pauses CPU until GDB sends "continue"
```
**2. Connect from a second terminal:**
```bash
xtensa-esp-elf-gdb build/esp32-csi-node.elf \
-ex "target remote :1234" \
-ex "b app_main" \
-ex "continue"
```
**3. Set a breakpoint on DSP processing and inspect state:**
```
(gdb) b edge_processing.c:dsp_task
(gdb) continue
# ...breakpoint hit...
(gdb) print g_nvs_config
(gdb) print ring->head - ring->tail
(gdb) continue
```
**4. Connect from VS Code** using the `launch.json` config below (set breakpoints in the editor gutter, then press F5).
**5. Dump gcov coverage data (requires `sdkconfig.coverage` overlay):**
```
(gdb) monitor gcov dump
# Writes .gcda files to the build directory.
# Then generate the HTML report on the host:
# lcov --capture --directory build --output-file coverage.info
# genhtml coverage.info --output-directory build/coverage_report
```
### Key Breakpoint Locations
| Breakpoint | Purpose |
@@ -862,3 +1001,32 @@ Alternative to QEMU with better peripheral modeling for some platforms.
- ADR-040: WASM programmable sensing runtime
- ADR-057: Build-time CSI guard (`CONFIG_ESP_WIFI_CSI_ENABLED`)
- ADR-060: Channel override and MAC address filter
---
## Optimization Log (2026-03-14)
### Bugs Fixed
1. **LFSR float bias** — `lfsr_float()` used divisor 32767.5 producing range [-1.0, 1.00002]; fixed to 32768.0 for exact [-1.0, +1.0)
2. **MAC filter initialization** — `gen_mac_filter()` compared `frame_count == scenario_start_ms` (count vs timestamp); replaced with boolean flag
3. **Scenario infinite loop** — `advance_scenario()` looped to scenario 0 when all completed; now sets `s_all_done=true` and timer callback exits early
4. **Boot check severity** — `validate_qemu_output.py` reported no-boot as ERROR; upgraded to FATAL (nothing works without boot)
5. **NVS boundary configs** — `boundary-max` used `vital_win=65535` which firmware silently rejects (valid: 32-256); fixed to 256
6. **NVS boundary-min** — `vital_win=1` also invalid; fixed to 32 (firmware min)
7. **edge-tier2-custom** — `vital_win=512` exceeded firmware max of 256; fixed to 256
8. **power-save config** — Described as "10% duty cycle" but didn't set `power_duty=10`; fixed
9. **wasm-signed/unsigned** — Both configs were identical; signed now includes pubkey blob, unsigned sets `wasm_verify=0`
### Optimizations Applied
1. **SLIRP networking** — QEMU runner now passes `-nic user,model=open_eth` for UDP testing
2. **Scenario completion tracking** — Validator now checks `All N scenarios complete` log marker (check 15)
3. **Frame rate monitoring** — Validator extracts `scenario=N frames=M` counters for rate analysis (check 16)
4. **Watchdog tuning** — `sdkconfig.qemu` relaxes WDT to 30s / INT_WDT to 800ms for QEMU timing variance
5. **Timer stack depth** — Increased `FREERTOS_TIMER_TASK_STACK_DEPTH=4096` to prevent overflow from math-heavy mock callback
6. **Display disabled** — `CONFIG_DISPLAY_ENABLE=n` in QEMU overlay (no I2C hardware)
7. **CI fuzz job** — Added `fuzz-test` job running all 3 fuzz targets for 60s each with crash artifact upload
8. **CI NVS validation** — Added `nvs-matrix-validate` job that generates all 14 binaries and verifies sizes
9. **CI matrix expanded** — Added `edge-tier1`, `boundary-max`, `boundary-min` to QEMU test matrix (4 → 7 configs)
10. **QEMU cache key** — Uses `github.run_id` with restore-keys fallback to prevent stale QEMU builds
docs/adr/ADR-062-qemu-swarm-configurator.md
+199
View File
@@ -0,0 +1,199 @@
# ADR-062: QEMU ESP32-S3 Swarm Configurator
| Field | Value |
|-------------|------------------------------------------------|
| **Status** | Accepted |
| **Date** | 2026-03-14 |
| **Authors** | RuView Team |
| **Relates** | ADR-061 (QEMU testing platform), ADR-060 (channel/MAC filter), ADR-018 (binary frame), ADR-039 (edge intel) |
## Glossary
| Term | Definition |
|------|-----------|
| Swarm | A group of N QEMU ESP32-S3 instances running simultaneously |
| Topology | How nodes are connected: star, mesh, line, ring |
| Role | Node function: `sensor` (collects CSI), `coordinator` (aggregates + forwards), `gateway` (bridges to host) |
| Scenario matrix | Cross-product of topology × node count × NVS config × mock scenario |
| Health oracle | Python process that monitors all node UART logs and declares swarm health |
## Context
ADR-061 Layer 3 provides a basic multi-node mesh test: N identical nodes with sequential TDM slots connected via a Linux bridge. This is useful but limited:
1. **All nodes are identical** — real deployments have heterogeneous roles (sensor, coordinator, gateway)
2. **Single topology** — only fully-connected bridge; no star, line, or ring topologies
3. **No scenario variation per node** — all nodes run the same mock CSI scenario
4. **Manual configuration** — each test requires hand-editing env vars and arguments
5. **No swarm-level health monitoring** — validation checks individual nodes, not collective behavior
6. **No cross-node timing validation** — TDM slot ordering and inter-frame gaps aren't verified
Real WiFi-DensePose deployments use 3-8 ESP32-S3 nodes in various topologies. A single coordinator aggregates CSI from multiple sensors. The firmware must handle TDM conflicts, missing nodes, role-based behavior differences, and network partitions — none of which ADR-061 Layer 3 tests.
## Decision
Build a **QEMU Swarm Configurator** — a YAML-driven tool that defines multi-node test scenarios declaratively and orchestrates them under QEMU with swarm-level validation.
### Architecture
```
┌─────────────────────────────────────────────────────┐
│ swarm_config.yaml │
│ nodes: [{role: sensor, scenario: 2, channel: 6}] │
│ topology: star │
│ duration: 60s │
│ assertions: [all_nodes_boot, tdm_no_collision, ...] │
└──────────────────────┬──────────────────────────────┘
│
┌────────────▼────────────┐
│ qemu_swarm.py │
│ (orchestrator) │
└───┬────┬────┬───┬──────┘
│ │ │ │
┌────▼┐ ┌▼──┐ ▼ ┌▼────┐
│Node0│ │N1 │... │N(n-1)│ QEMU instances
│sens │ │sen│ │coord │
└──┬──┘ └─┬─┘ └──┬───┘
│ │ │
┌──▼──────▼─────────▼──┐
│ Virtual Network │ TAP bridge / SLIRP
│ (topology-shaped) │
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ Aggregator (Rust) │ Collects frames
└──────────┬───────────┘
│
┌──────────▼───────────┐
│ Health Oracle │ Swarm-level assertions
│ (swarm_health.py) │
└──────────────────────┘
```
### YAML Configuration Schema
```yaml
# swarm_config.yaml
swarm:
name: "3-sensor-star"
duration_s: 60
topology: star # star | mesh | line | ring
aggregator_port: 5005
nodes:
- role: coordinator
node_id: 0
scenario: 0 # empty room (baseline)
channel: 6
edge_tier: 2
is_gateway: true # receives aggregated frames
- role: sensor
node_id: 1
scenario: 2 # walking person
channel: 6
tdm_slot: 1 # TDM slot index (auto-assigned from node position if omitted)
- role: sensor
node_id: 2
scenario: 3 # fall event
channel: 6
tdm_slot: 2
assertions:
- all_nodes_boot
- no_crashes
- tdm_no_collision
- all_nodes_produce_frames
- coordinator_receives_from_all
- fall_detected_by_node_2
- frame_rate_above: 15 # Hz minimum per node
- max_boot_time_s: 10
```
### Topologies
| Topology | Network | Description |
|----------|---------|-------------|
| `star` | All sensors connect to coordinator; coordinator has TAP to each sensor | Hub-and-spoke, most common |
| `mesh` | All nodes on same bridge (existing Layer 3 behavior) | Every node sees every other |
| `line` | Node 0 ↔ Node 1 ↔ Node 2 ↔ ... | Linear chain, tests multi-hop |
| `ring` | Like line but last connects to first | Circular, tests routing |
### Node Roles
| Role | Behavior | NVS Keys |
|------|----------|----------|
| `sensor` | Runs mock CSI, sends frames to coordinator | `node_id`, `tdm_slot`, `target_ip` |
| `coordinator` | Receives frames from sensors, runs edge aggregation | `node_id`, `tdm_slot=0`, `edge_tier=2` |
| `gateway` | Like coordinator but also bridges to host UDP | `node_id`, `target_ip=host`, `is_gateway=1` |
### Assertions (Swarm-Level)
| Assertion | What It Checks |
|-----------|---------------|
| `all_nodes_boot` | Every node's UART log shows boot indicators within timeout |
| `no_crashes` | No Guru Meditation, assert, panic in any log |
| `tdm_no_collision` | No two nodes transmit in the same TDM slot |
| `all_nodes_produce_frames` | Every sensor node's log contains CSI frame output |
| `coordinator_receives_from_all` | Coordinator log shows frames from each sensor's node_id |
| `fall_detected_by_node_N` | Node N's log reports a fall detection event |
| `frame_rate_above` | Each node produces at least N frames/second |
| `max_boot_time_s` | All nodes boot within N seconds |
| `no_heap_errors` | No OOM or heap corruption in any log |
| `network_partitioned_recovery` | After deliberate partition, nodes resume communication (future) |
### Preset Configurations
| Preset | Nodes | Topology | Purpose |
|--------|-------|----------|---------|
| `smoke` | 2 | star | Quick CI smoke test (15s) |
| `standard` | 3 | star | Default 3-node (sensor + sensor + coordinator) |
| `large-mesh` | 6 | mesh | Scale test with 6 fully-connected nodes |
| `line-relay` | 4 | line | Multi-hop relay chain |
| `ring-fault` | 4 | ring | Ring with fault injection mid-test |
| `heterogeneous` | 5 | star | Mixed scenarios: walk, fall, static, channel-sweep, empty |
| `ci-matrix` | 3 | star | CI-optimized preset (30s, minimal assertions) |
## File Layout
```
scripts/
├── qemu_swarm.py # Main orchestrator (CLI entry point)
├── swarm_health.py # Swarm-level health oracle
└── swarm_presets/
├── smoke.yaml
├── standard.yaml
├── large_mesh.yaml
├── line_relay.yaml
├── ring_fault.yaml
├── heterogeneous.yaml
└── ci_matrix.yaml
.github/workflows/
└── firmware-qemu.yml # MODIFIED: add swarm test job
```
## Consequences
### Benefits
1. **Declarative testing** — define swarm topology in YAML, not shell scripts
2. **Role-based nodes** — test coordinator/sensor/gateway interactions
3. **Topology variety** — star/mesh/line/ring match real deployment patterns
4. **Swarm-level assertions** — validate collective behavior, not just individual nodes
5. **Preset library** — quick CI smoke tests and thorough manual validation
6. **Reproducible** — YAML configs are version-controlled and shareable
### Limitations
1. **Still requires root** for TAP bridge topologies (star, line, ring); mesh can use SLIRP
2. **QEMU resource usage** — 6+ QEMU instances use ~2GB RAM, may slow CI runners
3. **No real RF** — inter-node communication is IP-based, not WiFi CSI multipath
## References
- ADR-061: QEMU ESP32-S3 firmware testing platform (Layers 1-9)
- ADR-060: Channel override and MAC address filter provisioning
- ADR-018: Binary CSI frame format (magic `0xC5110001`)
- ADR-039: Edge intelligence pipeline (biquad, vitals, fall detection)