feat: QEMU ESP32-S3 testing platform + swarm configurator (ADR-061/062) (#260)

9-layer QEMU testing platform (ADR-061) and YAML-driven swarm configurator (ADR-062) for ESP32-S3 firmware testing without hardware. 12 commits, 56 files, +9,500 lines. Tested on Windows with Espressif QEMU 9.0.0 — firmware boots, mock CSI generates frames, 14/16 validation checks pass. 39 bugs found and fixed across 2 deep code reviews. Closes #259 Co-Authored-By: claude-flow <ruv@ruv.net>
2026-08-01 19:01:42 +00:00 · 2026-03-14 13:39:51 -04:00
parent a467dfed9f
commit 523be943b0
57 changed files with 9532 additions and 8 deletions
@@ -2,8 +2,8 @@

 | Field       | Value                                          |
 |-------------|------------------------------------------------|
-| **Status**  | Proposed                                       |
-| **Date**    | 2026-03-13                                     |
+| **Status**  | Accepted                                       |
+| **Date**    | 2026-03-13 (updated 2026-03-14)                |
 | **Authors** | RuView Team                                    |
 | **Relates** | ADR-018 (binary frame), ADR-039 (edge intel), ADR-040 (WASM), ADR-057 (build guard), ADR-060 (channel/MAC filter) |

@@ -32,6 +32,98 @@ Currently, **every code change requires flashing to physical hardware** on COM7.

 Espressif maintains an official QEMU fork (`github.com/espressif/qemu`) with ESP32-S3 machine support, including dual-core Xtensa LX7, flash mapping, UART, GPIO, timers, and FreeRTOS.

+## Glossary
+
+| Term | Definition |
+|------|-----------|
+| CSI | Channel State Information — per-subcarrier amplitude/phase from WiFi |
+| NVS | Non-Volatile Storage — ESP-IDF key-value flash partition |
+| TDM | Time-Division Multiplexing — nodes transmit in assigned time slots |
+| UART | Universal Asynchronous Receiver-Transmitter — serial console output |
+| SLIRP | User-mode TCP/IP stack — enables networking without root/TAP |
+| QEMU | Quick Emulator — runs ESP32-S3 firmware without physical hardware |
+| QMP | QEMU Machine Protocol — JSON-based control interface |
+| LFSR | Linear Feedback Shift Register — deterministic pseudo-random generator |
+| SPSC | Single Producer Single Consumer — lock-free ring buffer pattern |
+| FreeRTOS | Real-time OS used by ESP-IDF for task scheduling |
+| gcov/lcov | GCC code coverage tools for line/branch analysis |
+| libFuzzer | LLVM coverage-guided fuzzer for finding crashes |
+| ASAN | AddressSanitizer — detects buffer overflows and use-after-free |
+| UBSAN | UndefinedBehaviorSanitizer — detects undefined C behavior |
+
+## Quick Start
+
+### Prerequisites
+
+Install required tools:
+
+```bash
+# QEMU (Espressif fork with ESP32-S3 support)
+git clone https://github.com/espressif/qemu.git
+cd qemu && ./configure --target-list=xtensa-softmmu && make -j$(nproc)
+export QEMU_PATH=/path/to/qemu/build/qemu-system-xtensa
+
+# ESP-IDF (for building firmware)
+# See https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/get-started/
+
+# Python tools
+pip install esptool esp-idf-nvs-partition-gen
+
+# Coverage tools (optional, Layer 5)
+sudo apt install lcov          # Debian/Ubuntu
+brew install lcov              # macOS
+
+# Fuzz testing (optional, Layer 6)
+sudo apt install clang         # Debian/Ubuntu
+
+# Mesh testing (optional, Layer 3 — requires root)
+sudo apt install socat bridge-utils iproute2
+```
+
+### Run the Full Test Suite
+
+```bash
+# Layer 2: Single-node test (build + run + validate)
+bash scripts/qemu-esp32s3-test.sh
+
+# Layer 3: Multi-node mesh (3 nodes, requires root)
+sudo bash scripts/qemu-mesh-test.sh 3
+
+# Layer 6: Fuzz testing (60 seconds per target)
+cd firmware/esp32-csi-node/test && make all CC=clang
+make run_serialize FUZZ_DURATION=60
+
+# Layer 7: Generate NVS test matrix
+python3 scripts/generate_nvs_matrix.py --output-dir build/nvs_matrix
+
+# Layer 8: Snapshot regression tests
+bash scripts/qemu-snapshot-test.sh --create
+bash scripts/qemu-snapshot-test.sh --restore csi-streaming
+
+# Layer 9: Chaos/fault injection
+bash scripts/qemu-chaos-test.sh --faults all --duration 120
+```
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `QEMU_PATH` | `qemu-system-xtensa` | Path to Espressif QEMU binary |
+| `QEMU_TIMEOUT` | `60` (single) / `45` (mesh) / `120` (chaos) | Test timeout in seconds |
+| `SKIP_BUILD` | unset | Set to `1` to skip firmware build step |
+| `NVS_BIN` | unset | Path to pre-built NVS partition binary |
+| `QEMU_NET` | `1` | Set to `0` to disable SLIRP networking |
+| `CHAOS_SEED` | current time | Seed for reproducible chaos testing |
+
+### Exit Codes (all scripts)
+
+| Code | Meaning | Action |
+|------|---------|--------|
+| 0 | PASS | All checks passed |
+| 1 | WARN | Non-critical issues; review output |
+| 2 | FAIL | Critical checks failed; fix and re-run |
+| 3 | FATAL | Build error, crash, or missing tool; check prerequisites |
+
 ## Decision

 Introduce a **comprehensive QEMU testing platform** for the ESP32-S3 CSI node firmware with nine capability layers:
@@ -145,7 +237,7 @@ This model exercises:
 | 5 | Channel sweep | 5s | Frames on channels 1, 6, 11 in sequence |
 | 6 | MAC filter test | 5s | Frames with wrong MAC are dropped (counter check) |
 | 7 | Ring buffer overflow | 3s | 1000 frames in 100ms burst, graceful drop |
-| 8 | Boundary RSSI | 5s | RSSI sweeps -127 to 0, no crash |
+| 8 | Boundary RSSI | 5s | RSSI sweeps -90 to -10 dBm, no crash |
 | 9 | Zero-length frame | 2s | `iq_len=0` frames, serialize returns 0 |

 ---
@@ -456,6 +548,53 @@ xtensa-esp-elf-gdb build/esp32-csi-node.elf \
  -ex "continue"
 ```

+### Debugging Walkthrough
+
+**1. Start QEMU with GDB stub (paused at reset vector):**
+
+```bash
+qemu-system-xtensa \
+  -machine esp32s3 \
+  -nographic \
+  -drive file=build/qemu_flash.bin,if=mtd,format=raw \
+  -serial mon:stdio \
+  -s -S
+# -s  opens GDB server on localhost:1234
+# -S  pauses CPU until GDB sends "continue"
+```
+
+**2. Connect from a second terminal:**
+
+```bash
+xtensa-esp-elf-gdb build/esp32-csi-node.elf \
+  -ex "target remote :1234" \
+  -ex "b app_main" \
+  -ex "continue"
+```
+
+**3. Set a breakpoint on DSP processing and inspect state:**
+
+```
+(gdb) b edge_processing.c:dsp_task
+(gdb) continue
+# ...breakpoint hit...
+(gdb) print g_nvs_config
+(gdb) print ring->head - ring->tail
+(gdb) continue
+```
+
+**4. Connect from VS Code** using the `launch.json` config below (set breakpoints in the editor gutter, then press F5).
+
+**5. Dump gcov coverage data (requires `sdkconfig.coverage` overlay):**
+
+```
+(gdb) monitor gcov dump
+# Writes .gcda files to the build directory.
+# Then generate the HTML report on the host:
+#   lcov --capture --directory build --output-file coverage.info
+#   genhtml coverage.info --output-directory build/coverage_report
+```
+
 ### Key Breakpoint Locations

 | Breakpoint | Purpose |
@@ -862,3 +1001,32 @@ Alternative to QEMU with better peripheral modeling for some platforms.
 - ADR-040: WASM programmable sensing runtime
 - ADR-057: Build-time CSI guard (`CONFIG_ESP_WIFI_CSI_ENABLED`)
 - ADR-060: Channel override and MAC address filter
+
+---
+
+## Optimization Log (2026-03-14)
+
+### Bugs Fixed
+
+1. **LFSR float bias** — `lfsr_float()` used divisor 32767.5 producing range [-1.0, 1.00002]; fixed to 32768.0 for exact [-1.0, +1.0)
+2. **MAC filter initialization** — `gen_mac_filter()` compared `frame_count == scenario_start_ms` (count vs timestamp); replaced with boolean flag
+3. **Scenario infinite loop** — `advance_scenario()` looped to scenario 0 when all completed; now sets `s_all_done=true` and timer callback exits early
+4. **Boot check severity** — `validate_qemu_output.py` reported no-boot as ERROR; upgraded to FATAL (nothing works without boot)
+5. **NVS boundary configs** — `boundary-max` used `vital_win=65535` which firmware silently rejects (valid: 32-256); fixed to 256
+6. **NVS boundary-min** — `vital_win=1` also invalid; fixed to 32 (firmware min)
+7. **edge-tier2-custom** — `vital_win=512` exceeded firmware max of 256; fixed to 256
+8. **power-save config** — Described as "10% duty cycle" but didn't set `power_duty=10`; fixed
+9. **wasm-signed/unsigned** — Both configs were identical; signed now includes pubkey blob, unsigned sets `wasm_verify=0`
+
+### Optimizations Applied
+
+1. **SLIRP networking** — QEMU runner now passes `-nic user,model=open_eth` for UDP testing
+2. **Scenario completion tracking** — Validator now checks `All N scenarios complete` log marker (check 15)
+3. **Frame rate monitoring** — Validator extracts `scenario=N frames=M` counters for rate analysis (check 16)
+4. **Watchdog tuning** — `sdkconfig.qemu` relaxes WDT to 30s / INT_WDT to 800ms for QEMU timing variance
+5. **Timer stack depth** — Increased `FREERTOS_TIMER_TASK_STACK_DEPTH=4096` to prevent overflow from math-heavy mock callback
+6. **Display disabled** — `CONFIG_DISPLAY_ENABLE=n` in QEMU overlay (no I2C hardware)
+7. **CI fuzz job** — Added `fuzz-test` job running all 3 fuzz targets for 60s each with crash artifact upload
+8. **CI NVS validation** — Added `nvs-matrix-validate` job that generates all 14 binaries and verifies sizes
+9. **CI matrix expanded** — Added `edge-tier1`, `boundary-max`, `boundary-min` to QEMU test matrix (4 → 7 configs)
+10. **QEMU cache key** — Uses `github.run_id` with restore-keys fallback to prevent stale QEMU builds
@@ -0,0 +1,199 @@
+# ADR-062: QEMU ESP32-S3 Swarm Configurator
+
+| Field       | Value                                          |
+|-------------|------------------------------------------------|
+| **Status**  | Accepted                                       |
+| **Date**    | 2026-03-14                                     |
+| **Authors** | RuView Team                                    |
+| **Relates** | ADR-061 (QEMU testing platform), ADR-060 (channel/MAC filter), ADR-018 (binary frame), ADR-039 (edge intel) |
+
+## Glossary
+
+| Term | Definition |
+|------|-----------|
+| Swarm | A group of N QEMU ESP32-S3 instances running simultaneously |
+| Topology | How nodes are connected: star, mesh, line, ring |
+| Role | Node function: `sensor` (collects CSI), `coordinator` (aggregates + forwards), `gateway` (bridges to host) |
+| Scenario matrix | Cross-product of topology × node count × NVS config × mock scenario |
+| Health oracle | Python process that monitors all node UART logs and declares swarm health |
+
+## Context
+
+ADR-061 Layer 3 provides a basic multi-node mesh test: N identical nodes with sequential TDM slots connected via a Linux bridge. This is useful but limited:
+
+1. **All nodes are identical** — real deployments have heterogeneous roles (sensor, coordinator, gateway)
+2. **Single topology** — only fully-connected bridge; no star, line, or ring topologies
+3. **No scenario variation per node** — all nodes run the same mock CSI scenario
+4. **Manual configuration** — each test requires hand-editing env vars and arguments
+5. **No swarm-level health monitoring** — validation checks individual nodes, not collective behavior
+6. **No cross-node timing validation** — TDM slot ordering and inter-frame gaps aren't verified
+
+Real WiFi-DensePose deployments use 3-8 ESP32-S3 nodes in various topologies. A single coordinator aggregates CSI from multiple sensors. The firmware must handle TDM conflicts, missing nodes, role-based behavior differences, and network partitions — none of which ADR-061 Layer 3 tests.
+
+## Decision
+
+Build a **QEMU Swarm Configurator** — a YAML-driven tool that defines multi-node test scenarios declaratively and orchestrates them under QEMU with swarm-level validation.
+
+### Architecture
+
+```
+┌─────────────────────────────────────────────────────┐
+│                 swarm_config.yaml                     │
+│  nodes: [{role: sensor, scenario: 2, channel: 6}]   │
+│  topology: star                                       │
+│  duration: 60s                                        │
+│  assertions: [all_nodes_boot, tdm_no_collision, ...]  │
+└──────────────────────┬──────────────────────────────┘
+                       │
+          ┌────────────▼────────────┐
+          │   qemu_swarm.py         │
+          │   (orchestrator)        │
+          └───┬────┬────┬───┬──────┘
+              │    │    │   │
+         ┌────▼┐ ┌▼──┐ ▼  ┌▼────┐
+         │Node0│ │N1 │... │N(n-1)│   QEMU instances
+         │sens │ │sen│    │coord │
+         └──┬──┘ └─┬─┘    └──┬───┘
+            │      │         │
+         ┌──▼──────▼─────────▼──┐
+         │  Virtual Network      │    TAP bridge / SLIRP
+         │  (topology-shaped)    │
+         └──────────┬───────────┘
+                    │
+         ┌──────────▼───────────┐
+         │  Aggregator (Rust)    │    Collects frames
+         └──────────┬───────────┘
+                    │
+         ┌──────────▼───────────┐
+         │  Health Oracle        │    Swarm-level assertions
+         │  (swarm_health.py)    │
+         └──────────────────────┘
+```
+
+### YAML Configuration Schema
+
+```yaml
+# swarm_config.yaml
+swarm:
+  name: "3-sensor-star"
+  duration_s: 60
+  topology: star          # star | mesh | line | ring
+  aggregator_port: 5005
+
+nodes:
+  - role: coordinator
+    node_id: 0
+    scenario: 0           # empty room (baseline)
+    channel: 6
+    edge_tier: 2
+    is_gateway: true       # receives aggregated frames
+
+  - role: sensor
+    node_id: 1
+    scenario: 2           # walking person
+    channel: 6
+    tdm_slot: 1           # TDM slot index (auto-assigned from node position if omitted)
+
+  - role: sensor
+    node_id: 2
+    scenario: 3           # fall event
+    channel: 6
+    tdm_slot: 2
+
+assertions:
+  - all_nodes_boot
+  - no_crashes
+  - tdm_no_collision
+  - all_nodes_produce_frames
+  - coordinator_receives_from_all
+  - fall_detected_by_node_2
+  - frame_rate_above: 15    # Hz minimum per node
+  - max_boot_time_s: 10
+```
+
+### Topologies
+
+| Topology | Network | Description |
+|----------|---------|-------------|
+| `star` | All sensors connect to coordinator; coordinator has TAP to each sensor | Hub-and-spoke, most common |
+| `mesh` | All nodes on same bridge (existing Layer 3 behavior) | Every node sees every other |
+| `line` | Node 0 ↔ Node 1 ↔ Node 2 ↔ ... | Linear chain, tests multi-hop |
+| `ring` | Like line but last connects to first | Circular, tests routing |
+
+### Node Roles
+
+| Role | Behavior | NVS Keys |
+|------|----------|----------|
+| `sensor` | Runs mock CSI, sends frames to coordinator | `node_id`, `tdm_slot`, `target_ip` |
+| `coordinator` | Receives frames from sensors, runs edge aggregation | `node_id`, `tdm_slot=0`, `edge_tier=2` |
+| `gateway` | Like coordinator but also bridges to host UDP | `node_id`, `target_ip=host`, `is_gateway=1` |
+
+### Assertions (Swarm-Level)
+
+| Assertion | What It Checks |
+|-----------|---------------|
+| `all_nodes_boot` | Every node's UART log shows boot indicators within timeout |
+| `no_crashes` | No Guru Meditation, assert, panic in any log |
+| `tdm_no_collision` | No two nodes transmit in the same TDM slot |
+| `all_nodes_produce_frames` | Every sensor node's log contains CSI frame output |
+| `coordinator_receives_from_all` | Coordinator log shows frames from each sensor's node_id |
+| `fall_detected_by_node_N` | Node N's log reports a fall detection event |
+| `frame_rate_above` | Each node produces at least N frames/second |
+| `max_boot_time_s` | All nodes boot within N seconds |
+| `no_heap_errors` | No OOM or heap corruption in any log |
+| `network_partitioned_recovery` | After deliberate partition, nodes resume communication (future) |
+
+### Preset Configurations
+
+| Preset | Nodes | Topology | Purpose |
+|--------|-------|----------|---------|
+| `smoke` | 2 | star | Quick CI smoke test (15s) |
+| `standard` | 3 | star | Default 3-node (sensor + sensor + coordinator) |
+| `large-mesh` | 6 | mesh | Scale test with 6 fully-connected nodes |
+| `line-relay` | 4 | line | Multi-hop relay chain |
+| `ring-fault` | 4 | ring | Ring with fault injection mid-test |
+| `heterogeneous` | 5 | star | Mixed scenarios: walk, fall, static, channel-sweep, empty |
+| `ci-matrix` | 3 | star | CI-optimized preset (30s, minimal assertions) |
+
+## File Layout
+
+```
+scripts/
+├── qemu_swarm.py              # Main orchestrator (CLI entry point)
+├── swarm_health.py            # Swarm-level health oracle
+└── swarm_presets/
+    ├── smoke.yaml
+    ├── standard.yaml
+    ├── large_mesh.yaml
+    ├── line_relay.yaml
+    ├── ring_fault.yaml
+    ├── heterogeneous.yaml
+    └── ci_matrix.yaml
+
+.github/workflows/
+└── firmware-qemu.yml          # MODIFIED: add swarm test job
+```
+
+## Consequences
+
+### Benefits
+
+1. **Declarative testing** — define swarm topology in YAML, not shell scripts
+2. **Role-based nodes** — test coordinator/sensor/gateway interactions
+3. **Topology variety** — star/mesh/line/ring match real deployment patterns
+4. **Swarm-level assertions** — validate collective behavior, not just individual nodes
+5. **Preset library** — quick CI smoke tests and thorough manual validation
+6. **Reproducible** — YAML configs are version-controlled and shareable
+
+### Limitations
+
+1. **Still requires root** for TAP bridge topologies (star, line, ring); mesh can use SLIRP
+2. **QEMU resource usage** — 6+ QEMU instances use ~2GB RAM, may slow CI runners
+3. **No real RF** — inter-node communication is IP-based, not WiFi CSI multipath
+
+## References
+
+- ADR-061: QEMU ESP32-S3 firmware testing platform (Layers 1-9)
+- ADR-060: Channel override and MAC address filter provisioning
+- ADR-018: Binary CSI frame format (magic `0xC5110001`)
+- ADR-039: Edge intelligence pipeline (biquad, vitals, fall detection)
@@ -38,8 +38,17 @@ WiFi DensePose turns commodity WiFi signals into real-time human pose estimation
    - [ESP32-S3 Mesh](#esp32-s3-mesh)
    - [Intel 5300 / Atheros NIC](#intel-5300--atheros-nic)
 15. [Docker Compose (Multi-Service)](#docker-compose-multi-service)
-16. [Troubleshooting](#troubleshooting)
-17. [FAQ](#faq)
+16. [Testing Firmware Without Hardware (QEMU)](#testing-firmware-without-hardware-qemu)
+    - [What You Need](#what-you-need)
+    - [Your First Test Run](#your-first-test-run)
+    - [Understanding the Test Output](#understanding-the-test-output)
+    - [Testing Multiple Nodes at Once (Swarm)](#testing-multiple-nodes-at-once-swarm)
+    - [Swarm Presets](#swarm-presets)
+    - [Writing Your Own Swarm Config](#writing-your-own-swarm-config)
+    - [Debugging Firmware in QEMU](#debugging-firmware-in-qemu)
+    - [Running the Full Test Suite](#running-the-full-test-suite)
+17. [Troubleshooting](#troubleshooting)
+18. [FAQ](#faq)

 ---

@@ -936,6 +945,288 @@ This starts:

 ---

+## Testing Firmware Without Hardware (QEMU)
+
+You can test the ESP32-S3 firmware on your computer without any physical hardware. The project uses **QEMU** — an emulator that pretends to be an ESP32-S3 chip, running the real firmware code inside a virtual machine on your PC.
+
+This is useful when:
+- You don't have an ESP32-S3 board yet
+- You want to test firmware changes before flashing to real hardware
+- You're running automated tests in CI/CD
+- You want to simulate multiple ESP32 nodes talking to each other
+
+### What You Need
+
+**Required:**
+- Python 3.8+ (you probably already have this)
+- QEMU with ESP32-S3 support (Espressif's fork)
+
+**Install QEMU (one-time setup):**
+
+```bash
+# Easiest: use the automated installer (installs QEMU + Python tools)
+bash scripts/install-qemu.sh
+
+# Or check what's already installed:
+bash scripts/install-qemu.sh --check
+```
+
+The installer detects your OS (Ubuntu, Fedora, macOS, etc.), installs build dependencies, clones Espressif's QEMU fork, builds it, and adds it to your PATH. It also installs the Python tools (`esptool`, `pyyaml`, `esp-idf-nvs-partition-gen`).
+
+<details>
+<summary>Manual installation (if you prefer)</summary>
+
+```bash
+# Build from source
+git clone https://github.com/espressif/qemu.git
+cd qemu
+./configure --target-list=xtensa-softmmu --enable-slirp
+make -j$(nproc)
+export QEMU_PATH=$(pwd)/build/qemu-system-xtensa
+
+# Install Python tools
+pip install esptool pyyaml esp-idf-nvs-partition-gen
+```
+
+</details>
+
+**For multi-node testing (optional):**
+
+```bash
+# Linux only — needed for virtual network bridges
+sudo apt install socat bridge-utils iproute2
+```
+
+### The `qemu-cli.sh` Command
+
+All QEMU testing is available through a single command:
+
+```bash
+bash scripts/qemu-cli.sh <command>
+```
+
+| Command | What it does |
+|---------|-------------|
+| `install` | Install QEMU (runs the installer above) |
+| `test` | Run single-node firmware test |
+| `swarm --preset smoke` | Quick 2-node swarm test |
+| `swarm --preset standard` | Standard 3-node test |
+| `mesh 3` | Multi-node mesh test |
+| `chaos` | Fault injection resilience test |
+| `fuzz --duration 60` | Run fuzz testing |
+| `status` | Show what's installed and ready |
+| `help` | Show all commands |
+
+### Your First Test Run
+
+The simplest way to test the firmware:
+
+```bash
+# Using the CLI:
+bash scripts/qemu-cli.sh test
+
+# Or directly:
+bash scripts/qemu-esp32s3-test.sh
+```
+
+**What happens behind the scenes:**
+1. The firmware is compiled with a "mock CSI" mode — instead of reading real WiFi signals, it generates synthetic test data that mimics real people walking, falling, or breathing
+2. The compiled firmware is loaded into QEMU, which boots it like a real ESP32-S3
+3. The emulator's serial output (what you'd see on a USB cable) is captured
+4. A validation script checks the output for expected behavior and errors
+
+If you already built the firmware and want to skip rebuilding:
+
+```bash
+SKIP_BUILD=1 bash scripts/qemu-esp32s3-test.sh
+```
+
+To give it more time (useful on slower machines):
+
+```bash
+QEMU_TIMEOUT=120 bash scripts/qemu-esp32s3-test.sh
+```
+
+### Understanding the Test Output
+
+The test runs 16 checks on the firmware's output. Here's what a successful run looks like:
+
+```
+=== QEMU ESP32-S3 Firmware Test (ADR-061) ===
+
+[PASS] Boot: Firmware booted successfully
+[PASS] NVS config: Configuration loaded from flash
+[PASS] Mock CSI: Synthetic WiFi data generator started
+[PASS] Edge processing: Signal analysis pipeline running
+[PASS] Frame serialization: Data packets formatted correctly
+[PASS] No crashes: No error conditions detected
+...
+
+16/16 checks passed
+=== Test Complete (exit code: 0) ===
+```
+
+**Exit codes explained:**
+
+| Code | Meaning | What to do |
+|------|---------|-----------|
+| 0 | **PASS** — everything works | Nothing, you're good! |
+| 1 | **WARN** — minor issues | Review the output; usually safe to continue |
+| 2 | **FAIL** — something broke | Check the `[FAIL]` lines for what went wrong |
+| 3 | **FATAL** — can't even start | Usually a missing tool or build failure; check error messages |
+
+### Testing Multiple Nodes at Once (Swarm)
+
+Real deployments use 3-8 ESP32 nodes. The **swarm configurator** lets you simulate multiple nodes on your computer, each with a different role:
+
+- **Sensor nodes** — generate WiFi signal data (like ESP32s placed around a room)
+- **Coordinator node** — collects data from all sensors and runs analysis
+- **Gateway node** — bridges data to your computer
+
+```bash
+# Quick 2-node smoke test (15 seconds)
+python3 scripts/qemu_swarm.py --preset smoke
+
+# Standard 3-node test: 2 sensors + 1 coordinator (60 seconds)
+python3 scripts/qemu_swarm.py --preset standard
+
+# See what's available
+python3 scripts/qemu_swarm.py --list-presets
+
+# Preview what would run (without actually running)
+python3 scripts/qemu_swarm.py --preset standard --dry-run
+```
+
+**Note:** Multi-node testing with virtual bridges requires Linux and `sudo`. On other systems, nodes use a simpler networking mode where each node can reach the coordinator but not each other.
+
+### Swarm Presets
+
+| Preset | Nodes | Duration | Best for |
+|--------|-------|----------|----------|
+| `smoke` | 2 | 15s | Quick check that things work |
+| `standard` | 3 | 60s | Normal development testing |
+| `ci_matrix` | 3 | 30s | CI/CD pipelines |
+| `large_mesh` | 6 | 90s | Testing at scale |
+| `line_relay` | 4 | 60s | Multi-hop relay testing |
+| `ring_fault` | 4 | 75s | Fault tolerance testing |
+| `heterogeneous` | 5 | 90s | Mixed scenario testing |
+
+### Writing Your Own Swarm Config
+
+Create a YAML file describing your test scenario:
+
+```yaml
+# my_test.yaml
+swarm:
+  name: my-custom-test
+  duration_s: 45
+  topology: star       # star, mesh, line, or ring
+  aggregator_port: 5005
+
+nodes:
+  - role: coordinator
+    node_id: 0
+    scenario: 0        # 0=empty room (baseline)
+    channel: 6
+    edge_tier: 2
+
+  - role: sensor
+    node_id: 1
+    scenario: 2        # 2=walking person
+    channel: 6
+    tdm_slot: 1
+
+  - role: sensor
+    node_id: 2
+    scenario: 3        # 3=fall event
+    channel: 6
+    tdm_slot: 2
+
+assertions:
+  - all_nodes_boot           # Did every node start up?
+  - no_crashes               # Any error/panic?
+  - all_nodes_produce_frames # Is each sensor generating data?
+  - fall_detected_by_node_2  # Did node 2 detect the fall?
+```
+
+**Available scenarios** (what kind of fake WiFi data to generate):
+
+| # | Scenario | Description |
+|---|----------|-------------|
+| 0 | Empty room | Baseline with just noise |
+| 1 | Static person | Someone standing still |
+| 2 | Walking | Someone walking across the room |
+| 3 | Fall | Someone falling down |
+| 4 | Multiple people | Two people in the room |
+| 5 | Channel sweep | Cycling through WiFi channels |
+| 6 | MAC filter | Testing device filtering |
+| 7 | Ring overflow | Stress test with burst of data |
+| 8 | RSSI sweep | Signal strength from weak to strong |
+| 9 | Zero-length | Edge case: empty data packet |
+
+**Topology options:**
+
+| Topology | Shape | When to use |
+|----------|-------|-------------|
+| `star` | All sensors connect to one coordinator | Most common setup |
+| `mesh` | Every node can talk to every other | Testing fully connected networks |
+| `line` | Nodes in a chain (A → B → C → D) | Testing relay/forwarding |
+| `ring` | Chain with ends connected | Testing circular routing |
+
+Run your custom config:
+
+```bash
+python3 scripts/qemu_swarm.py --config my_test.yaml
+```
+
+### Debugging Firmware in QEMU
+
+If something goes wrong, you can attach a debugger to the emulated ESP32:
+
+```bash
+# Terminal 1: Start QEMU with debug support (paused at boot)
+qemu-system-xtensa -machine esp32s3 -nographic \
+  -drive file=firmware/esp32-csi-node/build/qemu_flash.bin,if=mtd,format=raw \
+  -s -S
+
+# Terminal 2: Connect the debugger
+xtensa-esp-elf-gdb firmware/esp32-csi-node/build/esp32-csi-node.elf \
+  -ex "target remote :1234" \
+  -ex "break app_main" \
+  -ex "continue"
+```
+
+Or use VS Code: open the project, press **F5**, and select **"QEMU ESP32-S3 Debug"**.
+
+### Running the Full Test Suite
+
+For thorough validation before submitting a pull request:
+
+```bash
+# 1. Single-node test (2 minutes)
+bash scripts/qemu-esp32s3-test.sh
+
+# 2. Multi-node swarm test (1 minute)
+python3 scripts/qemu_swarm.py --preset standard
+
+# 3. Fuzz testing — finds edge-case crashes (1-5 minutes)
+cd firmware/esp32-csi-node/test
+make all CC=clang
+make run_serialize FUZZ_DURATION=60
+make run_edge FUZZ_DURATION=60
+make run_nvs FUZZ_DURATION=60
+
+# 4. NVS configuration matrix — tests 14 config combinations
+python3 scripts/generate_nvs_matrix.py --output-dir build/nvs_matrix
+
+# 5. Chaos testing — injects faults to test resilience (2 minutes)
+bash scripts/qemu-chaos-test.sh
+```
+
+All of these also run automatically in CI when you push changes to `firmware/`.
+
+---
+
 ## Troubleshooting

 ### Docker: "no matching manifest for linux/arm64" on macOS
@@ -1015,6 +1306,47 @@ The server applies a 3-stage smoothing pipeline (ADR-048). If readings are still
 - Hard refresh with Ctrl+Shift+R to clear cached settings
 - The auto-detect probes `/health` on the same origin — cross-origin won't work

+### QEMU: "qemu-system-xtensa: command not found"
+
+QEMU for ESP32-S3 must be built from Espressif's fork — it is not in standard package managers:
+
+```bash
+git clone https://github.com/espressif/qemu.git
+cd qemu && ./configure --target-list=xtensa-softmmu && make -j$(nproc)
+export QEMU_PATH=$(pwd)/build/qemu-system-xtensa
+```
+
+Or point to an existing build: `QEMU_PATH=/path/to/qemu-system-xtensa bash scripts/qemu-esp32s3-test.sh`
+
+### QEMU: Test times out with no output
+
+The emulator is slower than real hardware. Increase the timeout:
+
+```bash
+QEMU_TIMEOUT=120 bash scripts/qemu-esp32s3-test.sh
+```
+
+If there's truly no output at all, the firmware build may have failed. Rebuild without `SKIP_BUILD`:
+
+```bash
+bash scripts/qemu-esp32s3-test.sh   # without SKIP_BUILD
+```
+
+### QEMU: "esptool not found"
+
+Install it with pip: `pip install esptool`
+
+### QEMU Swarm: "Must be run as root"
+
+Multi-node swarm tests with virtual network bridges require root on Linux. Two options:
+
+1. Run with sudo: `sudo python3 scripts/qemu_swarm.py --preset standard`
+2. Skip bridges (nodes use simpler networking): the tool automatically falls back on non-root systems, but nodes can't communicate with each other (only with the aggregator)
+
+### QEMU Swarm: "yaml module not found"
+
+Install PyYAML: `pip install pyyaml`
+
 ---

 ## FAQ