mirror of
https://github.com/ruvnet/RuView
synced 2026-08-11 20:41:44 +00:00
docs: ADR-288/289/290 — benchmark harness, wideband CSI ingest, vitals ground-truth rig
Three ADRs closing the highest-impact gaps from the 2026 SOTA research sweep: public-benchmark comparability (Widar3.0 ingest + leakage-guarded split protocols), wideband 802.11ax CSI ingest (FeitCSI/AX210), and a vitals ground-truth rig (reference ingest, alignment, Bland-Altman agreement, evidence grading). Implementations follow in this branch. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_015TcKegTS7QqhWPC2L2SzaS
This commit is contained in:
@@ -0,0 +1,106 @@
|
|||||||
|
# ADR-288: Public-benchmark evaluation harness — Widar3.0 ingest, standard split protocols, leakage guards
|
||||||
|
|
||||||
|
- **Status**: Accepted — initial implementation (this PR)
|
||||||
|
- **Date**: 2026-08-10
|
||||||
|
- **Deciders**: ruv
|
||||||
|
- **Tags**: training, evaluation, benchmarks, widar, mm-fi, leakage, honesty
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
RuView implements the field's key techniques (CSI ratio, BVP features, MAE
|
||||||
|
pretraining, rapid adaptation) but reports results only on self-collected data
|
||||||
|
with self-defined metrics (e.g. the README's held-out temporal-triplet
|
||||||
|
accuracy). A 2026 deep-research sweep of the WiFi-sensing literature found:
|
||||||
|
|
||||||
|
1. Cross-domain generalization is the field's central unsolved problem; the
|
||||||
|
only widely reproduced cross-domain result is Widar3.0's BVP benchmark.
|
||||||
|
2. MM-Fi (NeurIPS 2023) is the standard WiFi-pose benchmark, with defined
|
||||||
|
cross-subject and cross-environment protocols.
|
||||||
|
3. The field had a documented leakage reckoning in 2024–2025: window-level
|
||||||
|
random splits on continuous recordings inflate accuracy (one dataset's F1
|
||||||
|
collapsed from ~90% to ~22% under subject-disjoint splits — Sensors
|
||||||
|
24(10):3159; Signals 6(4):59).
|
||||||
|
|
||||||
|
`wifi-densepose-train` already has an `MmFiDataset` NPY loader and a
|
||||||
|
deterministic `SyntheticCsiDataset`, but no Widar3.0 ingest, no standard split
|
||||||
|
protocols, and no structural leakage guard. CLAUDE.md already requires
|
||||||
|
mean-pose baselines and leakage-free held-out splits for pose PCK; nothing in
|
||||||
|
the code enforces this.
|
||||||
|
|
||||||
|
Without leaderboard-comparable numbers, RuView's claims cannot be ranked
|
||||||
|
against published systems, which blocks both scientific credibility and
|
||||||
|
commercial (OEM licensing) conversations.
|
||||||
|
|
||||||
|
## Options considered
|
||||||
|
|
||||||
|
1. **Do nothing; keep self-collected metrics.** Rejected: perpetuates the
|
||||||
|
comparability gap.
|
||||||
|
2. **Port a Python eval stack (SenseFi) alongside the Rust pipeline.**
|
||||||
|
Rejected: violates the v2 Rust-workspace direction and adds an unreviewed
|
||||||
|
dependency surface.
|
||||||
|
3. **Extend `wifi-densepose-train` with native loaders + protocol machinery.**
|
||||||
|
Chosen.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Extend `v2/crates/wifi-densepose-train` with three additions:
|
||||||
|
|
||||||
|
### 1. Widar3.0 ingest (`dataset::widar`)
|
||||||
|
|
||||||
|
- A parser for the Intel 5300 `.dat` CSI log format ("bfee" records) used by
|
||||||
|
the Widar3.0 raw distribution: framed records with a 3-byte header
|
||||||
|
(2-byte little-endian length + 1-byte code 0xBB), a 20-byte bfee header
|
||||||
|
(timestamp_low, bfee_count, Nrx, Ntx, RSSI a/b/c, noise, agc, antenna_sel,
|
||||||
|
len, rate), and a packed 10-bit-per-component complex CSI payload of
|
||||||
|
30 subcarrier groups. Invalid records are skipped with a warning, not a
|
||||||
|
panic — untrusted file input is validated at the boundary per CLAUDE.md.
|
||||||
|
- A `WidarDataset` implementing the existing `CsiDataset` trait, mapping
|
||||||
|
Widar's `Nrx × Ntx × 30` CSI into windowed `CsiSample`s via the existing
|
||||||
|
subcarrier interpolation, with domain metadata (user, room, orientation,
|
||||||
|
gesture) parsed from Widar's documented directory/file naming convention.
|
||||||
|
- No network access: the loader reads a local dataset root. Dataset download
|
||||||
|
remains a documented manual step.
|
||||||
|
|
||||||
|
### 2. Split protocols (`protocols`)
|
||||||
|
|
||||||
|
- A `SplitProtocol` type expressing the standard evaluations: cross-subject
|
||||||
|
(MM-Fi style), cross-environment/room, cross-orientation (Widar style), and
|
||||||
|
random-baseline (explicitly labelled as leakage-prone, for comparison only).
|
||||||
|
- Split assignment is a pure function of sample metadata + a seed — fully
|
||||||
|
deterministic, no RNG state.
|
||||||
|
|
||||||
|
### 3. Leakage guards (`protocols::leakage`)
|
||||||
|
|
||||||
|
- A structural `LeakageAudit` that, given a proposed train/test split,
|
||||||
|
verifies: (a) subject-disjointness, (b) environment-disjointness where the
|
||||||
|
protocol claims it, (c) no two windows from the same continuous recording
|
||||||
|
span both sides of the split. A failed audit is an `Err`, not a warning.
|
||||||
|
- PCK/accuracy reporting requires a `MeanPoseBaseline` computed from the
|
||||||
|
training split only, and reports model-vs-baseline together, enforcing the
|
||||||
|
CLAUDE.md rule in the type system rather than by convention.
|
||||||
|
- Evaluation output is an evidence-tagged report (`MEASURED` requires a
|
||||||
|
reproducer command line embedded in the report; anything else is emitted as
|
||||||
|
`SYNTHETIC` or `CLAIMED`).
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- RuView results become comparable to published numbers (Widar3.0 cross-domain
|
||||||
|
gesture; MM-Fi cross-subject pose) for the first time.
|
||||||
|
- The leakage audit will make some existing internal numbers look worse. That
|
||||||
|
is the point.
|
||||||
|
- Parsing a legacy binary format adds maintenance surface; mitigated by
|
||||||
|
fixture-based tests with synthetic, deterministically generated `.dat`
|
||||||
|
bytes (no dataset redistribution).
|
||||||
|
- Widar's raw distribution is Intel 5300-specific; ESP32-captured data
|
||||||
|
continues through existing loaders. The protocols/leakage machinery is
|
||||||
|
loader-agnostic.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
- `cargo test -p wifi-densepose-train` — unit tests for the bfee parser
|
||||||
|
(truncated, corrupt, and valid synthetic fixtures), split determinism,
|
||||||
|
leakage-audit rejection cases, and mean-pose baseline math.
|
||||||
|
- `cargo bench -p wifi-densepose-train` — criterion benchmark for parser
|
||||||
|
throughput and split assignment on synthetic corpora.
|
||||||
|
- No accuracy numbers are claimed by this ADR; it delivers the machinery to
|
||||||
|
produce MEASURED ones.
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
# ADR-289: Wideband 802.11ax CSI ingest — FeitCSI/AX210 adapter and subcarrier-agnostic plumbing
|
||||||
|
|
||||||
|
- **Status**: Accepted — initial implementation (this PR)
|
||||||
|
- **Date**: 2026-08-10
|
||||||
|
- **Deciders**: ruv
|
||||||
|
- **Tags**: hardware, csi, 80211ax, ax210, feitcsi, ingest, mat
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
RuView's CSI ingest (`wifi-densepose-mat/src/integration/hardware_adapter.rs`)
|
||||||
|
supports ESP32 serial streams, the legacy Intel 5300 tool, and Atheros/Nexmon
|
||||||
|
paths. All of these are 802.11n-class: ≤40 MHz bandwidth, ≤114 subcarriers,
|
||||||
|
2.4/5 GHz.
|
||||||
|
|
||||||
|
The 2026 research sweep found the field's center of gravity has moved to
|
||||||
|
Intel AX200/AX210 NICs via PicoScenes (closed-source core) and FeitCSI
|
||||||
|
(open-source, GPL): 802.11ax CSI at up to 160 MHz / 1992 subcarriers,
|
||||||
|
including the 6 GHz band. This is both the research-grade tier today and the
|
||||||
|
shape of the data 802.11bf silicon will deliver from ~2026 onward. RuView's
|
||||||
|
`wifi-densepose-hardware` crate already models 802.11bf session types, but no
|
||||||
|
ingest path can carry wideband CSI into the pipeline.
|
||||||
|
|
||||||
|
Without a wideband path, RuView cannot develop against the best available
|
||||||
|
signal, cannot compare ESP32-grade results to wideband upper bounds, and will
|
||||||
|
meet 802.11bf silicon with no tested plumbing for >114-subcarrier frames.
|
||||||
|
|
||||||
|
## Options considered
|
||||||
|
|
||||||
|
1. **PicoScenes `.csi` ingest.** Rejected for now: the format is produced by a
|
||||||
|
closed-source core and is versioned/complex; parsing it without a
|
||||||
|
maintained spec invites silent corruption.
|
||||||
|
2. **Raw pcap + radiotap parsing.** Rejected: duplicates what FeitCSI already
|
||||||
|
does on-device, and pulls a packet-capture dependency into the pipeline.
|
||||||
|
3. **FeitCSI file/stream ingest.** Chosen: FeitCSI is open-source (its header
|
||||||
|
layout is auditable against the source), targets AX200/AX210, covers
|
||||||
|
20–160 MHz including 6 GHz, and emits a compact binary record per frame.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Extend `v2/crates/wifi-densepose-mat/src/integration` with:
|
||||||
|
|
||||||
|
### 1. `feitcsi` record parser
|
||||||
|
|
||||||
|
- A validated parser for FeitCSI's binary CSI record layout (header with
|
||||||
|
CSI buffer length, rate/bandwidth/channel metadata, antenna counts, RSSI,
|
||||||
|
timestamp, followed by interleaved complex CSI). The parser is written
|
||||||
|
against the documented layout, is version-checked, and rejects
|
||||||
|
records whose declared dimensions disagree with the buffer length —
|
||||||
|
untrusted file/stream input is validated at the boundary.
|
||||||
|
- Bounded allocation: a hard cap on subcarrier count (4096) and antenna
|
||||||
|
count (8) so a corrupt length field cannot cause unbounded allocation.
|
||||||
|
|
||||||
|
### 2. `DeviceType::FeitCsi` in the hardware adapter
|
||||||
|
|
||||||
|
- File-replay mode (read a recorded FeitCSI capture deterministically) and a
|
||||||
|
streaming mode fed by an external process writing to a path/pipe. No
|
||||||
|
privileged operations inside the crate: RuView does not configure the NIC;
|
||||||
|
FeitCSI's own tooling owns that, per least-authority.
|
||||||
|
|
||||||
|
### 3. Subcarrier-agnostic plumbing
|
||||||
|
|
||||||
|
- Ingest carries native subcarrier dimensionality end-to-end and converts to
|
||||||
|
pipeline width explicitly via the existing interpolation/decimation stage,
|
||||||
|
recording the native → pipeline mapping in frame metadata so downstream
|
||||||
|
consumers know the true spectral resolution. Bandwidth (20–160 MHz) and
|
||||||
|
band (2.4/5/6 GHz) become first-class frame metadata.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- RuView gains a research-grade wideband development path and a tested
|
||||||
|
ingest shape for future 802.11bf reporting (truncated CIR is a natural
|
||||||
|
extension of the same plumbing).
|
||||||
|
- GPL FeitCSI is used as an external tool, never linked: only its output
|
||||||
|
format is parsed. No licensing contamination of the MIT workspace.
|
||||||
|
- The parser tracks an external project's format; version checks fail loudly
|
||||||
|
on mismatch rather than misparse.
|
||||||
|
- ESP32 remains the deployed sensor tier; wideband is a development/
|
||||||
|
validation tier. Accuracy claims from wideband captures must be tagged with
|
||||||
|
the capture hardware.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
- `cargo test -p wifi-densepose-mat` — parser tests over synthetic fixtures:
|
||||||
|
valid records at 20/80/160 MHz shapes, truncated buffer, dimension
|
||||||
|
mismatch, version mismatch, allocation-cap enforcement; adapter replay
|
||||||
|
determinism.
|
||||||
|
- `cargo bench -p wifi-densepose-mat` — criterion benchmark for record parse
|
||||||
|
throughput at 1992-subcarrier frames.
|
||||||
|
- Hardware validation on real AX210 silicon is explicitly out of scope for
|
||||||
|
this PR and remains required (per CLAUDE.md) before any capture-path
|
||||||
|
hardware claim; the file-replay path is testable without silicon.
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
# ADR-290: Vitals ground-truth rig — reference ingest, time alignment, and agreement metrics
|
||||||
|
|
||||||
|
- **Status**: Accepted — initial implementation (this PR)
|
||||||
|
- **Date**: 2026-08-10
|
||||||
|
- **Deciders**: ruv
|
||||||
|
- **Tags**: vitals, validation, ground-truth, bland-altman, evidence, honesty
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
`wifi-densepose-vitals` (ADR-021) extracts breathing (0.1–0.5 Hz) and heart
|
||||||
|
rate (0.8–2.0 Hz) from CSI. The 2026 research sweep found that every credible
|
||||||
|
vitals result in the literature ships with reference-sensor ground truth
|
||||||
|
(chest strap, pulse oximeter, ECG, or PSG), and that WiFi heart-rate numbers
|
||||||
|
without stated scope (single person, static, line-of-sight, short range) are
|
||||||
|
systematically misleading. RuView currently has no way to produce a MEASURED
|
||||||
|
vitals number: there is no reference-signal ingest, no time alignment between
|
||||||
|
CSI-derived estimates and a reference device, and no agreement statistics.
|
||||||
|
|
||||||
|
CLAUDE.md requires accuracy statements to be tagged MEASURED (with a
|
||||||
|
reproducer), CLAIMED, or SYNTHETIC. For vitals, MEASURED is currently
|
||||||
|
unreachable.
|
||||||
|
|
||||||
|
## Options considered
|
||||||
|
|
||||||
|
1. **Live BLE/ANT+ integration with reference devices.** Rejected for now:
|
||||||
|
drivers and pairing are a hardware/product concern; the blocking gap is
|
||||||
|
the evaluation math, not the radio link.
|
||||||
|
2. **File-based reference ingest + offline agreement analysis.** Chosen:
|
||||||
|
every consumer reference device (Polar, Garmin, oximeters) exports
|
||||||
|
timestamped series; a file boundary keeps the crate dependency-free and
|
||||||
|
the pipeline deterministic.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Add a `groundtruth` module to `v2/crates/wifi-densepose-vitals`:
|
||||||
|
|
||||||
|
### 1. Reference series ingest
|
||||||
|
|
||||||
|
- `ReferenceSeries`: timestamped samples (unix millis + value) for one
|
||||||
|
measurand (`HeartRateBpm` or `BreathingRateBrpm`), with device metadata
|
||||||
|
(make/model, measurement principle). Parsed from CSV (`timestamp_ms,value`
|
||||||
|
with a header line); malformed rows are rejected with row-numbered errors —
|
||||||
|
untrusted file input validated at the boundary. Non-monotonic timestamps
|
||||||
|
are an error, not silently sorted.
|
||||||
|
|
||||||
|
### 2. Time alignment
|
||||||
|
|
||||||
|
- Constant-offset estimation by maximizing normalized cross-correlation of
|
||||||
|
the estimate series against the reference over a bounded lag window
|
||||||
|
(default ±30 s), on a common resampled grid (nearest-sample, no
|
||||||
|
interpolation of physiological values across gaps larger than a
|
||||||
|
configurable limit).
|
||||||
|
- Optional linear clock-drift fit (offset + rate) for long sessions.
|
||||||
|
Alignment parameters are reported, never silently applied.
|
||||||
|
|
||||||
|
### 3. Agreement metrics
|
||||||
|
|
||||||
|
- `AgreementReport`: n paired samples, coverage fraction (time where both
|
||||||
|
series had valid samples), MAE, RMSE, mean error (bias), Bland–Altman
|
||||||
|
95% limits of agreement, and percentage-within-tolerance (configurable,
|
||||||
|
default ±2 bpm HR / ±1 brpm breathing).
|
||||||
|
- Session scope is mandatory metadata: subject count, motion state
|
||||||
|
(static/moving), line-of-sight (LOS/NLOS/through-wall), distance band.
|
||||||
|
A report without scope cannot be constructed.
|
||||||
|
|
||||||
|
### 4. Evidence tagging
|
||||||
|
|
||||||
|
- `EvidenceGrade::Measured` is only constructible when the report carries a
|
||||||
|
reference device, non-zero paired samples, minimum coverage, and a
|
||||||
|
reproducer command string; otherwise the report grades as `Claimed` (real
|
||||||
|
data, no reference) or `Synthetic` (generated input). This mirrors
|
||||||
|
ADR-288's enforcement-in-types approach and the CLAUDE.md tagging rule.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- RuView can convert vitals claims from CLAIMED to MEASURED with a
|
||||||
|
reproducible offline analysis, session by session, scope by scope.
|
||||||
|
- Honest reporting will likely show heart-rate performance below marketing
|
||||||
|
intuition, especially NLOS/moving — that is the purpose.
|
||||||
|
- CSV ingest means a manual export step per session; acceptable at current
|
||||||
|
scale, and the format is the de-facto export of consumer reference gear.
|
||||||
|
- No clinical claim is implied: agreement statistics against consumer
|
||||||
|
reference devices are engineering evidence, not medical validation.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
- `cargo test -p wifi-densepose-vitals` — CSV rejection cases, alignment
|
||||||
|
recovery of known synthetic offsets/drifts, agreement metrics against
|
||||||
|
hand-computed fixtures, evidence-grade constructibility rules.
|
||||||
|
- `cargo bench -p wifi-densepose-vitals` — criterion benchmark for alignment
|
||||||
|
over hour-scale synthetic sessions.
|
||||||
|
- Real-session validation (ESP32 capture + chest strap) remains a follow-up
|
||||||
|
requiring hardware evidence per CLAUDE.md.
|
||||||
@@ -145,6 +145,9 @@ Statuses: **Proposed** (under discussion), **Accepted** (approved and/or impleme
|
|||||||
| [ADR-287](ADR-287-coherent-wideband-rf-tomography-crate.md) | `wifi-densepose-sar` — coherent wideband RF tomography research crate | Accepted (implemented, published) |
|
| [ADR-287](ADR-287-coherent-wideband-rf-tomography-crate.md) | `wifi-densepose-sar` — coherent wideband RF tomography research crate | Accepted (implemented, published) |
|
||||||
| [ADR-285](ADR-285-homecore-wasm-first-metaharness.md) | WASM-first Homecore developer metaharness via `npx homecore` | Accepted (implemented and validated) |
|
| [ADR-285](ADR-285-homecore-wasm-first-metaharness.md) | WASM-first Homecore developer metaharness via `npx homecore` | Accepted (implemented and validated) |
|
||||||
| [ADR-286](ADR-286-wifi-densepose-sar-harness-via-metaharness.md) | `wifi-densepose-sar-harness` — MetaHarness with darwin/router/flywheel | Accepted (implemented, published) |
|
| [ADR-286](ADR-286-wifi-densepose-sar-harness-via-metaharness.md) | `wifi-densepose-sar-harness` — MetaHarness with darwin/router/flywheel | Accepted (implemented, published) |
|
||||||
|
| [ADR-288](ADR-288-public-benchmark-evaluation-harness.md) | Public-benchmark evaluation harness — Widar3.0 ingest, split protocols, leakage guards | Accepted (initial implementation) |
|
||||||
|
| [ADR-289](ADR-289-wideband-80211ax-csi-ingest.md) | Wideband 802.11ax CSI ingest — FeitCSI/AX210 adapter, subcarrier-agnostic plumbing | Accepted (initial implementation) |
|
||||||
|
| [ADR-290](ADR-290-vitals-ground-truth-rig.md) | Vitals ground-truth rig — reference ingest, alignment, agreement metrics | Accepted (initial implementation) |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user