ruvnet--RuView/docs/adr/ADR-291-public-benchmark-evaluation-harness.md

ADR-291: Public-benchmark evaluation harness — Widar3.0 ingest, standard split protocols, leakage guards

• Status: Accepted — initial implementation (this PR)
• Date: 2026-08-10
• Deciders: ruv
• Tags: training, evaluation, benchmarks, widar, mm-fi, leakage, honesty

Context

RuView implements the field's key techniques (CSI ratio, BVP features, MAE pretraining, rapid adaptation) but reports results only on self-collected data with self-defined metrics (e.g. the README's held-out temporal-triplet accuracy). A 2026 deep-research sweep of the WiFi-sensing literature found:

1. Cross-domain generalization is the field's central unsolved problem; the only widely reproduced cross-domain result is Widar3.0's BVP benchmark.
2. MM-Fi (NeurIPS 2023) is the standard WiFi-pose benchmark, with defined cross-subject and cross-environment protocols.
3. The field had a documented leakage reckoning in 2024–2025: window-level random splits on continuous recordings inflate accuracy (one dataset's F1 collapsed from ~90% to ~22% under subject-disjoint splits — Sensors 24(10):3159; Signals 6(4):59).

wifi-densepose-train already has an MmFiDataset NPY loader and a deterministic SyntheticCsiDataset, but no Widar3.0 ingest, no standard split protocols, and no structural leakage guard. CLAUDE.md already requires mean-pose baselines and leakage-free held-out splits for pose PCK; nothing in the code enforces this.

Without leaderboard-comparable numbers, RuView's claims cannot be ranked against published systems, which blocks both scientific credibility and commercial (OEM licensing) conversations.

Options considered

1. Do nothing; keep self-collected metrics. Rejected: perpetuates the comparability gap.
2. Port a Python eval stack (SenseFi) alongside the Rust pipeline. Rejected: violates the v2 Rust-workspace direction and adds an unreviewed dependency surface.
3. Extend wifi-densepose-train with native loaders + protocol machinery. Chosen.

Decision

Extend v2/crates/wifi-densepose-train with three additions:

1. Widar3.0 ingest (dataset::widar)

• A parser for the Intel 5300 .dat CSI log format ("bfee" records) used by the Widar3.0 raw distribution: framed records with a 3-byte header (2-byte little-endian length + 1-byte code 0xBB), a 20-byte bfee header (timestamp_low, bfee_count, Nrx, Ntx, RSSI a/b/c, noise, agc, antenna_sel, len, rate), and a packed 10-bit-per-component complex CSI payload of 30 subcarrier groups. Invalid records are skipped with a warning, not a panic — untrusted file input is validated at the boundary per CLAUDE.md.
• A WidarDataset implementing the existing CsiDataset trait, mapping Widar's Nrx × Ntx × 30 CSI into windowed CsiSamples via the existing subcarrier interpolation, with domain metadata (user, room, orientation, gesture) parsed from Widar's documented directory/file naming convention.
• No network access: the loader reads a local dataset root. Dataset download remains a documented manual step.

2. Split protocols (protocols)

• A SplitProtocol type expressing the standard evaluations: cross-subject (MM-Fi style), cross-environment/room, cross-orientation (Widar style), and random-baseline (explicitly labelled as leakage-prone, for comparison only).
• Split assignment is a pure function of sample metadata + a seed — fully deterministic, no RNG state.

3. Leakage guards (protocols::leakage)

• A structural LeakageAudit that, given a proposed train/test split, verifies: (a) subject-disjointness, (b) environment-disjointness where the protocol claims it, (c) no two windows from the same continuous recording span both sides of the split. A failed audit is an Err, not a warning.
• PCK/accuracy reporting requires a MeanPoseBaseline computed from the training split only, and reports model-vs-baseline together, enforcing the CLAUDE.md rule in the type system rather than by convention.
• Evaluation output is an evidence-tagged report (MEASURED requires a reproducer command line embedded in the report; anything else is emitted as SYNTHETIC or CLAIMED).

Consequences

• RuView results become comparable to published numbers (Widar3.0 cross-domain gesture; MM-Fi cross-subject pose) for the first time.
• The leakage audit will make some existing internal numbers look worse. That is the point.
• Parsing a legacy binary format adds maintenance surface; mitigated by fixture-based tests with synthetic, deterministically generated .dat bytes (no dataset redistribution).
• Widar's raw distribution is Intel 5300-specific; ESP32-captured data continues through existing loaders. The protocols/leakage machinery is loader-agnostic.

Validation

• cargo test -p wifi-densepose-train — unit tests for the bfee parser (truncated, corrupt, and valid synthetic fixtures), split determinism, leakage-audit rejection cases, and mean-pose baseline math.
• cargo bench -p wifi-densepose-train — criterion benchmark for parser throughput and split assignment on synthetic corpora.
• No accuracy numbers are claimed by this ADR; it delivers the machinery to produce MEASURED ones.