mirror of
https://github.com/ruvnet/RuView
synced 2026-06-25 12:53:19 +00:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c27d6cc98e |
@@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
- **Multistatic fusion guard interval is now operator-configurable — fixes permanent trust demotion with WiFi-synced ESP32 nodes (#1049).** Two independently-clocked ESP32-S3 boards on ESP-NOW sync drift 10–150 ms (typ. ~70 ms) — the 100 ms beacon + WiFi-MAC jitter cannot hold them within the published 60 ms default guard, so the governed-trust cycle permanently demoted to `Restricted`, suppressed all pose output, and spun the error counter to 200k+ with **no escape hatch but a container restart**. Added a **direct `WDP_GUARD_INTERVAL_US` override** (+ optional `WDP_SOFT_GUARD_US`) to `multistatic_guard_config_from_env`, so a deployment can lift the hard guard past its measured spread (e.g. `WDP_GUARD_INTERVAL_US=200000`) without having to know its exact TDM schedule. Precedence is most-specific-wins: a direct override beats the existing `WDP_TDM_SLOTS`+`WDP_TDM_SLOT_US` schedule-derived guard, which beats the 60 ms/20 ms default; the override is applied on top of whichever base is selected, the soft band is always clamped strictly below the hard guard, and a malformed/zero value is ignored (falls back to the base rather than breaking fusion). The effective guard is now logged at startup. Pinned by 6 new tests (`multistatic_guard_config_tests`): direct-override-wins / beats-TDM-derived / soft-clamped-below-hard / lowering-hard-pulls-soft-down / malformed-or-zero-falls-back / default-when-unset. `wifi-densepose-sensing-server` bin tests **449 → 455**, 0 failed; Python proof VERDICT PASS, hash unchanged (off the signal proof path).
|
||||
|
||||
### Security
|
||||
- **`wifi-densepose-occworld-candle` — beyond-SOTA security + correctness review (Milestone #9, crate 4/4).** (1) **HIGH (MEASURED) — checkpoint-load crash on any int32 tensor** (`model.rs::safetensor_dtype_to_candle`). `safetensors::Dtype::I32` was mapped to `candle_core::DType::I64` and the raw int32 byte buffer (4 bytes/elem) was then handed to `Tensor::from_raw_buffer(.., I64, shape, ..)`. Candle derives `elem_count = data.len() / dtype.size_in_bytes()`, so the I64 path halved the element count while keeping the *original* shape — yielding a tensor whose declared shape claims twice as many elements as its backing storage holds. Reading it **panics** (`range end index 6 out of range for slice of length 3` — slice OOB inside candle-core) on any attacker-supplied or PyTorch-exported checkpoint containing an int32 tensor (common: index/buffer tensors). Fixed by mapping `I32 → DType::I32` (and `I16 → DType::I16`), both first-class candle dtypes. Reproduction recorded on old code; pinned by `tests/checkpoint_loading.rs::int32_tensor_loads_with_consistent_shape_and_values` (panics on old, passes on new) plus F32/I64/corrupt-file control cases. (2) **LOW (MEASURED) — `predict()` lacked frame/batch validation at the input boundary** (`inference.rs`). It validated H/W/D but not the externally-supplied frame count; an `f_in > num_frames*2` over-indexed the temporal positional embedding deep in the transformer and surfaced as a cryptic candle "gather" `InvalidIndex` (returned error, not a panic — candle bounds-checks), and a zero frame/batch dim fed a zero-element tensor into the pipeline. Now rejected at the boundary with a clear `ShapeMismatch`. Pinned by `predict_rejects_zero_frames` / `predict_rejects_too_many_frames` / `predict_accepts_frame_count_at_capacity`. (3) **LOW (MEASURED) — divide-by-zero panic on a degenerate input to the public `VQCodebook::encode`** (`vqvae.rs`): a rank-0 / empty-last-dim tensor made `last == 0` and panicked on `elem_count() / last`. Now fails closed with a clear error. Pinned by `encode_rejects_scalar_without_panicking`. **Dimensions confirmed CLEAN with evidence:** panic surface — zero `unwrap()`/`expect()`/`panic!`/`unreachable!` in production code paths (grep evidence; all error handling via `?`/`map_err`); NaN-state-poisoning — N/A (engine is stateless between `predict` calls, input is `u8` class indices so non-finite input is structurally impossible, no persistent world-model buffer to latch into); unbounded-alloc / shape-data mismatch from malformed weights — defended upstream by `safetensors::validate()` (overflow-checked `nelements*dtype.size()` vs declared byte range, rejected before reaching candle); secrets — none (grep clean, only `token_h`/`token_w` config fields match). `unsafe_code = forbid` in the crate manifest. **Build/validation status (MEASURED on Windows):** crate builds and tests under `cargo test -p wifi-densepose-occworld-candle --no-default-features` — **29/29 pass** (20 unit + 4 checkpoint_loading + 3 predict_honesty + 2 doc) after fixes; `cargo test --workspace --no-default-features` = 0 failed across all crates (lone `wifi-densepose-desktop` `api_integration` failure was a Windows "Access is denied (os error 5)" file-lock flake — re-ran in isolation **21/21 pass**); Python proof VERDICT PASS, hash `f8e76f21…446f7a` unchanged. *Warrants ADR slot 179 (parent to author).*
|
||||
- **`wifi-densepose-wasm-edge` beyond-SOTA closing review — boundary NaN-state-poisoning guard + clean-with-evidence attestation (ADR-040 edge crate, ~70 modules).** Closing pass of the security campaign over the last untouched sizeable crate. **One real finding fixed (LOW / source-analysis + reproduced):** the two WASM↔host frame boundaries (`lib.rs::on_frame`/`on_timer` and `bin/ghost_hunter.rs::on_frame`) read raw IEEE-754 `f32` from the `csi_get_phase`/`csi_get_amplitude`/`csi_get_variance`/`csi_get_motion_energy` host imports **without any finiteness check** — the entire crate had **zero** `is_finite`/`is_nan` guards, and the in-crate `clamp` helpers propagate NaN (`NaN < lo` and `NaN > hi` are both false). A single non-finite value (firmware DSP bug, uninitialised buffer, or hostile host) latches NaN into the long-lived per-module accumulators (EMA, Welford, phasor sums, anomaly baselines); once latched, every downstream comparison evaluates `false`, so detectors fail **degraded** (stuck gate state, silently-disabled anomaly checks) — silent corruption, not a crash (WASM `panic=abort` is *not* tripped: no indexing/`unwrap` on the poisoned value). Threat model is a **semi-trusted** boundary (the Tier-2 DSP firmware supplies the imports, not direct network/JS), hence LOW severity / defense-in-depth. **Fix:** added `sanitize_host_f32()` (maps non-finite→`0.0`, `core`-only so it holds in `no_std`) applied at every `host_get_*` float read — a single chokepoint covering all ~70 downstream modules, mirroring the existing M-01 negative-`n_subcarriers` boundary clamp. **Pinned by** `boundary_tests::{sanitize_passes_finite_values_through, sanitize_maps_non_finite_to_zero, coherence_monitor_nan_latches_without_sanitize_but_not_with}` — the last asserts on the *current* `CoherenceMonitor` that a raw NaN frame latches the smoothed score (documents the hazard) while the boundary-sanitized path stays finite. **Dimensions attested CLEAN with evidence (source-analysis):** (a) **panic-on-input** — every non-test `unwrap()`/`expect()` is either `#[cfg(test)]` or in the `std`-gated RVF *builder* host tool writing to an in-memory `Vec` (infallible); no `panic!`/`unreachable!`/`todo!`/`get_unchecked` in any hot path. (b) **shape/bounds** — all frame-buffer access is `min()`-clamped (`MAX_SC=32`, `DTW_MAX_LEN`, `LCS_WINDOW`, `PATTERN_LEN`), all index-by-cast sites (`feature_id as usize`, `conclusion_id`, `minute_counter`, `plan_step`) are either compile-time-const-bounded or `if idx <`/`%`-guarded; negative `n_subcarriers` already mapped to 0 (M-01). (c) **memory/leak** — no `move ||` closures, no `mem::forget`/`Box::leak`/`.leak()`; the only `Box::new` is in the `std`-gated `skill_registry` (one-time init, bounded). (d) **secrets** — none (grep clean). **MEASURED build/test evidence:** host `cargo test --features std,medical-experimental` = **672 passed / 0 failed** (was 669 pre-fix; +3 new tests); the real deployment artifacts all build clean on the actual target — `cargo build --target wasm32-unknown-unknown --release` (no_std/panic=abort default lib), `--bin ghost_hunter --no-default-features --features standalone-bin`, and `--features medical-experimental` (toolchain 1.89 per `rust-toolchain.toml`). No ADR slot needed — a single LOW defense-in-depth boundary fix; CHANGELOG attestation suffices.
|
||||
|
||||
@@ -1,272 +0,0 @@
|
||||
# ADR-180: Through-Wall Camera↔CSI Hand-off Demo ("Behind the Wall")
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| **Status** | Proposed |
|
||||
| **Date** | 2026-06-15 |
|
||||
| **Deciders** | ruv |
|
||||
| **Codename** | **BEHIND-THE-WALL** |
|
||||
| **Builds on** | ADR-079 (camera ground-truth training), ADR-031 (sensing-first RF mode), ADR-134 (CSI→CIR multipath), ADR-029/030 (RuvSense multistatic + persistent field), ADR-024 (AETHER re-ID), ADR-151 (per-room calibration), ADR-173 (metric-locked PCK), ADR-095/096 (rvcsi nexmon) |
|
||||
|
||||
## Context
|
||||
|
||||
### The demo we want
|
||||
A single self-contained **HTML page** that tells one honest, visceral story:
|
||||
|
||||
1. You stand in front of the laptop. The camera tracks your **full skeletal pose**;
|
||||
the WiFi-CSI model, trained on *your* movements moments earlier, infers the **same
|
||||
skeleton** in parallel — a side-by-side "camera vs RF agree" view.
|
||||
2. You **walk out the door and behind the wall**. The camera **goes blind** (you are
|
||||
occluded — it honestly shows "no person in frame"). The CSI model **keeps inferring
|
||||
your skeleton** from the WiFi signal alone — the 3D figure keeps walking, behind the
|
||||
wall, smoothly. A badge flips from `CAMERA` to `RF-INFERRED (through-wall)`.
|
||||
3. You **walk back into view**. The camera **re-acquires**; the badge flips back to
|
||||
`CAMERA`, and the RF-inferred and camera skeletons reconverge.
|
||||
|
||||
This is the "WiFi sees through walls" demo — and the user explicitly wants the **inferred
|
||||
skeleton through the wall**, not just a blob. The project's "prove everything / no AI-slop"
|
||||
bar means we make that claim **only because we measure it**: a second camera on the far side
|
||||
of the wall records ground-truth pose *behind* the wall, so the through-wall skeleton's
|
||||
accuracy is a **reported, reproducible number** — never an unfalsifiable "trust me."
|
||||
|
||||
### Honest capability framing (the load-bearing section)
|
||||
Through-wall **per-joint skeletal inference from WiFi CSI is not a generally-validated
|
||||
capability** in open settings — WiFi-DensePose (CMU) is camera-*co-located*. What makes it
|
||||
defensible *here* is the tightly-controlled regime and the measurement:
|
||||
|
||||
- **Controlled regime:** one room, one subject (you), one doorway, a model **camera-supervised
|
||||
on your exact gait and your exact through-door transition** (ADR-079) minutes earlier. This
|
||||
is in-distribution for *this* demo, not a universal claim.
|
||||
- **Measured, not asserted:** a far-side camera (cognitum-v0 has 17 `/dev/video*` nodes — use
|
||||
one, or a phone) records ground-truth pose behind the wall. The through-wall CSI skeleton is
|
||||
scored against it with the metric-locked PCK harness (ADR-173). **We publish the number.**
|
||||
- **Uncertainty is rendered, not hidden:** the through-wall skeleton is drawn **translucent**,
|
||||
with a live **per-joint confidence** and an explicit `RF-INFERRED` badge. High-confidence
|
||||
joints render solid; low-confidence joints fade. It never masquerades as the camera's
|
||||
ground-truth pose.
|
||||
|
||||
| While… | Camera | WiFi CSI (S3 / Pi5 nexmon, fused) | 60 GHz mmWave (C6 + MR60BHA2) |
|
||||
|--------|--------|-----------------------------------|-------------------------------|
|
||||
| In frame | **Full 17-kpt pose** — ground truth | full skeleton (supervised model) — *agrees with camera* | presence + range + micro-motion |
|
||||
| Behind a **drywall** | nothing (occluded) | **inferred full skeleton** (camera-supervised model + multistatic fusion), confidence-scored, **measured vs far-side camera** | presence + range + breathing — independent through-thin-wall confirm |
|
||||
| Behind **brick/metal** | nothing | degrades to coarse motion/position only — report honestly | blocked |
|
||||
|
||||
**The claim — stated precisely:** *"A WiFi-CSI model, camera-supervised on this subject and
|
||||
room, infers a continuous skeletal pose that tracks the subject through a drywall partition;
|
||||
through-wall accuracy is measured at X% PCK@k against a far-side camera (declared, not
|
||||
claimed)."* If X turns out low, that is the **honest result we report** — the skeleton is still
|
||||
rendered (the user wants it) but flagged with its true confidence, and the headline number is
|
||||
whatever we measured, good or bad.
|
||||
|
||||
### Why multistatic + supervision is the enabler
|
||||
A single node behind a wall sees only "something moved." Three spatially-diverse vantage points
|
||||
around the doorway (RuvSense multistatic + cross-viewpoint fusion, ADR-029/030) triangulate the
|
||||
moving scatterer — drywall attenuates and diffracts 2.4/5 GHz but does not block it — giving the
|
||||
model a rich enough multipath signature to regress a skeleton it was *trained* to associate with
|
||||
your through-door motion. AETHER re-ID embeddings (ADR-024) keep it locked to **you** across the
|
||||
camera→RF→camera hand-off.
|
||||
|
||||
### Available hardware (the user's actual rig)
|
||||
| Role | Device | Where | Stream |
|
||||
|------|--------|-------|--------|
|
||||
| Near ground truth (visible) | Laptop / USB camera | front of workstation (ruvzen) | MediaPipe pose → keypoints |
|
||||
| **Far ground truth (validation)** | cognitum-v0 camera (1 of 17 `/dev/video*`) or a phone | **behind the wall** | MediaPipe pose → keypoints (for MEASURING the through-wall skeleton) |
|
||||
| CSI node A | ESP32-S3 (8 MB) | COM9 (ruvzen) | UDP CSI :5005 |
|
||||
| CSI + mmWave node B | ESP32-C6 + Seeed MR60BHA2 | COM12 (ruvzen) | WiFi CSI + 60 GHz FMCW presence/range |
|
||||
| CSI node C (through-wall vantage) | Pi 5, BCM43455c0 | cognitum-v0 (other room) | nexmon_csi `.pcap` → rvcsi → CsiFrame |
|
||||
| Fusion + serving | sensing-server | ruvzen :3000/:8765 | `/ws/sensing`, `/ws/pose`, new `/ws/handoff` |
|
||||
|
||||
Place **node C (Pi 5) and the far camera on the far side of the wall** — the Pi 5 gives the
|
||||
fuser a vantage the camera lacks, and the far camera turns the through-wall claim into a
|
||||
measurement.
|
||||
|
||||
## Decision
|
||||
|
||||
Build a **camera↔CSI hand-off demo** as a thin, additive layer over existing components (no new
|
||||
heavy crate). Five parts: a multi-source capture plane, a camera-supervised calibration walk
|
||||
that **learns to infer the skeleton through the wall**, a **hand-off state machine**, a
|
||||
**dead-reckoning smoother** so dropped CSI never makes the figure jump, and a single-file HTML
|
||||
viewer that renders the inferred skeleton with honest confidence.
|
||||
|
||||
### 1. Capture plane (reuse, don't rebuild)
|
||||
- **Near camera:** `scripts/collect-ground-truth.py` already does MediaPipe pose + ESP32 CSI
|
||||
paired capture (ADR-079). Extend it to also subscribe to the Pi 5 nexmon stream (rvcsi), the
|
||||
C6 mmWave presence, **and the far camera**, so every frame is
|
||||
`(near_pose|null, far_pose|null, csi_S3, csi_C6, mmwave_C6, csi_Pi5, t)`.
|
||||
- **CSI nodes:** S3 over UDP :5005, Pi 5 via `rvcsi` (vendor/rvcsi nexmon adapter → `CsiFrame`),
|
||||
C6 WiFi CSI + the MR60BHA2 60 GHz presence/range/breathing.
|
||||
- **Fusion:** all CSI sources into the existing `MultistaticFuser`
|
||||
(`signal/src/ruvsense/multistatic.rs`); node positions around the doorway via
|
||||
`--node-positions` (geometric-diversity index drives confidence). **#1049:** with 3
|
||||
independently-clocked nodes set `WDP_GUARD_INTERVAL_US` to the real inter-node spread or
|
||||
fusion demotes.
|
||||
|
||||
### 2. Calibration walk — "it learns my movements **and infers them through the wall**" (ADR-079)
|
||||
A 3–5 minute guided routine. The HTML page scripts the walk: stand, step left/right, walk to the
|
||||
door, **cross fully behind the wall and back**, repeat — covering the visible AND the occluded
|
||||
zone, because **both cameras label ground truth**:
|
||||
- **Visible-zone supervision:** near camera labels pose; synchronized CSI window is the input.
|
||||
- **Through-wall supervision (the key part):** while you are behind the wall, the **far camera**
|
||||
labels your pose. So the CSI→skeleton model is trained on *real behind-wall poses* paired with
|
||||
the *behind-wall multistatic CSI* — the model genuinely learns to infer your skeleton through
|
||||
the wall, supervised by ground truth, not extrapolated blindly.
|
||||
- Train/fine-tune on `ruvultra` (RTX 5080) if available, else the local recipe. Persist as a
|
||||
per-room calibration bank (ADR-151 `baseline → enroll → extract → train`). AETHER re-ID
|
||||
embeddings (ADR-024) bind the track to you across the hand-off.
|
||||
- **Held-out split:** reserve some behind-wall passes for evaluation so through-wall PCK is
|
||||
measured on data the model never trained on (no leakage — the ADR-152 measurement discipline).
|
||||
|
||||
### 3. Hand-off state machine (`sensing-server/src/handoff.rs`, < 300 lines)
|
||||
States: `CAMERA` → `HANDOFF_OUT` → `RF_INFERRED` → `HANDOFF_IN` → `CAMERA` (+ `LOST`).
|
||||
- **`CAMERA`** — near camera has a confident pose → render it; RF-inferred skeleton ghosted
|
||||
alongside for the "they agree" effect.
|
||||
- **`HANDOFF_OUT`** — near-camera confidence drops at the doorway **while** CSI motion stays high
|
||||
and the multistatic track heads into the door zone → cross-fade source camera→RF.
|
||||
- **`RF_INFERRED`** — no camera pose; the CSI model emits a **full 17-kpt skeleton** + per-joint
|
||||
confidence; AETHER confirms it is still you. Render the translucent skeleton + confidence,
|
||||
badge `RF-INFERRED (through-wall)`. (When fusion confidence is too low for a credible skeleton,
|
||||
degrade gracefully to a coarse marker rather than a flailing one — honest fallback.)
|
||||
- **`HANDOFF_IN`** — near camera re-acquires a pose positionally consistent with the last RF
|
||||
skeleton (continuity gate) → cross-fade RF→camera.
|
||||
- **`LOST`** — neither source for N cycles → "no track," never invented.
|
||||
|
||||
Fail-closed: `RF_INFERRED` requires real multistatic motion energy + an AETHER identity match
|
||||
above calibrated floors; absent that → `LOST`, never a phantom. Mirrors the governed-trust gate
|
||||
(ADR-031 / ADR-141).
|
||||
|
||||
### 4. Dead reckoning & smoothing — fluid, never jumpy (the user's requirement)
|
||||
CSI does **not** arrive cleanly: UDP frames drop, nexmon `.pcap` has gaps, the fuser skips
|
||||
cycles when the #1049 guard rejects a spread, and the model's per-frame skeleton jitters. Render
|
||||
only on real frames and the figure teleports and shakes — which also *reads as fake*. A
|
||||
**predict/correct (dead-reckoning) layer** keeps the skeleton continuous and smooth between
|
||||
measurements, with **bounded** extrapolation so we never invent motion that didn't happen:
|
||||
|
||||
- **Per-joint constant-velocity Kalman filter** — reuse `signal/src/ruvsense/pose_tracker.rs`
|
||||
(the project's existing 17-keypoint Kalman tracker with AETHER re-ID). The renderer runs at a
|
||||
**fixed ~30 Hz, decoupled from CSI arrival**:
|
||||
- **Measurement this tick** → Kalman *update* (correct) each joint with the new inferred pose.
|
||||
- **Dropped CSI this tick** → Kalman *predict* only: advance each joint by `x += v·dt`, so the
|
||||
skeleton keeps moving along its trajectory instead of freezing then snapping. **This is the
|
||||
dead reckoning** — the limbs keep their motion through a dropout.
|
||||
- **Confidence decay (honesty governor):** every predict-only tick multiplies confidence and
|
||||
widens covariance. Dead reckoning is trusted for a **bounded** horizon (default ≤ ~500 ms,
|
||||
`WDP_DEADRECKON_MAX_MS`); past it, confidence hits the floor → state machine → `LOST`. **We
|
||||
coast briefly to stay smooth; we never coast forever to fake a track.** Someone who actually
|
||||
stopped behind the wall converges to a still pose then `LOST`, not perpetual phantom walking.
|
||||
- **Re-acquire smoothing:** a returning measurement after a gap is blended in with a
|
||||
critically-damped step (no overshoot) over 2–3 ticks, so the skeleton eases onto truth.
|
||||
- **Client render smoothing (already present):** `ui/observatory/js/figure-pool.js`
|
||||
`applyKeypoints` already `lerp`s joints with a small velocity overshoot for secondary motion;
|
||||
the hand-off viewer reuses it. The camera↔RF cross-fade is an alpha-lerp over ~300 ms.
|
||||
|
||||
**Dead-reckoning honesty invariants (testable):**
|
||||
1. Predicted-only frames carry `"dead_reckoned": true` + `"age_ms"`; the UI dims them —
|
||||
extrapolation is never shown as a fresh measurement.
|
||||
2. Confidence is **monotonically non-increasing** across consecutive predict-only ticks.
|
||||
3. After `WDP_DEADRECKON_MAX_MS` of silence the state **must** become `LOST` (pinned test:
|
||||
measurements then silence → assert transition within the horizon; no perpetual motion).
|
||||
4. Dead reckoning extrapolates an **existing** track only — no measurement ever ⇒ no track ⇒
|
||||
`LOST`, never a phantom from zero.
|
||||
|
||||
### 5. The HTML demo (single file, vanilla — mirrors the Observatory)
|
||||
`ui/through-wall/index.html` (+ a small JS bundle, zero build step, like `ui/observatory/`):
|
||||
- **Left:** near camera feed with the MediaPipe skeleton overlaid while visible; greys to
|
||||
"CAMERA BLIND" when occluded. (Optional second tile: the far camera, shown only in a
|
||||
"validation" view, not the hero view.)
|
||||
- **Right:** a top-down 3D room (Three.js) with the **wall** drawn, the doorway, the three
|
||||
sensor positions, and the figure: a **solid skeleton** in `CAMERA`, a **translucent skeleton
|
||||
with per-joint confidence fade** in `RF_INFERRED`, eased by the dead-reckoning smoother.
|
||||
- **Banner / `BannerState`** (strict, mirrors rufield-viewer): `CAMERA` / `RF-INFERRED — through
|
||||
wall (conf X%, measured Y% PCK@k)` / `DEAD-RECKONED (age N ms)` / `LOST` — mutually exclusive,
|
||||
with a one-line honesty caption. The measured through-wall PCK is shown, not invented.
|
||||
- Consumes a new `GET /ws/handoff` WS/SSE topic of `HandoffFrame`s; `?demo=1` replays a recorded
|
||||
session badged `REPLAY`.
|
||||
|
||||
### Output contract (`HandoffFrame`, JSON)
|
||||
```jsonc
|
||||
{
|
||||
"t_ns": 1718400000000,
|
||||
"state": "RF_INFERRED", // CAMERA | HANDOFF_OUT | RF_INFERRED | HANDOFF_IN | LOST
|
||||
"source": "fused_csi", // camera | fused_csi | mmwave | dead_reckoned
|
||||
"pose": [[x,y,z,conf], …×17], // inferred skeleton WITH per-joint confidence (present in CAMERA/HANDOFF/RF_INFERRED)
|
||||
"pose_confidence": 0.58, // aggregate; the rendered translucency
|
||||
"identity_match": 0.81, // AETHER re-ID — is it still you?
|
||||
"coarse": { "cell":[x,y], "zone":"behind_wall", "heading_deg":95, "node_diversity":0.48 },
|
||||
"dead_reckoned": false, // true on predict-only (extrapolated) ticks
|
||||
"age_ms": 0, // ms since the last real measurement (0 = fresh)
|
||||
"camera_blind": true,
|
||||
"measured_pck": { "k": 20, "value": null }, // filled from the far-camera validation run; null until measured
|
||||
"caption": "RF-inferred skeleton — model camera-supervised on this room; through-wall PCK measured separately"
|
||||
}
|
||||
```
|
||||
|
||||
## Phased plan (each phase independently demoable + falsifiable)
|
||||
- **P1 — wiring (no claim):** 3-source CSI capture (S3+C6+Pi5) + near camera into the multistatic
|
||||
fuser. Gate: `/ws/sensing` shows ≥3 active nodes + a fused position with the camera running.
|
||||
- **P2 — supervised calibration + through-wall training:** the guided walk with **both cameras**;
|
||||
fine-tune CSI→skeleton on visible AND far-camera-labeled behind-wall poses (ADR-079). Gate:
|
||||
while-visible PCK declared (metric-locked, ADR-173) on a held-out segment.
|
||||
- **P3 — MEASURE the through-wall skeleton:** score the RF-inferred skeleton against the far
|
||||
camera on held-out behind-wall passes → **publish the through-wall PCK@k** (good or bad). Gate:
|
||||
a committed eval script reproduces the number; honest negative if low.
|
||||
- **P4 — hand-off + dead reckoning + HTML:** the camera→RF→camera transition renders end-to-end,
|
||||
smooth through dropped CSI. Gate: a recorded live walk where the camera goes blind, the inferred
|
||||
skeleton keeps walking fluidly behind the wall, dead-reckons through dropouts without jumps, and
|
||||
re-acquisition is position-continuous. **This is the demo.**
|
||||
- **P5 — multi-modal corroboration (optional):** overlay C6 60 GHz presence/range as an
|
||||
independent through-thin-wall confirm (two physics, one conclusion).
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- A genuinely compelling demo that does what the user asked — **infers and renders the skeleton
|
||||
through the wall** — while staying honest because the through-wall accuracy is **measured**
|
||||
against a far-side camera, not claimed. Reuses the multistatic fuser, ADR-079 supervision, the
|
||||
Kalman pose tracker, AETHER re-ID, the calibration crate, and the Observatory UI: the new code
|
||||
is a hand-off module + dead-reckoning smoother + an HTML page.
|
||||
|
||||
### Negative / Risks
|
||||
- **Through-wall skeletal accuracy may be modest or poor.** That is acceptable *iff* reported
|
||||
honestly — the headline is the measured PCK, whatever it is; the skeleton renders with its true
|
||||
per-joint confidence (low-confidence joints fade), never as fake certainty.
|
||||
- **Material dependence:** drywall good; brick/metal degrades to coarse-only — shoot on drywall
|
||||
and say so.
|
||||
- **3-node clock sync** is the #1049 hazard — tune `WDP_GUARD_INTERVAL_US`.
|
||||
- **Per-room, per-subject:** the model that "learned your movements" does not transfer without
|
||||
re-calibration — stated on the page.
|
||||
- **Over-claiming is the failure mode.** Mitigations baked in: translucent confidence-faded
|
||||
skeleton, `dead_reckoned`/`age_ms` flags, the measured-PCK banner, bounded extrapolation→`LOST`.
|
||||
|
||||
### Neutral
|
||||
- No new heavy crate; signal-path proof (`verify.py`) untouched — capture/fusion/UI orchestration
|
||||
over hardened, already-reviewed components.
|
||||
|
||||
## Acceptance criteria (falsifiable — "prove the haters wrong")
|
||||
On a recorded live session, all must hold:
|
||||
1. A contiguous window where the **near camera reports no person** (verifiable from raw frames)
|
||||
**and** the system renders an `RF_INFERRED` skeleton.
|
||||
2. The inferred skeleton's **gross motion matches reality** — direction of travel and rough gait
|
||||
phase — confirmed against the **far camera** (not eyeballed).
|
||||
3. **Through-wall per-joint accuracy is MEASURED** against the far camera and **reported** as
|
||||
PCK@k from a committed script. Low is fine *if* honestly published; fabricated is not.
|
||||
4. The figure is **smooth through dropped CSI** — no teleports/jitter — and every predicted-only
|
||||
frame is flagged `dead_reckoned`; after `WDP_DEADRECKON_MAX_MS` of silence it goes `LOST`.
|
||||
5. Re-acquisition is **position-continuous** (camera re-detects within a cell of the last RF
|
||||
position), and AETHER confirms identity across the hand-off.
|
||||
6. Every number (visible PCK, through-wall PCK, confidences) is MEASURED and reproducible — no
|
||||
hand-typed metrics.
|
||||
|
||||
A demo that cannot meet (1)–(2) and (4)–(5) on the available hardware is reported as a **negative
|
||||
result** (honest), not dressed up; a poor (3) is published as the real number.
|
||||
|
||||
## Links
|
||||
- ADR-079 — camera ground-truth training (supervision pipeline; extended here to a far camera)
|
||||
- ADR-031 — sensing-first RF mode / coherence gate (fail-closed honesty pattern)
|
||||
- ADR-134 — CSI→CIR multipath (through-wall multipath physics)
|
||||
- ADR-029 / ADR-030 — RuvSense multistatic + persistent field (the localization engine)
|
||||
- ADR-024 — AETHER contrastive re-ID (identity lock across the hand-off)
|
||||
- ADR-151 — per-room calibration crate (bank persistence)
|
||||
- ADR-152 / ADR-173 — measurement discipline + metric-locked PCK (the honest accuracy readout)
|
||||
- ADR-095 / ADR-096 — rvcsi nexmon (Pi 5 BCM43455c0 capture)
|
||||
- `signal/src/ruvsense/pose_tracker.rs` — 17-kpt Kalman tracker reused for dead reckoning
|
||||
- `ui/observatory/` — the vanilla-JS 3D viewer pattern this demo mirrors
|
||||
@@ -6391,32 +6391,71 @@ fn vitals_snapshots_from_sensing_json(
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the multistatic guard config, optionally derived from the TDM schedule
|
||||
/// declared in the environment (#1031).
|
||||
/// Build the multistatic guard config from the environment (#1031, #1049).
|
||||
///
|
||||
/// When both `WDP_TDM_SLOTS` and `WDP_TDM_SLOT_US` parse as positive integers,
|
||||
/// the guard is derived via [`MultistaticConfig::for_tdm_schedule`] so a
|
||||
/// deployment can match its exact schedule. Otherwise the published default
|
||||
/// (60 ms hard / 20 ms soft) is returned. `min_nodes` is *not* set here — the
|
||||
/// caller overrides it for single-node passthrough.
|
||||
/// Three precedence layers, most-specific wins:
|
||||
/// 1. `WDP_GUARD_INTERVAL_US` (+ optional `WDP_SOFT_GUARD_US`) — a **direct**
|
||||
/// hard-guard override. This is the #1049 escape hatch: WiFi/ESP-NOW-synced
|
||||
/// ESP32 nodes drift 10–150 ms (the 100 ms beacon + WiFi-MAC jitter cannot
|
||||
/// hold two independently-clocked boards within the published default), so a
|
||||
/// deployment can simply lift the guard past its measured spread (e.g.
|
||||
/// `WDP_GUARD_INTERVAL_US=200000`) without knowing its exact TDM schedule.
|
||||
/// 2. `WDP_TDM_SLOTS` + `WDP_TDM_SLOT_US` (both positive) — derive the guard
|
||||
/// from the declared schedule via [`MultistaticConfig::for_tdm_schedule`].
|
||||
/// 3. Otherwise the published default (60 ms hard / 20 ms soft).
|
||||
///
|
||||
/// The direct override (1) is applied **on top of** whichever base (2 or 3) is
|
||||
/// selected, so `WDP_GUARD_INTERVAL_US` always wins for the hard guard while a
|
||||
/// TDM-derived soft band is preserved unless it would exceed the new hard guard.
|
||||
/// `min_nodes` is *not* set here — the caller overrides it for single-node
|
||||
/// passthrough.
|
||||
fn multistatic_guard_config_from_env() -> MultistaticConfig {
|
||||
multistatic_guard_config_from(
|
||||
std::env::var("WDP_TDM_SLOTS").ok().as_deref(),
|
||||
std::env::var("WDP_TDM_SLOT_US").ok().as_deref(),
|
||||
std::env::var("WDP_GUARD_INTERVAL_US").ok().as_deref(),
|
||||
std::env::var("WDP_SOFT_GUARD_US").ok().as_deref(),
|
||||
)
|
||||
}
|
||||
|
||||
/// Pure core of [`multistatic_guard_config_from_env`] for testability.
|
||||
fn multistatic_guard_config_from(slots: Option<&str>, slot_us: Option<&str>) -> MultistaticConfig {
|
||||
match (
|
||||
fn multistatic_guard_config_from(
|
||||
slots: Option<&str>,
|
||||
slot_us: Option<&str>,
|
||||
guard_us: Option<&str>,
|
||||
soft_us: Option<&str>,
|
||||
) -> MultistaticConfig {
|
||||
// Base: TDM-schedule-derived when both slot params are valid, else default.
|
||||
let mut cfg = match (
|
||||
slots.and_then(|s| s.trim().parse::<usize>().ok()),
|
||||
slot_us.and_then(|s| s.trim().parse::<u64>().ok()),
|
||||
) {
|
||||
(Some(n), Some(us)) if n >= 1 && us >= 1 => {
|
||||
MultistaticConfig::for_tdm_schedule(n, us)
|
||||
}
|
||||
(Some(n), Some(us)) if n >= 1 && us >= 1 => MultistaticConfig::for_tdm_schedule(n, us),
|
||||
_ => MultistaticConfig::default(),
|
||||
};
|
||||
|
||||
// Direct hard-guard override (#1049). Ignored when unset/zero/unparseable so
|
||||
// a malformed env var falls back to the base rather than breaking fusion.
|
||||
if let Some(g) = guard_us
|
||||
.and_then(|s| s.trim().parse::<u64>().ok())
|
||||
.filter(|&g| g >= 1)
|
||||
{
|
||||
cfg.guard_interval_us = g;
|
||||
// Keep the soft band strictly below the (possibly lowered) hard guard.
|
||||
if cfg.soft_guard_us >= g {
|
||||
cfg.soft_guard_us = g.saturating_sub(1).max(1);
|
||||
}
|
||||
}
|
||||
|
||||
// Optional explicit soft-guard override, always clamped strictly below hard.
|
||||
if let Some(s) = soft_us
|
||||
.and_then(|s| s.trim().parse::<u64>().ok())
|
||||
.filter(|&s| s >= 1)
|
||||
{
|
||||
cfg.soft_guard_us = s.min(cfg.guard_interval_us.saturating_sub(1).max(1));
|
||||
}
|
||||
|
||||
cfg
|
||||
}
|
||||
|
||||
/// Turn a `ProgressiveLoader::new` failure into an actionable diagnostic (#894).
|
||||
@@ -7485,11 +7524,16 @@ async fn main() {
|
||||
pose_tracker: PoseTracker::new(),
|
||||
last_tracker_instant: None,
|
||||
multistatic_fuser: {
|
||||
// #1031: the default guard (60 ms hard / 20 ms soft) accommodates a
|
||||
// real TDM slot offset. A deployment can override it to match its
|
||||
// own schedule via WDP_TDM_SLOTS + WDP_TDM_SLOT_US (both set ⇒ derive
|
||||
// from the schedule), else the published default is used.
|
||||
// #1031/#1049: the default guard (60 ms hard / 20 ms soft)
|
||||
// accommodates a real TDM slot offset. A deployment overrides it via
|
||||
// WDP_GUARD_INTERVAL_US (direct, e.g. 200000 for WiFi/ESP-NOW sync —
|
||||
// #1049) or WDP_TDM_SLOTS + WDP_TDM_SLOT_US (derive from schedule).
|
||||
let cfg = multistatic_guard_config_from_env();
|
||||
info!(
|
||||
"Multistatic fusion guard: {} µs hard / {} µs soft (override via \
|
||||
WDP_GUARD_INTERVAL_US / WDP_SOFT_GUARD_US, or WDP_TDM_SLOTS+WDP_TDM_SLOT_US)",
|
||||
cfg.guard_interval_us, cfg.soft_guard_us
|
||||
);
|
||||
let mut fuser = MultistaticFuser::with_config(MultistaticConfig {
|
||||
min_nodes: 1, // single-node passthrough
|
||||
..cfg
|
||||
@@ -7797,6 +7841,72 @@ async fn main() {
|
||||
info!("Server shut down cleanly");
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod multistatic_guard_config_tests {
|
||||
//! #1049 — the multistatic guard interval must be operator-configurable so a
|
||||
//! WiFi/ESP-NOW deployment (10–150 ms inter-node clock drift) can lift the
|
||||
//! guard past its measured timestamp spread instead of being permanently
|
||||
//! demoted to Restricted with no escape hatch.
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn default_guard_when_nothing_set() {
|
||||
let cfg = multistatic_guard_config_from(None, None, None, None);
|
||||
assert_eq!(cfg.guard_interval_us, MultistaticConfig::default().guard_interval_us);
|
||||
assert_eq!(cfg.soft_guard_us, MultistaticConfig::default().soft_guard_us);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn direct_guard_override_wins_and_unblocks_wifi_spread() {
|
||||
// The #1049 reporter's measured ~70 ms spread exceeds the 60 ms default
|
||||
// → permanent demotion. A direct 200 ms override accepts it.
|
||||
let cfg = multistatic_guard_config_from(None, None, Some("200000"), None);
|
||||
assert_eq!(cfg.guard_interval_us, 200_000);
|
||||
assert!(cfg.soft_guard_us < cfg.guard_interval_us);
|
||||
// 70 ms spread now sits inside the guard.
|
||||
assert!(70_000 < cfg.guard_interval_us);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn direct_guard_override_beats_tdm_derived() {
|
||||
// Both TDM params AND a direct override set → the direct hard guard wins,
|
||||
// the TDM-derived soft band is preserved (still strictly below hard).
|
||||
let cfg = multistatic_guard_config_from(Some("2"), Some("18000"), Some("200000"), None);
|
||||
assert_eq!(cfg.guard_interval_us, 200_000);
|
||||
assert!(cfg.soft_guard_us < cfg.guard_interval_us);
|
||||
assert!(cfg.soft_guard_us >= 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn soft_override_is_clamped_strictly_below_hard() {
|
||||
// A soft guard ≥ hard would be nonsensical → clamped below the hard guard.
|
||||
let cfg = multistatic_guard_config_from(None, None, Some("50000"), Some("999999"));
|
||||
assert_eq!(cfg.guard_interval_us, 50_000);
|
||||
assert!(cfg.soft_guard_us < 50_000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lowering_hard_below_default_soft_pulls_soft_down() {
|
||||
// Override hard to 10 ms (< default 20 ms soft) → soft drops below it.
|
||||
let cfg = multistatic_guard_config_from(None, None, Some("10000"), None);
|
||||
assert_eq!(cfg.guard_interval_us, 10_000);
|
||||
assert!(cfg.soft_guard_us < 10_000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn malformed_or_zero_override_falls_back_to_base() {
|
||||
// Garbage / zero must not break fusion — fall back to the base config.
|
||||
for bad in ["", "abc", "0", "-5", "12.5"] {
|
||||
let cfg = multistatic_guard_config_from(None, None, Some(bad), None);
|
||||
assert_eq!(
|
||||
cfg.guard_interval_us,
|
||||
MultistaticConfig::default().guard_interval_us,
|
||||
"override {bad:?} should be ignored"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod node_sync_snapshot_serialization_tests {
|
||||
//! ADR-110 iter 24 — JSON public-API contract for the iter 23
|
||||
|
||||
Reference in New Issue
Block a user