mirror of
https://github.com/ruvnet/RuView
synced 2026-07-28 18:21:42 +00:00
Closes the two documentation gaps from #1456 (follow-up to #1401): - docs/calibration-guide.md: what calibrate/enroll/train-room actually enforce, grounded in v2/crates/wifi-densepose-calibration and wifi-densepose-cli source (not just ADR-135/151 aspirational prose). Covers the hard 600-frame baseline minimum, per-anchor quality gate thresholds, the unsolved pet/small-motion presence-detection gap, and what the empty-room baseline capture actually needs (steady vs silent). Flags that ADR-135's drift_score/BaselineDrift staleness system is not implemented in code — only bank.rs's baseline_id STALE check is real. - docs/trust-and-engine-errors.md: exact trigger conditions for engine_error_count vs the separate, non-sticky `demoted` privacy-class flag, where both are exposed (/health/ready and /api/v1/status share a handler), the real diagnostic gap (no per-cause breakdown, log line is the closest thing), the WDP_GUARD_INTERVAL_US recovery path for persistent clock-drift demotion, and an honest "no code path found" answer on whether a converted HuggingFace model explains engine errors. Also adds both docs to the README documentation table. No code changes.
This commit is contained in:
rUv
@@ -632,6 +632,8 @@ Verify the plugin structure: `bash plugins/ruview/scripts/smoke.sh`. Full detail
|
||||
|----------|-------------|
|
||||
| [User Guide](docs/user-guide.md) | Step-by-step guide: installation, first run, API usage, hardware setup, training |
|
||||
| [Build Guide](docs/build-guide.md) | Building from source (Rust and Python) |
|
||||
| [Calibration & Room Training Guide](docs/calibration-guide.md) | What `calibrate`/`enroll`/`train-room` actually enforce: minimum frame counts, per-anchor quality gates, the pet/small-motion presence-detection caveat, and empty-room baseline conditions — grounded in the real code, not just ADR-135/151 |
|
||||
| [Trust State & Engine Errors](docs/trust-and-engine-errors.md) | What `engine_error_count` and `demoted` mean on `/api/v1/status`, exact trigger conditions, the current diagnostic gap (no per-cause breakdown), the `WDP_GUARD_INTERVAL_US` recovery path, and why a converted Hugging Face model isn't shown to be the cause in code |
|
||||
| [**Home Assistant + Matter Integration**](docs/integrations/home-assistant.md) | **Works with Home Assistant** via MQTT auto-discovery + **Works with Matter** (Apple Home / Google Home / Alexa / SmartThings) — full entity catalog, 3 starter blueprints, Lovelace dashboards, privacy mode, threshold tuning ([ADR-115](docs/adr/ADR-115-home-assistant-integration.md)). |
|
||||
| [**BFLD — Beamforming Feedback Layer for Detection**](v2/crates/wifi-densepose-bfld/README.md) | New privacy-gated WiFi sensing layer that measures + structurally prevents identity leakage from 802.11ac/ax Beamforming Feedback Information. Three type-enforced invariants (raw BFI never exits node, identity embedding is in-RAM-only, cross-site correlation cryptographically impossible via per-site BLAKE3 keyed hash + daily rotation). Ships full operator surface (`BfldPipeline`, `BfldPipelineHandle`, the Soul Signature §3.6 per-channel matcher `EnrolledMatcher`/`SoulMatchOracle` — experimental; named identity is data-gated, **measured** as not-separable on WiFi-only channels alone), MQTT topic router + HA-DISCO + availability + LWT, 3 operator HA blueprints, two runnable examples, eclipse-mosquitto:2 CI service container. 327+ tests. [ADR-118](docs/adr/ADR-118-bfld-beamforming-feedback-layer-for-detection.md) umbrella + sub-ADRs [119](docs/adr/ADR-119-bfld-frame-format-and-wire-protocol.md)/[120](docs/adr/ADR-120-bfld-privacy-class-and-hash-rotation.md)/[121](docs/adr/ADR-121-bfld-identity-risk-scoring.md)/[122](docs/adr/ADR-122-bfld-ruview-ha-matter-exposure.md)/[123](docs/adr/ADR-123-bfld-capture-path-nexmon-and-esp32.md). Research dossier: [`docs/research/BFLD/`](docs/research/BFLD/) (11 files, 13,544 words). |
|
||||
| [**SENSE-BRIDGE — rvagent MCP server**](tools/ruview-mcp/README.md) | Dual-transport MCP server (`@ruvnet/rvagent`) bridging the RuView sensing stack to AI agents (Claude Code, Cursor, ruflo swarms). 6 tools wired: `ruview.presence.now`, `ruview.vitals.get_{breathing,heart_rate,all}`, `ruview.bfld.last_scan`, `ruview.bfld.subscribe`. stdio + Streamable HTTP (`POST /mcp`, Origin-validated, bearer-token auth, `127.0.0.1` bind). Full 20-tool Zod schema barrel + 5 RUVIEW-POLICY governance tools. 93 tests. [ADR-124](docs/adr/ADR-124-rvagent-mcp-ruvector-npm-integration.md). Try: `npx @ruvnet/rvagent stdio`. |
|
||||
|
||||
@@ -0,0 +1,300 @@
|
||||
# Calibration & Room Training Guide
|
||||
|
||||
This guide explains what actually happens — and what is actually *enforced* —
|
||||
when you run `wifi-densepose calibrate`, `enroll`, and `train-room`. It is
|
||||
written for the person setting up a room, not for developers.
|
||||
|
||||
Everything below was checked against the real Rust implementation in
|
||||
`v2/crates/wifi-densepose-calibration/`, `v2/crates/wifi-densepose-signal/src/ruvsense/calibration.rs`,
|
||||
and `v2/crates/wifi-densepose-cli/`, not just the design ADRs. Where the design
|
||||
documents (ADR-135, ADR-151) describe something that isn't actually built yet,
|
||||
this guide says so explicitly.
|
||||
|
||||
## The three-step pipeline
|
||||
|
||||
```
|
||||
wifi-densepose calibrate --port <PORT> # Stage 1: empty-room baseline (no people)
|
||||
wifi-densepose enroll --room <NAME> # Stage 2+3: 8 guided anchors (~4 minutes)
|
||||
wifi-densepose train-room --room <NAME> # Stage 4: fit the specialist bank
|
||||
wifi-densepose room-status --room <NAME> # check what trained / what's stale
|
||||
wifi-densepose room-watch --room <NAME> # live inference
|
||||
```
|
||||
|
||||
`calibrate` must run first — `enroll` refuses to start without a baseline file
|
||||
(`--baseline ./baseline.bin` by default), and `train-room` refuses to start
|
||||
without an enrollment file. Each step writes a file the next step reads; there
|
||||
is no way to skip a step.
|
||||
|
||||
---
|
||||
|
||||
## 1. Is there a minimum amount of data required?
|
||||
|
||||
**Yes, and for the empty-room baseline it is a hard, enforced minimum — not a
|
||||
recommendation.**
|
||||
|
||||
`wifi-densepose calibrate` will not produce a baseline file with fewer than
|
||||
**600 recorded frames** (the default for every PHY tier: HT20, HT40, HE20,
|
||||
HE40). This is `DEFAULT_MIN_FRAMES = 600` in
|
||||
`v2/crates/wifi-densepose-signal/src/ruvsense/calibration.rs:48`, and it is
|
||||
checked in `CalibrationRecorder::finalize()`
|
||||
(`ruvsense/calibration.rs:532-538`): if fewer than `config.min_frames` frames
|
||||
were recorded, `finalize()` returns
|
||||
`CalibrationError::InsufficientFrames { got, need }` and calibration fails
|
||||
outright — there is no partial/degraded baseline. This is pinned by a unit
|
||||
test (`finalize_requires_min_frames`, same file) so it isn't accidental
|
||||
behavior.
|
||||
|
||||
**Important subtlety:** the CLI's `--duration-s` flag (default 30 seconds)
|
||||
and the 600-frame minimum are checked *independently*. The capture loop in
|
||||
`v2/crates/wifi-densepose-cli/src/calibrate.rs:135-183` stops as soon as
|
||||
**either** the duration timer expires **or** 600 frames have been recorded,
|
||||
whichever comes first. If your node streams CSI slower than the assumed 20 Hz
|
||||
(e.g. congested WiFi, a busier ESP32), 30 seconds may not be enough to reach
|
||||
600 frames, and `calibrate` will fail with an explicit
|
||||
`"insufficient frames: have X, need 600"` error rather than silently
|
||||
producing a short baseline. If you hit this, raise `--duration-s` rather than
|
||||
overriding `--min-frames`.
|
||||
|
||||
You *can* override the 600-frame floor with `--min-frames <N>` (0 = use the
|
||||
tier default). The code prints an explicit warning when you do:
|
||||
|
||||
> `[calibrate] WARN: --min-frames=N overrides ADR-135 tier default (600 for
|
||||
> ht20). This relaxes the phase-concentration guarantee; do not use in
|
||||
> production.`
|
||||
|
||||
(`v2/crates/wifi-densepose-cli/src/calibrate.rs:112-119`). Treat this as a
|
||||
debugging escape hatch, not a supported way to shorten setup.
|
||||
|
||||
The CLI also independently rejects `--duration-s` below 10 seconds
|
||||
(`"Fewer frames produce unreliable phase-concentration estimates"`,
|
||||
`calibrate.rs:341-348`) and prints (but does not block on) a warning above 300
|
||||
seconds.
|
||||
|
||||
**Guided enrollment (`enroll`) has a much lower, per-anchor floor.** Each of
|
||||
the 8 guided anchors (`empty`, `stand_still`, `sit`, `lie_down`,
|
||||
`breathe_slow`, `breathe_normal`, `small_move`, `sleep_posture`) is captured
|
||||
for a fixed duration baked into the code — 20 seconds for the static/motion
|
||||
anchors, 30 seconds for the two breathing anchors and `sleep_posture`
|
||||
(`AnchorLabel::duration_s()`, `v2/crates/wifi-densepose-calibration/src/anchor.rs:98-104`).
|
||||
This is **not** a CLI flag — you cannot currently shorten or lengthen an
|
||||
individual anchor capture from the command line.
|
||||
|
||||
Underneath that fixed duration, the anchor is only *accepted* if it clears a
|
||||
quality gate (`AnchorQualityGate`, `v2/crates/wifi-densepose-calibration/src/enrollment.rs:43-53`):
|
||||
|
||||
| Threshold | Default | What it checks |
|
||||
|---|---|---|
|
||||
| `min_frames` | **60 frames** | Anchor is rejected if fewer than 60 frames were captured — mainly catches "the ESP32 stopped streaming" mid-capture, not a real duration requirement (60 frames is a fraction of a second of streaming at typical rates) |
|
||||
| `min_presence_z` | 1.5 | For anchors that expect a person, the mean amplitude z-score must exceed this or the anchor is rejected as "no person detected" |
|
||||
| `empty_max_z` | 1.0 | For the `empty` anchor, the z-score must stay under this or it's rejected as "room not empty" |
|
||||
| `max_still_motion` | 0.6 (60%) | For still anchors, motion-flagged frame fraction above this is rejected as "too much motion" |
|
||||
| `min_move_motion` | 0.3 (30%) | For `small_move`, motion-flagged fraction below this is rejected as "not enough motion" |
|
||||
|
||||
A rejected anchor is re-prompted, up to `--attempts` times (default **2**).
|
||||
If an anchor is still rejected after all attempts, `enroll` moves on without
|
||||
it and logs `"moving on without '<label>'"` — enrollment does **not** abort;
|
||||
you end up with a partial anchor set.
|
||||
|
||||
**`train-room` itself enforces almost nothing.** It only bails if the
|
||||
enrollment file has *zero* accepted anchors at all
|
||||
(`v2/crates/wifi-densepose-cli/src/room.rs:246-248`, `"no accepted anchors …
|
||||
re-run enroll"`). There is no minimum anchor count beyond that. What actually
|
||||
happens with a partial anchor set is that individual specialists silently
|
||||
fail to train and are simply absent from the resulting bank — for example
|
||||
(from `v2/crates/wifi-densepose-calibration/src/specialist.rs`):
|
||||
|
||||
- **presence** needs the `empty` anchor plus at least one anchor where a
|
||||
person was expected present — missing either, `PresenceSpecialist::train()`
|
||||
returns `None` and presence detection is unavailable in that bank.
|
||||
- **anomaly** needs at least 2 anchors total, of any kind.
|
||||
- **restlessness** needs `sleep_posture` (or `lie_down` as a fallback) *and*
|
||||
`small_move`.
|
||||
- **posture** needs at least one anchor that establishes a posture
|
||||
(`stand_still`, `sit`, `lie_down`, or `sleep_posture`).
|
||||
|
||||
So a "successful" `train-room` run can still produce a bank missing one or
|
||||
more specialists if enrollment didn't collect the anchors those specialists
|
||||
need. `room-status` (`v2/crates/wifi-densepose-cli/src/room.rs`) is the way
|
||||
to check what actually trained.
|
||||
|
||||
### What we could not verify
|
||||
|
||||
The ADR-151 design document (§2.2) claims total guided enrollment is
|
||||
"~4 minutes of wall-clock" — that arithmetic checks out against the coded
|
||||
per-anchor durations (5 × 20s + 3 × 30s = 190s ≈ 3.2 min, plus a 3-second
|
||||
countdown before each anchor ≈ +24s, so ~3.5–4 minutes is consistent with the
|
||||
code). But we found **no integration test or measurement showing that this
|
||||
duration is sufficient for reliable specialist accuracy** — the ADR's own
|
||||
status section says the full `baseline → enroll → train-room → infer` loop is
|
||||
proven only against **deterministic synthetic CSI** (`tests/full_loop.rs`),
|
||||
not yet run start-to-finish on real hardware in an empty room. Treat the
|
||||
default durations as reasonable code defaults, not as a validated minimum for
|
||||
real-world accuracy.
|
||||
|
||||
---
|
||||
|
||||
## 2. Recommended duration if there's no hard minimum
|
||||
|
||||
Where a hard minimum *does* exist (the 600-frame baseline, the 60-frame
|
||||
per-anchor floor), it's documented above. Beyond that:
|
||||
|
||||
- **Baseline capture**: the CLI default (`--duration-s 30`) is the number to
|
||||
use; it's what the 600-frame minimum is designed around at the assumed
|
||||
20 Hz sensing rate. ADR-135 §2.3 argues 30 s is the shortest duration that
|
||||
keeps the phase-concentration estimate's standard deviation under
|
||||
0.02 rad², citing published circular-statistics error bounds — but this is
|
||||
a paper-derived justification for the *default value*, not a code-enforced
|
||||
floor beyond the 600-frame check itself.
|
||||
- **Enrollment anchors**: use the built-in per-anchor durations (20s/30s) —
|
||||
there's currently no way to change them from the CLI anyway.
|
||||
|
||||
---
|
||||
|
||||
## 3. Will a pet get classified as "occupied"?
|
||||
|
||||
**Honest answer: the code has no way to distinguish a pet (or any small/animal-scale
|
||||
motion) from a person.** This is a real limitation, not a solved problem —
|
||||
flagging it here rather than guessing.
|
||||
|
||||
Presence detection (`PresenceSpecialist`,
|
||||
`v2/crates/wifi-densepose-calibration/src/specialist.rs:100-198`) is trained
|
||||
purely from two scalar channels measured during enrollment:
|
||||
|
||||
- **variance** of the CSI amplitude series, thresholded at the midpoint
|
||||
between the `empty` anchor's variance and the mean variance of the
|
||||
person-present anchors;
|
||||
- **mean shift** — `|mean − empty_mean|`, thresholded at half the
|
||||
empty→occupied mean distance.
|
||||
|
||||
Presence fires if **either** channel crosses its threshold. Both thresholds
|
||||
are learned entirely from the amplitude statistics of your enrollment
|
||||
anchors — there is no body-size, RCS (radar cross-section), Doppler-signature,
|
||||
or any other physical feature in this code that separates "a full-grown
|
||||
adult moved" from "a cat walked past" or "a dog jumped on the couch." If a
|
||||
pet's motion perturbs the CSI amplitude by roughly the same amount as the
|
||||
`small_move` anchor did during your enrollment, `PresenceSpecialist` will read
|
||||
it as occupied, because that's mechanically what the threshold measures.
|
||||
|
||||
The closest thing to a safeguard is `AnomalySpecialist`
|
||||
(`specialist.rs:386-448`), a generic novelty detector that flags a live
|
||||
window as "anomalous" when it's far (in embedding distance) from every
|
||||
enrolled anchor prototype. It is **not** a validated pet filter — it will
|
||||
flag *any* statistically unusual signal as anomalous or normal depending on
|
||||
how close it happens to land to your anchors, with no guarantee it
|
||||
distinguishes species or motion source. A pet whose motion pattern happens
|
||||
to resemble the `small_move` anchor would not be flagged as anomalous at all.
|
||||
|
||||
**Practical takeaway for a homeowner with pets:** expect presence/posture
|
||||
readings to occasionally trigger on pet motion, especially larger animals or
|
||||
motion near the sensor. There is currently no configuration option or code
|
||||
path to suppress this.
|
||||
|
||||
---
|
||||
|
||||
## 4. Does the empty-room baseline need "typical" conditions (HVAC running) or true silence?
|
||||
|
||||
The short answer, grounded in how the baseline is actually computed: **a
|
||||
stationary, continuously-running interferer (a fan, HVAC blower, humidifier)
|
||||
that is present for the *entire* capture window becomes part of what "empty"
|
||||
means, and gets subtracted out naturally** — that's a direct consequence of
|
||||
how the statistics are computed, not a documented feature you have to
|
||||
configure.
|
||||
|
||||
`CalibrationRecorder` uses Welford's online algorithm to accumulate a running
|
||||
mean and variance per subcarrier over however many frames you feed it
|
||||
(`ruvsense/calibration.rs`). If a fan is running steadily the whole time you
|
||||
capture the baseline, its contribution is baked into `amp_mean`/`amp_variance`
|
||||
for every frame equally, so the resulting baseline already represents "empty
|
||||
room with the fan on" — and at runtime, `BaselineCalibration::subtract()`
|
||||
removes exactly that reference, so a room in the same steady state reads as
|
||||
quiet. The design intent documented in ADR-135 §1.1 is explicit about this:
|
||||
the whole point of baseline subtraction is to remove "hardware-induced gain
|
||||
bias and environment-fixed multipath" so downstream motion detectors aren't
|
||||
tripped by things that are always there.
|
||||
|
||||
**What actually matters is consistency, not silence**: capture the baseline
|
||||
under whatever background conditions the room will normally be in during
|
||||
real use (HVAC/fans running as usual), and try to keep the room in that same
|
||||
steady state for the entire capture window. What the code cannot correct
|
||||
for is a background condition that **changes partway through** the capture
|
||||
(e.g. HVAC cycles on 15 seconds into a 30-second capture) — that would bias
|
||||
the Welford mean/variance toward an in-between state that matches neither
|
||||
"HVAC off" nor "HVAC on" well.
|
||||
|
||||
There is a **real-time guard during capture** that can catch gross problems:
|
||||
`--abort-z-threshold` (default `2.0`) aborts the capture if the per-frame
|
||||
amplitude z-score median stays above that threshold for 20 consecutive
|
||||
banner intervals (`v2/crates/wifi-densepose-cli/src/calibrate.rs:82-83,
|
||||
163-178`). This is designed to catch someone walking through mid-capture, not
|
||||
necessarily short-duration mechanical noise — we found no test exercising it
|
||||
against an HVAC-cycling scenario specifically, so how it behaves for
|
||||
"appliance turns on mid-capture" is unverified.
|
||||
|
||||
### What we could not verify — and a design gap worth knowing about
|
||||
|
||||
ADR-135 §2.5 describes a much more sophisticated staleness-detection system:
|
||||
a `drift_score` computed from ongoing z-scores, a `BaselineDrift` event fired
|
||||
after sustained drift, and a `baseline_stale` flag published over the
|
||||
sensing WebSocket. **We searched the actual `calibration.rs` implementation
|
||||
and none of that exists in code** — there is no `drift_score` field, no
|
||||
`BaselineDrift` event, and no `baseline_stale` flag anywhere in
|
||||
`v2/crates/wifi-densepose-signal/src/ruvsense/calibration.rs`. That part of
|
||||
ADR-135 is aspirational design, not shipped behavior.
|
||||
|
||||
What *is* implemented, at a different layer, is a much simpler check on the
|
||||
**trained specialist bank** (not the raw baseline): `SpecialistBank` stores
|
||||
the `baseline_id` it was trained against, and `SpecialistBank::is_stale()`
|
||||
(`v2/crates/wifi-densepose-calibration/src/bank.rs:102-104`) returns `true`
|
||||
whenever the *current* baseline's id doesn't match the id the bank was
|
||||
trained on. Re-running `calibrate` always produces a new baseline id, so
|
||||
**any** recalibration — whether because of furniture moving, a genuinely
|
||||
stale reference, or just re-running the command — immediately marks every
|
||||
previously trained specialist bank stale, and you'll need to re-run `enroll`
|
||||
and `train-room` afterward. There is no partial/graded staleness signal
|
||||
(no "how stale"), only this all-or-nothing id comparison.
|
||||
|
||||
**Practical guidance:**
|
||||
|
||||
1. Calibrate with the room in its normal, steady background state (HVAC,
|
||||
fans, fridge compressor, etc. running as they normally would) and keep
|
||||
that state constant for the whole `--duration-s` window.
|
||||
2. If you significantly change background conditions later (move furniture,
|
||||
add a permanent appliance, change HVAC routine) or notice the sensing
|
||||
quality degrade, re-run `calibrate` — this is an explicit, operator-driven
|
||||
step; there is no code path that recalibrates for you.
|
||||
3. Re-running `calibrate` invalidates every specialist bank trained against
|
||||
the old baseline (via the `baseline_id` mismatch above) — plan to re-run
|
||||
`enroll` and `train-room` right after.
|
||||
|
||||
---
|
||||
|
||||
## Quick reference: commands and defaults actually in the code
|
||||
|
||||
```bash
|
||||
# Stage 1 — empty-room baseline. Room must be empty for the whole window.
|
||||
wifi-densepose calibrate \
|
||||
--udp-port 5005 --duration-s 30 --tier ht20 --output ./baseline.bin
|
||||
# Hard requirement: >= 600 recorded frames, or calibration fails.
|
||||
|
||||
# Stage 2+3 — guided enrollment (8 fixed anchors, ~4 minutes total)
|
||||
wifi-densepose enroll --baseline ./baseline.bin --room living-room \
|
||||
--output ./enrollment.json --attempts 2
|
||||
|
||||
# Stage 4 — train the specialist bank from whatever anchors were accepted
|
||||
wifi-densepose train-room --enrollment ./enrollment.json \
|
||||
--output ./room-bank.json
|
||||
|
||||
# Check what actually trained (and whether the bank is stale)
|
||||
wifi-densepose room-status --room living-room
|
||||
```
|
||||
|
||||
Source references for everything above:
|
||||
- `v2/crates/wifi-densepose-cli/src/calibrate.rs`
|
||||
- `v2/crates/wifi-densepose-cli/src/room.rs`
|
||||
- `v2/crates/wifi-densepose-signal/src/ruvsense/calibration.rs`
|
||||
- `v2/crates/wifi-densepose-calibration/src/enrollment.rs`
|
||||
- `v2/crates/wifi-densepose-calibration/src/anchor.rs`
|
||||
- `v2/crates/wifi-densepose-calibration/src/specialist.rs`
|
||||
- `v2/crates/wifi-densepose-calibration/src/bank.rs`
|
||||
- `docs/adr/ADR-135-empty-room-baseline-calibration.md`
|
||||
- `docs/adr/ADR-151-room-calibration-specialist-training.md`
|
||||
@@ -0,0 +1,250 @@
|
||||
# Trust State & Engine Errors
|
||||
|
||||
If you've seen the sensing-server log a growing `engine_error_count`, or
|
||||
noticed your deployment reads as `"demoted": true` on the status endpoint,
|
||||
this page explains — from the actual code, not the design docs — what those
|
||||
two things mean, what triggers them, where you can see them, and what your
|
||||
real options are.
|
||||
|
||||
Everything below is grounded in
|
||||
`v2/crates/wifi-densepose-sensing-server/src/engine_bridge.rs`,
|
||||
`v2/crates/wifi-densepose-sensing-server/src/main.rs`,
|
||||
`v2/crates/wifi-densepose-engine/src/lib.rs`, and
|
||||
`v2/crates/wifi-densepose-signal/src/ruvsense/multistatic.rs`.
|
||||
|
||||
## Two different things, easy to conflate
|
||||
|
||||
The sensing-server runs a "governed trust cycle" every sensing tick
|
||||
(`StreamingEngine::process_cycle`, driven by `EngineBridge::observe_cycle` in
|
||||
`engine_bridge.rs:193-223`). Each cycle produces **one of two outcomes**, and
|
||||
they are tracked completely separately:
|
||||
|
||||
1. **The cycle fails outright** (`Result::Err(EngineError)`) — nothing is
|
||||
published for that tick. This increments `engine_error_count`, a
|
||||
monotonically increasing counter.
|
||||
2. **The cycle succeeds but under a demoted privacy class**
|
||||
(`Result::Ok(TrustedOutput { demoted: true, .. })`) — a belief *is*
|
||||
published, just at a more restricted privacy class than normal. This sets
|
||||
the `demoted` flag, which is recomputed fresh on every successful cycle.
|
||||
|
||||
A deployment can have a high `engine_error_count` with `demoted: false` (lots
|
||||
of failed cycles, but the ones that succeed are clean), or `demoted: true`
|
||||
with `engine_error_count: 0` (every cycle succeeds, but under a downgraded
|
||||
privacy class), or both at once — which is what the issue reporter saw.
|
||||
|
||||
## 1. Exact conditions for each
|
||||
|
||||
### Engine errors (`engine_error_count`)
|
||||
|
||||
`engine_error_count` increments only when
|
||||
`StreamingEngine::process_cycle` returns `Err(EngineError::Fusion(..))`
|
||||
(`engine_bridge.rs:198-222`). `EngineError` wraps
|
||||
`wifi_densepose_signal::ruvsense::multistatic::MultistaticError`
|
||||
(`wifi-densepose-engine/src/lib.rs:54-69`), which has exactly four variants
|
||||
(`multistatic.rs:36-56`):
|
||||
|
||||
| Variant | Condition |
|
||||
|---|---|
|
||||
| `NoFrames` | No node frames were passed to fusion. In practice not reachable through the bridge: `process_cycle_from_states` returns `None` (not an error, not counted) before calling the engine at all if there are no frames (`engine_bridge.rs:168-171`). |
|
||||
| `InsufficientNodes(n)` | Fewer than 2 nodes contributing in multistatic mode. |
|
||||
| `TimestampMismatch { spread_us, guard_us }` | The spread between contributing nodes' frame timestamps exceeds the **hard guard interval**, default **60,000 µs (60 ms)** (`MultistaticConfig::default()`, `multistatic.rs:133`). |
|
||||
| `DimensionMismatch { node_idx, expected, got }` | A node's subcarrier count doesn't match the others. As of #1170 the live bridge canonicalizes every node onto a common 56-tone grid before fusion, so this is now rare on real hardware — see the comment on `observe_cycle_counts_engine_errors` in `engine_bridge.rs`. |
|
||||
|
||||
Regardless of cause, an error is **rate-limited in the log** to one
|
||||
`tracing::warn!` line per 10 seconds (`ENGINE_ERROR_WARN_INTERVAL`,
|
||||
`engine_bridge.rs:50, 206-219`) — errors are still counted every cycle, only
|
||||
the *log line* is throttled, so a 20 Hz loop failing continuously won't flood
|
||||
your log with 20 lines/second.
|
||||
|
||||
### Trust demotion (`demoted`)
|
||||
|
||||
This is a **separate mechanism** from engine errors: it happens on cycles
|
||||
that *succeed*, and it downgrades the privacy class the output is emitted
|
||||
under, one step, rather than failing the cycle. From
|
||||
`wifi-densepose-engine/src/lib.rs:514-515`:
|
||||
|
||||
```rust
|
||||
let demoted = quality.forces_privacy_demotion() || array_contradiction || mesh_at_risk;
|
||||
let effective_class = if demoted { demote_one(base_class) } else { base_class };
|
||||
```
|
||||
|
||||
Three independent conditions can trigger it:
|
||||
|
||||
- **`quality.forces_privacy_demotion()`** — true whenever the fusion
|
||||
quality record carries any non-empty `contradiction_flags`
|
||||
(`fusion_quality.rs:111-118`). These are *tolerated* disagreements, distinct
|
||||
from a hard fusion failure:
|
||||
- `TimestampMismatch` — spread within the **hard** guard but beyond the
|
||||
**soft** guard (default **20,000 µs / 20 ms**, `soft_guard_us`,
|
||||
`multistatic.rs:104-113`) — i.e. loose-but-tolerable timing alignment.
|
||||
- `CalibrationIdMismatch` — contributing frames disagree on which
|
||||
calibration epoch (baseline) they were captured under.
|
||||
- `PhaseAlignmentFailed`, `DriftProfileConflict`, `CoherenceDrop`,
|
||||
`GeometryInsufficient` — raised upstream by the array coordinator /
|
||||
baseline drift checks.
|
||||
- **`array_contradiction`** — a separate array-level directional-fusion
|
||||
contradiction check.
|
||||
- **`mesh_at_risk`** — the mesh is close to partitioning (`mesh_guard.rs`).
|
||||
|
||||
`demote_one()` (`lib.rs:688-690`) steps the privacy class exactly one notch
|
||||
toward `Restricted` (it never jumps more than one step, and never relaxes a
|
||||
class in the same cycle — proven by the `forced_contradiction_never_relaxes_class`
|
||||
test). At `PrivacyClass::Restricted`, `EngineBridge::suppress_raw_outputs()`
|
||||
becomes true and `main.rs` strips per-node raw amplitude vectors from the
|
||||
published `SensingUpdate` (`engine_bridge.rs:251-260`).
|
||||
|
||||
**Crucially, `demoted` is not sticky.** It is overwritten on every
|
||||
successful cycle to reflect *that cycle's* outcome
|
||||
(`self.demoted = trust.demoted;`, `engine_bridge.rs:203`). If the
|
||||
contradiction that caused demotion was transient, the very next clean cycle
|
||||
reports `demoted: false` again with no action from you. If you see
|
||||
`demoted: true` *persistently*, that means the underlying condition (usually
|
||||
clock drift beyond the guard, or a geometry/calibration disagreement) is
|
||||
itself persistent, not that something got "stuck."
|
||||
|
||||
## 2. Where this is exposed
|
||||
|
||||
Both `GET /health/ready` and `GET /api/v1/status` are wired to the same
|
||||
handler (`health_ready`, `main.rs:8128,8133`) and return a `trust` block
|
||||
(`main.rs:4589-4606`):
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ready",
|
||||
"trust": {
|
||||
"last_witness": "…64 hex chars or null…",
|
||||
"effective_class": "Anonymous | Restricted | …",
|
||||
"demoted": false,
|
||||
"recalibration_recommended": false,
|
||||
"engine_error_count": 0,
|
||||
"raw_outputs_suppressed": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**This is a real, currently-shipped diagnostic surface — but it is honestly
|
||||
limited.** It tells you *that* errors are occurring and *that* the current
|
||||
class is demoted, and the total count, but not *why* for your specific run:
|
||||
|
||||
- `engine_error_count` is a single running total. There is **no breakdown by
|
||||
error type** anywhere in the API or in `EngineBridge`'s state — you cannot
|
||||
tell from `/health/ready` whether your 20,000 errors are 20,000
|
||||
`TimestampMismatch`es or 20,000 `DimensionMismatch`es.
|
||||
- `demoted` is a boolean with no accompanying list of which
|
||||
`ContradictionFlag`s actually fired. The underlying `contradiction_flags`
|
||||
vector exists in `QualityScore` (`fusion_quality.rs:106`) but is not
|
||||
surfaced over the wire anywhere we found.
|
||||
- There's no error history/timeline, and no per-node breakdown (which node
|
||||
is the one whose clock is drifting, for instance).
|
||||
|
||||
**The closest thing to a real diagnostic today is the rate-limited log
|
||||
line itself.** Unlike the API, the log message includes the `Display` text
|
||||
of the actual `EngineError`, which for `TimestampMismatch` and
|
||||
`DimensionMismatch` includes the concrete numbers (e.g. `"Timestamp spread
|
||||
87000 us exceeds guard interval 60000 us"`, `"Dimension mismatch: node 2 has
|
||||
114 subcarriers, expected 56"`). If you're trying to diagnose a specific
|
||||
demotion/error episode today, grepping the sensing-server log for
|
||||
`"governed trust cycle failed"` is the most concrete answer available — the
|
||||
status endpoint alone will not tell you the underlying cause. Treat this as
|
||||
the honest state of the diagnostics, not a missing feature we're pretending
|
||||
exists.
|
||||
|
||||
## 3. Is a demoted / errored state permanent? Is there a reset?
|
||||
|
||||
**`demoted` never needs resetting** — as described above, it's recomputed
|
||||
every successful cycle from that cycle's own contradiction/mesh state. There
|
||||
is no persistence, no counter, no cooldown timer for it in the code.
|
||||
|
||||
**`engine_error_count` has no reset mechanism at all.** It is a plain `u64`
|
||||
field on `EngineBridge`, initialized to `0` in `EngineBridge::new`
|
||||
(`engine_bridge.rs:111`) and only ever incremented
|
||||
(`self.engine_error_count += 1;`, line 207) — there is no method, admin
|
||||
endpoint, or timer anywhere in the crate that decrements or clears it. The
|
||||
only way to bring it back to zero is to **restart the sensing-server
|
||||
process**, which constructs a brand-new `EngineBridge`. If your count is
|
||||
growing and you want to confirm whether a fix actually worked, restart the
|
||||
server and watch whether the count starts climbing again — there is
|
||||
currently no lighter-weight way to "clear the counter" without a restart.
|
||||
|
||||
**If demotion (or errors) are persistent rather than one-off**, the
|
||||
documented, real fix for the most common cause — clock drift between nodes
|
||||
exceeding the fixed 60 ms hard guard — is an environment-variable override,
|
||||
not a restart or a wait:
|
||||
|
||||
- `WDP_GUARD_INTERVAL_US` — directly overrides the hard guard (e.g.
|
||||
`WDP_GUARD_INTERVAL_US=200000` for a 200 ms guard). This is the escape
|
||||
hatch a real deployment (issue #1049) needed: WiFi/ESP-NOW-synced ESP32
|
||||
nodes were measured drifting 10–150 ms, which the published 60 ms default
|
||||
could not absorb, causing **every** cycle to demote with "no escape hatch"
|
||||
(see the comment at `main.rs:8336-8339`).
|
||||
- `WDP_SOFT_GUARD_US` — optionally overrides the soft (tolerated-contradiction)
|
||||
guard, always clamped below the hard guard.
|
||||
- `WDP_TDM_SLOTS` + `WDP_TDM_SLOT_US` — derive the guard from your actual TDM
|
||||
schedule instead of setting it directly.
|
||||
|
||||
See `multistatic_guard_config_from_env` / `multistatic_guard_config_from`
|
||||
(`main.rs:6791-6856`) for the exact precedence rules (a direct
|
||||
`WDP_GUARD_INTERVAL_US` always wins over the TDM-derived value).
|
||||
|
||||
## 4. Does a converted Hugging Face model explain this?
|
||||
|
||||
**We could not find a code path connecting `--convert-model` to engine
|
||||
errors or trust demotion — they appear to be entirely separate subsystems.**
|
||||
Saying this plainly rather than speculating:
|
||||
|
||||
- `--convert-model` (`main.rs:6976-7028`, `run_convert_model` /
|
||||
`load_or_convert_model` at `main.rs:6925-6974`) converts a **pose-model
|
||||
weights file** — Hugging Face `safetensors` or a `jsonl` manifest — into
|
||||
this project's own RVF binary container format, so it can be loaded via
|
||||
`--model`. This is entirely about which neural-network weights the pose
|
||||
estimator uses.
|
||||
- `engine_error_count` and `demoted` come from `StreamingEngine::process_cycle`
|
||||
in `wifi-densepose-engine`, which performs **multistatic CSI sensor
|
||||
fusion** — checking node count, per-node timestamp spread, and per-node
|
||||
subcarrier dimensions across your ESP32 nodes. This code path has no
|
||||
dependency on which pose model is loaded, and `load_or_convert_model` /
|
||||
`run_convert_model` never call into `engine_bridge` or
|
||||
`StreamingEngine` at all.
|
||||
|
||||
Because the code shows no coupling between the two, we are not going to
|
||||
invent one. Two possibilities that the code doesn't rule out, but also
|
||||
doesn't confirm, if you hit both symptoms together:
|
||||
|
||||
- **Coincidence** — the deployment that had trouble loading/using a
|
||||
converted model separately had a fusion-timing or node-count problem
|
||||
(e.g. the #1049-style clock-drift issue, or fewer than 2 active nodes),
|
||||
unrelated to the model conversion itself.
|
||||
- **A configuration change made alongside the model swap** — e.g. changing
|
||||
node count, geometry, or guard settings at the same time as switching
|
||||
models — could produce both symptoms together without the model itself
|
||||
being the cause.
|
||||
|
||||
If you're hitting this, the actionable step from the code is to check
|
||||
`engine_error_count` and the log line's error text (per §2 above)
|
||||
**independently** of whatever model you have loaded — if the errors are
|
||||
`TimestampMismatch`/`DimensionMismatch`/`InsufficientNodes`, the fix is on
|
||||
the sensor-fusion side (§3), not the model side, regardless of which model
|
||||
produced the report.
|
||||
|
||||
## Quick reference
|
||||
|
||||
```bash
|
||||
# Check current trust state
|
||||
curl -s http://localhost:3000/api/v1/status | jq .trust
|
||||
|
||||
# Watch for the rate-limited error log line (most specific diagnostic today)
|
||||
# — look for "governed trust cycle failed" in the sensing-server's stderr/log.
|
||||
|
||||
# If demotion/errors are persistent due to node clock drift, raise the guard:
|
||||
WDP_GUARD_INTERVAL_US=200000 wifi-densepose-sensing-server ...
|
||||
|
||||
# The only way to reset engine_error_count is a process restart.
|
||||
```
|
||||
|
||||
Source references for everything above:
|
||||
- `v2/crates/wifi-densepose-sensing-server/src/engine_bridge.rs`
|
||||
- `v2/crates/wifi-densepose-sensing-server/src/main.rs` (search `trust`, `health_ready`, `multistatic_guard_config_from`, `convert_model`)
|
||||
- `v2/crates/wifi-densepose-engine/src/lib.rs`
|
||||
- `v2/crates/wifi-densepose-engine/src/mesh_guard.rs`
|
||||
- `v2/crates/wifi-densepose-signal/src/ruvsense/multistatic.rs`
|
||||
- `v2/crates/wifi-densepose-signal/src/ruvsense/fusion_quality.rs`
|
||||
Reference in New Issue
Block a user