ruvnet--RuView/docs/trust-and-engine-errors.md

Trust State & Engine Errors

If you've seen the sensing-server log a growing engine_error_count, or noticed your deployment reads as "demoted": true on the status endpoint, this page explains — from the actual code, not the design docs — what those two things mean, what triggers them, where you can see them, and what your real options are.

Everything below is grounded in v2/crates/wifi-densepose-sensing-server/src/engine_bridge.rs, v2/crates/wifi-densepose-sensing-server/src/main.rs, v2/crates/wifi-densepose-engine/src/lib.rs, and v2/crates/wifi-densepose-signal/src/ruvsense/multistatic.rs.

Two different things, easy to conflate

The sensing-server runs a "governed trust cycle" every sensing tick (StreamingEngine::process_cycle, driven by EngineBridge::observe_cycle in engine_bridge.rs:193-223). Each cycle produces one of two outcomes, and they are tracked completely separately:

1. The cycle fails outright (Result::Err(EngineError)) — nothing is published for that tick. This increments engine_error_count, a monotonically increasing counter.
2. The cycle succeeds but under a demoted privacy class (Result::Ok(TrustedOutput { demoted: true, .. })) — a belief is published, just at a more restricted privacy class than normal. This sets the demoted flag, which is recomputed fresh on every successful cycle.

A deployment can have a high engine_error_count with demoted: false (lots of failed cycles, but the ones that succeed are clean), or demoted: true with engine_error_count: 0 (every cycle succeeds, but under a downgraded privacy class), or both at once — which is what the issue reporter saw.

1. Exact conditions for each

Engine errors (engine_error_count)

engine_error_count increments only when StreamingEngine::process_cycle returns Err(EngineError::Fusion(..)) (engine_bridge.rs:198-222). EngineError wraps wifi_densepose_signal::ruvsense::multistatic::MultistaticError (wifi-densepose-engine/src/lib.rs:54-69), which has exactly four variants (multistatic.rs:36-56):

Variant	Condition
NoFrames	No node frames were passed to fusion. In practice not reachable through the bridge: process_cycle_from_states returns None (not an error, not counted) before calling the engine at all if there are no frames (engine_bridge.rs:168-171).
InsufficientNodes(n)	Fewer than 2 nodes contributing in multistatic mode.
TimestampMismatch { spread_us, guard_us }	The spread between contributing nodes' frame timestamps exceeds the hard guard interval, default 60,000 µs (60 ms) (MultistaticConfig::default(), multistatic.rs:133).
DimensionMismatch { node_idx, expected, got }	A node's subcarrier count doesn't match the others. As of #1170 the live bridge canonicalizes every node onto a common 56-tone grid before fusion, so this is now rare on real hardware — see the comment on observe_cycle_counts_engine_errors in engine_bridge.rs.

Regardless of cause, an error is rate-limited in the log to one tracing::warn! line per 10 seconds (ENGINE_ERROR_WARN_INTERVAL, engine_bridge.rs:50, 206-219) — errors are still counted every cycle, only the log line is throttled, so a 20 Hz loop failing continuously won't flood your log with 20 lines/second.

Trust demotion (demoted)

This is a separate mechanism from engine errors: it happens on cycles that succeed, and it downgrades the privacy class the output is emitted under, one step, rather than failing the cycle. From wifi-densepose-engine/src/lib.rs:514-515:

let demoted = quality.forces_privacy_demotion() || array_contradiction || mesh_at_risk;
let effective_class = if demoted { demote_one(base_class) } else { base_class };

Three independent conditions can trigger it:

• quality.forces_privacy_demotion() — true whenever the fusion quality record carries any non-empty contradiction_flags (fusion_quality.rs:111-118). These are tolerated disagreements, distinct from a hard fusion failure:
  • TimestampMismatch — spread within the hard guard but beyond the soft guard (default 20,000 µs / 20 ms, soft_guard_us, multistatic.rs:104-113) — i.e. loose-but-tolerable timing alignment.
  • CalibrationIdMismatch — contributing frames disagree on which calibration epoch (baseline) they were captured under.
  • PhaseAlignmentFailed, DriftProfileConflict, CoherenceDrop, GeometryInsufficient — raised upstream by the array coordinator / baseline drift checks.
• array_contradiction — a separate array-level directional-fusion contradiction check.
• mesh_at_risk — the mesh is close to partitioning (mesh_guard.rs).

demote_one() (lib.rs:688-690) steps the privacy class exactly one notch toward Restricted (it never jumps more than one step, and never relaxes a class in the same cycle — proven by the forced_contradiction_never_relaxes_class test). At PrivacyClass::Restricted, EngineBridge::suppress_raw_outputs() becomes true and main.rs strips per-node raw amplitude vectors from the published SensingUpdate (engine_bridge.rs:251-260).

Crucially, demoted is not sticky. It is overwritten on every successful cycle to reflect that cycle's outcome (self.demoted = trust.demoted;, engine_bridge.rs:203). If the contradiction that caused demotion was transient, the very next clean cycle reports demoted: false again with no action from you. If you see demoted: true persistently, that means the underlying condition (usually clock drift beyond the guard, or a geometry/calibration disagreement) is itself persistent, not that something got "stuck."

2. Where this is exposed

Both GET /health/ready and GET /api/v1/status are wired to the same handler (health_ready, main.rs:8128,8133) and return a trust block (main.rs:4589-4606):

{
  "status": "ready",
  "trust": {
    "last_witness": "…64 hex chars or null…",
    "effective_class": "Anonymous | Restricted | …",
    "demoted": false,
    "recalibration_recommended": false,
    "engine_error_count": 0,
    "raw_outputs_suppressed": false
  }
}

This is a real, currently-shipped diagnostic surface — but it is honestly limited. It tells you that errors are occurring and that the current class is demoted, and the total count, but not why for your specific run:

• engine_error_count is a single running total. There is no breakdown by error type anywhere in the API or in EngineBridge's state — you cannot tell from /health/ready whether your 20,000 errors are 20,000 TimestampMismatches or 20,000 DimensionMismatches.
• demoted is a boolean with no accompanying list of which ContradictionFlags actually fired. The underlying contradiction_flags vector exists in QualityScore (fusion_quality.rs:106) but is not surfaced over the wire anywhere we found.
• There's no error history/timeline, and no per-node breakdown (which node is the one whose clock is drifting, for instance).

The closest thing to a real diagnostic today is the rate-limited log line itself. Unlike the API, the log message includes the Display text of the actual EngineError, which for TimestampMismatch and DimensionMismatch includes the concrete numbers (e.g. "Timestamp spread 87000 us exceeds guard interval 60000 us", "Dimension mismatch: node 2 has 114 subcarriers, expected 56"). If you're trying to diagnose a specific demotion/error episode today, grepping the sensing-server log for "governed trust cycle failed" is the most concrete answer available — the status endpoint alone will not tell you the underlying cause. Treat this as the honest state of the diagnostics, not a missing feature we're pretending exists.

3. Is a demoted / errored state permanent? Is there a reset?

demoted never needs resetting — as described above, it's recomputed every successful cycle from that cycle's own contradiction/mesh state. There is no persistence, no counter, no cooldown timer for it in the code.

engine_error_count has no reset mechanism at all. It is a plain u64 field on EngineBridge, initialized to 0 in EngineBridge::new (engine_bridge.rs:111) and only ever incremented (self.engine_error_count += 1;, line 207) — there is no method, admin endpoint, or timer anywhere in the crate that decrements or clears it. The only way to bring it back to zero is to restart the sensing-server process, which constructs a brand-new EngineBridge. If your count is growing and you want to confirm whether a fix actually worked, restart the server and watch whether the count starts climbing again — there is currently no lighter-weight way to "clear the counter" without a restart.

If demotion (or errors) are persistent rather than one-off, the documented, real fix for the most common cause — clock drift between nodes exceeding the fixed 60 ms hard guard — is an environment-variable override, not a restart or a wait:

• WDP_GUARD_INTERVAL_US — directly overrides the hard guard (e.g. WDP_GUARD_INTERVAL_US=200000 for a 200 ms guard). This is the escape hatch a real deployment (issue #1049) needed: WiFi/ESP-NOW-synced ESP32 nodes were measured drifting 10–150 ms, which the published 60 ms default could not absorb, causing every cycle to demote with "no escape hatch" (see the comment at main.rs:8336-8339).
• WDP_SOFT_GUARD_US — optionally overrides the soft (tolerated-contradiction) guard, always clamped below the hard guard.
• WDP_TDM_SLOTS + WDP_TDM_SLOT_US — derive the guard from your actual TDM schedule instead of setting it directly.

See multistatic_guard_config_from_env / multistatic_guard_config_from (main.rs:6791-6856) for the exact precedence rules (a direct WDP_GUARD_INTERVAL_US always wins over the TDM-derived value).

4. Does a converted Hugging Face model explain this?

We could not find a code path connecting --convert-model to engine errors or trust demotion — they appear to be entirely separate subsystems. Saying this plainly rather than speculating:

• --convert-model (main.rs:6976-7028, run_convert_model / load_or_convert_model at main.rs:6925-6974) converts a pose-model weights file — Hugging Face safetensors or a jsonl manifest — into this project's own RVF binary container format, so it can be loaded via --model. This is entirely about which neural-network weights the pose estimator uses.
• engine_error_count and demoted come from StreamingEngine::process_cycle in wifi-densepose-engine, which performs multistatic CSI sensor fusion — checking node count, per-node timestamp spread, and per-node subcarrier dimensions across your ESP32 nodes. This code path has no dependency on which pose model is loaded, and load_or_convert_model / run_convert_model never call into engine_bridge or StreamingEngine at all.

Because the code shows no coupling between the two, we are not going to invent one. Two possibilities that the code doesn't rule out, but also doesn't confirm, if you hit both symptoms together:

• Coincidence — the deployment that had trouble loading/using a converted model separately had a fusion-timing or node-count problem (e.g. the #1049-style clock-drift issue, or fewer than 2 active nodes), unrelated to the model conversion itself.
• A configuration change made alongside the model swap — e.g. changing node count, geometry, or guard settings at the same time as switching models — could produce both symptoms together without the model itself being the cause.

If you're hitting this, the actionable step from the code is to check engine_error_count and the log line's error text (per §2 above) independently of whatever model you have loaded — if the errors are TimestampMismatch/DimensionMismatch/InsufficientNodes, the fix is on the sensor-fusion side (§3), not the model side, regardless of which model produced the report.

Quick reference

# Check current trust state
curl -s http://localhost:3000/api/v1/status | jq .trust

# Watch for the rate-limited error log line (most specific diagnostic today)
# — look for "governed trust cycle failed" in the sensing-server's stderr/log.

# If demotion/errors are persistent due to node clock drift, raise the guard:
WDP_GUARD_INTERVAL_US=200000 wifi-densepose-sensing-server ...

# The only way to reset engine_error_count is a process restart.

Source references for everything above:

• v2/crates/wifi-densepose-sensing-server/src/engine_bridge.rs
• v2/crates/wifi-densepose-sensing-server/src/main.rs (search trust, health_ready, multistatic_guard_config_from, convert_model)
• v2/crates/wifi-densepose-engine/src/lib.rs
• v2/crates/wifi-densepose-engine/src/mesh_guard.rs
• v2/crates/wifi-densepose-signal/src/ruvsense/multistatic.rs
• v2/crates/wifi-densepose-signal/src/ruvsense/fusion_quality.rs