ADR-304: Evidence engine — MLflow for physical sensing

Status: Accepted — initial implementation planned (ADR-300 phase 1)
Date: 2026-08-11
Deciders: ruv
Tags: evidence, provenance, ledger, accuracy, drift, benchmark, honesty, substrate

Context

This ADR is primitive 4 of the perception-substrate program (ADR-300) and a central pillar of the phase-1 certificate spine. It consumes the domain state from out-of-distribution detection (ADR-302) and the calibration age from the calibration certificate (ADR-301), it is the store that capability certificates (ADR-318) are minted from, and it is the accuracy source the real benchmark service (ADR-317) reads. In phase 2 it ingests agreement reports from the ground-truth plane (ADR-303).

The strategic assessment (ADR-300) judged this primitive more commercially important than another pose architecture: what unblocks OEM and integrator conversations is not a higher headline number but a defensible, auditable record of how a model actually performs, per room, per device, per subject, over time. MLflow made ML experiments trackable; physical sensing needs the equivalent for deployed accuracy, drift, and evidence level — an append-only ledger, not a dashboard that overwrites yesterday's number.

RuView already has the constituent evidence types; what is missing is the ledger that unifies them per deployment context:

RuField provenance/signature types (ADR-260/262/277/279) — the signed, provenance-bearing record types to reuse rather than reinvent.
The AetherArena witness-ledger pattern (ADR-149) — an append-only, witness-anchored ledger of scored results, the structural template here.
frame::EvidenceLevel L0–L5 (ADR-282) — the mandatory evidence tag every record carries.
ADR-302 DomainState, ADR-137 QualityScore, ADR-301 certificate version and age — the per-inference signals to accumulate.

Options considered

Log accuracy to flat files / metrics dashboards. Rejected: mutable, un-signed, un-scoped, and not comparable over time — the exact gap.
Reuse a general experiment tracker (MLflow itself). Rejected: it is experiment-time, not deployment-time; it has no notion of room/device/ subject context, calibration age, evidence level, or signed provenance, and it would add an external service dependency contrary to the substrate's edge-first, dependency-light direction.
A native append-only evidence ledger reusing RuField record types and the AetherArena ledger pattern. Chosen.

Decision

Build an evidence engine: a per-(room, device, subject) append-only accuracy ledger that every model automatically writes to.

1. The evidence record

An EvidenceRecord keyed by context — space id (ADR-306), signed device id (ADR-305), and subject id where consented and available — carrying: model version; calibration certificate version and age (ADR-301); the ADR-302 DomainState (KNOWN/DEGRADED/UNKNOWN) and its four inputs; the ADR-137 quality signals; predictive uncertainty; and, when a reference is present (ADR-303), the agreement result (accuracy, false-positive rate). Each record carries exactly one EvidenceLevel (L0–L5, ADR-282).
Records are append-only and signed with RuField signature types (ADR-260/262/277/279); the ledger is anchored in the witness chain (ADR-319), following the AetherArena witness-ledger pattern (ADR-149). No record is ever mutated in place — a correction is a new record.

2. Per-context accuracy accounting

The engine maintains, per (room, device, subject) context: measured accuracy (only where an ADR-303 reference backs it — otherwise the record is CLAIMED/SYNTHETIC, never MEASURED), false-positive rate, drift trajectory (fingerprint distance over time from ADR-301), the fraction of inferences in each domain state, calibration age distribution, and model-version history.
Aggregation is a pure function over the append-only log at a queried time — the ledger is the source of truth; summaries are derived, never authoritative (mirroring CLAUDE.md's "source over summaries" rule).

3. Honesty enforced in the record

The engine cannot upgrade an evidence level; a level is set by the record's provenance at write time (synthetic input → L0/Synthetic; no reference → CLAIMED; reference + reproducer → MEASURED), reusing the ADR-282/ADR-291/ ADR-293 constructor discipline. A benchmark or certificate reading the ledger gets the honest level, not an optimistic rollup.
No benchmark numbers are invented by this ADR; it delivers the ledger and the accounting. Empty contexts report "no evidence," which downstream (ADR-318) must treat as no capability.

Consequences

RuView gains a single auditable answer to "how well does this model actually work, here, on this device, for this subject, and how fresh is the calibration?" — the artifact OEM/integrator diligence actually asks for.
ADR-318 capability certificates become derivable (a certificate is a signed attestation over a slice of the ledger) and ADR-317 gains a real accuracy source per PR instead of self-reported numbers.
The append-only, signed design has storage and key-management cost; bounded by per-context retention policy and by reusing the existing RuField/witness infrastructure rather than a new store.
Some contexts will show sparse or unflattering evidence. Surfacing that is the point; the engine must never paper over a thin context with a global average.

Validation

cargo test on the evidence-engine crate — append-only invariant (no in-place mutation; corrections are new records); per-context aggregation math against fixtures; evidence-level is set by provenance and cannot be upgraded; signature round-trip and tamper rejection; witness anchoring; empty-context queries return "no evidence" not a fabricated number.
Cross-ADR: ingests ADR-302 DomainState and (phase 2) ADR-303 agreement reports; an ADR-318 test mints a certificate from a ledger slice and an ADR-317 test reads accuracy from the ledger.
Real-deployment evidence (a populated ledger from live ESP32 captures with ADR-303 references) is the maturity milestone and requires hardware evidence per CLAUDE.md; a synthetic ledger is L0 by construction.

6.2 KiB Raw Blame History Unescape Escape