Claude
2cafa1fdcc
Turns the review's code-implementable P0/P1 items into ADRs: source provenance state machine (synthetic never presents as live), UDP data-plane bind hardening (loopback default + allowlist, step one), multi-node semantic correctness (per-node inference, node-keyed rate limiter, stale state), model release sanity gates (block degenerate/mislabeled heads), and CSI data-incident repo controls. Also fixes the stale .gitignore rule so the active recordings directories and CSI globs are covered going forward. Tree removal of existing recordings, history rewrite, and withdrawal of the published presence head are gated on maintainer sign-off (outward-facing / destructive) and intentionally not done here. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_015TcKegTS7QqhWPC2L2SzaS
2.8 KiB
ADR-295: Model release sanity gates — block degenerate and mislabeled model artifacts
- Status: Accepted — initial implementation (this PR)
- Date: 2026-08-11
- Deciders: ruv
- Tags: models, evaluation, release-gate, honesty, presence
Context
The external review (corroborating issue 1521) showed the published presence head is mathematically degenerate: with L2-normalized embeddings, a weight norm ≈ 3.67 against a bias ≈ 8.19 makes the smallest possible logit positive, so predicted presence probability is ≥ ~0.989 for every valid input — the decision boundary is unreachable and the head is effectively constant. The README then labeled a temporal-triplet accuracy (a representation-ordering metric) as "presence accuracy" — a category error.
Nothing in the release path catches a constant classifier, an unreachable boundary, or a metric-name mismatch. A machine check would have.
Decision
Add a model_gates module (in wifi-densepose-train) plus a CI gate that,
for any classifier artifact proposed for release, fails on:
- Constant output — output variance below a threshold across a diverse probe set (including the degenerate-embedding probe from issue 1521).
- Unreachable decision boundary — for a normalized-embedding linear head,
check whether
biassign dominates‖weight‖so the logit cannot change sign; fail if the boundary is analytically unreachable. - Degenerate class balance — predicted-positive rate at/above a ceiling (e.g. > 99%) on a balanced probe set.
- Missing/blank baseline — a report without a paired mean-pose/majority
baseline (ties into ADR-288
EvaluationReport). - Metric-name provenance — a metric may not be surfaced under a task name that does not match its computed kind (temporal-triplet ≠ presence); enforced by making the metric carry its kind and the label derive from it.
Each gate emits a structured, human-readable failure explaining the defect and the offending numbers.
Consequences
- The specific degenerate presence head cannot ship again, and the temporal-triplet-as-presence mislabel is structurally prevented.
- Some existing artifacts will fail the gate on introduction — intended; they should fail.
- The gate is heuristic, not a correctness proof; it catches the known failure shapes, not all bad models.
Validation
- Unit tests: the issue-1521 weights fail the unreachable-boundary and constant-output gates; a healthy synthetic head passes; a temporal-triplet metric cannot be constructed with a presence label.
cargo test -p wifi-densepose-train; the CI gate runs in the model-check workflow.- This ADR does not withdraw the already-published artifact (an outward-facing action requiring maintainer sign-off) — it prevents recurrence and documents the model-card correction.