mirror of
https://github.com/ruvnet/RuView
synced 2026-08-11 20:41:44 +00:00
Claude
2cafa1fdcc
Turns the review's code-implementable P0/P1 items into ADRs: source provenance state machine (synthetic never presents as live), UDP data-plane bind hardening (loopback default + allowlist, step one), multi-node semantic correctness (per-node inference, node-keyed rate limiter, stale state), model release sanity gates (block degenerate/mislabeled heads), and CSI data-incident repo controls. Also fixes the stale .gitignore rule so the active recordings directories and CSI globs are covered going forward. Tree removal of existing recordings, history rewrite, and withdrawal of the published presence head are gated on maintainer sign-off (outward-facing / destructive) and intentionally not done here. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_015TcKegTS7QqhWPC2L2SzaS
61 lines
2.8 KiB
Markdown
61 lines
2.8 KiB
Markdown
# ADR-295: Model release sanity gates — block degenerate and mislabeled model artifacts
|
|
|
|
- **Status**: Accepted — initial implementation (this PR)
|
|
- **Date**: 2026-08-11
|
|
- **Deciders**: ruv
|
|
- **Tags**: models, evaluation, release-gate, honesty, presence
|
|
|
|
## Context
|
|
|
|
The external review (corroborating issue 1521) showed the published presence
|
|
head is mathematically degenerate: with L2-normalized embeddings, a weight
|
|
norm ≈ 3.67 against a bias ≈ 8.19 makes the smallest possible logit positive,
|
|
so predicted presence probability is ≥ ~0.989 for every valid input — the
|
|
decision boundary is unreachable and the head is effectively constant. The
|
|
README then labeled a temporal-triplet accuracy (a representation-ordering
|
|
metric) as "presence accuracy" — a category error.
|
|
|
|
Nothing in the release path catches a constant classifier, an unreachable
|
|
boundary, or a metric-name mismatch. A machine check would have.
|
|
|
|
## Decision
|
|
|
|
Add a `model_gates` module (in `wifi-densepose-train`) plus a CI gate that,
|
|
for any classifier artifact proposed for release, fails on:
|
|
|
|
- **Constant output** — output variance below a threshold across a diverse
|
|
probe set (including the degenerate-embedding probe from issue 1521).
|
|
- **Unreachable decision boundary** — for a normalized-embedding linear head,
|
|
check whether `bias` sign dominates `‖weight‖` so the logit cannot change
|
|
sign; fail if the boundary is analytically unreachable.
|
|
- **Degenerate class balance** — predicted-positive rate at/above a ceiling
|
|
(e.g. > 99%) on a balanced probe set.
|
|
- **Missing/blank baseline** — a report without a paired mean-pose/majority
|
|
baseline (ties into ADR-288 `EvaluationReport`).
|
|
- **Metric-name provenance** — a metric may not be surfaced under a task name
|
|
that does not match its computed kind (temporal-triplet ≠ presence);
|
|
enforced by making the metric carry its kind and the label derive from it.
|
|
|
|
Each gate emits a structured, human-readable failure explaining the defect and
|
|
the offending numbers.
|
|
|
|
## Consequences
|
|
|
|
- The specific degenerate presence head cannot ship again, and the
|
|
temporal-triplet-as-presence mislabel is structurally prevented.
|
|
- Some existing artifacts will fail the gate on introduction — intended; they
|
|
should fail.
|
|
- The gate is heuristic, not a correctness proof; it catches the known
|
|
failure shapes, not all bad models.
|
|
|
|
## Validation
|
|
|
|
- Unit tests: the issue-1521 weights fail the unreachable-boundary and
|
|
constant-output gates; a healthy synthetic head passes; a temporal-triplet
|
|
metric cannot be constructed with a presence label.
|
|
- `cargo test -p wifi-densepose-train`; the CI gate runs in the model-check
|
|
workflow.
|
|
- This ADR does **not** withdraw the already-published artifact (an
|
|
outward-facing action requiring maintainer sign-off) — it prevents
|
|
recurrence and documents the model-card correction.
|