mirror of
https://github.com/ruvnet/RuView
synced 2026-07-24 17:43:20 +00:00
feat(train): metric-locked PCK/MPJPE accuracy harness + ADR-173 (resolve PCK-definition ambiguity) (#1092)
* feat(train): metric-locked PCK/MPJPE accuracy harness — resolve PCK-definition ambiguity
The SOTA brief (docs/research/sota-nn-train-benchmark-brief.md §1/§3.1/§4)
identifies metric ambiguity as the single biggest threat to any beyond-SOTA
claim: three PCK@20 numbers (96.09% WiFlow-STD image-normalized, 81.63%
AetherArena torso-PCK, 61.1% GraphPose-Fi standard PCK) cannot be lined up
because each silently uses a different normalization. The project was retracted
twice over this (a withdrawn 92.9% used absolute pixels, not torso).
New src/accuracy.rs makes the normalizer explicit, selectable, and carried with
every reported number:
- PckNormalization enum: TorsoDiameter (standard MM-Fi/GraphPose-Fi hip↔hip),
BoundingBoxDiagonal (looser WiFlow-STD image-normalized), AbsolutePixels(t)
(retracted convention, reproducible + clearly non-comparable).
- pck_at(pred, gt, vis, k, normalization) — one canonical PCK reusing the
metrics_core geometric primitives (no duplicate kernel).
- mpjpe(pred, gt, vis) — 2D/3D, mm.
- PoseAccuracy { pck_at: BTreeMap<u8,f32>, mpjpe, normalization, n_keypoints,
n_frames } via accuracy_report(frames, ks, normalization) — an unlabeled PCK
number is structurally impossible.
17 hand-computed deterministic tests (no GPU, no datasets) prove the harness
arithmetic, including the key proof that identical predictions score
0.50 / 1.00 / 0.75 under the three normalizations, plus graceful degenerate
handling (zero torso, empty frames, NaN coords — no panic, never false-perfect).
This is measurement infrastructure, NOT an accuracy claim. Public API worth an
ADR — needs ADR slot 173 (parent to write).
wifi-densepose-train lib 191→206, test_metrics 12→14, 0 failed; full workspace
green (exit 0); Python deterministic proof unchanged
(f8e76f21a0f9852b70b6d9dd5318239f6b20cbcb4cdd995863263cecdc446f7a).
Co-Authored-By: claude-flow <ruv@ruv.net>
* docs(adr): ADR-173 — metric-locked PCK/MPJPE accuracy harness
Documents the accuracy harness (committed 3a8b2ed13) that resolves the
PCK-definition ambiguity flagged as the #1 beyond-SOTA risk in the SOTA brief
(#1090): three historical numbers (96/81.6/61) used three unstated
normalizations. The harness makes normalization explicit + selectable
(PckNormalization enum) and every reported number carries its definition.
Key proof: identical predictions → 0.50/1.00/0.75 under torso/bbox/abs.
Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
rUv
@@ -0,0 +1,123 @@
|
||||
# ADR-173: Metric-Locked PCK/MPJPE Accuracy Harness
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| **Status** | Accepted — implemented, deterministically tested |
|
||||
| **Date** | 2026-06-15 |
|
||||
| **Deciders** | ruv |
|
||||
| **Codename** | **METRIC-LOCK** |
|
||||
| **Amends** | ADR-155 (generalizes the torso-only `metrics_core::pck_canonical` to a selectable normalization) |
|
||||
| **Motivated by** | `docs/research/sota-nn-train-benchmark-brief.md` (PR #1090) |
|
||||
|
||||
## Context
|
||||
|
||||
The beyond-SOTA SOTA-research brief (PR #1090) identified the single biggest
|
||||
threat to any "beyond-SOTA" accuracy claim this project makes: **metric
|
||||
ambiguity**. Three PCK@20 numbers circulate, computed under three *different and
|
||||
unstated* normalizations, so they cannot be compared:
|
||||
|
||||
- **96.09–96.61%** — WiFlow-STD reproduction, **image/bounding-box-normalized** PCK (the looser convention).
|
||||
- **81.63%** — an internal MM-Fi number reported as **"torso-PCK"** (tighter).
|
||||
- **61.1%** — GraphPose-Fi (arXiv 2511.19105), **standard torso-diameter** PCK on the MM-Fi random split (the academic frontier).
|
||||
|
||||
The project has been burned by this twice: a previously-published 92.9% was
|
||||
retracted because it used **absolute-pixel** normalization, not torso. Until
|
||||
there is *one canonical, documented, tested* PCK definition — and every reported
|
||||
number carries the definition it was computed under — no accuracy comparison is
|
||||
credible, and the "prove everything" bar cannot be met for the benchmark half of
|
||||
the work.
|
||||
|
||||
This is measurement infrastructure, not an accuracy claim. The deliverable's job
|
||||
is to make the metric **unambiguous and reproducible**, so future numbers are
|
||||
comparable and an unlabeled PCK is structurally impossible.
|
||||
|
||||
## Decision
|
||||
|
||||
Add a metric-locked accuracy harness as a new module
|
||||
`v2/crates/wifi-densepose-train/src/accuracy.rs` (404 non-test lines; inline
|
||||
deterministic tests bring the file to 708), re-exported at the crate root. It
|
||||
**extends, not duplicates** — it reuses `metrics_core`'s geometric primitives
|
||||
(`bounding_box_diagonal`, canonical hip indices `CANON_LEFT_HIP/RIGHT_HIP`), so
|
||||
there remains exactly one implementation of each geometric reference; the
|
||||
existing ADR-155 `pck_canonical` (torso-only) is unchanged and this generalizes
|
||||
it.
|
||||
|
||||
### Public API
|
||||
|
||||
- `enum PckNormalization { TorsoDiameter, BoundingBoxDiagonal, AbsolutePixels(f32) }`
|
||||
— the three conventions the three historical numbers used, now **explicit and
|
||||
selectable**. `.label()` / `.tolerance(...)`.
|
||||
- `pck_at(pred, gt, vis, k, norm) -> (correct, total, pck)` — PCK@k =
|
||||
fraction of *visible* keypoints whose predicted-vs-GT distance ≤ the tolerance,
|
||||
where tolerance = `k%` of the chosen normalizer (or an absolute threshold for
|
||||
`AbsolutePixels`).
|
||||
- `mpjpe(pred, gt, vis) -> f32` — mean per-joint position error (2D/3D, coordinate
|
||||
units; mm for mm inputs). Re-exported crate-root as `pck_mpjpe` to avoid
|
||||
colliding with the existing `eval::mpjpe`.
|
||||
- `struct PoseAccuracy { pck_at: BTreeMap<u8,f32>, mpjpe, normalization, n_keypoints, n_frames }`
|
||||
— **a reported number always carries its `normalization`**; an unlabeled PCK is
|
||||
structurally impossible to produce through this surface.
|
||||
- `struct PoseFrame { pred, gt, visibility }` + `accuracy_report(frames, ks, norm) -> PoseAccuracy`
|
||||
(micro-averaged over keypoints).
|
||||
|
||||
### Correctness is proven by hand-computed deterministic tests (no GPU, no data)
|
||||
|
||||
The tests construct synthetic keypoint sets whose PCK/MPJPE can be computed by
|
||||
hand, and assert the harness matches. Highlights (all pass):
|
||||
|
||||
| Test | Construction | Expected |
|
||||
|------|--------------|----------|
|
||||
| perfect_prediction | pred==gt | PCK=1.0 (all 3 norms), MPJPE=0 |
|
||||
| all_just_outside | every error just past τ@20 | PCK=0.0 |
|
||||
| half_in_half_out | 2 exact, 2 just outside | PCK=0.5 |
|
||||
| **three_normalizations (KEY PROOF)** | identical pred; nose err .06, shoulder .10, hips exact | torso=**0.50**, bbox=**1.00**, abs(.08)=**0.75** |
|
||||
| mpjpe_2d / mpjpe_3d | (3,4)→5 / (1,2,2)→3 | 2.5 / 3.0 |
|
||||
| mpjpe_excludes_invisible | invisible joint err 100 ignored | 5.0 |
|
||||
| zero_torso_unscoreable | coincident hips | `(0,0,0.0)`, **not** false-perfect |
|
||||
| no_visible_keypoints | vis=∅ | `(0,0,0.0)` |
|
||||
| nan_coords | one NaN pred coord | counted wrong, **no panic** |
|
||||
| empty report | no frames | 0.0, **not** NaN |
|
||||
| bbox≥torso ordering | same frames | bbox-PCK ≥ torso-PCK |
|
||||
|
||||
### The key proof (the ambiguity is real and quantified)
|
||||
|
||||
Identical predictions, three declared normalizations → **0.50 / 1.00 / 0.75**.
|
||||
Mechanism: the bbox diagonal `√(0.20² + 0.80²) = 0.825` is ~4× the hip-span torso
|
||||
`0.20`, so τ@20 is 0.165 (bbox) vs 0.040 (torso) — the looser image-normalized
|
||||
convention passes joints the strict torso convention rejects. This is *exactly*
|
||||
why 96% / 81.6% / 61% cannot be lined up without declaring the enum, demonstrated
|
||||
in-code.
|
||||
|
||||
## Validation
|
||||
|
||||
- `cargo test -p wifi-densepose-train --no-default-features` → lib **191 → 206**
|
||||
(+15), `test_metrics` **12 → 14** (+2), doc-tests 8 — **0 failed**.
|
||||
- `cargo test --workspace --no-default-features` → **exit 0**, 0 failed.
|
||||
- `python archive/v1/data/proof/verify.py` → **VERDICT: PASS**, hash
|
||||
`f8e76f21a0f9852b70b6d9dd5318239f6b20cbcb4cdd995863263cecdc446f7a` **unchanged**
|
||||
(off the signal proof path — confirms no pipeline alteration).
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- The three historical PCK numbers can now be **recomputed under one declared
|
||||
definition** and compared honestly. The retracted-number class of error
|
||||
(silent normalization mismatch) is structurally prevented going forward.
|
||||
- Establishes the measurement substrate for the beyond-SOTA target: GraphPose-Fi
|
||||
cross-environment **PCK@20 = 12.9%** (standard torso PCK) is now a number this
|
||||
harness can produce comparably.
|
||||
|
||||
### Negative
|
||||
- None functional. The harness is additive; no existing metric path changed.
|
||||
|
||||
### Neutral
|
||||
- Producing actual model numbers under this harness requires the trained models +
|
||||
datasets (MM-Fi) and, for cross-domain splits, is the next sub-deliverable of
|
||||
the benchmark/optimization milestone — out of scope here (this ADR is the
|
||||
*instrument*, not the *reading*).
|
||||
|
||||
## Links
|
||||
- ADR-155 — metric core (`pck_canonical`, torso-only) — generalized here
|
||||
- ADR-152 — WiFi-Pose SOTA 2026 intake / WiFlow-STD benchmark
|
||||
- `docs/research/sota-nn-train-benchmark-brief.md` — the motivating gap analysis
|
||||
- GraphPose-Fi — arXiv 2511.19105 (verified cross-env PCK@20 = 12.9% anchor)
|
||||
Reference in New Issue
Block a user