frank/ruvnet--RuView
1
0
Fork 0
mirror of https://github.com/ruvnet/RuView synced 2026-07-25 17:51:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(benchmarks): edge optimization measured + measurement (b) blocked + 92.9% retraction

Browse Source 
Edge optimization (ADR-152 optimize track): ONNX Runtime fp32 is the CPU
latency win (3.2 ms/window, ~3.4x faster than torch, parity 2.4e-7); ORT
dynamic int8 reaches 2.44 MB (paper's ~2.2 MB claim plausible only via
conv-capable toolchains; -0.16pt PCK@20, +18% MPJPE, 2x slower); torch
dynamic quant converts 0% of this conv-only model; fp16 halves storage free
but is slower on CPU.

Measurement (b) BLOCKED-ON-DATA: only 1,077 paired ESP32 windows exist
(stop rule <2k). Forensic recheck of the surviving April holdout RETRACTS
the ADR-079 '92.9% PCK@20' figure: constant-output model, absolute (not
torso) threshold, 69 near-static frames — mean predictor scores 100% under
that protocol; torso-PCK@20 is 19.1%. Corroborates PR #535. Stale citations
removed from user-guide, readme-details, ADR-152 §2.1.3; no-citation rule
extended to ADR-079 accuracy claims. Unblock: >=2k-window multi-pose paired
session + torso-PCK re-baseline.

Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
ruv
2026-06-10 22:05:39 -04:00
parent 0e4b596803
commit 42b261f807
9 changed files with 981 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/user-guide.md
+8 -1
View File
@@ -1747,7 +1747,14 @@ See [ADR-071](adr/ADR-071-ruvllm-training-pipeline.md) and the [pretraining tuto
For significantly higher accuracy, use a webcam as a **temporary teacher** during training. The camera captures real 17-keypoint poses via MediaPipe, paired with simultaneous ESP32 CSI data. After training, the camera is no longer needed — the model runs on CSI only.
**Result: 92.9% PCK@20** from a 5-minute collection session.
> **Accuracy note (2026-06-10):** the previously cited "92.9% PCK@20" figure is
> retracted — a forensic recheck of the surviving eval holdout showed it came
> from a constant-output model scored with an absolute (non-torso-normalized)
> threshold on 69 nearly-static frames, a protocol under which a trivial
> mean-pose predictor scores 100%. No measured camera-supervised PCK@20 is
> currently published (see CHANGELOG, PR #535). Treat this workflow as a data
> collection mechanism; accuracy claims will follow a ≥35-minute multi-pose
> collection session evaluated with torso-normalized PCK.
### Requirements