frank/ruvnet--RuView
1
0
Fork 0
mirror of https://github.com/ruvnet/RuView synced 2026-07-24 17:43:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(benchmarks): edge optimization measured + measurement (b) blocked + 92.9% retraction

Browse Source 
Edge optimization (ADR-152 optimize track): ONNX Runtime fp32 is the CPU
latency win (3.2 ms/window, ~3.4x faster than torch, parity 2.4e-7); ORT
dynamic int8 reaches 2.44 MB (paper's ~2.2 MB claim plausible only via
conv-capable toolchains; -0.16pt PCK@20, +18% MPJPE, 2x slower); torch
dynamic quant converts 0% of this conv-only model; fp16 halves storage free
but is slower on CPU.

Measurement (b) BLOCKED-ON-DATA: only 1,077 paired ESP32 windows exist
(stop rule <2k). Forensic recheck of the surviving April holdout RETRACTS
the ADR-079 '92.9% PCK@20' figure: constant-output model, absolute (not
torso) threshold, 69 near-static frames — mean predictor scores 100% under
that protocol; torso-PCK@20 is 19.1%. Corroborates PR #535. Stale citations
removed from user-guide, readme-details, ADR-152 §2.1.3; no-citation rule
extended to ADR-079 accuracy claims. Unblock: >=2k-window multi-pose paired
session + torso-PCK re-baseline.

Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
ruv
2026-06-10 22:05:39 -04:00
parent 0e4b596803
commit 42b261f807
9 changed files with 981 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/ADR-152-wifi-pose-sota-2026-intake.md
+1 -1
View File
@@ -47,7 +47,7 @@ Adopt four changes, ordered by effort-vs-gain:
1. **Record transceiver geometry at enrollment.** `EnrollmentProtocol` gains an optional `NodeGeometry` record per node (position estimate, antenna orientation, inter-node distances where known). Stored alongside the room baseline in the bank; schema-versioned so existing banks remain readable.
2. **Fuse geometry embeddings into specialist training.** Where a specialist head consumes the (future, ADR-150) backbone embedding, concatenate a small learned embedding of `NodeGeometry` — the PerceptAlign mechanism, transplanted to our per-room banks. Statistical specialists (current) ignore it; LoRA heads (ADR-151 P6) consume it.
3. **Adopt the two-checkerboard alignment for the camera-supervised path (ADR-079).** When MediaPipe supervision is used, calibrate camera↔WiFi into one shared 3D frame before regression (<5 min, two checkerboards, a few photos). This is the direct defense against F1 for our 92.9%-PCK@20 pipeline.
3. **Adopt the two-checkerboard alignment for the camera-supervised path (ADR-079).** When MediaPipe supervision is used, calibrate camera↔WiFi into one shared 3D frame before regression (<5 min, two checkerboards, a few photos). This is the direct defense against F1 for our camera-supervised pipeline. ~~92.9%-PCK@20~~ — *that figure was retracted during measurement (b) (2026-06-10): the surviving holdout shows a constant-output model under an absolute (non-torso) threshold on 69 near-static frames; mean predictor scores 100% under the same protocol. The §2.2 no-citation rule now applies to it.*
4. **Evaluate on the PerceptAlign cross-domain dataset** (21 subjects / 7 layouts) as the MERIDIAN cross-layout benchmark — *gated on confirming its license and downloadability* (open question; repo per paper: github.com/Trymore-lab/PerceptAlign).
> **Gate resolved (2026-06-10, MEASURED by repo inspection):** repo exists, **MIT license**, dataset downloadable from HuggingFace (5 per-scene repos, raw CSI + separate vision keypoints; Intel 5300, 1TX×3RX×3 ant, 57 subcarriers — same order as ESP32 subcarrier counts; Scene3 ships 3 distinct layouts). Code present, no pretrained weights. Benchmark adoption unblocked; dataset-side license terms inherit HF dataset terms (not separately stated — check at download time).
docs/readme-details.md
+15 -10
View File
@@ -50,7 +50,7 @@ See [PR #405](https://github.com/ruvnet/RuView/pull/405) for full details.
### What's New in v0.7.0
<details>
<summary><strong>Camera Ground-Truth Training — 92.9% PCK@20</strong></summary>
<summary><strong>Camera Ground-Truth Training</strong></summary>
**v0.7.0 adds camera-supervised pose training** using MediaPipe + real ESP32 CSI data:
@@ -76,15 +76,20 @@ node scripts/train-wiflow-supervised.js --data data/paired/*.jsonl --scale lite
node scripts/eval-wiflow.js --model models/wiflow-real/wiflow-v1.json --data data/paired/*.jsonl
```
**Result: 92.9% PCK@20** from a 5-minute data collection session with one ESP32-S3 and one webcam.
> **Accuracy retraction (2026-06-10):** the "92.9% PCK@20" figure previously
> shown here is retracted. A forensic recheck of the surviving eval holdout
> (69 samples) found a constant-output model scored with an absolute
> (non-torso-normalized) threshold on nearly-static frames — a protocol under
> which a trivial mean-pose predictor scores 100%. Torso-normalized PCK@20 on
> the same holdout is ~19% (from that degenerate predictor). No measured
> camera-supervised PCK@20 is currently published (CHANGELOG, PR #535).
| Metric | Before (proxy) | After (camera-supervised) |
|--------|----------------|--------------------------|
| PCK@20 | 0% | **92.9%** |
| Eval loss | 0.700 | **0.082** |
| Bone constraint | N/A | **0.008** |
| Training time | N/A | **19 minutes** |
| Model size | N/A | **974 KB** |
| Metric | Camera-supervised run (protocol retracted) |
|--------|--------------------------------------------|
| Eval loss | 0.082 |
| Bone constraint | 0.008 |
| Training time | 19 minutes |
| Model size | 974 KB |
Pre-trained model: [HuggingFace ruv/ruview/wiflow-v1](https://huggingface.co/ruv/ruview)
@@ -868,7 +873,7 @@ Download a pre-built binary — no build toolchain needed:
| Release | What's included | Tag |
|---------|-----------------|-----|
| [v0.7.0](https://github.com/ruvnet/RuView/releases/tag/v0.7.0) | **Latest** — Camera-supervised WiFlow model (92.9% PCK@20), ground-truth training pipeline, ruvector optimizations | `v0.7.0` |
| [v0.7.0](https://github.com/ruvnet/RuView/releases/tag/v0.7.0) | **Latest** — Camera-supervised WiFlow model (accuracy figure retracted 2026-06-10, see above), ground-truth training pipeline, ruvector optimizations | `v0.7.0` |
| [v0.6.0](https://github.com/ruvnet/RuView/releases/tag/v0.6.0-esp32) | [Pre-trained models on HuggingFace](https://huggingface.co/ruv/ruview), 17 sensing apps, 51.6% contrastive improvement, 0.008ms inference | `v0.6.0-esp32` |
| [v0.5.5](https://github.com/ruvnet/RuView/releases/tag/v0.5.5-esp32) | SNN + MinCut (#348 fix) + CNN spectrogram + WiFlow + multi-freq mesh + graph transformer | `v0.5.5-esp32` |
| [v0.5.4](https://github.com/ruvnet/RuView/releases/tag/v0.5.4-esp32) | Cognitum Seed integration ([ADR-069](docs/adr/ADR-069-cognitum-seed-csi-pipeline.md)), 8-dim feature vectors, RVF store, witness chain, security hardening | `v0.5.4-esp32` |
docs/user-guide.md
+8 -1
View File
@@ -1747,7 +1747,14 @@ See [ADR-071](adr/ADR-071-ruvllm-training-pipeline.md) and the [pretraining tuto
For significantly higher accuracy, use a webcam as a **temporary teacher** during training. The camera captures real 17-keypoint poses via MediaPipe, paired with simultaneous ESP32 CSI data. After training, the camera is no longer needed — the model runs on CSI only.
**Result: 92.9% PCK@20** from a 5-minute collection session.
> **Accuracy note (2026-06-10):** the previously cited "92.9% PCK@20" figure is
> retracted — a forensic recheck of the surviving eval holdout showed it came
> from a constant-output model scored with an absolute (non-torso-normalized)
> threshold on 69 nearly-static frames, a protocol under which a trivial
> mean-pose predictor scores 100%. No measured camera-supervised PCK@20 is
> currently published (see CHANGELOG, PR #535). Treat this workflow as a data
> collection mechanism; accuracy claims will follow a ≥35-minute multi-pose
> collection session evaluated with torso-normalized PCK.
### Requirements