frank/ruvnet--RuView
1
0
Fork 0
mirror of https://github.com/ruvnet/RuView synced 2026-07-25 17:51:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

deploy: 55f6a74e1e

Browse Source
This commit is contained in:
ruvnet
2026-06-02 15:46:25 +00:00
parent e3c245b45b
commit 62f74a1ea3
340 changed files with 119915 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
api-docs/benchmarks/homecore-vs-home-assistant.md
+98
View File
@@ -0,0 +1,98 @@
# RuView HOMECORE vs Home Assistant — Performance & Capability Benchmark
**Measured:** 2026-05-31 · Windows 11, Docker Desktop 28.5.1 (WSL2 Linux engine) · single host.
**Reproduce:** `python aether-arena/staging/run_homecore_bench.py` and `python aether-arena/staging/run_ha_bench.py`.
HOMECORE is RuView's **wire-compatible Rust port of Home Assistant's core** (ADR-125…ADR-134): the
same `/api` REST + WebSocket surface, the same SQLite recorder schema, an automation engine, a
HomeKit bridge, a WASM plugin runtime, and a voice/assist pipeline — plus **native WiFi/RF sensing
entities** (presence, breathing, heart-rate, pose) that Home Assistant can only get through external
add-ons. Because the API is wire-compatible, the two can be measured head-to-head on the same client.
> **Read this honestly.** HOMECORE (`0.1.0-alpha`) is a young, focused core; Home Assistant is a
> mature platform with ~3,000 integrations and a decade of ecosystem. HOMECORE's thesis is **not**
> "more features" — it is **the same control plane at 1/35th the memory and 18× the startup speed,
> with RF sensing built in.** The numbers below quantify exactly that trade.
## Performance (measured)
| Metric | RuView HOMECORE `0.1.0-alpha` | Home Assistant `stable` | Advantage |
|--------|------------------------------:|------------------------:|-----------|
| **Cold start → API/web ready** | **0.55 s** | 9.72 s | **18× faster** |
| **Idle resident memory (RSS)** | **10.1 MB** | 359 MB | **35× leaner** |
| **Distribution size** | **4.7 MB** (single static binary) | 610 MB (container image) | **130× smaller** |
| **Idle CPU** | 0.0 % | 0.0 % | tie |
| **REST latency p50** | 2.13 ms | 2.95 ms | comparable¹ |
| **REST latency p95** | 22.9 ms | 27.3 ms | comparable¹ |
| **REST latency p99** | 26.2 ms | 28.3 ms | comparable¹ |
| **REST throughput (1 conn, sequential)** | **1,599 req/s** | 716 req/s | **2.2×** |
| **Recorder DB after boot** | 36.9 KB | 4.1 KB | — (HOMECORE seeds 10 demo entities + history) |
| **Process threads (idle)** | 22 | n/a (containerized Python) | — |
¹ **Latency caveat — read before quoting.** The two latency rows are *not* the same endpoint.
HOMECORE is measured on **authenticated `/api/states`** (returns 10 live entities). Home Assistant's
`/api/*` requires a completed onboarding flow + long-lived access token, so HA is measured on the
**unauthenticated `/manifest.json`** served by the same aiohttp stack. Both are single-connection,
300-sample, sequential. Treat latency as "same order of magnitude"; treat **memory, startup, and
size as the decisive, apples-to-apples results.** Throughput is endpoint-confounded the same way —
the 2.2× is directional, not a controlled isolate.
### What the deltas mean in practice
- **10 MB vs 359 MB RSS:** HOMECORE runs comfortably on a Pi Zero 2 W or an ESP32-class gateway
alongside the sensing pipeline; HA effectively needs a Pi 4/5 or x86 to itself.
- **0.55 s vs 9.7 s start:** HOMECORE can be cold-started per-request or restarted on config change
without a noticeable outage; HA's ~10 s boot (longer with real integrations) makes it a
long-lived daemon only.
- **4.7 MB vs 610 MB:** OTA-updating the whole control plane over a metered/rural link is trivial
for HOMECORE; HA ships as a ~250 MB compressed image.
## Capability & feature comparison
| Capability | RuView HOMECORE | Home Assistant |
|-----------|-----------------|----------------|
| HA-compatible REST `/api` | ✅ wire-compatible subset (ADR-130) | ✅ reference implementation |
| HA-compatible WebSocket API | ✅ (ADR-130) | ✅ |
| State machine + event bus + service registry | ✅ 13 seeded services (ADR-127) | ✅ |
| SQLite recorder (history) | ✅ HA-compat schema **+ ruvector semantic search** (ADR-132) | ✅ (no vector search) |
| Automation engine + Jinja templates | ✅ MiniJinja trigger/condition/action (ADR-129) | ✅ (full Jinja2) |
| HomeKit (Apple Home) bridge | ✅ scaffold (ADR-125) | ✅ mature |
| Plugin/integration runtime | ✅ **sandboxed WASM** plugins (ADR-128) | ✅ Python integrations (in-process, unsandboxed) |
| Voice / intent / "Assist" | ✅ 5 built-in intents **+ ruflo agent bridge** (ADR-133) | ✅ Assist + LLM agents |
| Migration from existing HA | ✅ reads HA `.storage/` + `automations.yaml` (ADR-134) | n/a |
| **Native WiFi/RF sensing entities** | ✅ **presence, breathing, HR, 17-kp pose, fall** as first-class sensors | ⚠️ only via external add-on/MQTT |
| Integration ecosystem breadth | ⚠️ early — core + WASM plugins | ✅ ~3,000 integrations, HACS |
| Mature web UI / dashboards (Lovelace) | ❌ not yet | ✅ extensive |
| Add-on store / supervised OS | ❌ | ✅ HAOS + Supervisor |
| Community / docs maturity | ⚠️ alpha | ✅ very large |
| Memory / startup / footprint | ✅✅ (see table) | ⚠️ heavy |
| Language / safety | Rust (memory-safe, single static binary) | Python (interpreted, large dep tree) |
### Where each wins
- **HOMECORE wins:** resource footprint, cold-start, distribution size, throughput-per-MB, memory
safety, sandboxed (WASM) plugins, and — uniquely — **WiFi/RF sensing as native entities**. Ideal
for edge gateways, battery/solar nodes, and shipping the control plane *with* the sensor.
- **Home Assistant wins:** integration breadth, UI/dashboard maturity, add-on ecosystem, community
support, and production track record. Ideal as a full-house hub on a Pi 4/5+ or x86.
## Honest summary
For the **shared, wire-compatible HA control plane**, HOMECORE delivers it at **~35× less RAM,
~18× faster startup, and ~130× smaller footprint**, with WiFi sensing built in and HA-config
migration on the way. What it does **not** yet match is Home Assistant's enormous integration
catalog and UI maturity. The right read is **"HA-compatible core, edge-class resource budget,
RF-native"** — not "HA replacement." For a sensing node that also needs to *be* a smart-home hub,
HOMECORE's efficiency is decisive; for a feature-complete whole-home hub today, Home Assistant
remains the broader platform.
## Reproduction & method
- **HOMECORE:** `v2/target/release/homecore-server.exe` (`0.1.0-alpha.0`), bound to `127.0.0.1:8124`,
SQLite file recorder, dev-token auth (`Authorization: Bearer …`). Startup = `Popen` → first `200`
on `/api/`. RSS/CPU via `psutil` after a 2 s settle. 300-sample sequential latency on `/api/states`.
- **Home Assistant:** `ghcr.io/home-assistant/home-assistant:stable` in Docker, `-p 8125:8123`,
fresh `/config`. Startup = container start → first `<500` on `/manifest.json`. RSS/CPU via
`docker stats --no-stream` after a 20 s settle. 300-sample sequential latency on `/manifest.json`.
- Both runs are single-host, single-connection, no concurrency tuning. Numbers are indicative of
the **resource/startup class**, which is the property that differs by orders of magnitude;
latency/throughput are reported with the endpoint caveat above and should not be over-read.
- Harness scripts: `aether-arena/staging/run_homecore_bench.py`, `aether-arena/staging/run_ha_bench.py`.
api-docs/benchmarks/mmfi-wifi-sensing-study.md
+166
View File
@@ -0,0 +1,166 @@
# WiFi-CSI Sensing on MM-Fi — a complete, honest study
**Scope:** what works, what doesn't, and what actually ships — for 2D human **pose** and **action
recognition** from WiFi Channel State Information on the public [MM-Fi](https://github.com/ybhbingo/MMFi_dataset)
benchmark (40 subjects × 4 environments, 27 activities, `[3 antennas, 114 subcarriers, 10 frames]`
CSI amplitude). All numbers measured on an RTX 5080; reproduction scripts referenced throughout.
> **One-line takeaway:** we beat published pose SOTA *and* shrank it to a 20 KB edge model, but the
> deeper result is that **WiFi sensing doesn't generalize zero-shot to new people/rooms — and a
> ~30-second in-room calibration fixes that completely, for *both* tasks.** Few-shot calibration, not
> zero-shot invariance, is the deployment answer.
>
> **Sharpest finding (§7):** WiFi-CSI sensing is largely a **random-features + target-trained-readout**
> problem — a *random frozen* encoder + a trained head gets within ~2–4 pts of a fully-trained encoder
> (and within <2 pts cross-subject). The encoder barely learns anything transferable; the signal is in
> the readout. This single fact explains the zero-shot collapse, the no-transfer results, the
> foundation-encoder failure, *and* why per-room calibration works.
## 1. Pose estimation
### 1.1 In-domain accuracy (beats SOTA)
Metric: torso-normalized PCK@20 (MultiFormer's definition). Protocol: MM-Fi `random_split` (the
dataset default).
| Model | torso-PCK@20 |
|-------|-------------:|
| CSI2Pose (prior) | 68.41% |
| MultiFormer (prior SOTA, 2025) | 72.25% |
| **Ours (single)** | **82.69%** |
| **Ours (graph + 3-ensemble + TTA)** | **83.59%** |
Architecture: linear projection → 4-layer/8-head Transformer over the 10 temporal tokens →
**temporal attention pooling** (the single biggest lever) → MLP head → skeleton-graph refinement.
The headline was *self-corrected down* from an inflated 91.86% (loose bbox normalization) to 82.69%
under the matched torso metric before publishing.
### 1.2 Efficiency frontier (beats SOTA at a fraction of the size)
Every model from `micro` (75 K params) up is **Pareto-dominant** — smaller *and* more accurate than
prior SOTA. A **75 K-param model tops MultiFormer**; deployed **int4 is ~20 KB at 74.08% (QAT)**,
0.135 ms single-thread CPU. (int8 is lossless at 74.7%; naïve int4 PTQ drops to 70.2% — QAT recovers
it.) Full curve: [`wifi-pose-efficiency-frontier.md`](wifi-pose-efficiency-frontier.md).
Published: [`ruvnet/wifi-densepose-mmfi-pose`](https://huggingface.co/ruvnet/wifi-densepose-mmfi-pose).
## 2. Action recognition (27 classes)
MM-Fi's own paper **does not benchmark WiFi-CSI action recognition** (its HAR is skeleton-based,
RGB/LiDAR/mmWave only). The only published WiFi-CSI-on-MM-Fi number is WiDistill (2024): 34.0%
(ResNet-18, unspecified split). We establish:
| Protocol | top-1 |
|----------|------:|
| random_split (in-domain) | 88.08% |
| cross-subject (official), zero-shot | **10.0%** (near-chance) |
The 88% is **leakage-inflated** (see §3); the honest cross-subject zero-shot is ~10%.
## 3. The generalization story (the real result)
Random-split numbers are inflated by temporal/subject adjacency. Under leakage-free protocols, WiFi
sensing **collapses**:
| Task | in-domain | cross-subject (zero-shot) | cross-environment (zero-shot) |
|------|----------:|--------------------------:|------------------------------:|
| Pose | 83.6% | 64% | ~10% |
| Action | 88.1% | 10% | — |
### 3.1 What does NOT close the gap (all measured, all negative)
- **CORAL** (deep feature-cov alignment): no cross-subject gain; only marginal on cross-env (~17%).
- **DANN** (subject-adversarial): ~0, loss-imbalance fragile.
- **Per-antenna instance-norm + SpecAugment**: −4.6 (destroys cross-antenna pose structure).
- **Pose-contrastive foundation pretraining**: −2.3 — and the SupCon loss *never left the `ln(B)`
random floor*, i.e. same-pose CSI is **not contrastively alignable across subjects**: the invariance
the objective wants isn't present in the data.
- **Knowledge distillation** (flagship→tiny): no gain; direct training wins.
- **More training subjects**: saturates — 4→8 subjects = +21 pts, but 24→32 = +0.45 pts (asymptote ~64%).
Only **mixup + TTA + ensemble** helps cross-subject, and by <1 pt. The gap is *fundamental
distribution shift*, not a tunable/algorithmic gap.
### 3.2 What DOES close it: few-shot in-room calibration
A handful of labeled frames from the actual deployment room recovers most of the gap — and the
*biggest* zero-shot gap gives the *biggest* gain (an unseen room is one coherent shift a few frames
pin down):
| Calibration samples/subject | Pose cross-subj | Pose cross-env | Action cross-subj |
|----------------------------:|----------------:|---------------:|------------------:|
| 0 (zero-shot) | 64% | ~10% | 10% |
| 5 | — | **60%** | 13% |
| 50 | 70% | 70% | 36% |
| 200 | 76% | 73% | 59% |
| 1000 | 78% | 75% | 76% |
**Confirmed task-general:** the identical pattern holds for pose regression *and* 27-class action
classification. Few-shot in-room calibration is the **universal** WiFi-sensing deployment mechanism.
(Action needs more calibration than pose — classification vs regression.)
### 3.3 Deployable as a ~11 KB adapter
Full fine-tune means a 2.3 MB model copy per room. A **rank-8 LoRA adapter (~11 KB)** recovers most
of the gain (cross-subject 64→72.5% at 0.5% the size). Calibration data budget: **~100–200 labeled
samples** (knee at ~50 → 70%; below ~20 it can hurt).
| Calibration method @200 samples | PCK@20 | adapter |
|---------------------------------|-------:|--------:|
| LoRA rank-8 | 72.5% | ~11 KB |
| head + graph only | 72.7% | 119 KB |
| frozen-trunk | 73.5% | 207 KB |
| full finetune | 76.2% | 2.3 MB |
## 4. The calibration service (shipped)
The mechanism is implemented end-to-end: a Python reference
([`aether-arena/calibration/`](../../aether-arena/calibration/) — `calibrate.py` fits an adapter from
a labeled clip, verified 3.09%→74.29% on an unseen MM-Fi room) **and** in the Rust product engine
(`cog-pose-estimation`: `InferenceEngine::with_adapter()`, `run --adapter <room.safetensors>`,
architecture-agnostic LoRA on the pose head, tested).
## 5. Honest limitations
- Most generalization numbers are within MM-Fi (one dataset, one hardware setup). **Cross-*dataset***
transfer was tested against **NTU-Fi HAR** (same 3×114 layout, different lab/hardware/rooms): an
MM-Fi-trained representation does **not** transfer beneficially — a frozen MM-Fi trunk probes NTU-Fi
at 91.5%, *no better than random features* (93%), and full fine-tuning (75%) underperforms a linear
probe. CSI representations are **distribution-locked** (same root cause as the within-MM-Fi
cross-subject/-environment collapse); the practical answer is on-target training/few-shot, not
transferable zero-shot features. Caveat: NTU-Fi's 6 coarse activities are an *easy* target (random
features → 93%), so it weakly stresses representation quality — but re-running on the harder
**NTU-Fi-HumanID** task (14-class gait person-ID, chance 7.1%) gave the *same* result (MM-Fi
pretrain 91.7% ≈ random 92.8%). **Unified root cause:** for CSI, in-domain classification lives in
the *target-trained readout* (a random 256-d projection of 3,420-d CSI is already linearly
separable), while the *learned representation* fails to transfer across subjects, rooms, and
datasets alike. WiFi-CSI sensing is **distribution-locked**; the answer is on-target few-shot
calibration, not transferable features. A harder cross-dataset *pose* benchmark (vs classification)
remains the one open variant.
- Random-split numbers are reported only to compare to prior work on the same protocol; they are
in-domain and partly leaky. The cross-subject / cross-environment numbers are the honest ones.
- Action-recognition accuracy is window-level (MM-Fi's own HAR experiment is clip-level); not directly
comparable to sequence-level reports.
- On-device (ARM/Hailo) latency is pending hardware; CPU latency (0.135 ms x86 single-thread) is the
current proxy.
## 6. Reproduction
Pose: `aether-arena/staging/train_save.py` (flagship), `train_efficiency_pareto.py`,
`quant_micro.py`, `train_fewshot_adapt.py`, `train_adapter_calib.py`. Action: `train_action.py`,
`train_action_fewshot.py`. Calibration service: `aether-arena/calibration/`. Decision record + full
empirical chain: [ADR-150 §3.2–3.6](../adr/ADR-150-rf-foundation-encoder.md). Leaderboard + witness
ledger: [AetherArena](https://huggingface.co/spaces/ruvnet/aether-arena) (ADR-149).
## 7. The sharpest result: the encoder barely matters
A random *frozen* transformer encoder + a trained pose head matches a fully-trained encoder to within
2–4 points (cross-subject: <2 points):
| Pose protocol | fully-trained encoder | random-frozen encoder + head |
|---------------|----------------------:|-----------------------------:|
| in-domain | 78.2% | 73.8% |
| cross-subject | 63.9% | 62.1% |
(Same fair-comparison config; absolute numbers below the 83.6% flagship — the *delta* is the point.)
**Almost all the task signal lives in the readout** (pose head + skeleton-graph refinement on a
random high-dim CSI projection), not in the learned encoder. This is the unifying explanation for the
whole study: there is barely a *learned representation* to transfer (hence the cross-subject/-env/
-dataset collapses and the foundation-encoder failure), and per-room calibration works precisely
because it re-fits the readout where the signal is. **Practical upshot:** for WiFi-CSI sensing, spend
compute on the readout + per-room calibration, not on expensive encoder pretraining. Reproduce:
`aether-arena/staging/train_pose_randomfeat.py`.
api-docs/benchmarks/person-count-cog.md
+185
View File
@@ -0,0 +1,185 @@
# `cog-person-count` — Benchmark Log
Append-only log of every published count_v1 training run per ADR-103. New runs add a section; never overwrite history.
## v0.0.2 — K-fold validated, random split + label smoothing + early stop + temp scale (2026-05-21)
### Why a new release
A 5-fold stratified CV on the same 1,077 samples proved the v0.0.1 result was driven by an unlucky temporal split — the trailing window was class-0-heavy, and a degenerate "always predict 0" classifier hit the class-0 fraction (65.1%) trivially.
| Metric | v0.0.1 (temporal) | **5-fold random CV** (diagnostic) |
|---|---|---|
| Overall accuracy | 65.1% | 62.2% ± 1.9% |
| Class 1 accuracy | **0%** | **57.1%** ✓ |
| Confidence Spearman | 0.023 | 0.160 ± 0.029 |
The architecture has real ~57% class-1 capacity under fair splits.
### v0.0.2 results
Architecture unchanged. Training changes only:
- **Random 80/20 split** (seed=42) — temporal split eliminated.
- **Label smoothing 0.1** on cross-entropy.
- **Class-balanced multinomial sampler** with replacement.
- **Early stopping** with patience 20 (exited at epoch 29 of 400 max).
- **Temperature scaling** of the conf head via LBFGS — T = **0.9262**, shipped as a `count_v1.temperature` sidecar.
| Metric | v0.0.1 | **v0.0.2** | K-fold ref |
|---|---|---|---|
| Overall accuracy | 65.1% | **62.3%** | 62.2% ± 1.9% |
| Class 0 accuracy | 100% (cheating) | **86.2%** | 67.4% |
| **Class 1 accuracy** | **0%** | **34.3%** ✓ | 57.1% |
| MAE | 0.349 | 0.377 | 0.378 |
| Confidence Spearman (post-temp) | 0.023 | 0.013 | 0.160 |
| Wall time | 5.6 s (400 ep) | **0.7 s (29 ep)** | 7.5 s (5×100) |
### Honest read
**Class-1 accuracy 0% → 34.3% is the headline.** The cog now reports `count = 1` honestly when a person is present, instead of always-zero cheating. Single random draw lands below the K-fold mean of 57% — that gap is run-to-run variance, not a missing improvement. Reaching 57% on a fixed eval set needs averaging over independent draws, which means more independent recordings — i.e. multi-room data (#645), not another training trick.
Confidence calibration didn't move. Temperature scaling alone can't fix a confidence head trained against a noisy `argmax==truth` indicator over a 62%-accurate classifier — its training signal is the bottleneck.
### Release artifacts (live on cognitum-v0)
```
gs://cognitum-apps/cogs/arm/cog-person-count-count_v1.safetensors
sha256: 32996433516891a37c63c600db8b95e42192a53bd538c088c82cd6a85e55513c
bytes: 392,088
```
Binaries themselves unchanged from v0.0.1 — weights load at runtime via mmap. Per-arch manifests under `cog/artifacts/manifests/{arm,x86_64}/` bumped to `version: 0.0.2`, weights_sha256 + build_metadata caveats updated.
### Reproducibility
```bash
python3 scripts/train-count.py --paired data/paired/wiflow-p7-1779210883.paired.jsonl \
--k-fold 5 --epochs 100 --out-results kfold_results.json
python3 scripts/train-count.py --paired data/paired/wiflow-p7-1779210883.paired.jsonl \
--v2 --epochs 400 \
--out-safetensors count_v1.safetensors --out-onnx count_v1.onnx \
--out-results count_train_results.json
```
## v0.0.1 — first measured run (2026-05-21)
### Setup
| Component | Value |
|-----------|-------|
| Training host | `ruvultra` (Ubuntu, x86_64, RTX 5080) |
| Backend | PyTorch 2.12 + CUDA |
| Data | `data/paired/wiflow-p7-1779210883.paired.jsonl` — 1,077 paired samples, single 30-min session, label distribution `{0: 533, 1: 544}` |
| Train/eval split | 80/20 stratified on `ts_start` (held-out tail of the recording) |
| Architecture | Conv1d encoder (56→64→128→128, dilations 1/2/4) + Linear(128→64→8) count head + Linear(128→32→1) confidence head — bit-identical to `v2/crates/cog-person-count/src/inference.rs::CountNet` |
| Loss | `cross_entropy(count) + 0.3·BCE(conf) + 0.1·Brier(conf)` with per-class weighting |
| Optimizer | AdamW, lr 1e-3, cosine warm restarts (T_0=50) |
| Z-score normalisation | per-subcarrier on train statistics, applied to eval |
| Epochs | 400 |
| Wall time | **5.6 s** |
### Accuracy (held-out 215-sample tail of the 30-min recording)
| Metric | Value |
|--------|-------|
| Best eval accuracy | **65.1%** |
| Final eval accuracy | 65.1% |
| Within ±1 | **100%** (labels are all in `{0, 1}`, predictions trivially within ±1) |
| MAE | 0.349 persons |
| Class 0 ("empty") accuracy | **100%** (140 samples) |
| Class 1 ("1 person") accuracy | **0%** (75 samples) |
| Confidence↔correctness Spearman | 0.023 |
### Honest read
The model overfit hard. By epoch 100 train_acc reached 1.0 and eval_loss climbed from 0.67 → 7.8. The "best" checkpoint (epoch ~2-3) is the snapshot that happened to predict mostly class-0 across eval, which matches the held-out window's class distribution (140/215 = 65.1%) — i.e. it learned the **distribution of the tail of the recording**, not a real empty-vs-occupied classifier.
Why: the training data is one continuous 30-minute solo recording. The held-out tail captures a stretch where the operator stepped away from the desk for stretches at a time, so the eval set is class-0-heavy and the model finds a degenerate "always predict 0" minimum that gets the eval distribution exactly right. Class 1 accuracy = 0 is the smoking gun.
Same data-bound failure mode as `pose_v1` (#645). Same fix path: multi-room paired recordings.
### What v0.0.1 still validates
- **Pipeline correctness end-to-end.** The Rust cog loaded the PyTorch-trained safetensors successfully on first try (`backend: candle-cpu` reported by `cog-person-count health`), confirming the architecture in `src/inference.rs` is byte-compatible with `train-count.py`.
- **ONNX parity.** 16 KB ONNX, exports cleanly under opset 18 with dynamic batch axis.
- **Fast iteration loop.** 5.6 s end-to-end training means we can sweep hyperparameters or retrain on new data in seconds, not hours.
- **Cog binary size.** Same 2.36 MB stripped release binary (no change — model loads at runtime via mmap'd safetensors).
### Comparison to ADR-103 v0.1.0 targets
| Gate | Target | Today | Status |
|------|--------|-------|--------|
| Day-0 same-room accuracy within ±1 | ≥ 80% | 100% (trivially — labels span {0,1}) | met |
| Cross-room accuracy within ±1 | ≥ 60% | Not measured (no cross-room data) | deferred to v0.2.0 |
| MAE | ≤ 0.6 | 0.349 | met |
| Per-frame confidence reflects accuracy (Spearman) | r ≥ 0.5 | 0.023 | **NOT MET** |
| Inference latency on Pi 5 | < 5 ms / frame | Not yet measured (cross-compile pending) | deferred |
| Binary size on GCS | ≤ 4 MB | 2.36 MB | met |
The accuracy ones look "met" only because the labels collapse to {0, 1} and "within ±1" with 8 classes is trivially satisfied. The **confidence calibration is the real failure** for v0.0.1 — Spearman 0.023 means the confidence head is essentially random noise. That's also bounded by data scarcity; multi-session training should sharpen it.
### Artifacts
- `v2/crates/cog-person-count/cog/artifacts/count_v1.safetensors` — 392 KB
- `v2/crates/cog-person-count/cog/artifacts/count_v1.onnx` — 16 KB
- `v2/crates/cog-person-count/cog/artifacts/count_train_results.json` — full per-epoch loss curve + hyperparameters + per-class breakdown
### Reproducibility
```bash
# On any host with PyTorch + CUDA (cargo path not needed for training):
scp data/paired/wiflow-p7-1779210883.paired.jsonl <host>:/tmp/
scp scripts/train-count.py <host>:/tmp/
ssh <host> "cd /tmp && python3 train-count.py --paired wiflow-p7-1779210883.paired.jsonl --epochs 400"
```
Loads in the Rust cog with no translation step (safetensors layout matches `cog-person-count::inference::CountNet` exactly):
```bash
cp count_v1.safetensors v2/crates/cog-person-count/cog/artifacts/
cargo run -p cog-person-count --release -- health
# → {"backend":"candle-cpu", "synthetic_count": <int>, "synthetic_confidence": <float>, ...}
```
### Live appliance install (cognitum-v0 Pi 5)
Installed at `/var/lib/cognitum/apps/person-count/` with the same on-disk shape as `cog-pose-estimation`, `anomaly-detect`, `seizure-detect`, etc.:
```
$ ls -la /var/lib/cognitum/apps/person-count/
-rwxr-xr-x cog-person-count-arm 2,168,816 B (sha matches GCS)
-rw-r--r-- count_v1.safetensors 392,088 B
-rw-r--r-- manifest.json 1,073 B
-rw-r--r-- config.json 160 B
```
```
$ ./cog-person-count-arm health
{"ts": ..., "event": "health.ok",
"fields": {"backend": "candle-cpu", "synthetic_count": 0,
"synthetic_confidence": 0.49, "synthetic_p95_range": [0, 7]}}
```
Cold-start on real Pi 5 hardware: **9.2 ms / invocation** (30 sequential `health` invocations in 0.276 s). Slightly slower than the pose cog (8.4 ms) because the dual-head inference (count softmax + confidence sigmoid) does ~2× the work after the shared encoder; still comfortably inside ADR-103's < 5 ms warm-path budget once the long-running `run` loop lands and the safetensors stay mmapped between frames.
### Signed GCS release artifacts (publicly downloadable)
```
gs://cognitum-apps/cogs/arm/cog-person-count-arm 2,168,816 B
sha256: 36bc0bb0ece894350377d5f93d46cd29378cb289b3773530611c0d47b507b3c3
signature: R/00xdzHriyr/2rzr4wmPJ/Ken60A+RNdi8r0g2HYJNTXBaFtr46ExfNbiHlgYWadQXzTZdfJoyJK+a6k71NDg==
gs://cognitum-apps/cogs/x86_64/cog-person-count-x86_64 2,615,528 B
sha256: 76cdd1ec40211add90b4942a09f79939aa28210a27e931de67122357392b01db
signature: QB+8cnGSMQmubSt/KWVu1+JMg37AKnQXDsFQi/vi+jqpW9rVrGMtnxQpWEWZPeWU1AJ6pl3O2V+7ZtTNIQ2rDg==
gs://cognitum-apps/cogs/arm/cog-person-count-count_v1.safetensors 392,088 B
sha256: dacb0551fd3887958db19696d90d811ab08faa44703e6e04ff56d15c3a65a9ff
```
All signed with `COGNITUM_OWNER_SIGNING_KEY` (Ed25519). SHAs verified via public anonymous `https://storage.googleapis.com/...` download.
Manifests at:
- `v2/crates/cog-person-count/cog/artifacts/manifests/arm/manifest.json`
- `v2/crates/cog-person-count/cog/artifacts/manifests/x86_64/manifest.json
api-docs/benchmarks/pose-estimation-cog.md
+176
View File
@@ -0,0 +1,176 @@
# `cog-pose-estimation` — Benchmark Log
This file tracks every published benchmark for the pose-estimation Cog. New runs append; never overwrite history. Per ADR-101 §"Acceptance gates".
## v0.0.1 — first measured run (2026-05-19)
### Setup
| Component | Value |
|-----------|-------|
| Training host | `ruvultra` (Ubuntu 6.17, x86_64, RTX 5080) |
| Backend | `candle-core 0.9` with `cuda` feature |
| Data | `data/paired/wiflow-p7-1779210883.paired.jsonl` — 1,077 paired samples, 30-min seated-at-desk recording, avg conf 0.44 |
| Train/eval split | 80/20 stratified on `ts_start` (eval is a held-out time window, not random) |
| Architecture | Conv1d encoder (56 → 64 → 128, dilations 1/2/4) + MLP head (128 → 256 → 34 → sigmoid → [17, 2]) |
| Encoder init | random — HF presence model is MLP `8→64→128`, incompatible with this Conv1d shape |
| Optimizer | AdamW, lr 1e-3, weight_decay 0.01 |
| LR schedule | Cosine with 50-epoch warm restarts |
| Loss | SmoothL1 (Huber β=0.1), confidence-weighted by `record.conf` |
| Augmentation | Subcarrier dropout 10% (final 50 epochs) |
| Epochs | 400 (full-batch) |
| Wall time | **2.1 s** total |
### Accuracy
| Metric | Value |
|--------|-------|
| **PCK@20** (overall) | **3.0%** |
| **PCK@50** (overall) | **18.5%** |
| **MPJPE** (normalized) | **0.0931** |
| Final eval loss | 0.0101 |
| Loss reduction | 0.181 → 0.014 (13×) |
### Per-joint PCK
| Joint | PCK@20 | PCK@50 | | Joint | PCK@20 | PCK@50 |
|-------|-------:|-------:|--|-------|-------:|-------:|
| nose | 0.5% | 5.1% | | l_hip | 0.0% | 27.3% |
| l_eye | 2.8% | 8.3% | | **r_hip** | **25.0%** | **76.9%** |
| r_eye | 1.9% | 15.7% | | l_knee | 2.3% | 20.8% |
| l_ear | 0.0% | 3.2% | | r_knee | 0.9% | 35.2% |
| r_ear | 1.9% | 9.7% | | l_ankle | 1.4% | 7.9% |
| l_shoulder | 4.6% | 8.8% | | r_ankle | 0.9% | 9.3% |
| r_shoulder | 1.9% | 19.9% | | l_elbow | 1.9% | 26.4% |
| l_wrist | 3.2% | 24.1% | | r_elbow | 0.0% | 4.2% |
| r_wrist | 1.4% | 12.0% | | | | |
Strongest signal at right-side proximal joints (`r_hip` 77% PCK@50, `r_knee` 35%, `r_shoulder` 20%) — consistent with the camera framing during data collection (operator's right side most consistently in frame).
### Comparison to prior baseline
| Run | Backend | Train time | PCK@20 | PCK@50 | MPJPE |
|-----|---------|-----------:|-------:|-------:|------:|
| pre-2026-05-19 | pure-JS SPSA, lite TCN (#645) | ~20 min | 0.0% | 0.0% | 0.66 |
| **v0.0.1** (this run) | **candle-cuda, Conv1d TCN** | **2.1 s** | **3.0%** | **18.5%** | **0.093** |
**7× MPJPE improvement, 570× faster training, signal-bearing PCK at all proximal joints.** The remaining gap to ADR-079's PCK@20 ≥ 35% target is data-bound, not infra-bound (see Issue #645).
### Inference latency
Measured on Windows host (x86_64, no GPU — `candle-cpu` backend) running the release binary:
| Mode | Measurement | Notes |
|------|-------------|-------|
| Cold start | **76.2 ms / invocation** (avg over 100 sequential `health` invocations) | Includes safetensors load + 1 synthetic forward pass. Most of the cost is process startup + mmap. |
| Long-running `run` warm inference | sub-millisecond per frame (estimated) | The model is 125K params / 507 KB; once loaded, a single forward at batch=1 is essentially memory-bandwidth bound. To be measured precisely against a live sensing-server feed. |
### ONNX export
`pose_v1.onnx` is produced from `pose_v1.safetensors` by `scripts/export-onnx.py`, which mirrors the Candle architecture in PyTorch, loads the safetensors weights, and uses `torch.onnx.export` with opset 18 + dynamic batch axis. Verified end-to-end:
| Check | Result |
|-------|--------|
| `onnx.checker.check_model` | ✅ ok |
| Parity vs torch reference | **max \|torch − onnx\| = 8.94e−8** (1e−5 threshold) |
| File size | 12,059 bytes |
| Dynamic axes | `batch` on input and output |
The ONNX artifact is the input to the Hailo Dataflow Compiler (HEF cross-compile) and to ONNX Runtime CPU/GPU benchmarks on each target arch — both still pending.
### Real-hardware smoke (cognitum-v0 Pi 5)
Cross-compiled to `aarch64-unknown-linux-gnu` on ruvultra and run on a live Cognitum-V0 appliance:
| Host | Mode | Result |
|------|------|--------|
| ruvultra (under `qemu-aarch64-static`) | `health` | `backend: candle-cpu`, `confidence: 0.185` — real weights loaded under emulation |
| **cognitum-v0** (Raspberry Pi 5, Cortex-A76) | `health` | `backend: candle-cpu`, `confidence: 0.185` — real weights, real hardware |
| cognitum-v0 | 30× sequential `health` invocations | **0.251 s total → 8.4 ms / invocation** (cold) |
8.4 ms cold-start on real Pi 5 hardware vs 76 ms on the x86_64 Windows host. The Pi 5 has tighter NVMe I/O + the candle CPU path benefits from the in-cache safetensors mmap. Long-running `run` warm inference will still be sub-millisecond.
### Release artifacts (signed + published to GCS)
```
gs://cognitum-apps/cogs/arm/cog-pose-estimation-arm 3,741,976 bytes
gs://cognitum-apps/cogs/arm/cog-pose-estimation-pose_v1.safetensors 507,032 bytes
binary_sha256: 1e1a7d3dd01ca05d5bfc5dbb142a5941b7866ed9f3224a21edc04d3f09a99bf5
weights_sha256: eb249b9a6b2e10130437a10976ed0230b0d085f86a0553d7226e1ae6eae4b9e5
signature: LUN7xqLPYD3MFzm5dKB5MnYU0LvoRtek5ci5KiKPHBg+Xo6xuazwokn2Dw2JPMaLYJzmWn/SpT4djuR7hYvVDw== (Ed25519, signed with COGNITUM_OWNER_SIGNING_KEY)
```
Full manifest at `cog/artifacts/manifest.json`. Verified via public anonymous GET against `https://storage.googleapis.com/cognitum-apps/cogs/arm/cog-pose-estimation-arm` — downloaded SHA matches the locally-computed SHA.
### Live appliance install
Installed on `cognitum-v0` (the V0 cluster leader) at `/var/lib/cognitum/apps/pose-estimation/`:
```
$ ls -la /var/lib/cognitum/apps/pose-estimation/
-rwxr-xr-x cog-pose-estimation-arm 3,741,976 B (matches GCS sha256)
-rw-r--r-- pose_v1.safetensors 507,032 B
-rw-r--r-- manifest.json 989 B
-rw-r--r-- config.json 187 B
-rw-r--r-- output.log 28,438 B (5-sec smoke run)
```
Layout matches the existing `anomaly-detect`, `presence`, `seizure-detect`, etc. cogs on the same appliance — the Cogs dashboard at `http://cognitum-v0:9000/cogs` auto-discovers entries under this dir.
`cog-pose-estimation run` ran cleanly in the background for 5 seconds with the default config. It correctly:
- Emitted a `run.started` event with the configured `sensing_url`, `model_path`, and `poll_ms`.
- Started its 40 ms poll loop.
- **Gracefully handled the missing local sensing-server on port 3000** by logging structured WARN events (`{"level":"WARN","fields":{"message":"sensing-server fetch failed","error":"...Connection refused..."}}`) without crashing, leaking, or producing NaN output.
- Exited cleanly on SIGTERM.
0 `pose.frame` events fired during the smoke run — expected, since `127.0.0.1:3000` isn't serving CSI on the appliance. The appliance's actual CSI source is `ruview-vitals-worker` on `:50054` plus the `/api/v1/v0/system/...` endpoints behind the appliance's bearer auth on `:9000`. Wiring `sensing_url` to the appliance-native source is a Day-2 integration task — separate from the cog binary itself.
Pending separately:
- Hailo HEF cross-compile (gated on Hailo SDK on a self-hosted runner) — uses `pose_v1.onnx` as input.
- Appliance-native sensing-source integration (`config.sensing_url` should point at the cog-gateway's CSI tap on `:9000`, not the dev-loopback `:3000`).
### x86_64 release (2026-05-19)
Built on ruvultra (native, no cross-compile):
```
gs://cognitum-apps/cogs/x86_64/cog-pose-estimation-x86_64 4,548,856 bytes
sha256: a434739a24415b34e1aff50e5e1c3c32e568db96af473bbb3e5ecc9b95fe71fa
signature: pNNuxhgM18PztN8BSZdfw5oAShG2pV3na5T/q2QdlJWX/5FJgo4QTiUCbcTAxI2Uiva8VURSOlRzMU3xoQPqCQ==
```
Manifest at `cog/artifacts/manifests/x86_64/manifest.json`. Re-uses the same `pose_v1.safetensors` weights as the arm release (architecture is arch-independent).
**Cold-start: 5.4 ms / invocation** on ruvultra (30× sequential `health` in 0.162 s) — faster than the Pi 5's 8.4 ms (faster NVMe + wider CPU), slower than the Windows 76 ms (less mature Windows release toolchain).
| Host | arch | rust | binary | cold-start |
|------|------|------|--------|------------|
| Windows (ruvzen) | x86_64 | 1.95.0 | (built locally, not published) | 76.2 ms |
| ruvultra (Ubuntu) | x86_64 | 1.89.0 | 4,548,856 B (GCS x86_64) | **5.4 ms** |
| cognitum-v0 (Pi 5) | aarch64 | (cross-built) | 3,741,976 B (GCS arm) | 8.4 ms |
### Artifacts
- `v2/crates/cog-pose-estimation/cog/artifacts/pose_v1.safetensors` — 507 KB
- `v2/crates/cog-pose-estimation/cog/artifacts/train_results.json` — full per-epoch loss curve + hyperparameters + per-joint PCK
### Reproducibility
```bash
# On any host with cargo + a CUDA-capable GPU:
cd ~/work/cog-pose-train
mkdir -p ./
# Stage the same inputs (1,077 paired samples + HF encoder, see scripts/align-ground-truth.js for regeneration)
cp paired.jsonl ./paired.jsonl
cp encoder.safetensors ./encoder.safetensors
# Build & train (no Python, no pip)
cargo new --bin pose-trainer && cd pose-trainer
# Edit Cargo.toml deps: candle-core 0.9 (cuda), candle-nn 0.9 (cuda), safetensors, serde, serde_json, anyhow
# Drop the training script into src/main.rs (see this repo's training-tooling examples for reference)
cargo run --release
```
`candle-core 0.8.4 + 0.9.2` are typically already in `~/.cargo/registry/cache/` on any developer host, so the build completes in seconds.
api-docs/benchmarks/wifi-pose-efficiency-frontier.md
+91
View File
@@ -0,0 +1,91 @@
# WiFi-CSI Pose — Efficiency Frontier (beyond SOTA at a fraction of the size)
**Measured:** 2026-05-31 · MM-Fi `random_split` (ratio 0.8, seed 0) · RTX 5080 · torso-normalized
PCK@20 (MultiFormer Table VII metric: `‖pred−gt‖ ≤ 0.2·‖R-shoulder − L-hip‖`).
The flagship [`ruvnet/wifi-densepose-mmfi-pose`](https://huggingface.co/ruvnet/wifi-densepose-mmfi-pose)
reaches **83.59%** torso-PCK@20 (vs MultiFormer 72.25%, CSI2Pose 68.41%). But the headline number
isn't the whole story for **edge deployment** — on a Raspberry Pi / ESP32-class target, *params and
latency* matter as much as accuracy. So we swept model size to map the **accuracy-per-parameter
frontier**: how small can a WiFi-CSI pose model be and still beat the prior published SOTA?
## The frontier
| Model | Params | Latency (batch=1) | torso-PCK@20 | vs SOTA (72.25%) |
|-------|-------:|------------------:|-------------:|------------------|
| nano | 39,971 | 0.126 ms | 71.76% | −0.49 (58× smaller than flagship) |
| **micro** | **75,237** | 0.224 ms | **74.30%** | **✅ +2.05 — beats SOTA at 31× fewer params** |
| tiny | 210,949 | 0.299 ms | 76.82% | ✅ +4.57 |
| small | 348,005 | 0.287 ms | 77.87% | ✅ +5.62 |
| base | 726,437 | 0.344 ms | 79.38% | ✅ +7.13 (3.2× smaller) |
| flagship | 2,320,869 | — | 83.59% | +11.34 |
**Every configuration from `micro` (75K params) upward beats the prior published state of the art**,
and even `nano` (40K params, 0.13 ms) lands within half a point of it — at ~1/58th the flagship's
parameter count. A **75,237-parameter** model tops MultiFormer's 72.25%.
### Deployable footprint AND deployed accuracy (quantized `micro`)
Size alone isn't the claim — what matters is **accuracy at the deployed precision**. Measured
(weight-only, per-tensor symmetric):
| Precision | Size | torso-PCK@20 | vs SOTA 72.25 |
|-----------|-----:|-------------:|---------------|
| fp32 | 294 KB | 74.73% | ✅ +2.5 |
| **int8 (PTQ)** | **73.5 KB** | **74.70%** | ✅ +2.5 — **essentially lossless** |
| int4 (naïve PTQ) | 36.7 KB | 70.21% | ❌ −2.0 — drops below SOTA |
| **int4 (QAT)** | **36.7 KB** | **74.46%** | ✅ **+2.2 — recovered, still beats SOTA** |
**The honest edge result:** `micro` is **lossless at int8 (73.5 KB, 74.70%)**, and at **int4 (36.7 KB)
naïve post-training quantization falls below SOTA (70.21%) — but quantization-aware training fully
recovers it to 74.46%**, still beating MultiFormer. So a **SOTA-beating WiFi-pose model genuinely runs
in ~37 KB int4** (with QAT) or **~73 KB int8** (no retraining) — deployable on the sensing node itself.
`nano` (40K params) sits at the SOTA line in fp32 and is best treated as int8.
(We also tested flagship→tiny **knowledge distillation**: it did *not* help — the tiny students reach
equal or higher accuracy from ground truth alone, so regression-KD on keypoints only adds teacher
noise. Direct training wins.)
**Shipped as a usable artifact.** The int4-QAT `micro` model is published and downloadable at
[`ruvnet/wifi-densepose-mmfi-pose/edge`](https://huggingface.co/ruvnet/wifi-densepose-mmfi-pose/tree/main/edge)
(`pose_micro_int4.npz` + `load_int4.py`): **verified deployed int4 accuracy 74.08%** (beats SOTA),
~20 KB int4 weight payload, sha256 `c03eeb…`. It runs in **0.135 ms single-thread on x86 CPU**
(no GPU) — i.e. real-time pose with no accelerator; a Raspberry-Pi-class ARM core would be slower
but still comfortably real-time. (Latency measured on ruvultra x86; on-device ARM validation pending
the Pi fleet coming back online.)
## Why this matters
- **Edge-native pose.** `micro`/`tiny` (75–210K params, sub-0.3 ms on a discrete GPU) are small
enough to quantize and run on a Pi-class / Hailo edge node next to the sensing pipeline — no cloud
round-trip, no camera.
- **Pareto-dominant, not just smaller.** These aren't accuracy-traded-for-size compromises *below*
SOTA; they are simultaneously **smaller than MultiFormer and more accurate than it**.
- **Orthogonal to the accuracy frontier.** Unlike cross-subject/cross-environment generalization
(which is data-bound — see [ADR-150 §3.2](../adr/ADR-150-rf-foundation-encoder.md)), the efficiency
frontier responded immediately to optimization. This is the lever that's still open.
## Method & reproduction
Same architecture family as the flagship — input `[3,114,10]` CSI amplitude → linear projection →
`L`-layer / `H`-head Transformer encoder over the 10 temporal tokens → **temporal attention
pooling** → MLP head → **skeleton-graph refinement** (COCO bone topology) — with width `d`, depth
`L`, heads `H` swept. Training: mixup (Beta(0.2,0.2)), 4-view test-time augmentation, EMA, cosine LR.
| Model | d | L | H | graph head |
|-------|--:|--:|--:|:----------:|
| nano | 48 | 1 | 2 | — |
| micro | 64 | 1 | 2 | ✓ |
| tiny | 96 | 2 | 4 | ✓ |
| small | 128 | 2 | 4 | ✓ |
| base | 160 | 3 | 4 | ✓ |
Reproduce: `python aether-arena/staging/train_efficiency_pareto.py npy/X.npy npy/Y.npy npy/split_random.npy`
(MM-Fi parsed via `aether-arena/staging/parse_mmfi_zips.py`). Latency is mean of 200 batch-1 forward
passes after 10 warmups on an RTX 5080; expect different absolute numbers on edge hardware but the
same param/accuracy ordering.
> **Controlled claim.** In-domain `random_split` (the dataset's documented default) — the same
> protocol on which MultiFormer reports 72.25%. Random split has temporal/subject-adjacency effects
> common to this benchmark family; it is in-domain accuracy, not solved cross-subject/-environment
> generalization (those remain ~65% / ~17% — the honest frontier, tracked in ADR-150).