mirror of
https://github.com/ruvnet/RuView
synced 2026-08-04 19:31:42 +00:00
test(wasm-edge): synthetic-ground-truth validation harness for edge skills (ADR-160)
Plant signals with known answers, run the real detector, MEASURE detection accuracy / precision / recall / rate-error — synthetic-ground-truth ONLY, not field accuracy. MEASURED-on-synthetic (12 tests, all green): - vital_trend, exo_ghost_hunter(hidden breathing), occupancy, intrusion, exo_rain_detect, sig_optimal_transport: acc 1.000 - exo_time_crystal: 1.000 on periodic-vs-aperiodic (its sub-harmonic-vs-clean- period claim is NOT separable by autocorrelation — recorded honestly) - sig_flash_attention: 8/8 peak localization; spt_spiking_tracker: 4/4 zone localization (sparse plant); sig_mincut_person_match: 0 id-swaps/40 frames - lrn_dtw_gesture_learn: enrollment validated (replay-match reported, not asserted) - sig_sparse_recovery: trigger validated; recovery accuracy reported NEGATIVE (-2.2% vs unrecovered baseline) — only its detect/trigger path is validated DATA-GATED (listed, NOT faked): med_seizure/apnea/cardiac/respiratory/gait, sec_weapon_detect, exo_emotion/happiness/dream_stage/gesture_language — each needs real labelled clinical/affect/ASL/metal-object data; no number claimed. benchmarks/edge-skills/RESULTS.md documents every result + reproduce command and the explicit honesty boundary. ADR-160 deferred 'per-skill accuracy validation' item updated to PARTIALLY MEASURED-on-synthetic + DATA-GATED. Suite: 631 passed default / 669 medical, 0 failed. Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
@@ -178,10 +178,33 @@ label or behavior change, consistent with leaving their claim surface intact.)
|
||||
|
||||
## Deferred Backlog (Nothing Dropped)
|
||||
|
||||
- **Per-skill accuracy validation** — **DATA-GATED**. Validating any med_*/affect/
|
||||
sign-language claim requires labelled clinical/affective/ASL data and reference
|
||||
standards that do not exist in this repo. The disclaimers + feature gate are the
|
||||
honest stand-in. Nothing is claimed that is not measured.
|
||||
- **Per-skill accuracy validation** — **PARTIALLY MEASURED-on-synthetic**
|
||||
(2026-06-13). For the subset of skills whose detection target is *constructible*
|
||||
with known ground truth, a synthetic-ground-truth harness
|
||||
(`tests/synthetic_validation.rs`, 12 tests) plants signals with known answers,
|
||||
runs the real detector, and **measures** detection accuracy / rate-error:
|
||||
`vital_trend`, `exo_time_crystal` (periodic-vs-aperiodic — its sub-harmonic-vs-
|
||||
clean-period claim is NOT separable, recorded honestly), `exo_ghost_hunter`
|
||||
(hidden breathing), `occupancy`, `intrusion`, `exo_rain_detect`,
|
||||
`sig_flash_attention` (8/8 peak localization), `spt_spiking_tracker` (4/4 zone
|
||||
localization, sparse plant), `sig_optimal_transport`, `sig_mincut_person_match`
|
||||
(0 id-swaps), `lrn_dtw_gesture_learn` (enrollment) — all 1.000 where claimed;
|
||||
`sig_sparse_recovery`'s recovery accuracy is reported **negative** (−2.2% vs
|
||||
unrecovered baseline) — only its trigger path is validated. Full numbers +
|
||||
reproduce commands in `benchmarks/edge-skills/RESULTS.md`.
|
||||
The **med_*/affect/sign-language/weapon** claims remain **DATA-GATED**:
|
||||
validating them requires labelled clinical/affective/ASL/metal-object data and
|
||||
reference standards that do not exist in this repo. Planting a "seizure-/weapon-/
|
||||
happy-like" synthetic signal validates nothing real and is explicitly refused;
|
||||
RESULTS.md lists each with the real data it needs. The disclaimers + feature gate
|
||||
are the honest stand-in. Nothing is claimed that is not measured.
|
||||
- **Unified edge pipeline** — **MEASURED** (2026-06-13). `src/pipeline_all.rs`
|
||||
(`EdgePipeline`) + `src/skill_registry.rs` register **every** runtime skill
|
||||
behind one uniform `EdgeSkill` trait and run them all per CSI frame; `med_*` are
|
||||
registered only under `--features medical-experimental` (preserves the §A1 gate).
|
||||
`tests/pipeline_all.rs` (4 tests) proves all 59 default / 64 medical skills run
|
||||
without panic over 300 synthetic frames with a well-formed aggregated event
|
||||
stream. `examples/run_all_skills.rs` is a runnable demo. No skill DSP changed.
|
||||
- **Criterion benches for `process_frame` budget claims** — **DONE (host)**
|
||||
(ADR-163, 2026-06-12). `benches/process_frame_bench.rs` benches the heaviest
|
||||
hot paths (`exo_time_crystal` 256×128 autocorrelation, `exo_ghost_hunter`
|
||||
|
||||
Reference in New Issue
Block a user