frank/ruvnet--RuView
1
0
Fork 0
mirror of https://github.com/ruvnet/RuView synced 2026-07-20 17:03:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(train): falsifiable occupancy benchmark with anti-overfitting gate

Browse Source 
Makes the presence/person-count "beyond SOTA" claim falsifiable in code
instead of aspirational (the unfalsifiability gap from the beyond-SOTA system
review). occupancy_bench grades predictions vs ground truth and gates a SOTA
claim behind one claim_allowed invariant requiring ALL of:

- DataProvenance::Measured — synthetic/mock data is scorable for regression
  but never claimable (anti-mock-contamination; the CLAUDE.md Kconfig-bug
  lesson made structural).
- A leak-free EvalSplit — validate() refuses any split where a subject OR
  environment id appears in both train and test (subject leakage /
  per-environment overfitting).
- n_test >= min_test_samples (small-N guard).
- Presence F1 whose bootstrap-CI lower bound (deterministic seeded splitmix64)
  clears the threshold — not the point estimate.
- Count MAE within threshold.

The claim string is unreadable except through the gate (NO_CLAIM otherwise),
same discipline as the ruview-gamma acceptance gate. What remains is data, not
method: a frozen, SHA-pinned, subject/environment-disjoint measured replay set
turns the claim into a passing/failing test.

Lives in wifi-densepose-train (the eval bounded context, alongside ablation/
eval/metrics). 10 tests cover each refusal path; warning-clean under the
crate's missing_docs lint. Workspace gate 2,914 passed / 0 failed. Doc 03
updated.

https://claude.ai/code/session_01MjBucx95K4BuUxZi8NWwRH
This commit is contained in:
Claude
2026-06-10 20:43:42 +00:00
parent a8291d2535
commit d925538773
4 changed files with 577 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/ruview-beyond-sota/03-benchmark-validation-methodology.md
+27
View File
@@ -344,6 +344,33 @@ Anyone outside the project must be able to re-run every claimed result:
---
## Update — falsifiable occupancy benchmark implemented
`wifi-densepose-train::occupancy_bench` (added this branch) makes the
presence/person-count claim **falsifiable in code**, directly enforcing the L3
discipline above. It grades predictions vs ground truth and gates a SOTA claim
behind a single `claim_allowed` invariant that requires **all** of:
1. `DataProvenance::Measured` — synthetic/mock data is scorable for regression
but **never claimable** (anti-mock-contamination; the CLAUDE.md Kconfig-bug
lesson made structural).
2. A leak-free `EvalSplit` — `validate()` refuses any split where a subject *or*
environment id appears in both train and test (subject leakage / per-env
overfitting).
3. `n_test ≥ min_test_samples` (small-N guard).
4. Presence F1 whose **bootstrap-CI lower bound** (deterministic splitmix64,
seeded) clears the threshold — not the point estimate.
5. Count MAE within threshold.
The claim string is unreadable except through the gate (returns `NO_CLAIM`
otherwise) — same discipline as the `ruview-gamma` acceptance gate. 10 tests
cover each refusal path. What remains is *data*, not *method*: feed it a frozen,
SHA-pinned, subject/environment-disjoint **measured** replay set (the curated
room-A/room-B item above) and the "beyond SOTA" claim becomes a passing or
failing test, not a slogan.
---
*All values cited from: `benchmark_baseline.json`, `v2/crates/*/benches/*.rs` (15
files), `docs/adr/ADR-147-benchmark-proof.md`,
`docs/adr/ADR-149-swarm-benchmarking-evaluation-methodology.md`,