mirror of
https://github.com/ruvnet/RuView
synced 2026-07-21 17:13:19 +00:00
rUv
b209b8b778
* ci(bench): wire v2 criterion benches into CI as a compile-verify regression gate
Sub-deliverable 8.3 of the benchmark/optimization milestone (needs ADR slot 174).
The v2/ workspace ships 26 criterion benches across 18 crates, but benches are
not part of `cargo test`, so nothing in CI compiled them and they silently rot
when a public API they call changes.
Add `.github/workflows/bench-regression.yml`:
- bench-compile (HARD GATE): `cargo bench --workspace --no-default-features
--no-run` compiles + links every default-feature bench (no measurement) plus
the cir-gated cir_bench — a real, deterministic regression guard against
bench bit-rot.
- bench-fast-run (INFORMATIONAL, continue-on-error, never gates): runs a
curated pure-CPU subset (nvsim, ruvector sketch/fusion) in criterion
quick-mode and uploads logs as an artifact.
No timing-regression gate, by design: wall-clock on shared GitHub runners varies
2-3x run-to-run, so a hard threshold or cross-runner `criterion --baseline`
compare would manufacture false failures. The honest scope is compile-verify +
informational-run; the workflow header documents the self-hosted-runner
condition under which true timing-gating becomes honest. The crv-gated crv_bench
is excluded because its crates.io dep ruvector-crv 0.1.1 fails to build upstream.
Running the gate immediately caught one already-bit-rotted bench:
wifi-densepose-mat/detection_bench failed to compile (E0063: missing field
last_rssi in SensorPosition). Fixed (last_rssi: None) and re-verified.
Validation (MEASURED): mat detection_bench + cir_bench + nvsim + ruvector +
vitals + swarm benches compile under --no-default-features; fast subset runs;
`cargo test -p wifi-densepose-mat --no-default-features` 174 passed / 0 failed;
Python proof PASS, hash f8e76f21...46f7a unchanged.
Co-Authored-By: claude-flow <ruv@ruv.net>
* docs(adr): ADR-174 — CI bench-regression compile-verify gate
Records sub-deliverable 8.3 (bench-regression.yml, committed c4c59e085):
a hard compile-verify gate over all 26 v2 criterion benches (caught + fixed
one real bit-rotted bench, mat/detection_bench E0063) + an informational
fast-run. Documents the honest scope — no timing-regression gate, since
shared-runner wall-clock varies 2-3x; states the self-hosted-runner condition
under which timing gating becomes honest.
Co-Authored-By: claude-flow <ruv@ruv.net>
111 lines
5.7 KiB
Markdown
111 lines
5.7 KiB
Markdown
# ADR-174: CI Bench-Regression Gate (Compile-Verify)
|
||
|
||
| Field | Value |
|
||
|-------|-------|
|
||
| **Status** | Accepted — implemented, caught one real bit-rotted bench |
|
||
| **Date** | 2026-06-15 |
|
||
| **Deciders** | ruv |
|
||
| **Codename** | **BENCH-GATE** |
|
||
| **Milestone** | benchmark/optimization re-balance — sub-deliverable 8.3 |
|
||
| **Motivated by** | `docs/research/sota-nn-train-benchmark-brief.md` (target 3: criterion benches as CI regression baselines) |
|
||
|
||
## Context
|
||
|
||
The v2/ workspace ships **26 criterion benches across 18 crates** (e.g.
|
||
`nvsim/pipeline_throughput`, `wifi-densepose-ruvector/{ann,sketch,fusion}_bench`,
|
||
`wifi-densepose-signal/{signal,dsp_perf,features,calibration,cir,…}_bench`,
|
||
`wifi-densepose-mat/detection_bench`, `wifi-densepose-nn/{inference,native_conv}_bench`,
|
||
`wifi-densepose-engine/engine_cycle`, …). Because **benches are not part of
|
||
`cargo test`**, nothing in CI compiled them — so they bit-rot silently the moment
|
||
a public API they call changes, and the rot is invisible until someone manually
|
||
runs `cargo bench` months later.
|
||
|
||
The SOTA brief named "wire existing criterion benches into CI as regression
|
||
baselines" as a concrete benchmark-hygiene target. The honest difficulty: true
|
||
*timing*-regression gating on shared GitHub runners is unreliable — wall-clock
|
||
varies 2–3× run-to-run (a captured 10-sample run showed `float_l2/512` ranging
|
||
307–444 ns), so a hard threshold or a cross-runner `criterion --baseline` compare
|
||
(baseline and PR land on different physical machines) would manufacture false
|
||
regressions. A gate that cries wolf gets disabled.
|
||
|
||
## Decision
|
||
|
||
Add `.github/workflows/bench-regression.yml` with **two jobs of explicitly
|
||
different authority** — and do NOT pretend to gate on timing.
|
||
|
||
### `bench-compile` — HARD GATE (real regression detection)
|
||
`cargo bench --workspace --no-default-features --no-run` compiles + links every
|
||
default-feature bench (no measurement → fully deterministic), plus a
|
||
`--features cir` compile of the gated `cir_bench`. Benches aren't in `cargo test`,
|
||
so this is the genuine guard: **the build fails the moment a bench stops
|
||
compiling.**
|
||
|
||
### `bench-fast-run` — INFORMATIONAL (`continue-on-error: true`, never gates)
|
||
Runs a curated pure-CPU subset (`nvsim/pipeline_throughput`,
|
||
`ruvector/{sketch,fusion}_bench`) in criterion quick-mode (1 s warm-up / 2 s
|
||
measure / 10 samples), targeted per-`--bench`, and uploads logs as an artifact.
|
||
Every number it produces is **informational only** — explicitly stated in the
|
||
workflow header.
|
||
|
||
### What is NOT done, and why (honest scope)
|
||
No timing-regression gate, no committed baseline JSON. The workflow header
|
||
documents the exact condition under which true timing-gating becomes honest: a
|
||
frequency-pinned **self-hosted** runner with a generous (>2×) floor. A
|
||
cross-runner baseline would be dishonest, so none is committed.
|
||
|
||
### Proof it matters (MEASURED)
|
||
Running the new gate on the current tree immediately caught
|
||
`wifi-densepose-mat/detection_bench` failing to compile:
|
||
`error[E0063]: missing field last_rssi in initializer of SensorPosition` — the
|
||
struct gained a field; the bench was never updated. **Fixed** in the same change
|
||
(`last_rssi: None`, the simulated-zone convention) and re-verified
|
||
(`cargo bench -p wifi-densepose-mat --no-default-features --bench detection_bench --no-run`
|
||
→ `Finished`). The gate paid for itself on its first run.
|
||
|
||
### Exclusions (documented in-workflow)
|
||
- `ruvector/crv_bench` — its crates.io dep `ruvector-crv 0.1.1` fails to build on
|
||
stable (upstream `E0308` in `stage_iii.rs`); excluded with a re-add condition.
|
||
- `onnx_bench` / `mqtt_throughput` — feature-gated (ort / mqtt), left to their
|
||
crates' own workflows. `wasm-edge/process_frame_bench` — workspace-excluded.
|
||
|
||
Conventions mirror existing workflows: `submodules: recursive` (the workspace
|
||
path-deps `vendor/rufield`), Swatinem/rust-cache `workspaces: v2`, Tauri/GTK apt
|
||
deps (a `--workspace` bench link pulls the whole graph), path-filtered triggers.
|
||
|
||
## Validation
|
||
|
||
- **Bit-rot caught + fixed** (above), re-verified `--no-run`.
|
||
- **MEASURED locally** (`--no-default-features`, Windows): nvsim, ruvector
|
||
(sketch/fusion/ann), signal/cir_bench, mat/detection_bench (post-fix),
|
||
vitals, ruview-swarm/swarm_bench all compile; fast subset runs (`nvsim
|
||
pipeline_run/d1/256` ≈ 55 µs; `ruvector sketch_hamming` ≈ 3–7 ns vs `float_l2`
|
||
≈ 63–371 ns).
|
||
- `cargo test -p wifi-densepose-mat --no-default-features` → 166/6/2 passed, 0 failed.
|
||
- `python archive/v1/data/proof/verify.py` → **VERDICT: PASS**, hash
|
||
`f8e76f21…46f7a` unchanged.
|
||
- **Honest limitation:** the full `--workspace --no-run` could not be
|
||
end-to-end validated on this Windows box (`desktop` needs GTK, `candle-core`
|
||
fails on MSVC, `swarm_bench` LTO-links OOM under parallel pressure — all
|
||
Windows-env artifacts; each affected bench compiles standalone here). **The
|
||
first green Linux CI run on the PR is the authoritative proof of the
|
||
`--workspace` step.**
|
||
|
||
## Consequences
|
||
|
||
### Positive
|
||
- Bench bit-rot is now a hard CI failure, not a silent surprise — the 26 benches
|
||
stay compilable as the APIs they exercise evolve.
|
||
- The benchmark-infrastructure half of the DoD (step 5) is satisfied honestly,
|
||
setting up the next sub-deliverable (QAT-int8 measurement) to be
|
||
regression-protected.
|
||
|
||
### Negative / Neutral
|
||
- No automated timing-regression detection (deliberate — see scope). Revisit only
|
||
with a frequency-pinned self-hosted runner.
|
||
- One bench (`crv_bench`) excluded pending an upstream dep fix.
|
||
|
||
## Links
|
||
- ADR-173 — metric-locked accuracy harness (sub-deliverable 8.1)
|
||
- `docs/research/sota-nn-train-benchmark-brief.md` — motivating target
|
||
- ADR-134 (CIR), ADR-135 (calibration), ADR-154 (signal DSP benches) — benched paths
|