mirror of https://github.com/ruvnet/RuView synced 2026-07-20 17:03:24 +00:00

Files

T

rUv b209b8b778 ci(bench): compile-verify regression gate for v2 criterion benches + ADR-174 (#1094 )

* ci(bench): wire v2 criterion benches into CI as a compile-verify regression gate

Sub-deliverable 8.3 of the benchmark/optimization milestone (needs ADR slot 174).

The v2/ workspace ships 26 criterion benches across 18 crates, but benches are
not part of `cargo test`, so nothing in CI compiled them and they silently rot
when a public API they call changes.

Add `.github/workflows/bench-regression.yml`:
  - bench-compile (HARD GATE): `cargo bench --workspace --no-default-features
    --no-run` compiles + links every default-feature bench (no measurement) plus
    the cir-gated cir_bench — a real, deterministic regression guard against
    bench bit-rot.
  - bench-fast-run (INFORMATIONAL, continue-on-error, never gates): runs a
    curated pure-CPU subset (nvsim, ruvector sketch/fusion) in criterion
    quick-mode and uploads logs as an artifact.

No timing-regression gate, by design: wall-clock on shared GitHub runners varies
2-3x run-to-run, so a hard threshold or cross-runner `criterion --baseline`
compare would manufacture false failures. The honest scope is compile-verify +
informational-run; the workflow header documents the self-hosted-runner
condition under which true timing-gating becomes honest. The crv-gated crv_bench
is excluded because its crates.io dep ruvector-crv 0.1.1 fails to build upstream.

Running the gate immediately caught one already-bit-rotted bench:
wifi-densepose-mat/detection_bench failed to compile (E0063: missing field
last_rssi in SensorPosition). Fixed (last_rssi: None) and re-verified.

Validation (MEASURED): mat detection_bench + cir_bench + nvsim + ruvector +
vitals + swarm benches compile under --no-default-features; fast subset runs;
`cargo test -p wifi-densepose-mat --no-default-features` 174 passed / 0 failed;
Python proof PASS, hash f8e76f21...46f7a unchanged.

Co-Authored-By: claude-flow <ruv@ruv.net>

* docs(adr): ADR-174 — CI bench-regression compile-verify gate

Records sub-deliverable 8.3 (bench-regression.yml, committed c4c59e085):
a hard compile-verify gate over all 26 v2 criterion benches (caught + fixed
one real bit-rotted bench, mat/detection_bench E0063) + an informational
fast-run. Documents the honest scope — no timing-regression gate, since
shared-runner wall-clock varies 2-3x; states the self-hosted-runner condition
under which timing gating becomes honest.

Co-Authored-By: claude-flow <ruv@ruv.net>

2026-06-15 08:26:38 -04:00

5.7 KiB

Raw Blame History

ADR-174: CI Bench-Regression Gate (Compile-Verify)

Field	Value
Status	Accepted — implemented, caught one real bit-rotted bench
Date	2026-06-15
Deciders	ruv
Codename	BENCH-GATE
Milestone	benchmark/optimization re-balance — sub-deliverable 8.3
Motivated by	`docs/research/sota-nn-train-benchmark-brief.md` (target 3: criterion benches as CI regression baselines)

Context

The v2/ workspace ships 26 criterion benches across 18 crates (e.g. nvsim/pipeline_throughput, wifi-densepose-ruvector/{ann,sketch,fusion}_bench, wifi-densepose-signal/{signal,dsp_perf,features,calibration,cir,…}_bench, wifi-densepose-mat/detection_bench, wifi-densepose-nn/{inference,native_conv}_bench, wifi-densepose-engine/engine_cycle, …). Because benches are not part of cargo test, nothing in CI compiled them — so they bit-rot silently the moment a public API they call changes, and the rot is invisible until someone manually runs cargo bench months later.

The SOTA brief named "wire existing criterion benches into CI as regression baselines" as a concrete benchmark-hygiene target. The honest difficulty: true timing-regression gating on shared GitHub runners is unreliable — wall-clock varies 2–3× run-to-run (a captured 10-sample run showed float_l2/512 ranging 307–444 ns), so a hard threshold or a cross-runner criterion --baseline compare (baseline and PR land on different physical machines) would manufacture false regressions. A gate that cries wolf gets disabled.

Decision

Add .github/workflows/bench-regression.yml with two jobs of explicitly different authority — and do NOT pretend to gate on timing.

`bench-compile` — HARD GATE (real regression detection)

cargo bench --workspace --no-default-features --no-run compiles + links every default-feature bench (no measurement → fully deterministic), plus a --features cir compile of the gated cir_bench. Benches aren't in cargo test, so this is the genuine guard: the build fails the moment a bench stops compiling.

`bench-fast-run` — INFORMATIONAL (`continue-on-error: true`, never gates)

Runs a curated pure-CPU subset (nvsim/pipeline_throughput, ruvector/{sketch,fusion}_bench) in criterion quick-mode (1 s warm-up / 2 s measure / 10 samples), targeted per---bench, and uploads logs as an artifact. Every number it produces is informational only — explicitly stated in the workflow header.

What is NOT done, and why (honest scope)

No timing-regression gate, no committed baseline JSON. The workflow header documents the exact condition under which true timing-gating becomes honest: a frequency-pinned self-hosted runner with a generous (>2×) floor. A cross-runner baseline would be dishonest, so none is committed.

Proof it matters (MEASURED)

Running the new gate on the current tree immediately caught wifi-densepose-mat/detection_bench failing to compile: error[E0063]: missing field last_rssi in initializer of SensorPosition — the struct gained a field; the bench was never updated. Fixed in the same change (last_rssi: None, the simulated-zone convention) and re-verified (cargo bench -p wifi-densepose-mat --no-default-features --bench detection_bench --no-run → Finished). The gate paid for itself on its first run.

Exclusions (documented in-workflow)

ruvector/crv_bench — its crates.io dep ruvector-crv 0.1.1 fails to build on stable (upstream E0308 in stage_iii.rs); excluded with a re-add condition.
onnx_bench / mqtt_throughput — feature-gated (ort / mqtt), left to their crates' own workflows. wasm-edge/process_frame_bench — workspace-excluded.

Conventions mirror existing workflows: submodules: recursive (the workspace path-deps vendor/rufield), Swatinem/rust-cache workspaces: v2, Tauri/GTK apt deps (a --workspace bench link pulls the whole graph), path-filtered triggers.

Validation

Bit-rot caught + fixed (above), re-verified --no-run.
MEASURED locally (--no-default-features, Windows): nvsim, ruvector (sketch/fusion/ann), signal/cir_bench, mat/detection_bench (post-fix), vitals, ruview-swarm/swarm_bench all compile; fast subset runs (nvsim pipeline_run/d1/256 ≈ 55 µs; ruvector sketch_hamming ≈ 3–7 ns vs float_l2 ≈ 63–371 ns).
cargo test -p wifi-densepose-mat --no-default-features → 166/6/2 passed, 0 failed.
python archive/v1/data/proof/verify.py → VERDICT: PASS, hash f8e76f21…46f7a unchanged.
Honest limitation: the full --workspace --no-run could not be end-to-end validated on this Windows box (desktop needs GTK, candle-core fails on MSVC, swarm_bench LTO-links OOM under parallel pressure — all Windows-env artifacts; each affected bench compiles standalone here). The first green Linux CI run on the PR is the authoritative proof of the --workspace step.

Consequences

Positive

Bench bit-rot is now a hard CI failure, not a silent surprise — the 26 benches stay compilable as the APIs they exercise evolve.
The benchmark-infrastructure half of the DoD (step 5) is satisfied honestly, setting up the next sub-deliverable (QAT-int8 measurement) to be regression-protected.

Negative / Neutral

No automated timing-regression detection (deliberate — see scope). Revisit only with a frequency-pinned self-hosted runner.
One bench (crv_bench) excluded pending an upstream dep fix.

5.7 KiB Raw Blame History Unescape Escape