mirror of
https://github.com/ruvnet/RuView
synced 2026-08-09 20:21:43 +00:00
d32a30a5e2ee53a0bae512312811a98c379de4d2
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dragan Spiridonov
|
788d685b9f |
fix(python,training): validate PyO3 inputs; make the single-training-job guard atomic
Closes the two findings from the adversarial review that were verified but left unfixed. Both now have proper tests, each proven to fail against the old code. 1. PyO3 bindings panicked / over-allocated on caller input (aether.rs). `EmbeddingExtractor(n_heads=0)` reached `d_model % n_heads` in the transformer and panicked (surfacing to Python as an opaque PanicException); a non-divisor head count tripped the native assert; and `AetherConfig(d_model=100_000)` allocated multi-gigabyte weight matrices that abort the interpreter. The binding did no validation. Now both constructors return `PyResult` and validate at the boundary — positive dims, `d_model % n_heads == 0`, and a generous MAX_DIM/MAX_LAYERS cap — raising `ValueError`. Proven: `n_heads=0` -> "n_heads must be positive", `d_model=100_000` -> a clean ValueError, both previously a PanicException / abort. +10 pytest cases (test_aether.py); a valid config still constructs and embeds. 2. The single-training-job guard was a TOCTOU race (training_api.rs). `spawn_training_job` checked `is_active()` under a `state` READ lock, released it, then set `active` later. A tokio RwLock read lock is SHARED, so two concurrent `POST /train/start` could both hold it, both see the slot free, and both spawn jobs — sharing/overwriting one status+cancel and orphaning a task handle. Extracted `claim_training_slot`, which does the check-and-set in ONE `status` mutex scope — the atomicity lives on the status mutex, not the coarse state lock — so concurrent starts serialise and exactly one wins. This also makes it unit-testable without a full AppState. Test: 32 threads hit a barrier and race to claim; asserts EXACTLY ONE wins. Mutation-proven — reverting to the split check-then-set makes it fail (`left: 3, right: 1`), and it returns to 1 with the fix. Verified on aarch64/macOS: training_api 28 pass (26 existing + 2 new), full python/tests suite 237 pass (227 + 10). The native module keeps its internal assert as a defence-in-depth invariant; the binding now enforces it at the edge. Co-Authored-By: Ruflo & AQE |
||
|
ruv
|
8409f434c9 |
fix(adr-186): make exported .rvf filenames process-unique (fixes CI flake)
Root cause: the trained-model filename was `trained-{type}-{%Y%m%d_%H%M%S}` — a
**second-resolution** timestamp. Two training runs of the same type that finish
in the same wall-clock second compute the SAME path and silently overwrite each
other's artifact. This is a real robustness gap (a second rapid run clobbers the
first model), and it surfaced as a flaky test on the Linux CI runner:
`http_train_start_produces_model_and_streams` asserts (via a before/after
directory diff) that a new `.rvf` appeared, but the concurrent
`training_job_streams_real_progress_and_writes_model` test — same `supervised`
type, same second — wrote to the identical path and then deleted it in its own
cleanup, so the diff saw no new file. Windows scheduling happened to avoid the
same-second overlap; Linux CI hit it (the sibling `TRAIN_ENV_LOCK` holders then
failed via std::sync::Mutex poison-cascade after the first panic).
Reproduced locally by isolating the three model-writing tests (which forces the
overlap): ~10/20 runs flaked before; 0/30 after.
Fix: `next_model_id()` appends microseconds (`%6f`) plus a monotonic per-process
`AtomicU64` counter, so every run gets a distinct path even for same-microsecond
concurrent completions. Pinned by `model_ids_are_unique_per_call` (1000 ids, all
distinct) so the scheme can't regress to non-unique. No test-level band-aid
(no added sleeps / serialization) — the collision is removed at the source.
Verified: cargo test -p wifi-densepose-sensing-server -p wifi-densepose-train
--no-default-features — bin 218 passed / 0 failed, all suites 0 failed; the
isolated-tests stress loop is 0/15 flakes post-fix.
|
||
|
ruv
|
65648fe4c2 |
feat(adr-186): P5/P6 + acceptance verification — dashboard honesty, HTTP tests, Accepted
Completes ADR-186 phases P5 (dashboard honesty / fallback guarantee) and P6
(tests + witness) on top of the P1–P4 reconnection, and flips the ADR to Accepted.
P5 — dashboard honesty:
- Backend (training_api): a runtime enablement gate (`RUVIEW_DISABLE_SERVER_TRAINING`)
makes the three start endpoints return a structured `{enabled:false, reason,
cli:"wifi-densepose train-room"}` HTTP 409 (never a silent success) when server
training is disabled; `/api/v1/train/status` now carries an `enabled` flag. A runtime
flag (not a Cargo feature) so `--no-default-features` builds keep training ON (resolves
§9.4). Handlers return `Response` to carry the 409.
- Frontend (TrainingPanel.js): reads `enabled` off the status payload and disables the
Start/Pretrain/LoRA buttons with a CLI tooltip when server training is off; a rejected
start tears down the optimistically-opened WS and refreshes. The enabled path was
already fully wired (opens the WS before POST, renders live epoch/loss/ETA + terminal
state) — verified.
P6 — tests & witness:
- New HTTP-level tests in a `#[cfg(test)] mod adr186_http_tests` built on a minimal
`AppStateInner::minimal()` test-state helper: a live-socket test that completes a
genuine 101 WebSocket handshake (tokio-tungstenite) and receives a real progress frame
after a POST start; a 426-not-404 route-wired check; a full POST→poll-status→`.rvf`-exists
round-trip; and the disabled-409 structured-response test. Added `tokio-tungstenite`
dev-dep (version already in the workspace lock).
- CHANGELOG updated; ADR §6 ledger + §7 acceptance criteria checked off with evidence;
Status → Accepted. README/CLAUDE have no training route table (no edit needed).
Fixed a test-only parallelism race found under `cargo test --workspace`: two
model-writing tests deleted `.rvf`s by directory-diff and could remove a file a third
test asserted existed. Each test now cleans only its own artifact (data/models is
gitignored).
Verified this session: `cargo test -p wifi-densepose-sensing-server -p wifi-densepose-train
--no-default-features` — sensing-server bin 217 passed / 0 failed, all train suites 0
failed. Full `--workspace` re-run in progress to reconfirm untouched crates.
|
||
|
ruv
|
569cd237fb |
feat(adr-186): reconnect orphaned in-server trainer + real /ws/train/progress
The training pipeline in `training_api.rs` was written and committed but never declared as a module (no `mod training_api;`), so it was dead, uncompiled code. The live `/api/v1/train/start` was a stub that flipped a status string and logged one line without starting any job, and `/ws/train/progress` 404'd (issue #1233). This wires the real trainer into the live server. - main.rs: declare `mod training_api;` (+ `mod path_safety;` for its load guard), replace the `training_status`/`training_config` stub fields on AppStateInner with `training_state: training_api::TrainingState` + `training_progress_tx`, delete the three stub handlers, and `.merge(training_api::routes())` before `.with_state` so `/api/v1/train/*` (bearer-gated) and `/ws/train/progress` resolve against the shared state. - training_api.rs: decouple the training core (`run_training_job`) from the ~60-field AppStateInner via a shared status handle (`Arc<Mutex<TrainingStatus>>`) + cooperative cancel flag (`Arc<AtomicBool>`) + an owned frame_history snapshot, making it unit-testable. Self-contained input schema (local `RecordedFrame`/`RECORDINGS_DIR`) instead of coupling to the still-orphaned `recording.rs`. Consolidate the single-job guard + spawn into `spawn_training_job`. - Tests: real end-to-end test (synthetic CSI dataset -> real progress frames stream + an actual `.rvf` artifact is written), plus path-traversal-rejection and cancellation tests. Verified: cargo test -p wifi-densepose-sensing-server -p wifi-densepose-train --no-default-features -> 0 failed. P5 (feature-flagged disabled-build honesty) deferred as follow-up; the default/enabled build no longer silently no-ops. |
||
|
rUv
|
9b07dff298 |
feat(beyond-sota): ADR-155 metric unification + ADR-156 RaBitQ Pass-2 (honest negative + latent topk bugfix) (#1053)
* refactor(train): hoist canonical PCK/OKS to un-gated metrics_core; fold test_metrics onto production (ADR-155 M1 §8) ADR-155 §8 deferred item: test_metrics.rs reference kernels validated production against their OWN reimplementation — a test that cannot catch a canonical-impl bug (both could be wrong the same way). - Extract canonical_torso_size / pck_canonical / oks_canonical / sigmas / bounding_box_diagonal into a new NON-tch-gated `metrics_core` module, so the single metric definition is reachable under `cargo test --no-default-features` (the `metrics` module is tch-gated). `metrics` re-exports every item → still exactly ONE implementation. - Rewrite tests/test_metrics.rs to assert the PRODUCTION pck_canonical / oks_canonical equal hand-computed fixtures (not a reimplementation): canonical_pck_matches_hand_computed_fixture (corr=3/total=4/pck=0.75), hip↔hip normalizer pin, zero-visible⇒0.0, OKS perfect⇒1.0, fake-Gold pin. - Keep an INDEPENDENT raw-threshold reference kernel only as a differential cross-check: test_kernel_agrees_with_canonical asserts it AGREES with canonical where torso==1.0 (genuine cross-check, not duplication). Grade: MEASURED. test_metrics 10→12 tests, 0 failed. Co-Authored-By: claude-flow <ruv@ruv.net> * fix(sensing-server): relabel divergent live PCK/OKS so they're never conflated with canonical (ADR-155 M1 §2.1/§8 Goal C) Goal C named training_api.rs:804 (torso-HEIGHT PCK). Auditing it surfaced TWO findings the ADR-155 §1 table missed: 1. training_api.rs is an ORPHAN file — not declared `mod` in lib.rs OR main.rs, so it does NOT compile into the crate. It does not drive the live server. 2. The REAL live `best_pck`/`best_oks` (main.rs training path → RVF metadata JSON read by model_manager.rs) come from trainer.rs: - `pck_at_threshold` = RAW-threshold PCK, NO torso normalization (the most divergent kind), printed/serialized as bare "PCK@0.2". - `oks_map` calls `oks_single(area=1.0)` = the EXACT fake-Gold pattern ADR-155 §2.1 claimed closed elsewhere — still live here, inflating best_oks. Resolution = RELABEL (torso/raw math is load-bearing on different data; the pub fns can't be renamed without breaking API; sensing-server has no train/ ndarray dep). Honest unify is a tracked §8 backlog item. - training_api.rs: `compute_pck` → `compute_pck_torso_height` + divergence doc; val_pck/best_pck/val_oks struct fields documented as torso-HEIGHT proxies; logs say `pck_torso_h@0.2`. Test torso_pck_is_labelled_distinctly_from_canonical. - trainer.rs (LIVE): `pck_at_threshold` documented raw-unnormalized; `oks_map` area=1.0 flagged fake-Gold; test pck_at_threshold_is_raw_unnormalized_not_canonical. - main.rs: live print relabelled `pck_raw@0.2` / `oks_map(area=1.0 proxy)`. No wire-format field renames (back-compat); no pub-API rename (no silent break). Grade: MEASURED (relabel + divergence pinned). sensing-server 450→451 lib tests, 0 failed. Co-Authored-By: claude-flow <ruv@ruv.net> * docs(adr-155): mark §8 metric items RESOLVED + audit map + honest §1 under-count correction (M1b Goals A/D) - §8.1: full PCK/OKS audit map (every def: file:line, basis, canonical/ legacy/distinct), the two §8 items marked RESOLVED with resolution+why. - Honest finding: §1's "seven divergent metrics" was an UNDER-count — sensing-server's LIVE trainer.rs has a raw-unnormalized PCK and an area=1.0 fake-Gold OKS the table omitted, and the file §8 named (training_api.rs) is orphaned dead code. §9 honest-limits updated. - Goal D: metrics.rs *_v2 variants confirmed caller-less + deprecated; noted for future cleanup, NOT deleted (public API, tch-gated). - CHANGELOG [Unreleased] Fixed entry. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(ruvector): RaBitQ Pass-2 randomized rotation + topk bugfix (ADR-156 §8) Implements the deferred "Multi-bit / Extended RaBitQ Pass 2" backlog item from ADR-156 §8: a deterministic randomized orthogonal rotation applied before sign-quantization, the published RaBitQ construction (Gao & Long, SIGMOD 2024). Rotation construction: Fast Hadamard Transform + seeded ±1 sign flips ("HD" / randomized Hadamard), O(d log d) time and O(d) memory — a dense d×d rotation is O(d²) and infeasible at the 65,535-d the wire format provisions for. Pads to the next power of two; SplitMix64 seeds the sign stream so index-time and query-time rotations are bit-identical. API is additive and backward-compatible: Pass 1 (`from_embedding`) is untouched; Pass 2 is opt-in via `Sketch::from_embedding_rotated` and `SketchBank::with_rotation` (+ `insert_embedding` / `topk_embedding` / `novelty_embedding` helpers that rotate consistently). Default behaviour is unchanged. While building the Pass-2 coverage harness, found and fixed a PRE-EXISTING correctness bug in `SketchBank::topk`: the n>k heap path used `BinaryHeap<Reverse<(d,id)>>` (a min-heap) but treated its peek as the max, so it returned the k FARTHEST sketches as "nearest". The shipped unit tests only exercised the n≤k fast path, so it went unnoticed. Fixed to a plain max-heap; pinned by `topk_heap_path_returns_nearest` and `tight_clusters_give_high_coverage_with_overfetch` (the latter measured 0.072 on the old code). New tests (+17, 100→117 in the crate): rotation determinism/norm-preservation (`rotation_is_deterministic_for_seed`, `rotation_preserves_norm`), Pass-2 shape-compatibility, `pass2_coverage_not_worse_than_pass1`, and a deterministic coverage report. MEASURED top-K coverage (anisotropic planted-cluster fixture, cosine ground truth; dim=128 N=2048 K=8 64 clusters noise=0.35 128 queries): candidate_k=K=8 : Pass1 36.13% -> Pass2 46.39% (both << 90% bar) candidate_k=24 : Pass1 83.89% -> Pass2 91.60% (Pass2 clears 90%) candidate_k=32 : Pass1/Pass2 100% Honest result: rotation consistently helps (+10pp at strict K), but neither pass clears the ADR-084 90% bar at candidate_k==K on this distribution. Pass 2 reaches 90% only with ~3x over-fetch (the ADR-084 "candidate set" deployment pattern). Multi-bit Pass 3 evaluated separately. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(ruvector): multi-bit Pass-3 experiment + ADR-156/084 measured results Adds the multi-bit half of the ADR-156 §8 "Multi-bit / Extended RaBitQ" item as a MEASURED experiment (coverage::measure_multibit): rotate, then b-bit uniform scalar-quantize each coord, rank by L1 over codes — the natural multi-bit generalization of hamming. Measures the bit/coverage tradeoff the backlog item asked for. MEASURED at the strict bar (candidate_k=K=8, anisotropic planted-cluster fixture, cosine ground truth): Pass1 (1-bit, no rot) 36.13% 16 B/vec Pass2 (1-bit, rot) 46.39% 16 B/vec Pass3 (rot, 2-bit) 54.39% 32 B/vec Pass3 (rot, 3-bit) 66.70% 48 B/vec Pass3 (rot, 4-bit) 74.22% 64 B/vec Honest: multi-bit monotonically helps but even 4-bit (4x memory) reaches only 74% at the strict bar — neither rotation nor <=4-bit multi-bit clears the strict-K 90% bar on this distribution. The bar is met via over-fetch (Pass2 @ candidate_k=24). Tests: multibit_tradeoff_report, multibit_1bit_matches_pass2_approx (+ sanity that 1-bit ~= Pass-2). Docs: - ADR-156 §8 item #2 marked RESOLVED-PARTIAL; §5 #2 grade CLAIMED -> MEASURED-on-our-hardware; new §10 with full measured tables, the topk bugfix disclosure, and graded deferred sub-items. - ADR-084: "Pass 2" section answering the rotation open-question with measured numbers + the topk bug note. - CHANGELOG [Unreleased]: Added (Pass-2 milestone) + Fixed (topk heap). Co-Authored-By: claude-flow <ruv@ruv.net> |
||
|
ruv
|
50b657459f |
fix(train): unify 7 divergent PCK/OKS into one canonical metric (ADR-155 §Tier-1.1)
Collapse the four PCK and three OKS implementations into a single source of
truth — pck_canonical (torso hip↔hip, COCO/ADR-152 convention validated at
~96% PCK@20 in benchmarks/wiflow-std) and oks_canonical (scale from GT pose
extent). MetricsAccumulator, compute_pck/_per_joint/_oks, aggregate_metrics and
the deprecated *_v2 path all route through them, so Trainer::evaluate() and the
bench definition agree.
Fixes two claim-inflating bugs, each pinned by a regression test:
- zero-visible-joint PCK was 1.0 (false-perfect) -> now 0.0
- OKS s=1.0 on normalized coords made OKS~=1.0 for any pose ("fake Gold tier")
-> scale now derived from the pose; a 3x-torso-wrong pose yields OKS<0.2
Divergent local kernels (training_bench raw-threshold, sensing-server
torso-height) annotated "DO NOT USE for reported metrics". Legitimately changed
test expectations (all-coincident "perfect" fixtures are correctly unscoreable;
all-invisible -> 0.0) updated with comments citing the finding.
Co-Authored-By: claude-flow <ruv@ruv.net>
|
||
|
rUv
|
72bbd256e7 |
fix(security): path-traversal guard on 5 sensing-server endpoints (closes #615) (#616)
Reported by @bannned-bit. Five endpoints in
v2/crates/wifi-densepose-sensing-server embedded user-controlled
identifiers in format!() paths with no sanitization:
recording.rs POST /api/v1/recording/start (session_name)
recording.rs GET /api/v1/recording/download/:id (id)
recording.rs DELETE /api/v1/recording/delete/:id (id)
model_manager.rs POST /api/v1/models/load (model_id)
training_api.rs load_recording_frames (dataset_ids[])
Each unauthenticated caller could:
- READ arbitrary files via ../../etc/passwd, ../../.env, etc.
- WRITE attacker-controlled JSONL via recording/start
- LOAD attacker-controlled .rvf model files
- DELETE arbitrary files the server process can touch
New `path_safety` module exports `safe_id(&str) -> Result<&str, PathSafetyError>`
that enforces the rejection envelope BEFORE any user input reaches a
format!() that builds a path:
- Allowed character set: [A-Za-z0-9._-]
- Reject leading '.' (rules out '.', '..', '.env', hidden files)
- Reject empty strings
- Reject anything > 64 bytes
- Reject all whitespace, path separators, null bytes, non-ASCII
Applied at all 5 sites. Errors return 400 Bad Request (download) /
status:"error" JSON (others) — not panics.
9 unit tests in path_safety::tests cover:
- accepts simple alphanumeric / hyphen / underscore / dot
- rejects empty, leading dot, path separators ('/', '\'),
null byte, whitespace, shell specials, non-ASCII (including
fullwidth slash U+FF0F), too-long, boundary at MAX_ID_LEN
test result: ok. 9 passed; 0 failed
cargo build -p wifi-densepose-sensing-server --no-default-features: 33s
Fix-marker RuView#615 in scripts/fix-markers.json prevents removing the
guard at any of the 5 call sites. CHANGELOG entry under [Unreleased] /
Security documents the patched endpoints and the rejection envelope.
Severity: critical per reporter — five remotely-reachable paths to read,
write, or delete arbitrary files. Hot per-request paths, not edge cases.
|
||
|
rUv
|
f49c722764 |
chore(repo): rename rust-port/wifi-densepose-rs → v2/ (flatten to one level) (#427)
The Rust port lived two directories deep (rust-port/wifi-densepose-rs/) without any sibling under rust-port/ that warranted the extra level. Move the whole workspace up to v2/ to match v1/ (Python) at the same depth and shorten every cd / build command across the repo. git mv preserves history for all tracked files. 60 files updated for path references (CI workflows, ADRs, docs, scripts, READMEs, internal .claude-flow state). Two manual fixes for relative-cd paths in CLAUDE.md and ADR-043 that became wrong after the depth change (cd ../.. → cd ..). Validated: - cargo check --workspace --no-default-features → clean (after target/ nuke; the gitignored target/ was carried by the OS rename and had hard-coded old paths in build scripts) - cargo test --workspace --no-default-features → 1,539 passed, 0 failed, 8 ignored (same totals as pre-rename) - ESP32-S3 on COM7 → still streaming live CSI (cb #40300, RSSI -64 dBm) After-merge follow-up: contributors should `rm -rf v2/target` once and let cargo regenerate from the new path. |