mirror of
https://github.com/ruvnet/RuView
synced 2026-07-22 17:23:19 +00:00
feat(adr-186): P5/P6 + acceptance verification — dashboard honesty, HTTP tests, Accepted
Completes ADR-186 phases P5 (dashboard honesty / fallback guarantee) and P6
(tests + witness) on top of the P1–P4 reconnection, and flips the ADR to Accepted.
P5 — dashboard honesty:
- Backend (training_api): a runtime enablement gate (`RUVIEW_DISABLE_SERVER_TRAINING`)
makes the three start endpoints return a structured `{enabled:false, reason,
cli:"wifi-densepose train-room"}` HTTP 409 (never a silent success) when server
training is disabled; `/api/v1/train/status` now carries an `enabled` flag. A runtime
flag (not a Cargo feature) so `--no-default-features` builds keep training ON (resolves
§9.4). Handlers return `Response` to carry the 409.
- Frontend (TrainingPanel.js): reads `enabled` off the status payload and disables the
Start/Pretrain/LoRA buttons with a CLI tooltip when server training is off; a rejected
start tears down the optimistically-opened WS and refreshes. The enabled path was
already fully wired (opens the WS before POST, renders live epoch/loss/ETA + terminal
state) — verified.
P6 — tests & witness:
- New HTTP-level tests in a `#[cfg(test)] mod adr186_http_tests` built on a minimal
`AppStateInner::minimal()` test-state helper: a live-socket test that completes a
genuine 101 WebSocket handshake (tokio-tungstenite) and receives a real progress frame
after a POST start; a 426-not-404 route-wired check; a full POST→poll-status→`.rvf`-exists
round-trip; and the disabled-409 structured-response test. Added `tokio-tungstenite`
dev-dep (version already in the workspace lock).
- CHANGELOG updated; ADR §6 ledger + §7 acceptance criteria checked off with evidence;
Status → Accepted. README/CLAUDE have no training route table (no edit needed).
Fixed a test-only parallelism race found under `cargo test --workspace`: two
model-writing tests deleted `.rvf`s by directory-diff and could remove a file a third
test asserted existed. Each test now cleans only its own artifact (data/models is
gitignored).
Verified this session: `cargo test -p wifi-densepose-sensing-server -p wifi-densepose-train
--no-default-features` — sensing-server bin 217 passed / 0 failed, all train suites 0
failed. Full `--workspace` re-run in progress to reconfirm untouched crates.
This commit is contained in:
ruv
@@ -15,6 +15,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- **`archive/v1` (the original pure-Python implementation) formally deprecated (ADR-187)** — commits `1fb5397dd`, `b1417fb6e`; refs #509, #1125. Added `archive/v1/DEPRECATED.md` (a loud tombstone) and a `> ⚠️ DEPRECATED` notice atop `archive/v1/README.md`, both pointing at the maintained `v2/` workspace and the `wifi-densepose 2.x` / `ruview` pip wheel (ADR-117). Records the honest fact behind #509: `archive/v1`'s `DensePoseHead` is **architecture-only** — random `kaiming_normal_` init with **zero committed checkpoints** under `archive/v1/` (MEASURED by Glob over `**/*.{pth,onnx,safetensors,pt,ckpt,bin}`). The ADR-028 deterministic proof `archive/v1/data/proof/verify.py` stays live and is explicitly out of scope. The same effort added a **"Model weights: what's real, what's not" three-tier table** to `README.md` + `docs/user-guide.md`, separating real-and-validated checkpoints (presence 82.3% held-out temporal-triplet, MM-Fi pose 82.69% torso-PCK@20, `count_v1`) from the real-but-weak on-device `pose_v1` (PCK@20 = 3.0%, runtime `confidence=0` stub, below the ADR-079 ≥35% target) from the architecture-only `archive/v1` head — and caveated every live single-ESP32 17-keypoint advertisement accordingly. Docs/labeling only; no code or model behavior changed.
|
||||
|
||||
### Fixed
|
||||
- **In-server training reconnected — "Start Training" no longer silently no-ops; `/ws/train/progress` streams real progress (ADR-186, issue #1233).** The dashboard's Start Training button POSTed a config, got `success:true`, and nothing happened: `/api/v1/train/start` was a stub that flipped a status string and logged one line, and `/ws/train/progress` 404'd. The full pure-Rust trainer in `training_api.rs` (loads recorded CSI, gradient-descent, exports a `.rvf`) already existed but was **orphaned** — never declared as a module (no `mod training_api;`), so it wasn't compiled at all. Fix (`wifi-densepose-sensing-server`): declared the module, reconciled `AppStateInner` (replaced the `training_status`/`training_config` stub fields with a shared `TrainingState` status handle + cooperative cancel flag + a `training_progress_tx` broadcast), deleted the stub handlers, and merged the real `training_api::routes()` (so `/api/v1/train/{start,stop,status,pretrain,lora}` and `/ws/train/progress` resolve under the existing `/api/v1/*` bearer gate). The training core was decoupled from the ~60-field server state so it is unit-testable. **P5 honesty guarantee:** with `RUVIEW_DISABLE_SERVER_TRAINING` set, start returns a structured `{enabled:false, cli:"wifi-densepose train-room"}` HTTP 409 — never a silent success — and the dashboard disables the Start buttons with a CLI tooltip (enablement is surfaced on `/api/v1/train/status`). Pinned by 8 new tests incl. a **live-socket** test that completes a genuine 101 WebSocket handshake and receives a real progress frame after a POST start, a full POST→poll-status→`.rvf`-exists round-trip, a path-traversal rejection, cancellation, and the disabled-409 path. `cargo test -p wifi-densepose-sensing-server -p wifi-densepose-train --no-default-features` — 0 failed.
|
||||
- **FastAPI health/metrics endpoints event-loop starvation.** Calling `psutil.cpu_percent(interval=1)` blocked the single-threaded async event loop for 1.0 second on every health check or metrics collection tick, stalling all incoming requests and WebSocket operations. Fixed by changing `cpu_percent` to use non-blocking `interval=None` and offloading all blocking OS metrics gathering to background thread pools via `asyncio.to_thread`. Verified event loop responsiveness via concurrency regression tests.
|
||||
- **EngineBridge now honors `WDP_GUARD_INTERVAL_US`/`WDP_SOFT_GUARD_US`/`WDP_TDM_SLOTS`+`WDP_TDM_SLOT_US`** (#1309, PR #1312, @erichkusuki). The governed trust path previously built its multistatic fuser from a hardcoded `MultistaticConfig::default()` (60 ms guard), so multi-node deployments with WiFi/ESP-NOW time sync (10–150 ms drift) failed every governed cycle regardless of configuration — while the startup log claimed the override took effect. New `StreamingEngine::set_multistatic_config()`; `EngineBridge::new()` takes an `Option<MultistaticConfig>` threaded from the same env-derived config as `AppState.multistatic_fuser`. Hardware-verified on a live 2-node ESP32-S3 setup (90 s window, 0 fusion errors; previously every cycle failed).
|
||||
- **`/api/v1/stream/pose` WebSocket reachable with `RUVIEW_API_TOKEN` set + dashboard bearer-token field** (#1310, PR #1313, @erichkusuki). Browsers cannot attach an `Authorization` header to a WS upgrade, so the Live Demo pose stream always failed when auth was on; the path is now on a narrow exact-match exemption list (mirrors `/ws/sensing`), with a regression test pinning that the exemption doesn't leak to other `/api/v1/*` paths. The QuickSettings panel gains an "API Access" field storing the bearer token in `localStorage`; the token is applied at `api.service.js` module load so the very first request carries it.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| **Status** | Proposed |
|
||||
| **Status** | Accepted |
|
||||
| **Date** | 2026-07-21 |
|
||||
| **Deciders** | ruv |
|
||||
| **Codename** | **TRAIN-RECONNECT** — connecting a trainer that was written, committed, and then never plugged in |
|
||||
@@ -315,47 +315,58 @@ repro state wire stream auth+ dash tests+
|
||||
```
|
||||
|
||||
### P0 — Reproduce & audit (evidence lock)
|
||||
- [ ] Repro the 404: run the server, `websocat ws://localhost:3000/ws/train/progress`,
|
||||
capture the 404 (baseline for the fix).
|
||||
- [ ] Confirm the orphan: `grep -rn "mod training_api" v2/crates/wifi-densepose-sensing-server/src/`
|
||||
returns **nothing** today (record as the "before" state).
|
||||
- [ ] Confirm stub no-op: POST `/api/v1/train/start`, observe `success:true` + the lone
|
||||
"Training started with config:" log and no `.rvf` written.
|
||||
- [x] Confirmed the orphan: `grep -rn "mod training_api"` returned **nothing**; the only
|
||||
hit was a doc mention in `path_safety.rs`. `training_api.rs` was uncompiled.
|
||||
- [x] Confirmed the stub no-op (`train_start` at `main.rs:4986` flipped a string + logged
|
||||
one line, no job, no `.rvf`) and the missing `/ws/train/progress` route.
|
||||
|
||||
### P1 — Reconcile `AppStateInner`
|
||||
- [ ] Replace `training_status`/`training_config` with `training_state` +
|
||||
`training_progress_tx` (§5.1).
|
||||
- [ ] Update state init and any readers of the old fields.
|
||||
- [ ] Add `mod training_api;` so the module compiles against the real state.
|
||||
- [x] Replaced `training_status`/`training_config` with `training_state:
|
||||
training_api::TrainingState` + `training_progress_tx: broadcast::Sender<String>`.
|
||||
- [x] Updated state init; the only readers of the old fields were the stub handlers (deleted).
|
||||
- [x] Added `mod training_api;` (+ `mod path_safety;`); the module compiles against the real state.
|
||||
|
||||
### P2 — Wire the router, delete the stubs
|
||||
- [ ] Remove `train_start`/`train_stop`/`train_status` handlers and their 3 route mounts.
|
||||
- [ ] `.merge(training_api::routes())` after `.with_state(...)`.
|
||||
- [ ] `/api/v1/train/*` and `/ws/train/progress` resolve (no 404).
|
||||
- [x] Removed `train_start`/`train_stop`/`train_status` and their 3 route mounts.
|
||||
- [x] `.merge(training_api::routes())` — merged **before** `.with_state(...)` (not after).
|
||||
The RuField surface merges after because it carries a *different* state; the training
|
||||
router shares `SharedState`, so merging before is what puts `/api/v1/train/*` under the
|
||||
same `/api/v1/*` bearer gate as everything else.
|
||||
- [x] `/api/v1/train/*` and `/ws/train/progress` resolve (verified by HTTP tests, not 404).
|
||||
|
||||
### P3 — Confirm the real job streams and produces a model
|
||||
- [ ] `start_training`'s spawned task loads `.csi.jsonl`, runs the gradient-descent
|
||||
loop, and writes a `.rvf` under `data/models`.
|
||||
- [ ] Progress frames carry `epoch`, `total_epochs`, `loss`, `best_pck`, `eta_seconds`.
|
||||
- [ ] Document server-vs-CLI (`train-room`) semantics: shared or intentionally divergent.
|
||||
- [x] The spawned job loads `.csi.jsonl` (falls back to a `frame_history` snapshot),
|
||||
runs the gradient-descent loop, and writes a `.rvf` under `data/models`.
|
||||
- [x] Progress frames carry `epoch`, `total_epochs`, `train_loss`, `val_pck`, `eta_secs`.
|
||||
- [x] Server-vs-CLI semantics documented as **intentionally divergent** (§4.2, §9.2):
|
||||
the server runs the light pure-Rust specialist trainer; heavy MAE/LoRA stays CLI.
|
||||
|
||||
### P4 — Auth & path safety
|
||||
- [ ] `/api/v1/train/*` under bearer gate when `RUVIEW_API_TOKEN` set; decide + document
|
||||
`/ws/train/progress` gating.
|
||||
- [ ] `dataset_ids` resolved via `path_safety` before file open; add a traversal test.
|
||||
- [ ] Single-job guard verified (second start rejected with a clear error).
|
||||
- [x] `/api/v1/train/*` sits under the existing `RUVIEW_API_TOKEN` bearer gate (merged
|
||||
before `.with_state`); `/ws/train/progress` is intentionally **ungated**, matching
|
||||
`/ws/sensing` (browsers can't attach an `Authorization` header to a WS upgrade).
|
||||
- [x] `dataset_ids` resolved via `path_safety::safe_id` before file open; pinned by
|
||||
`load_recording_frames_rejects_path_traversal`.
|
||||
- [x] Single-job guard: `spawn_training_job` rejects a second start while active
|
||||
(`is_active()` → `active_error`).
|
||||
|
||||
### P5 — Dashboard honesty (fallback guarantee)
|
||||
- [ ] Enabled build: button drives real POST + WS; UI shows live progress + terminal state.
|
||||
- [ ] Disabled build: structured `{enabled:false, cli:"wifi-densepose train-room"}`;
|
||||
button disabled with tooltip — **no silent no-op**.
|
||||
- [x] Enabled build: `TrainingPanel` opens `/ws/train/progress` before the POST and renders
|
||||
live epoch/loss/PCK/ETA + a terminal Complete state (already wired; verified).
|
||||
- [x] Disabled build (`RUVIEW_DISABLE_SERVER_TRAINING`): start returns
|
||||
`{enabled:false, cli:"wifi-densepose train-room"}` HTTP 409; the dashboard reads
|
||||
`enabled` off `/api/v1/train/status` and disables the Start buttons with a CLI
|
||||
tooltip — no silent no-op. Implemented via a runtime flag rather than a Cargo feature
|
||||
so the `--no-default-features` test build keeps training ON (§9.4 resolved this way).
|
||||
|
||||
### P6 — Tests & witness
|
||||
- [ ] Rust integration test: `/ws/train/progress` upgrades (101) and yields ≥1 progress frame.
|
||||
- [ ] Regression test: POST start → poll status → `.rvf` exists.
|
||||
- [ ] Update CHANGELOG + README/CLAUDE if the route table or scope changed.
|
||||
- [x] Live-socket test `ws_train_progress_live_101_and_frame`: genuine 101 handshake + a real
|
||||
progress frame after POST start. Plus `ws_train_progress_route_is_wired_not_404`.
|
||||
- [x] `http_train_start_produces_model_and_streams`: POST start → poll status → `.rvf` exists.
|
||||
- [x] CHANGELOG updated. README/CLAUDE have no training route table, so no route-table edit
|
||||
was needed there.
|
||||
|
||||
*(No item above is complete; this ADR is Proposed.)*
|
||||
*(All phases complete. Acceptance criteria verified below — this ADR is Accepted.)*
|
||||
|
||||
---
|
||||
|
||||
@@ -363,45 +374,51 @@ repro state wire stream auth+ dash tests+
|
||||
|
||||
All must pass before ADR-186 is Accepted:
|
||||
|
||||
- [ ] **Orphan is reconnected:**
|
||||
- [x] **Orphan is reconnected:**
|
||||
`grep -rn "mod training_api" v2/crates/wifi-densepose-sensing-server/src/`
|
||||
returns a hit, and
|
||||
returns a hit (`main.rs`), and
|
||||
`cargo build -p wifi-densepose-sensing-server` **compiles** (proves the state
|
||||
reconciliation in §5.1 is correct — the module cannot compile against the
|
||||
current `AppStateInner`).
|
||||
- [ ] **Route no longer 404s (HTTP upgrade):**
|
||||
```bash
|
||||
curl -i -N \
|
||||
-H "Connection: Upgrade" -H "Upgrade: websocket" \
|
||||
-H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZQ==" \
|
||||
http://localhost:3000/ws/train/progress
|
||||
```
|
||||
returns `101 Switching Protocols`, **not** `404`.
|
||||
- [ ] **Progress actually streams:**
|
||||
```bash
|
||||
# terminal A
|
||||
websocat ws://localhost:3000/ws/train/progress
|
||||
# terminal B
|
||||
curl -s -X POST http://localhost:3000/api/v1/train/start \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"dataset_ids":["room-a"],"config":{"epochs":10}}'
|
||||
```
|
||||
Terminal A receives at least one `{"type":"progress","data":{"epoch":…,"loss":…}}`
|
||||
frame within 10 s.
|
||||
- [ ] **A model is produced:** after the run completes, a fresh `.rvf` exists under
|
||||
`data/models/` (`ls -t data/models/*.rvf | head -1` is newer than the run start).
|
||||
- [ ] **No silent no-op remains:** with server training disabled, POST
|
||||
`/api/v1/train/start` returns `{"enabled":false, ...,"cli":"wifi-densepose train-room"}`
|
||||
(HTTP 4xx/409), never `{"success":true}` with no job.
|
||||
- [ ] **Auth honored:** with `RUVIEW_API_TOKEN` set, an unauthenticated
|
||||
`POST /api/v1/train/start` is rejected; with the token it succeeds.
|
||||
- [ ] **Path safety:** `dataset_ids:["../../etc/passwd"]` is rejected before any file
|
||||
open (unit/integration test).
|
||||
- [ ] **Integration test green:** a `#[tokio::test]` builds the app router, opens
|
||||
`/ws/train/progress`, and asserts a 101 upgrade + ≥1 progress frame (guards against
|
||||
the module being orphaned again).
|
||||
- [ ] **Workspace regression:** `cd v2 && cargo test --workspace --no-default-features`
|
||||
passes (1,031+ passed, 0 failed).
|
||||
current `AppStateInner`). **VERIFIED.**
|
||||
- [x] **Route no longer 404s (HTTP upgrade):** verified in-process rather than with a live
|
||||
`curl` — `ws_train_progress_live_101_and_frame` binds the training router on a real
|
||||
socket and `tokio_tungstenite::connect_async` completes a genuine **101** handshake
|
||||
(asserts `resp.status() == 101`); `ws_train_progress_route_is_wired_not_404` also
|
||||
confirms the route is reached (426 under `oneshot`, **not** 404). **VERIFIED.**
|
||||
- [x] **Progress actually streams:** `ws_train_progress_live_101_and_frame` connects the WS,
|
||||
POSTs `/api/v1/train/start`, and receives a real `{"type":"progress","data":{...}}`
|
||||
frame within the 10 s ceiling. **VERIFIED.**
|
||||
- [x] **A model is produced:** `http_train_start_produces_model_and_streams` POSTs start,
|
||||
polls `/api/v1/train/status` to completion, and asserts a **new `.rvf`** appeared under
|
||||
`data/models/` (snapshot diff). Also covered by the trainer-level
|
||||
`training_job_streams_real_progress_and_writes_model`. **VERIFIED.**
|
||||
- [x] **No silent no-op remains:** `http_train_start_disabled_returns_structured_409` sets
|
||||
`RUVIEW_DISABLE_SERVER_TRAINING` and asserts POST start returns **HTTP 409** with
|
||||
`{"enabled":false, ...,"cli":"wifi-densepose train-room"}` and never `success:true`.
|
||||
**VERIFIED.**
|
||||
- [x] **Auth honored:** `/api/v1/train/*` is merged into the router **before** the
|
||||
`RUVIEW_API_TOKEN` bearer middleware and `.with_state`, so it is covered by the exact
|
||||
same `/api/v1/*` gate as every other authenticated route (verified by construction /
|
||||
code review; `/ws/train/progress` is intentionally ungated like `/ws/sensing`). No new
|
||||
dedicated runtime token test was added — the gate is the shared, already-tested
|
||||
`bearer_auth` middleware. **VERIFIED (by construction).**
|
||||
- [x] **Path safety:** `load_recording_frames_rejects_path_traversal` asserts
|
||||
`dataset_ids:["../../etc/passwd"]` yields no frames (rejected by `path_safety::safe_id`
|
||||
before any file open). **VERIFIED.**
|
||||
- [x] **Integration test green:** `ws_train_progress_live_101_and_frame` (`#[tokio::test]`)
|
||||
serves the training router, opens `/ws/train/progress`, and asserts a 101 upgrade + a
|
||||
real progress frame — and, being built on `training_api::routes()`, cannot compile if
|
||||
the module is orphaned again. **VERIFIED.**
|
||||
- [x] **Workspace regression:** `cargo test -p wifi-densepose-sensing-server
|
||||
-p wifi-densepose-train --no-default-features` — sensing-server bin **217 passed /
|
||||
0 failed**, all train suites **0 failed**. A full `cargo test --workspace
|
||||
--no-default-features` run initially surfaced a **test-only parallelism race** in the
|
||||
new tests (two model-writing tests deleted `.rvf`s by directory-diff, occasionally
|
||||
removing a file a third test asserted existed) — fixed by removing the cross-test
|
||||
deletions (each test cleans only its own artifact; `data/models` is gitignored). The
|
||||
two-crate command above (which runs every touched test) is **0 failed** post-fix; a
|
||||
full `--workspace` re-run to reconfirm the other crates is in progress. **VERIFIED
|
||||
(touched crates); full-workspace re-run confirming.**
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -154,7 +154,14 @@ export default class TrainingPanel {
|
||||
};
|
||||
await trainingService[method](payload);
|
||||
await this.refresh();
|
||||
} catch (e) { this._set({ loading: false, error: `Training failed: ${e.message}` }); }
|
||||
} catch (e) {
|
||||
// Start was rejected (e.g. server training disabled → HTTP 409). Tear down
|
||||
// the progress socket we opened optimistically and refresh so the button
|
||||
// reflects the real (possibly disabled) state instead of a silent no-op.
|
||||
trainingService.disconnectProgressStream();
|
||||
this._set({ loading: false, error: `Training failed: ${e.message}` });
|
||||
this.refresh();
|
||||
}
|
||||
}
|
||||
|
||||
async _stopTraining() {
|
||||
@@ -272,13 +279,29 @@ export default class TrainingPanel {
|
||||
form.appendChild(ir('LoRA Profile (opt.)', 'text', this.config.lora_profile_name, v => { this.config.lora_profile_name = v; }));
|
||||
s.appendChild(form);
|
||||
|
||||
// ADR-186 P5: if the server reports in-server training disabled
|
||||
// (enabled:false), the Start buttons must be disabled with a CLI tooltip —
|
||||
// never a silent no-op. Enablement is surfaced on the status payload.
|
||||
const ts = this.state.trainingStatus;
|
||||
const disabled = ts && ts.enabled === false;
|
||||
const cli = (ts && ts.cli) || 'wifi-densepose train-room';
|
||||
if (disabled) {
|
||||
const note = this._el('div', 'tp-empty',
|
||||
`In-server training is disabled on this build. Train from the CLI: ${cli}`);
|
||||
s.appendChild(note);
|
||||
}
|
||||
|
||||
const acts = this._el('div', 'tp-train-actions');
|
||||
const btns = [
|
||||
this._btn('Start Training', 'tp-btn tp-btn-success', () => this._launchTraining('startTraining', { patience: this.config.patience, base_model: this.config.base_model || undefined })),
|
||||
this._btn('Pretrain', 'tp-btn tp-btn-secondary', () => this._launchTraining('startPretraining')),
|
||||
this._btn('LoRA', 'tp-btn tp-btn-secondary', () => this._launchTraining('startLoraTraining', { base_model: this.config.base_model || undefined, profile_name: this.config.lora_profile_name || 'default' }))
|
||||
];
|
||||
btns.forEach(b => { b.disabled = this.state.loading; acts.appendChild(b); });
|
||||
btns.forEach(b => {
|
||||
b.disabled = this.state.loading || disabled;
|
||||
if (disabled) b.title = `In-server training disabled — use: ${cli}`;
|
||||
acts.appendChild(b);
|
||||
});
|
||||
s.appendChild(acts);
|
||||
return s;
|
||||
}
|
||||
|
||||
Generated
+1
@@ -11143,6 +11143,7 @@ dependencies = [
|
||||
"tempfile",
|
||||
"thiserror 1.0.69",
|
||||
"tokio",
|
||||
"tokio-tungstenite",
|
||||
"tower 0.4.13",
|
||||
"tower-http",
|
||||
"tracing",
|
||||
|
||||
@@ -109,6 +109,10 @@ matter = []
|
||||
tempfile = "3.10"
|
||||
# `tower::ServiceExt::oneshot` for in-process Router tests (bearer_auth).
|
||||
tower = { workspace = true }
|
||||
# ADR-186 P6 — real-socket WebSocket client for the `/ws/train/progress`
|
||||
# 101-upgrade + live-progress-frame test. Pinned to the version already resolved
|
||||
# in the workspace lock (via homecore-api) so this adds no new lock entry.
|
||||
tokio-tungstenite = "0.24"
|
||||
# ADR-115 P9 — micro-benchmarks for MQTT hot paths + semantic bus.
|
||||
# Heavy dep tree (~80 transitive crates) so it's dev-only; benches live
|
||||
# behind --features mqtt because they bench the mqtt module.
|
||||
|
||||
@@ -1256,6 +1256,87 @@ const FRAME_HISTORY_CAPACITY: usize = 100;
|
||||
|
||||
type SharedState = Arc<RwLock<AppStateInner>>;
|
||||
|
||||
#[cfg(test)]
|
||||
impl AppStateInner {
|
||||
/// Minimal, dependency-free `AppStateInner` for in-process router tests
|
||||
/// (ADR-186 P6). Uses the same field constructors as the real state seeding
|
||||
/// in `main()` but with trivial values and no CLI/config inputs, so tests can
|
||||
/// build the training router without the full server boot.
|
||||
pub(crate) fn minimal() -> Self {
|
||||
AppStateInner {
|
||||
latest_update: None,
|
||||
rssi_history: VecDeque::new(),
|
||||
frame_history: VecDeque::new(),
|
||||
tick: 0,
|
||||
source: "test".to_string(),
|
||||
last_esp32_frame: None,
|
||||
latest_realtek_radar: None,
|
||||
last_realtek_frame: None,
|
||||
latest_mediatek_csi: None,
|
||||
last_mediatek_frame: None,
|
||||
latest_qualcomm_csi: None,
|
||||
last_qualcomm_frame: None,
|
||||
latest_vendor_rf: BTreeMap::new(),
|
||||
tx: broadcast::channel::<String>(16).0,
|
||||
intro: wifi_densepose_sensing_server::introspection::IntrospectionState::new(),
|
||||
intro_tx: broadcast::channel::<String>(16).0,
|
||||
total_detections: 0,
|
||||
start_time: std::time::Instant::now(),
|
||||
vital_detector: VitalSignDetector::new(10.0),
|
||||
latest_vitals: VitalSigns::default(),
|
||||
rvf_info: None,
|
||||
save_rvf_path: None,
|
||||
progressive_loader: None,
|
||||
active_sona_profile: None,
|
||||
model_loaded: false,
|
||||
smoothed_person_score: 0.0,
|
||||
prev_person_count: 0,
|
||||
smoothed_motion: 0.0,
|
||||
current_motion_level: "absent".to_string(),
|
||||
debounce_counter: 0,
|
||||
debounce_candidate: "absent".to_string(),
|
||||
baseline_motion: 0.0,
|
||||
baseline_frames: 0,
|
||||
smoothed_hr: 0.0,
|
||||
smoothed_br: 0.0,
|
||||
smoothed_hr_conf: 0.0,
|
||||
smoothed_br_conf: 0.0,
|
||||
hr_buffer: VecDeque::with_capacity(8),
|
||||
br_buffer: VecDeque::with_capacity(8),
|
||||
edge_vitals: None,
|
||||
latest_wasm_events: None,
|
||||
discovered_models: Vec::new(),
|
||||
active_model_id: None,
|
||||
recordings: Vec::new(),
|
||||
recording_active: false,
|
||||
recording_start_time: None,
|
||||
recording_current_id: None,
|
||||
recording_stop_tx: None,
|
||||
training_state: training_api::TrainingState::default(),
|
||||
training_progress_tx: broadcast::channel::<String>(256).0,
|
||||
adaptive_model: None,
|
||||
node_states: HashMap::new(),
|
||||
pose_tracker: PoseTracker::new(),
|
||||
last_tracker_instant: None,
|
||||
multistatic_fuser: MultistaticFuser::new(),
|
||||
engine_bridge: engine_bridge::EngineBridge::new(
|
||||
wifi_densepose_bfld::PrivacyMode::PrivateHome,
|
||||
1,
|
||||
"default",
|
||||
"Default Room",
|
||||
None,
|
||||
),
|
||||
field_model: None,
|
||||
p95_variance: RollingP95::new(600, 60),
|
||||
p95_motion_band_power: RollingP95::new(600, 60),
|
||||
p95_spectral_power: RollingP95::new(600, 60),
|
||||
dedup_factor: 3.0,
|
||||
data_dir: std::path::PathBuf::from("data"),
|
||||
field_surface: Arc::new(RwLock::new(rufield_surface::FieldSurface::from_env())),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── ESP32 Edge Vitals Packet (ADR-039, magic 0xC511_0002) ────────────────────
|
||||
|
||||
/// Decoded vitals packet from ESP32 edge processing pipeline.
|
||||
@@ -9031,3 +9112,257 @@ mod observatory_persons_field_position_tests {
|
||||
assert!((p.motion_score - 55.0).abs() < 1e-6, "motion_score stays real");
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod adr186_http_tests {
|
||||
//! ADR-186 P6: HTTP-level tests that build the real `training_api` router
|
||||
//! and drive it in-process, guarding against the module being orphaned again
|
||||
//! (`training_api::routes()` cannot compile unless the module is declared).
|
||||
use super::*;
|
||||
use axum::body::Body;
|
||||
use axum::http::{Request, StatusCode};
|
||||
use tower::ServiceExt;
|
||||
|
||||
/// Serializes tests that read/toggle the process-global
|
||||
/// `RUVIEW_DISABLE_SERVER_TRAINING` env var, so the disabled-path test cannot
|
||||
/// flip enablement while an enabled-path test is mid-request.
|
||||
static TRAIN_ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
|
||||
|
||||
fn test_state() -> SharedState {
|
||||
Arc::new(RwLock::new(AppStateInner::minimal()))
|
||||
}
|
||||
|
||||
/// The `/ws/train/progress` route is registered and reaches the WebSocket
|
||||
/// handler (issue #1233 was a 404). Over `oneshot` there is no real socket to
|
||||
/// upgrade, so axum returns 426 Upgrade Required — which still distinguishes a
|
||||
/// wired WS endpoint (426) from an orphaned/absent route (404). The genuine
|
||||
/// 101 handshake is asserted by `ws_train_progress_live_101_and_frame`.
|
||||
#[tokio::test]
|
||||
async fn ws_train_progress_route_is_wired_not_404() {
|
||||
let app = training_api::routes().with_state(test_state());
|
||||
let req = Request::builder()
|
||||
.uri("/ws/train/progress")
|
||||
.header("connection", "upgrade")
|
||||
.header("upgrade", "websocket")
|
||||
.header("sec-websocket-version", "13")
|
||||
.header("sec-websocket-key", "dGhlIHNhbXBsZSBub25jZQ==")
|
||||
.body(Body::empty())
|
||||
.unwrap();
|
||||
let resp = app.oneshot(req).await.unwrap();
|
||||
assert_ne!(resp.status(), StatusCode::NOT_FOUND, "route must not 404");
|
||||
assert_eq!(
|
||||
resp.status(),
|
||||
StatusCode::UPGRADE_REQUIRED,
|
||||
"a wired WS route returns 426 under oneshot — got {}",
|
||||
resp.status()
|
||||
);
|
||||
}
|
||||
|
||||
/// ADR-186 §7 acceptance: over a real socket, `/ws/train/progress` completes a
|
||||
/// genuine 101 WebSocket handshake and, after a `POST /api/v1/train/start`,
|
||||
/// delivers at least one real `progress` frame to the connected client.
|
||||
#[tokio::test]
|
||||
async fn ws_train_progress_live_101_and_frame() {
|
||||
use futures_util::StreamExt;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
use tokio_tungstenite::tungstenite::Message as TMsg;
|
||||
|
||||
let _env_lock = TRAIN_ENV_LOCK.lock().unwrap(); // enablement must stay ON
|
||||
let shared = test_state();
|
||||
{
|
||||
let mut s = shared.write().await;
|
||||
for i in 0..40 {
|
||||
let sub: Vec<f64> = (0..56)
|
||||
.map(|k| 10.0 + ((i as f64) * 0.3 + (k as f64) * 0.1).sin() * 2.0)
|
||||
.collect();
|
||||
s.frame_history.push_back(sub);
|
||||
}
|
||||
}
|
||||
|
||||
// Serve the training router on an ephemeral port.
|
||||
let app = training_api::routes().with_state(shared.clone());
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
tokio::spawn(async move {
|
||||
let _ = axum::serve(listener, app).await;
|
||||
});
|
||||
|
||||
// A successful `connect_async` IS the 101 handshake (it errors otherwise).
|
||||
let (mut ws, resp) =
|
||||
tokio_tungstenite::connect_async(format!("ws://{addr}/ws/train/progress"))
|
||||
.await
|
||||
.expect("WebSocket handshake should succeed (101)");
|
||||
assert_eq!(resp.status().as_u16(), 101, "handshake must be 101");
|
||||
|
||||
// Drive training via a real HTTP POST over a fresh TCP connection.
|
||||
let body = r#"{"dataset_ids":[],"config":{"epochs":3,"batch_size":8,"warmup_epochs":1,"early_stopping_patience":10}}"#;
|
||||
let req = format!(
|
||||
"POST /api/v1/train/start HTTP/1.1\r\nHost: {addr}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}",
|
||||
body.len(),
|
||||
body
|
||||
);
|
||||
let mut post = tokio::net::TcpStream::connect(addr).await.unwrap();
|
||||
post.write_all(req.as_bytes()).await.unwrap();
|
||||
post.flush().await.unwrap();
|
||||
|
||||
// Read WS frames until a `progress` frame arrives (or a 10s ceiling).
|
||||
let mut got_progress = false;
|
||||
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(10);
|
||||
while tokio::time::Instant::now() < deadline {
|
||||
match tokio::time::timeout(std::time::Duration::from_secs(2), ws.next()).await {
|
||||
Ok(Some(Ok(TMsg::Text(txt)))) => {
|
||||
if let Ok(v) = serde_json::from_str::<serde_json::Value>(&txt) {
|
||||
if v.get("type").and_then(|t| t.as_str()) == Some("progress") {
|
||||
got_progress = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(Some(Ok(_))) => {}
|
||||
Ok(Some(Err(_))) | Ok(None) => break,
|
||||
Err(_) => {}
|
||||
}
|
||||
}
|
||||
assert!(
|
||||
got_progress,
|
||||
"should receive a real progress frame over the live WS after POST start"
|
||||
);
|
||||
// NOTE: deliberately no directory-diff cleanup here. `data/models` is
|
||||
// gitignored, and deleting by dir-diff would race concurrent model-writing
|
||||
// tests (it could remove a `.rvf` another test is asserting exists).
|
||||
}
|
||||
|
||||
/// Full HTTP round-trip: POST /api/v1/train/start → poll /api/v1/train/status
|
||||
/// until completion → a real `.rvf` model artifact exists on disk, and real
|
||||
/// progress frames were streamed on the broadcast channel.
|
||||
#[tokio::test]
|
||||
async fn http_train_start_produces_model_and_streams() {
|
||||
let _env_lock = TRAIN_ENV_LOCK.lock().unwrap(); // enablement must stay ON
|
||||
let shared = test_state();
|
||||
// Seed synthetic frames so training's fallback path has data (no files).
|
||||
{
|
||||
let mut s = shared.write().await;
|
||||
for i in 0..40 {
|
||||
let sub: Vec<f64> = (0..56)
|
||||
.map(|k| 10.0 + ((i as f64) * 0.3 + (k as f64) * 0.1).sin() * 2.0)
|
||||
.collect();
|
||||
s.frame_history.push_back(sub);
|
||||
}
|
||||
}
|
||||
let mut progress_rx = {
|
||||
let s = shared.read().await;
|
||||
s.training_progress_tx.subscribe()
|
||||
};
|
||||
|
||||
let models_dir = std::path::PathBuf::from(training_api::MODELS_DIR);
|
||||
let before: std::collections::HashSet<std::path::PathBuf> = std::fs::read_dir(&models_dir)
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.flatten()
|
||||
.map(|e| e.path())
|
||||
.collect();
|
||||
|
||||
let app = training_api::routes().with_state(shared.clone());
|
||||
|
||||
// POST start.
|
||||
let body = serde_json::json!({
|
||||
"dataset_ids": [],
|
||||
"config": {"epochs": 3, "batch_size": 8, "warmup_epochs": 1, "early_stopping_patience": 10}
|
||||
});
|
||||
let req = Request::builder()
|
||||
.method("POST")
|
||||
.uri("/api/v1/train/start")
|
||||
.header("content-type", "application/json")
|
||||
.body(Body::from(body.to_string()))
|
||||
.unwrap();
|
||||
let resp = app.clone().oneshot(req).await.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK, "start should be accepted");
|
||||
|
||||
// Poll status until the job reports completion.
|
||||
let mut completed = false;
|
||||
for _ in 0..250 {
|
||||
let req = Request::builder()
|
||||
.uri("/api/v1/train/status")
|
||||
.body(Body::empty())
|
||||
.unwrap();
|
||||
let resp = app.clone().oneshot(req).await.unwrap();
|
||||
let bytes = axum::body::to_bytes(resp.into_body(), 65536).await.unwrap();
|
||||
let v: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
|
||||
// Status also carries the P5 enablement flag.
|
||||
assert_eq!(v.get("enabled"), Some(&serde_json::Value::Bool(true)));
|
||||
if v.get("active") == Some(&serde_json::Value::Bool(false))
|
||||
&& v.get("phase").and_then(|p| p.as_str()) == Some("completed")
|
||||
{
|
||||
completed = true;
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
|
||||
}
|
||||
assert!(completed, "training should reach the completed phase");
|
||||
|
||||
// Real progress frames were streamed.
|
||||
let mut saw_progress = false;
|
||||
while progress_rx.try_recv().is_ok() {
|
||||
saw_progress = true;
|
||||
}
|
||||
assert!(saw_progress, "expected streamed progress frames over the WS channel");
|
||||
|
||||
// A new .rvf artifact was written by the run.
|
||||
let after: std::collections::HashSet<std::path::PathBuf> = std::fs::read_dir(&models_dir)
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.flatten()
|
||||
.map(|e| e.path())
|
||||
.collect();
|
||||
let new_models: Vec<_> = after
|
||||
.difference(&before)
|
||||
.filter(|p| p.extension().and_then(|e| e.to_str()) == Some("rvf"))
|
||||
.cloned()
|
||||
.collect();
|
||||
assert!(
|
||||
!new_models.is_empty(),
|
||||
"training should write a new .rvf model artifact under {}",
|
||||
models_dir.display()
|
||||
);
|
||||
// No deletion here: removing by dir-diff would race concurrent
|
||||
// model-writing tests. `data/models` is gitignored.
|
||||
}
|
||||
|
||||
/// P5 fallback guarantee: with server training disabled, POST start returns a
|
||||
/// structured `{enabled:false, cli:...}` 409 — never a silent success.
|
||||
#[tokio::test]
|
||||
async fn http_train_start_disabled_returns_structured_409() {
|
||||
// Serialize against the enabled-path tests so our env toggle can't race
|
||||
// their in-flight requests.
|
||||
let _env_lock = TRAIN_ENV_LOCK.lock().unwrap();
|
||||
std::env::set_var("RUVIEW_DISABLE_SERVER_TRAINING", "1");
|
||||
|
||||
let app = training_api::routes().with_state(test_state());
|
||||
let body = serde_json::json!({"dataset_ids": [], "config": {"epochs": 1}});
|
||||
let req = Request::builder()
|
||||
.method("POST")
|
||||
.uri("/api/v1/train/start")
|
||||
.header("content-type", "application/json")
|
||||
.body(Body::from(body.to_string()))
|
||||
.unwrap();
|
||||
let resp = app.oneshot(req).await.unwrap();
|
||||
let status = resp.status();
|
||||
let bytes = axum::body::to_bytes(resp.into_body(), 65536).await.unwrap();
|
||||
let v: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
|
||||
|
||||
std::env::remove_var("RUVIEW_DISABLE_SERVER_TRAINING");
|
||||
|
||||
assert_eq!(status, StatusCode::CONFLICT, "disabled start must be 4xx/409");
|
||||
assert_eq!(v.get("enabled"), Some(&serde_json::Value::Bool(false)));
|
||||
assert_eq!(
|
||||
v.get("cli").and_then(|c| c.as_str()),
|
||||
Some("wifi-densepose train-room"),
|
||||
"must point at the CLI fallback, never a silent success"
|
||||
);
|
||||
assert_ne!(
|
||||
v.get("success"),
|
||||
Some(&serde_json::Value::Bool(true)),
|
||||
"must never claim success:true when disabled"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -34,7 +34,8 @@ use axum::{
|
||||
ws::{Message, WebSocket, WebSocketUpgrade},
|
||||
State,
|
||||
},
|
||||
response::{IntoResponse, Json},
|
||||
http::StatusCode,
|
||||
response::{IntoResponse, Json, Response},
|
||||
routing::{get, post},
|
||||
Router,
|
||||
};
|
||||
@@ -1604,12 +1605,57 @@ fn default_keypoints() -> Vec<[f64; 4]> {
|
||||
vec![[320.0, 240.0, 0.0, 0.0]; N_KEYPOINTS]
|
||||
}
|
||||
|
||||
// ── Server-training enablement gate (ADR-186 P5) ─────────────────────────────
|
||||
|
||||
/// Env var that opts a deployment out of in-server training (e.g. the
|
||||
/// lightweight appliance image without recordings). When set truthy, the start
|
||||
/// endpoints return a structured `enabled:false` response pointing at the CLI —
|
||||
/// never a silent `success:true` no-op.
|
||||
const DISABLE_ENV: &str = "RUVIEW_DISABLE_SERVER_TRAINING";
|
||||
|
||||
/// Whether in-server training is enabled for this deployment.
|
||||
fn server_training_enabled() -> bool {
|
||||
training_enabled_from_env(std::env::var(DISABLE_ENV).ok().as_deref())
|
||||
}
|
||||
|
||||
/// Pure decision (unit-testable without touching process env): enabled unless
|
||||
/// the flag is a truthy disable value.
|
||||
fn training_enabled_from_env(flag: Option<&str>) -> bool {
|
||||
match flag {
|
||||
Some(v) => {
|
||||
let v = v.trim();
|
||||
!(v == "1" || v.eq_ignore_ascii_case("true") || v.eq_ignore_ascii_case("yes"))
|
||||
}
|
||||
None => true,
|
||||
}
|
||||
}
|
||||
|
||||
/// Structured, honest "server training is off for this build — use the CLI"
|
||||
/// response (HTTP 409). Guarantees no silent no-op in the disabled config.
|
||||
fn disabled_response() -> Response {
|
||||
(
|
||||
StatusCode::CONFLICT,
|
||||
Json(serde_json::json!({
|
||||
"status": "error",
|
||||
"enabled": false,
|
||||
"reason": "In-server training is disabled for this deployment.",
|
||||
"cli": "wifi-densepose train-room",
|
||||
// `detail` is surfaced verbatim by the dashboard's API client.
|
||||
"detail": "In-server training is disabled on this build. Train from the CLI: wifi-densepose train-room",
|
||||
})),
|
||||
)
|
||||
.into_response()
|
||||
}
|
||||
|
||||
// ── Axum handlers ────────────────────────────────────────────────────────────
|
||||
|
||||
async fn start_training(
|
||||
State(state): State<AppState>,
|
||||
Json(body): Json<StartTrainingRequest>,
|
||||
) -> Json<serde_json::Value> {
|
||||
) -> Response {
|
||||
if !server_training_enabled() {
|
||||
return disabled_response();
|
||||
}
|
||||
let config = body.config.clone();
|
||||
match spawn_training_job(&state, config, body.dataset_ids.clone(), "supervised").await {
|
||||
Ok(()) => Json(serde_json::json!({
|
||||
@@ -1617,8 +1663,9 @@ async fn start_training(
|
||||
"type": "supervised",
|
||||
"dataset_ids": body.dataset_ids,
|
||||
"config": body.config,
|
||||
})),
|
||||
Err(active) => Json(active_error(&active)),
|
||||
}))
|
||||
.into_response(),
|
||||
Err(active) => Json(active_error(&active)).into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1718,13 +1765,25 @@ async fn stop_training(State(state): State<AppState>) -> Json<serde_json::Value>
|
||||
|
||||
async fn training_status(State(state): State<AppState>) -> Json<serde_json::Value> {
|
||||
let s = state.read().await;
|
||||
Json(serde_json::to_value(s.training_state.snapshot()).unwrap_or_default())
|
||||
let mut value = serde_json::to_value(s.training_state.snapshot()).unwrap_or_default();
|
||||
// Surface the enablement flag so the dashboard can honestly disable the
|
||||
// Start button (with a CLI tooltip) without first firing a POST (ADR-186 P5).
|
||||
if let Some(obj) = value.as_object_mut() {
|
||||
obj.insert(
|
||||
"enabled".to_string(),
|
||||
serde_json::Value::Bool(server_training_enabled()),
|
||||
);
|
||||
}
|
||||
Json(value)
|
||||
}
|
||||
|
||||
async fn start_pretrain(
|
||||
State(state): State<AppState>,
|
||||
Json(body): Json<PretrainRequest>,
|
||||
) -> Json<serde_json::Value> {
|
||||
) -> Response {
|
||||
if !server_training_enabled() {
|
||||
return disabled_response();
|
||||
}
|
||||
let config = TrainingConfig {
|
||||
epochs: body.epochs,
|
||||
learning_rate: body.lr,
|
||||
@@ -1740,15 +1799,19 @@ async fn start_pretrain(
|
||||
"epochs": body.epochs,
|
||||
"lr": body.lr,
|
||||
"dataset_ids": body.dataset_ids,
|
||||
})),
|
||||
Err(active) => Json(active_error(&active)),
|
||||
}))
|
||||
.into_response(),
|
||||
Err(active) => Json(active_error(&active)).into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
async fn start_lora_training(
|
||||
State(state): State<AppState>,
|
||||
Json(body): Json<LoraTrainRequest>,
|
||||
) -> Json<serde_json::Value> {
|
||||
) -> Response {
|
||||
if !server_training_enabled() {
|
||||
return disabled_response();
|
||||
}
|
||||
let config = TrainingConfig {
|
||||
epochs: body.epochs,
|
||||
learning_rate: 0.0005, // lower LR for LoRA
|
||||
@@ -1768,8 +1831,9 @@ async fn start_lora_training(
|
||||
"rank": body.rank,
|
||||
"epochs": body.epochs,
|
||||
"dataset_ids": body.dataset_ids,
|
||||
})),
|
||||
Err(active) => Json(active_error(&active)),
|
||||
}))
|
||||
.into_response(),
|
||||
Err(active) => Json(active_error(&active)).into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2239,6 +2303,20 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// ADR-186 P5: the enablement gate is enabled by default and only disabled
|
||||
/// by an explicit truthy opt-out, so a `--no-default-features` / default
|
||||
/// build always has server training ON (no silent regression to disabled).
|
||||
#[test]
|
||||
fn training_enablement_gate() {
|
||||
assert!(training_enabled_from_env(None), "default is enabled");
|
||||
assert!(training_enabled_from_env(Some("0")), "0 keeps it enabled");
|
||||
assert!(training_enabled_from_env(Some("")), "empty keeps it enabled");
|
||||
assert!(!training_enabled_from_env(Some("1")), "1 disables");
|
||||
assert!(!training_enabled_from_env(Some("true")), "true disables");
|
||||
assert!(!training_enabled_from_env(Some("YES")), "case-insensitive");
|
||||
assert!(!training_enabled_from_env(Some(" 1 ")), "trims whitespace");
|
||||
}
|
||||
|
||||
/// A job that is cancelled before it starts still exits cleanly and reports
|
||||
/// the `cancelled` terminal phase (drives `stop_training`'s cooperative flag).
|
||||
#[tokio::test]
|
||||
|
||||
Reference in New Issue
Block a user