ruvnet--RuView/docs/adr/ADR-186-training-progress-api.md

# ADR-186: Training progress API — wire the orphaned in-server trainer to `/ws/train/progress`

| Field | Value |
|-------|-------|
| **Status** | Proposed |
| **Date** | 2026-07-21 |
| **Deciders** | ruv |
| **Codename** | **TRAIN-RECONNECT** — connecting a trainer that was written, committed, and then never plugged in |
| **Relates to** | [ADR-051](ADR-051-sensing-server-decomposition.md) (main.rs decomposition into ~14 modules), [ADR-151](ADR-151-per-room-calibration.md) (`train-room` specialist bank), [ADR-152](ADR-152-wifi-pose-sota-2026.md) (MAE recipe / geometry conditioning), [ADR-166](ADR-166-quality-engineering-security-hardening.md) (WS auth + god-object decomposition) |
| **Tracking issue** | [#1233](https://github.com/ruvnet/wifi-densepose/issues/1233) — "Training does not start – /ws/train/progress returns 404 and no model is generated" (open) |

---

## 1. Context

### 1.1 The reported gap

A user starting training from the web dashboard hits
`ws://localhost:3000/ws/train/progress`, which **404s**, and the backend never
produces a trained `.rvf` model or any further log output beyond a single
"Training started" line. Issue #1233 is open, and the repo owner's own comment on
it states:

> The `/ws/train/progress` WebSocket endpoint is not yet exposed in the stable
> server — the training pipeline (room-calibration specialists, MAE pretraining)
> runs via the CLI (`wifi-densepose train-room`) rather than through the
> HTTP/WebSocket API, which is why the Docker image returns 404 for that path.

So the dashboard has a **"Start Training" button that silently no-ops**: it POSTs a
config, receives a `success: true` response, and then nothing happens — no error is
surfaced, no model is produced, no progress stream exists. A button that appears to
work but does nothing is the definition of slop, and this ADR exists to close that
gap honestly.

### 1.2 What the live server actually does today (evidence)

The stable server mounts **stub** training handlers. The POST handler flips a string
flag, logs one line, and returns success — it starts no job:

```rust
// v2/crates/wifi-densepose-sensing-server/src/main.rs:4986–5006
async fn train_start(
    State(state): State<SharedState>,
    Json(body): Json<serde_json::Value>,
) -> Json<serde_json::Value> {
    let mut s = state.write().await;
    if s.training_status == "running" { /* ... */ }
    s.training_status = "running".to_string();
    s.training_config = Some(body.clone());
    info!("Training started with config: {}", body);   // ← the one log line the issue reports
    Json(serde_json::json!({
        "success": true,
        "status": "running",
        "message": "Training pipeline started. Use GET /api/v1/train/status to monitor.",
    }))
}
```

These three stubs — and **nothing else training-related** — are wired into the live
router:

```rust
// v2/crates/wifi-densepose-sensing-server/src/main.rs:8068–8071
// Training endpoints
.route("/api/v1/train/status", get(train_status))
.route("/api/v1/train/start",  post(train_start))
.route("/api/v1/train/stop",   post(train_stop))
```

There is **no `/ws/train/progress` route in the live app** — hence the 404 that
issue #1233 reports. The stub state fields backing them are just:

```rust
// v2/crates/wifi-densepose-sensing-server/src/main.rs:1125–1127
training_status: String,               // "idle" | "running" | ...
training_config: Option<serde_json::Value>,
```

### 1.3 The surprising finding: a real trainer already exists, orphaned

The gap is **not** that training was never built for the server. A complete
in-server training pipeline **already exists in the tree** at
`v2/crates/wifi-densepose-sensing-server/src/training_api.rs` (1,860 lines). Its own
module doc describes what it does (`training_api.rs:1–25`):

- Loads recorded CSI from `.csi.jsonl` files, extracts signal features (subcarrier
  variance, temporal gradients, Goertzel frequency-domain power).
- Trains a regularised linear model via batch gradient descent.
- Exports a calibrated `.rvf` model container via `RvfBuilder` on completion.
- **"No PyTorch / `tch` dependency is required. All linear algebra is implemented
  inline using standard Rust math."** (`training_api.rs:11–13`)

It runs training on a **background tokio task** and streams progress over a
`tokio::sync::broadcast` channel to a real WebSocket handler:

- `start_training` spawns the job: `tokio::spawn(async move { ... })`
  (`training_api.rs:1564`, spawn at `:1610`).
- `ws_train_progress_handler` subscribes to `training_progress_tx` and forwards
  `{"type":"progress", "data": …}` frames (`training_api.rs:1778–1836`).
- A `routes()` factory wires the whole surface, **including the missing route**:

```rust
// v2/crates/wifi-densepose-sensing-server/src/training_api.rs:1841–1849
pub fn routes() -> Router<AppState> {
    Router::new()
        .route("/api/v1/train/start",    post(start_training))
        .route("/api/v1/train/stop",     post(stop_training))
        .route("/api/v1/train/status",   get(training_status))
        .route("/api/v1/train/pretrain", post(start_pretrain))
        .route("/api/v1/train/lora",     post(start_lora_training))
        .route("/ws/train/progress",     get(ws_train_progress_handler))
}
```

**This module is dead code.** There is no `mod training_api;` declaration anywhere
in the crate — a repo-wide search for `training_api` returns only a doc-comment
mention in `path_safety.rs:9`. Because Rust never sees the file without a `mod`
declaration, `training_api.rs` is **not compiled into the binary at all**, and
`training_api::routes()` is never merged into the app. It was written, committed
(last touched by commit `9b07dff29`), and then orphaned.

### 1.4 Why it would not even compile if naively wired in

The orphan was written against a **different state shape than the one that shipped**.
`training_api.rs` expects its parent to expose an `AppStateInner` carrying a training
sub-state and a broadcast sender:

```rust
// v2/crates/wifi-densepose-sensing-server/src/training_api.rs:249
pub type AppState = Arc<RwLock<super::AppStateInner>>;
// handlers read s.training_state.status, s.training_state.task_handle,
// s.training_progress_tx  (e.g. training_api.rs:1588, :1610, :1788)
```

But the **real** `AppStateInner` (`main.rs:1024`, aliased `SharedState` at
`main.rs:1249`) has none of those fields — only the `training_status: String` /
`training_config` stubs from §1.2. `training_state: TrainingState` is defined
locally in `training_api.rs:232`, and `training_progress_tx` exists nowhere on the
live state. So adding `mod training_api;` today produces a compile error: the module
references `AppStateInner` fields that do not exist. Wiring it in requires
**reconciling the state struct first**, not merely uncommenting a route.

### 1.5 The working path today

The path that actually trains a model is the CLI, exactly as the maintainer's
comment says:

- `wifi-densepose train-room` → `room.rs:241` `train_room(...)`, the ADR-151
  Stage-2–5 per-room specialist-bank trainer (`enroll → train-room → room-watch`).
- The heavier `wifi-densepose-train` crate exposes epoch-level metrics
  (`trainer.rs:43` `pub epoch: usize`, `trainer.rs:64` `best_epoch`) that a progress
  stream could surface directly — the data a WebSocket needs already exists in the
  training loop.

### 1.6 What this ADR is *not*

- Not a rewrite of the trainer. The pipeline in `training_api.rs` already exists;
  this ADR reconnects and hardens it.
- Not a move of GPU/`tch`-backed training into the Axum server. The in-server
  trainer is deliberately `tch`-free (§1.3). Heavy MAE/LoRA training stays in the
  CLI / `wifi-densepose-train` crate; the server streams progress for the light,
  pure-Rust specialist trainer and (optionally) proxies status for CLI-launched runs.
- Not a change to the `train-room` CLI contract (ADR-151). The CLI remains the
  authoritative path for offline / batch training.

---

## 2. Current state — evidence

| Artifact | Value | Source |
|---|---|---|
| Live POST handler | `train_start` — flips a flag, logs, returns `success:true`, starts no job | `main.rs:4986–5006` |
| The "Training started" log line from the issue | `info!("Training started with config: {}", body)` | `main.rs:5000` |
| Live training routes | `train/status`, `train/start`, `train/stop` (stubs only) | `main.rs:8068–8071` |
| `/ws/train/progress` in live app | **Absent** → 404 | (no route in `main.rs` router) |
| Live training state fields | `training_status: String`, `training_config: Option<Value>` | `main.rs:1125–1127` |
| Real in-server trainer | 1,860-line implemented pipeline, `tch`-free, exports `.rvf` | `training_api.rs:1–25` |
| Real WS progress handler | subscribes to broadcast, streams `progress` frames | `training_api.rs:1778–1836` |
| Real route factory (has the missing route) | `routes()` incl. `/ws/train/progress` | `training_api.rs:1841–1849` |
| Background job spawn | `tokio::spawn` of the training task | `training_api.rs:1564`, spawn `:1610` |
| `mod training_api;` declaration | **None in the crate** (only a doc mention) | `path_safety.rs:9` |
| State-shape mismatch | expects `super::AppStateInner.{training_state, training_progress_tx}` | `training_api.rs:249`, `:232` |
| Real `AppStateInner` / `SharedState` | has neither field | `main.rs:1024`, `:1249` |
| Working training path | CLI `train-room` (ADR-151 specialist bank) | `room.rs:241` |
| Epoch metrics available to stream | `TrainMetrics.epoch`, `best_epoch` | `train/src/trainer.rs:43`, `:64` |

---

## 3. Gap analysis

| Capability | Desired | Today | Gap severity |
|---|---|---|---|
| `/ws/train/progress` resolves | 101 Switching Protocols, streams epoch/loss/eta | 404 (route absent) | **Critical** — the reported bug |
| "Start Training" produces a model | background job trains and writes `.rvf` | flag flip + one log line, no job, no model | **Critical** |
| Error surfaced to the user | button reflects real state / disabled with reason | silent no-op, `success:true` | **Critical** (slop) |
| In-server trainer compiled | part of the crate, unit-tested | orphaned; not compiled (no `mod`) | **High** |
| State supports progress streaming | `training_state` + `training_progress_tx` on `AppStateInner` | absent — orphan won't compile as-is | **High** |
| WS auth on the training surface | `/ws/train/progress` under bearer gate (ADR-166 §Sprint-1) | n/a (route absent) | **High** |
| `dataset_ids` path safety | validated before file open | `path_safety.rs` exists but unreached by live routes | **Medium** |
| Server ↔ CLI parity | shared/consistent training semantics | two divergent trainers (stub vs CLI vs orphan) | **Medium** |

---

## 4. Decision

**Chosen path: wire the existing in-server trainer into the live server** — reconcile
the state struct, declare the module, merge `training_api::routes()`, delete the
stub handlers, and expose a real `/ws/train/progress` that streams epoch/loss/eta
events from the already-implemented background job.

This is called **TRAIN-RECONNECT**.

### 4.1 Why this path, and not "make the button honestly say CLI-only"

The task framing offered two honest options. Investigation decided it:

| Consideration | Evidence | Implication |
|---|---|---|
| Is server-side training genuinely GPU/`tch`-bound (→ keep CLI-only)? | The in-server trainer is explicitly **`tch`-free**, pure Rust, exports `.rvf` (`training_api.rs:11–13`) | The "too heavy for Axum" argument is contradicted by the code |
| Does a real streaming implementation already exist? | Full pipeline + broadcast + WS handler + `routes()` present (`training_api.rs:1564,1778,1841`) | The impressive-sounding option is also the *least* new code — it already exists |
| Why does it 404 then? | No `mod training_api;`; state-shape mismatch (`:249` vs `main.rs:1024`) | The fix is reconnection + reconciliation, not new invention |

Because the honest, code-supported reality is "a working trainer was written and left
unplugged," the right decision is to plug it in — this is not choosing the flashier
option over the code; it *is* what the code says.

**However**, path B is retained as a **mandatory fallback guarantee** (Phase P5): if,
for a given build/deployment, server-side training is disabled (e.g. behind a
feature flag, or on the lightweight appliance image where recordings aren't
available), the dashboard button MUST be disabled with a tooltip pointing at
`wifi-densepose train-room` — never a silent `success:true` no-op again. The slop is
eliminated in both the enabled and disabled configurations.

### 4.2 Scope boundary — light trainer streams, heavy trainer proxies

- The **pure-Rust specialist trainer** (`training_api.rs`, ADR-151 flavour) runs
  in-process and streams live epoch/loss/eta over `/ws/train/progress`.
- **Heavy MAE/LoRA training** (`wifi-densepose-train`, `tch`/GPU) stays CLI-launched.
  The server does not host it; at most `/api/v1/train/status` reports on a
  CLI-launched run if one registers itself. Streaming heavy training is out of scope
  for this ADR (noted as an open question, §8).

---

## 5. Detailed design

### 5.1 Reconcile `AppStateInner`

Replace the two stub fields (`main.rs:1125–1127`) with the sub-state the trainer
expects, so `training_api.rs` compiles against `super::AppStateInner`:

```rust
// main.rs — inside AppStateInner (replacing training_status / training_config)
training_state: training_api::TrainingState,          // status, epoch, best_pck, task_handle
training_progress_tx: tokio::sync::broadcast::Sender<String>,  // progress fan-out
```

`train_status` consumers that read `s.training_status` / `s.training_config` are
updated to read `s.training_state.status`. The broadcast sender is created at state
init (`main.rs:7826` region, where the stubs are seeded today).

### 5.2 Declare and merge the module

- Add `mod training_api;` to `main.rs` (or `pub mod` in `lib.rs` if the router is
  assembled there).
- Delete the stub handlers `train_start` / `train_stop` / `train_status`
  (`main.rs:4977–5023`) and their three route mounts (`main.rs:8069–8071`).
- Merge the real router **after** `.with_state(state.clone())`, the same pattern the
  RuField surface already uses (`main.rs:8104–8111`):

```rust
// main.rs router assembly
.merge(training_api::routes())
```

so that `/api/v1/train/*` and `/ws/train/progress` resolve against the shared state.

### 5.3 Auth and safety (ADR-166 alignment)

- `/api/v1/train/*` sits under the existing opt-in bearer gate (`main.rs:8095–8102`,
  `RUVIEW_API_TOKEN`). `/ws/train/progress` follows the same policy decision made for
  `/ws/sensing` — document explicitly whether the training WS is gated (recommended:
  gated when a token is set, since training reads/writes recordings and models).
- `dataset_ids` from `StartTrainingRequest` (`training_api.rs:126–130`) are resolved
  through `path_safety` before any file open — `path_safety.rs:9` already anticipates
  `{dataset_id}.csi.jsonl` under `RECORDINGS_DIR`; wire it in the load path.
- Single-job concurrency guard: `start_training` already rejects a second run while
  `training_state.status.active` (`training_api.rs:1571`) — keep it.

### 5.4 Progress event schema (already emitted)

The WS handler already frames messages as `{"type":"status"|"progress", "data": …}`
(`training_api.rs:1796–1815`). Confirm the `data` payload carries at minimum
`epoch`, `total_epochs`, `loss`, `best_pck`, and an `eta_seconds`; these map onto the
`TrainMetrics`/`TrainingStatus` fields already populated by the loop
(`training_api.rs:1251`, `train/src/trainer.rs:43,64`).

### 5.5 Dashboard honesty (both configurations)

- **Enabled build:** button POSTs `/api/v1/train/start`, then opens
  `/ws/train/progress`; the UI renders live epoch/loss/eta and a terminal
  success/failure with the output `.rvf` path.
- **Disabled build:** `/api/v1/train/start` returns a structured
  `{"enabled": false, "reason": "...", "cli": "wifi-densepose train-room"}` and the
  button renders disabled with a tooltip — no silent `success:true`.

---

## 6. Phase ledger

```
P0 ──► P1 ──► P2 ──► P3 ──► P4 ──► P5 ──► P6
repro   state   wire    stream  auth+   dash    tests+
+audit  recon   router  job     safety  honesty witness
```

### P0 — Reproduce & audit (evidence lock)
- [ ] Repro the 404: run the server, `websocat ws://localhost:3000/ws/train/progress`,
      capture the 404 (baseline for the fix).
- [ ] Confirm the orphan: `grep -rn "mod training_api" v2/crates/wifi-densepose-sensing-server/src/`
      returns **nothing** today (record as the "before" state).
- [ ] Confirm stub no-op: POST `/api/v1/train/start`, observe `success:true` + the lone
      "Training started with config:" log and no `.rvf` written.

### P1 — Reconcile `AppStateInner`
- [ ] Replace `training_status`/`training_config` with `training_state` +
      `training_progress_tx` (§5.1).
- [ ] Update state init and any readers of the old fields.
- [ ] Add `mod training_api;` so the module compiles against the real state.

### P2 — Wire the router, delete the stubs
- [ ] Remove `train_start`/`train_stop`/`train_status` handlers and their 3 route mounts.
- [ ] `.merge(training_api::routes())` after `.with_state(...)`.
- [ ] `/api/v1/train/*` and `/ws/train/progress` resolve (no 404).

### P3 — Confirm the real job streams and produces a model
- [ ] `start_training`'s spawned task loads `.csi.jsonl`, runs the gradient-descent
      loop, and writes a `.rvf` under `data/models`.
- [ ] Progress frames carry `epoch`, `total_epochs`, `loss`, `best_pck`, `eta_seconds`.
- [ ] Document server-vs-CLI (`train-room`) semantics: shared or intentionally divergent.

### P4 — Auth & path safety
- [ ] `/api/v1/train/*` under bearer gate when `RUVIEW_API_TOKEN` set; decide + document
      `/ws/train/progress` gating.
- [ ] `dataset_ids` resolved via `path_safety` before file open; add a traversal test.
- [ ] Single-job guard verified (second start rejected with a clear error).

### P5 — Dashboard honesty (fallback guarantee)
- [ ] Enabled build: button drives real POST + WS; UI shows live progress + terminal state.
- [ ] Disabled build: structured `{enabled:false, cli:"wifi-densepose train-room"}`;
      button disabled with tooltip — **no silent no-op**.

### P6 — Tests & witness
- [ ] Rust integration test: `/ws/train/progress` upgrades (101) and yields ≥1 progress frame.
- [ ] Regression test: POST start → poll status → `.rvf` exists.
- [ ] Update CHANGELOG + README/CLAUDE if the route table or scope changed.

*(No item above is complete; this ADR is Proposed.)*

---

## 7. Acceptance criteria (concrete verification)

All must pass before ADR-186 is Accepted:

- [ ] **Orphan is reconnected:**
      `grep -rn "mod training_api" v2/crates/wifi-densepose-sensing-server/src/`
      returns a hit, and
      `cargo build -p wifi-densepose-sensing-server` **compiles** (proves the state
      reconciliation in §5.1 is correct — the module cannot compile against the
      current `AppStateInner`).
- [ ] **Route no longer 404s (HTTP upgrade):**
      ```bash
      curl -i -N \
        -H "Connection: Upgrade" -H "Upgrade: websocket" \
        -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZQ==" \
        http://localhost:3000/ws/train/progress
      ```
      returns `101 Switching Protocols`, **not** `404`.
- [ ] **Progress actually streams:**
      ```bash
      # terminal A
      websocat ws://localhost:3000/ws/train/progress
      # terminal B
      curl -s -X POST http://localhost:3000/api/v1/train/start \
        -H 'Content-Type: application/json' \
        -d '{"dataset_ids":["room-a"],"config":{"epochs":10}}'
      ```
      Terminal A receives at least one `{"type":"progress","data":{"epoch":…,"loss":…}}`
      frame within 10 s.
- [ ] **A model is produced:** after the run completes, a fresh `.rvf` exists under
      `data/models/` (`ls -t data/models/*.rvf | head -1` is newer than the run start).
- [ ] **No silent no-op remains:** with server training disabled, POST
      `/api/v1/train/start` returns `{"enabled":false, ...,"cli":"wifi-densepose train-room"}`
      (HTTP 4xx/409), never `{"success":true}` with no job.
- [ ] **Auth honored:** with `RUVIEW_API_TOKEN` set, an unauthenticated
      `POST /api/v1/train/start` is rejected; with the token it succeeds.
- [ ] **Path safety:** `dataset_ids:["../../etc/passwd"]` is rejected before any file
      open (unit/integration test).
- [ ] **Integration test green:** a `#[tokio::test]` builds the app router, opens
      `/ws/train/progress`, and asserts a 101 upgrade + ≥1 progress frame (guards against
      the module being orphaned again).
- [ ] **Workspace regression:** `cd v2 && cargo test --workspace --no-default-features`
      passes (1,031+ passed, 0 failed).

---

## 8. Consequences

### Positive
- Closes issue #1233: the dashboard button either trains-and-streams or honestly says
  "use the CLI" — the silent no-op is gone in every configuration.
- Reclaims 1,860 lines of already-written, already-committed trainer that were dead
  (uncompiled) code, and adds a test that keeps them wired.
- `/ws/train/progress` gives the UI real epoch/loss/eta, matching the maintainer's
  stated intent.
- Forces the state-shape reconciliation that the orphan implied but never landed,
  removing a latent "two competing training designs" trap in `AppStateInner`.

### Negative
- Editing `AppStateInner` (`main.rs:1024`) and the router (`main.rs:8068`) touches the
  large `main.rs`; merge-conflict risk with concurrent work on the same file (the
  ADR-166 decomposition is relevant here).
- Adds a live training code path to the server's attack surface — mitigated by the
  bearer gate and `path_safety`, but it must be reviewed (network/hardware boundary,
  per the pre-merge security checklist).
- Server and CLI now have two trainers that must be kept semantically consistent, or
  their divergence explicitly documented.

### Neutral
- Heavy MAE/LoRA/`tch` training remains CLI-only; the server streams only the
  light pure-Rust specialist trainer. Streaming heavy runs is deferred.
- The progress event schema (`epoch/loss/best_pck/eta`) is already emitted by the
  orphan; no new schema is invented, only confirmed and documented.

---

## 9. Open questions

1. **WS auth policy for `/ws/train/progress`:** gate it whenever `RUVIEW_API_TOKEN`
   is set (like `/api/v1/*`), or leave it open like `/ws/sensing`? *Tentative: gate
   it — training reads recordings and writes models.*
2. **Server ↔ CLI trainer parity:** should the in-server trainer and
   `wifi-densepose train-room` (ADR-151) share one code path, or remain deliberately
   separate (server = quick UI-driven specialist fit; CLI = full bank + geometry
   conditioning)? *Tentative: keep separate, document the split, share feature
   extraction where cheap.*
3. **Heavy-training progress:** can a CLI-launched `wifi-densepose-train` (`tch`)
   run register itself so `/api/v1/train/status` and the WS can report on it without
   hosting it in-process? *Tentative: out of scope here; a follow-on ADR.*
4. **Feature-flagging server training:** should in-server training be behind a Cargo
   feature (off on the lightweight appliance image), making the P5 disabled-button
   path the default there? *Tentative: yes — flag it; default the UI to the honest
   disabled state on images without recordings.*

---

## 10. References

- **Issue #1233**: https://github.com/ruvnet/wifi-densepose/issues/1233 — the reported bug.
- **Live stubs**: `v2/crates/wifi-densepose-sensing-server/src/main.rs:4977–5023` (handlers),
  `:8068–8071` (routes), `:1125–1127` (state fields), `:1024`/`:1249` (`AppStateInner`/`SharedState`).
- **Orphaned trainer**: `v2/crates/wifi-densepose-sensing-server/src/training_api.rs` —
  module doc `:1–25`, `TrainingState` `:232`, `AppState` alias `:249`, `start_training` `:1564`
  (spawn `:1610`), WS handler `:1778–1836`, `routes()` `:1841–1849`.
- **Not-a-module proof**: repo-wide `training_api` only in `path_safety.rs:9` (doc comment).
- **CLI working path**: `v2/crates/wifi-densepose-cli/src/room.rs:241` `train_room` (ADR-151).
- **Epoch metrics**: `v2/crates/wifi-densepose-train/src/trainer.rs:43`, `:64`.
- **ADR-166**: WebSocket authentication + `main.rs` decomposition (security context for this change).
- **ADR-151**: per-room calibration / `train-room` specialist bank.