mirror of
https://github.com/ruvnet/RuView
synced 2026-06-18 11:43:19 +00:00
Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 ruv
|
a4d5ea88f3 | ||
|
ruv
|
ebe217569b |
@@ -1,272 +0,0 @@
|
||||
# ADR-180: Through-Wall Camera↔CSI Hand-off Demo ("Behind the Wall")
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| **Status** | Proposed |
|
||||
| **Date** | 2026-06-15 |
|
||||
| **Deciders** | ruv |
|
||||
| **Codename** | **BEHIND-THE-WALL** |
|
||||
| **Builds on** | ADR-079 (camera ground-truth training), ADR-031 (sensing-first RF mode), ADR-134 (CSI→CIR multipath), ADR-029/030 (RuvSense multistatic + persistent field), ADR-024 (AETHER re-ID), ADR-151 (per-room calibration), ADR-173 (metric-locked PCK), ADR-095/096 (rvcsi nexmon) |
|
||||
|
||||
## Context
|
||||
|
||||
### The demo we want
|
||||
A single self-contained **HTML page** that tells one honest, visceral story:
|
||||
|
||||
1. You stand in front of the laptop. The camera tracks your **full skeletal pose**;
|
||||
the WiFi-CSI model, trained on *your* movements moments earlier, infers the **same
|
||||
skeleton** in parallel — a side-by-side "camera vs RF agree" view.
|
||||
2. You **walk out the door and behind the wall**. The camera **goes blind** (you are
|
||||
occluded — it honestly shows "no person in frame"). The CSI model **keeps inferring
|
||||
your skeleton** from the WiFi signal alone — the 3D figure keeps walking, behind the
|
||||
wall, smoothly. A badge flips from `CAMERA` to `RF-INFERRED (through-wall)`.
|
||||
3. You **walk back into view**. The camera **re-acquires**; the badge flips back to
|
||||
`CAMERA`, and the RF-inferred and camera skeletons reconverge.
|
||||
|
||||
This is the "WiFi sees through walls" demo — and the user explicitly wants the **inferred
|
||||
skeleton through the wall**, not just a blob. The project's "prove everything / no AI-slop"
|
||||
bar means we make that claim **only because we measure it**: a second camera on the far side
|
||||
of the wall records ground-truth pose *behind* the wall, so the through-wall skeleton's
|
||||
accuracy is a **reported, reproducible number** — never an unfalsifiable "trust me."
|
||||
|
||||
### Honest capability framing (the load-bearing section)
|
||||
Through-wall **per-joint skeletal inference from WiFi CSI is not a generally-validated
|
||||
capability** in open settings — WiFi-DensePose (CMU) is camera-*co-located*. What makes it
|
||||
defensible *here* is the tightly-controlled regime and the measurement:
|
||||
|
||||
- **Controlled regime:** one room, one subject (you), one doorway, a model **camera-supervised
|
||||
on your exact gait and your exact through-door transition** (ADR-079) minutes earlier. This
|
||||
is in-distribution for *this* demo, not a universal claim.
|
||||
- **Measured, not asserted:** a far-side camera (cognitum-v0 has 17 `/dev/video*` nodes — use
|
||||
one, or a phone) records ground-truth pose behind the wall. The through-wall CSI skeleton is
|
||||
scored against it with the metric-locked PCK harness (ADR-173). **We publish the number.**
|
||||
- **Uncertainty is rendered, not hidden:** the through-wall skeleton is drawn **translucent**,
|
||||
with a live **per-joint confidence** and an explicit `RF-INFERRED` badge. High-confidence
|
||||
joints render solid; low-confidence joints fade. It never masquerades as the camera's
|
||||
ground-truth pose.
|
||||
|
||||
| While… | Camera | WiFi CSI (S3 / Pi5 nexmon, fused) | 60 GHz mmWave (C6 + MR60BHA2) |
|
||||
|--------|--------|-----------------------------------|-------------------------------|
|
||||
| In frame | **Full 17-kpt pose** — ground truth | full skeleton (supervised model) — *agrees with camera* | presence + range + micro-motion |
|
||||
| Behind a **drywall** | nothing (occluded) | **inferred full skeleton** (camera-supervised model + multistatic fusion), confidence-scored, **measured vs far-side camera** | presence + range + breathing — independent through-thin-wall confirm |
|
||||
| Behind **brick/metal** | nothing | degrades to coarse motion/position only — report honestly | blocked |
|
||||
|
||||
**The claim — stated precisely:** *"A WiFi-CSI model, camera-supervised on this subject and
|
||||
room, infers a continuous skeletal pose that tracks the subject through a drywall partition;
|
||||
through-wall accuracy is measured at X% PCK@k against a far-side camera (declared, not
|
||||
claimed)."* If X turns out low, that is the **honest result we report** — the skeleton is still
|
||||
rendered (the user wants it) but flagged with its true confidence, and the headline number is
|
||||
whatever we measured, good or bad.
|
||||
|
||||
### Why multistatic + supervision is the enabler
|
||||
A single node behind a wall sees only "something moved." Three spatially-diverse vantage points
|
||||
around the doorway (RuvSense multistatic + cross-viewpoint fusion, ADR-029/030) triangulate the
|
||||
moving scatterer — drywall attenuates and diffracts 2.4/5 GHz but does not block it — giving the
|
||||
model a rich enough multipath signature to regress a skeleton it was *trained* to associate with
|
||||
your through-door motion. AETHER re-ID embeddings (ADR-024) keep it locked to **you** across the
|
||||
camera→RF→camera hand-off.
|
||||
|
||||
### Available hardware (the user's actual rig)
|
||||
| Role | Device | Where | Stream |
|
||||
|------|--------|-------|--------|
|
||||
| Near ground truth (visible) | Laptop / USB camera | front of workstation (ruvzen) | MediaPipe pose → keypoints |
|
||||
| **Far ground truth (validation)** | cognitum-v0 camera (1 of 17 `/dev/video*`) or a phone | **behind the wall** | MediaPipe pose → keypoints (for MEASURING the through-wall skeleton) |
|
||||
| CSI node A | ESP32-S3 (8 MB) | COM9 (ruvzen) | UDP CSI :5005 |
|
||||
| CSI + mmWave node B | ESP32-C6 + Seeed MR60BHA2 | COM12 (ruvzen) | WiFi CSI + 60 GHz FMCW presence/range |
|
||||
| CSI node C (through-wall vantage) | Pi 5, BCM43455c0 | cognitum-v0 (other room) | nexmon_csi `.pcap` → rvcsi → CsiFrame |
|
||||
| Fusion + serving | sensing-server | ruvzen :3000/:8765 | `/ws/sensing`, `/ws/pose`, new `/ws/handoff` |
|
||||
|
||||
Place **node C (Pi 5) and the far camera on the far side of the wall** — the Pi 5 gives the
|
||||
fuser a vantage the camera lacks, and the far camera turns the through-wall claim into a
|
||||
measurement.
|
||||
|
||||
## Decision
|
||||
|
||||
Build a **camera↔CSI hand-off demo** as a thin, additive layer over existing components (no new
|
||||
heavy crate). Five parts: a multi-source capture plane, a camera-supervised calibration walk
|
||||
that **learns to infer the skeleton through the wall**, a **hand-off state machine**, a
|
||||
**dead-reckoning smoother** so dropped CSI never makes the figure jump, and a single-file HTML
|
||||
viewer that renders the inferred skeleton with honest confidence.
|
||||
|
||||
### 1. Capture plane (reuse, don't rebuild)
|
||||
- **Near camera:** `scripts/collect-ground-truth.py` already does MediaPipe pose + ESP32 CSI
|
||||
paired capture (ADR-079). Extend it to also subscribe to the Pi 5 nexmon stream (rvcsi), the
|
||||
C6 mmWave presence, **and the far camera**, so every frame is
|
||||
`(near_pose|null, far_pose|null, csi_S3, csi_C6, mmwave_C6, csi_Pi5, t)`.
|
||||
- **CSI nodes:** S3 over UDP :5005, Pi 5 via `rvcsi` (vendor/rvcsi nexmon adapter → `CsiFrame`),
|
||||
C6 WiFi CSI + the MR60BHA2 60 GHz presence/range/breathing.
|
||||
- **Fusion:** all CSI sources into the existing `MultistaticFuser`
|
||||
(`signal/src/ruvsense/multistatic.rs`); node positions around the doorway via
|
||||
`--node-positions` (geometric-diversity index drives confidence). **#1049:** with 3
|
||||
independently-clocked nodes set `WDP_GUARD_INTERVAL_US` to the real inter-node spread or
|
||||
fusion demotes.
|
||||
|
||||
### 2. Calibration walk — "it learns my movements **and infers them through the wall**" (ADR-079)
|
||||
A 3–5 minute guided routine. The HTML page scripts the walk: stand, step left/right, walk to the
|
||||
door, **cross fully behind the wall and back**, repeat — covering the visible AND the occluded
|
||||
zone, because **both cameras label ground truth**:
|
||||
- **Visible-zone supervision:** near camera labels pose; synchronized CSI window is the input.
|
||||
- **Through-wall supervision (the key part):** while you are behind the wall, the **far camera**
|
||||
labels your pose. So the CSI→skeleton model is trained on *real behind-wall poses* paired with
|
||||
the *behind-wall multistatic CSI* — the model genuinely learns to infer your skeleton through
|
||||
the wall, supervised by ground truth, not extrapolated blindly.
|
||||
- Train/fine-tune on `ruvultra` (RTX 5080) if available, else the local recipe. Persist as a
|
||||
per-room calibration bank (ADR-151 `baseline → enroll → extract → train`). AETHER re-ID
|
||||
embeddings (ADR-024) bind the track to you across the hand-off.
|
||||
- **Held-out split:** reserve some behind-wall passes for evaluation so through-wall PCK is
|
||||
measured on data the model never trained on (no leakage — the ADR-152 measurement discipline).
|
||||
|
||||
### 3. Hand-off state machine (`sensing-server/src/handoff.rs`, < 300 lines)
|
||||
States: `CAMERA` → `HANDOFF_OUT` → `RF_INFERRED` → `HANDOFF_IN` → `CAMERA` (+ `LOST`).
|
||||
- **`CAMERA`** — near camera has a confident pose → render it; RF-inferred skeleton ghosted
|
||||
alongside for the "they agree" effect.
|
||||
- **`HANDOFF_OUT`** — near-camera confidence drops at the doorway **while** CSI motion stays high
|
||||
and the multistatic track heads into the door zone → cross-fade source camera→RF.
|
||||
- **`RF_INFERRED`** — no camera pose; the CSI model emits a **full 17-kpt skeleton** + per-joint
|
||||
confidence; AETHER confirms it is still you. Render the translucent skeleton + confidence,
|
||||
badge `RF-INFERRED (through-wall)`. (When fusion confidence is too low for a credible skeleton,
|
||||
degrade gracefully to a coarse marker rather than a flailing one — honest fallback.)
|
||||
- **`HANDOFF_IN`** — near camera re-acquires a pose positionally consistent with the last RF
|
||||
skeleton (continuity gate) → cross-fade RF→camera.
|
||||
- **`LOST`** — neither source for N cycles → "no track," never invented.
|
||||
|
||||
Fail-closed: `RF_INFERRED` requires real multistatic motion energy + an AETHER identity match
|
||||
above calibrated floors; absent that → `LOST`, never a phantom. Mirrors the governed-trust gate
|
||||
(ADR-031 / ADR-141).
|
||||
|
||||
### 4. Dead reckoning & smoothing — fluid, never jumpy (the user's requirement)
|
||||
CSI does **not** arrive cleanly: UDP frames drop, nexmon `.pcap` has gaps, the fuser skips
|
||||
cycles when the #1049 guard rejects a spread, and the model's per-frame skeleton jitters. Render
|
||||
only on real frames and the figure teleports and shakes — which also *reads as fake*. A
|
||||
**predict/correct (dead-reckoning) layer** keeps the skeleton continuous and smooth between
|
||||
measurements, with **bounded** extrapolation so we never invent motion that didn't happen:
|
||||
|
||||
- **Per-joint constant-velocity Kalman filter** — reuse `signal/src/ruvsense/pose_tracker.rs`
|
||||
(the project's existing 17-keypoint Kalman tracker with AETHER re-ID). The renderer runs at a
|
||||
**fixed ~30 Hz, decoupled from CSI arrival**:
|
||||
- **Measurement this tick** → Kalman *update* (correct) each joint with the new inferred pose.
|
||||
- **Dropped CSI this tick** → Kalman *predict* only: advance each joint by `x += v·dt`, so the
|
||||
skeleton keeps moving along its trajectory instead of freezing then snapping. **This is the
|
||||
dead reckoning** — the limbs keep their motion through a dropout.
|
||||
- **Confidence decay (honesty governor):** every predict-only tick multiplies confidence and
|
||||
widens covariance. Dead reckoning is trusted for a **bounded** horizon (default ≤ ~500 ms,
|
||||
`WDP_DEADRECKON_MAX_MS`); past it, confidence hits the floor → state machine → `LOST`. **We
|
||||
coast briefly to stay smooth; we never coast forever to fake a track.** Someone who actually
|
||||
stopped behind the wall converges to a still pose then `LOST`, not perpetual phantom walking.
|
||||
- **Re-acquire smoothing:** a returning measurement after a gap is blended in with a
|
||||
critically-damped step (no overshoot) over 2–3 ticks, so the skeleton eases onto truth.
|
||||
- **Client render smoothing (already present):** `ui/observatory/js/figure-pool.js`
|
||||
`applyKeypoints` already `lerp`s joints with a small velocity overshoot for secondary motion;
|
||||
the hand-off viewer reuses it. The camera↔RF cross-fade is an alpha-lerp over ~300 ms.
|
||||
|
||||
**Dead-reckoning honesty invariants (testable):**
|
||||
1. Predicted-only frames carry `"dead_reckoned": true` + `"age_ms"`; the UI dims them —
|
||||
extrapolation is never shown as a fresh measurement.
|
||||
2. Confidence is **monotonically non-increasing** across consecutive predict-only ticks.
|
||||
3. After `WDP_DEADRECKON_MAX_MS` of silence the state **must** become `LOST` (pinned test:
|
||||
measurements then silence → assert transition within the horizon; no perpetual motion).
|
||||
4. Dead reckoning extrapolates an **existing** track only — no measurement ever ⇒ no track ⇒
|
||||
`LOST`, never a phantom from zero.
|
||||
|
||||
### 5. The HTML demo (single file, vanilla — mirrors the Observatory)
|
||||
`ui/through-wall/index.html` (+ a small JS bundle, zero build step, like `ui/observatory/`):
|
||||
- **Left:** near camera feed with the MediaPipe skeleton overlaid while visible; greys to
|
||||
"CAMERA BLIND" when occluded. (Optional second tile: the far camera, shown only in a
|
||||
"validation" view, not the hero view.)
|
||||
- **Right:** a top-down 3D room (Three.js) with the **wall** drawn, the doorway, the three
|
||||
sensor positions, and the figure: a **solid skeleton** in `CAMERA`, a **translucent skeleton
|
||||
with per-joint confidence fade** in `RF_INFERRED`, eased by the dead-reckoning smoother.
|
||||
- **Banner / `BannerState`** (strict, mirrors rufield-viewer): `CAMERA` / `RF-INFERRED — through
|
||||
wall (conf X%, measured Y% PCK@k)` / `DEAD-RECKONED (age N ms)` / `LOST` — mutually exclusive,
|
||||
with a one-line honesty caption. The measured through-wall PCK is shown, not invented.
|
||||
- Consumes a new `GET /ws/handoff` WS/SSE topic of `HandoffFrame`s; `?demo=1` replays a recorded
|
||||
session badged `REPLAY`.
|
||||
|
||||
### Output contract (`HandoffFrame`, JSON)
|
||||
```jsonc
|
||||
{
|
||||
"t_ns": 1718400000000,
|
||||
"state": "RF_INFERRED", // CAMERA | HANDOFF_OUT | RF_INFERRED | HANDOFF_IN | LOST
|
||||
"source": "fused_csi", // camera | fused_csi | mmwave | dead_reckoned
|
||||
"pose": [[x,y,z,conf], …×17], // inferred skeleton WITH per-joint confidence (present in CAMERA/HANDOFF/RF_INFERRED)
|
||||
"pose_confidence": 0.58, // aggregate; the rendered translucency
|
||||
"identity_match": 0.81, // AETHER re-ID — is it still you?
|
||||
"coarse": { "cell":[x,y], "zone":"behind_wall", "heading_deg":95, "node_diversity":0.48 },
|
||||
"dead_reckoned": false, // true on predict-only (extrapolated) ticks
|
||||
"age_ms": 0, // ms since the last real measurement (0 = fresh)
|
||||
"camera_blind": true,
|
||||
"measured_pck": { "k": 20, "value": null }, // filled from the far-camera validation run; null until measured
|
||||
"caption": "RF-inferred skeleton — model camera-supervised on this room; through-wall PCK measured separately"
|
||||
}
|
||||
```
|
||||
|
||||
## Phased plan (each phase independently demoable + falsifiable)
|
||||
- **P1 — wiring (no claim):** 3-source CSI capture (S3+C6+Pi5) + near camera into the multistatic
|
||||
fuser. Gate: `/ws/sensing` shows ≥3 active nodes + a fused position with the camera running.
|
||||
- **P2 — supervised calibration + through-wall training:** the guided walk with **both cameras**;
|
||||
fine-tune CSI→skeleton on visible AND far-camera-labeled behind-wall poses (ADR-079). Gate:
|
||||
while-visible PCK declared (metric-locked, ADR-173) on a held-out segment.
|
||||
- **P3 — MEASURE the through-wall skeleton:** score the RF-inferred skeleton against the far
|
||||
camera on held-out behind-wall passes → **publish the through-wall PCK@k** (good or bad). Gate:
|
||||
a committed eval script reproduces the number; honest negative if low.
|
||||
- **P4 — hand-off + dead reckoning + HTML:** the camera→RF→camera transition renders end-to-end,
|
||||
smooth through dropped CSI. Gate: a recorded live walk where the camera goes blind, the inferred
|
||||
skeleton keeps walking fluidly behind the wall, dead-reckons through dropouts without jumps, and
|
||||
re-acquisition is position-continuous. **This is the demo.**
|
||||
- **P5 — multi-modal corroboration (optional):** overlay C6 60 GHz presence/range as an
|
||||
independent through-thin-wall confirm (two physics, one conclusion).
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- A genuinely compelling demo that does what the user asked — **infers and renders the skeleton
|
||||
through the wall** — while staying honest because the through-wall accuracy is **measured**
|
||||
against a far-side camera, not claimed. Reuses the multistatic fuser, ADR-079 supervision, the
|
||||
Kalman pose tracker, AETHER re-ID, the calibration crate, and the Observatory UI: the new code
|
||||
is a hand-off module + dead-reckoning smoother + an HTML page.
|
||||
|
||||
### Negative / Risks
|
||||
- **Through-wall skeletal accuracy may be modest or poor.** That is acceptable *iff* reported
|
||||
honestly — the headline is the measured PCK, whatever it is; the skeleton renders with its true
|
||||
per-joint confidence (low-confidence joints fade), never as fake certainty.
|
||||
- **Material dependence:** drywall good; brick/metal degrades to coarse-only — shoot on drywall
|
||||
and say so.
|
||||
- **3-node clock sync** is the #1049 hazard — tune `WDP_GUARD_INTERVAL_US`.
|
||||
- **Per-room, per-subject:** the model that "learned your movements" does not transfer without
|
||||
re-calibration — stated on the page.
|
||||
- **Over-claiming is the failure mode.** Mitigations baked in: translucent confidence-faded
|
||||
skeleton, `dead_reckoned`/`age_ms` flags, the measured-PCK banner, bounded extrapolation→`LOST`.
|
||||
|
||||
### Neutral
|
||||
- No new heavy crate; signal-path proof (`verify.py`) untouched — capture/fusion/UI orchestration
|
||||
over hardened, already-reviewed components.
|
||||
|
||||
## Acceptance criteria (falsifiable — "prove the haters wrong")
|
||||
On a recorded live session, all must hold:
|
||||
1. A contiguous window where the **near camera reports no person** (verifiable from raw frames)
|
||||
**and** the system renders an `RF_INFERRED` skeleton.
|
||||
2. The inferred skeleton's **gross motion matches reality** — direction of travel and rough gait
|
||||
phase — confirmed against the **far camera** (not eyeballed).
|
||||
3. **Through-wall per-joint accuracy is MEASURED** against the far camera and **reported** as
|
||||
PCK@k from a committed script. Low is fine *if* honestly published; fabricated is not.
|
||||
4. The figure is **smooth through dropped CSI** — no teleports/jitter — and every predicted-only
|
||||
frame is flagged `dead_reckoned`; after `WDP_DEADRECKON_MAX_MS` of silence it goes `LOST`.
|
||||
5. Re-acquisition is **position-continuous** (camera re-detects within a cell of the last RF
|
||||
position), and AETHER confirms identity across the hand-off.
|
||||
6. Every number (visible PCK, through-wall PCK, confidences) is MEASURED and reproducible — no
|
||||
hand-typed metrics.
|
||||
|
||||
A demo that cannot meet (1)–(2) and (4)–(5) on the available hardware is reported as a **negative
|
||||
result** (honest), not dressed up; a poor (3) is published as the real number.
|
||||
|
||||
## Links
|
||||
- ADR-079 — camera ground-truth training (supervision pipeline; extended here to a far camera)
|
||||
- ADR-031 — sensing-first RF mode / coherence gate (fail-closed honesty pattern)
|
||||
- ADR-134 — CSI→CIR multipath (through-wall multipath physics)
|
||||
- ADR-029 / ADR-030 — RuvSense multistatic + persistent field (the localization engine)
|
||||
- ADR-024 — AETHER contrastive re-ID (identity lock across the hand-off)
|
||||
- ADR-151 — per-room calibration crate (bank persistence)
|
||||
- ADR-152 / ADR-173 — measurement discipline + metric-locked PCK (the honest accuracy readout)
|
||||
- ADR-095 / ADR-096 — rvcsi nexmon (Pi 5 BCM43455c0 capture)
|
||||
- `signal/src/ruvsense/pose_tracker.rs` — 17-kpt Kalman tracker reused for dead reckoning
|
||||
- `ui/observatory/` — the vanilla-JS 3D viewer pattern this demo mirrors
|
||||
@@ -0,0 +1,135 @@
|
||||
# WiFlow Browser Trainer (`wiflow_browser.html`)
|
||||
|
||||
A **single self-contained HTML page** that does the entire camera-supervised
|
||||
WiFi-pose loop **in your browser, in your laptop camera's coordinate frame**, as
|
||||
a **4-stage gated flow** with a progress stepper (each stage unlocks the next):
|
||||
|
||||
0. **CALIBRATE** *(ADR-151 empty-room baseline)* — you step OUT of the space; the
|
||||
page captures ~10 s of the quiescent CSI and computes a per-feature running
|
||||
**mean + std (Welford)** over the 410-d vector. Every CSI vector afterwards is
|
||||
expressed as **deviation from baseline**
|
||||
(`x_norm = (x − base_mean) / (base_std + ε)`), so a body's perturbation stands
|
||||
out from the static channel. Persisted to IndexedDB. *Can't capture without it.*
|
||||
1. **CAPTURE** — MediaPipe Pose runs on your laptop camera → 17 COCO keypoints
|
||||
(the *label*), paired with the **baseline-normalized** 410-d ESP32 CSI vector
|
||||
(the *input*). A **guided, balanced routine** cycles big on-screen prompts
|
||||
(stand / turn / walk / arms / crouch / sit / reach) with a countdown, and a
|
||||
**per-pose coverage meter** so you build a balanced dataset, not 2 000 frames
|
||||
of standing.
|
||||
2. **TRAIN** — a TensorFlow.js MLP learns `CSI → pose` in-browser. Honest
|
||||
held-out PCK@0.10 / PCK@0.05 / MPJPE, plus a **mean-pose baseline** the model
|
||||
must beat (the project's whole ethos — no baseline-beating signal, it says so).
|
||||
*Can't train with <200 samples.*
|
||||
3. **INFER** — the trained model drives a skeleton **from WiFi CSI only**
|
||||
(baseline-normalized → standardized → model), drawn over the **same** camera
|
||||
frame it trained in — so the inferred skeleton **aligns** with the camera
|
||||
image. That alignment is the entire point of doing this in-browser instead of
|
||||
with a separate Python camera. *Can't infer without a model.*
|
||||
|
||||
## Why in-browser
|
||||
|
||||
The Python pipeline (`wiflow_capture.py` → `wiflow_train.py` → `wiflow_infer.py`)
|
||||
proved the signal is real (held-out PCK@0.10 ≈ 59.5% vs a 50% mean-pose baseline
|
||||
= +9.4 pp). But it trained in a *different* camera's frame, so the inferred
|
||||
skeleton never lined up with the laptop camera. Doing capture + train + infer all
|
||||
in the browser with the **same** camera makes the training frame and the
|
||||
inference frame identical → the skeleton aligns.
|
||||
|
||||
## Compute backends (WebGPU / WASM / WebGL)
|
||||
|
||||
Training and inference run on TensorFlow.js. The page selects the backend at
|
||||
startup, preferring the fastest available:
|
||||
|
||||
- **WebGPU** (Chrome / Edge, secure context — `localhost` qualifies) — GPU compute.
|
||||
- **WASM-SIMD** fallback (`tfjs-backend-wasm`, SIMD enabled, `.wasm` from the CDN).
|
||||
- **WebGL** last-resort fallback (ships inside tfjs core).
|
||||
|
||||
The **active backend is shown as a badge in the header** (`compute: WebGPU` /
|
||||
`WASM-SIMD` / `WebGL`) so it's honest about what's actually running. The model
|
||||
code is backend-agnostic — tf.js abstracts the device.
|
||||
|
||||
## Honesty (baked in)
|
||||
|
||||
- The **CAPTURE** skeleton (blue) is the camera = ground truth, labeled as such.
|
||||
- The **INFER** skeleton (green) is **CSI-only**, labeled, and **coarse** — the
|
||||
real measured held-out PCK is shown, not a marketing number.
|
||||
- The **mean-pose baseline** is always computed and shown in TRAIN; the verdict
|
||||
states plainly whether the model **beats** it (real signal) or **does not**
|
||||
(no usable signal). This guards against the project's retracted 92.9% that
|
||||
failed exactly this check.
|
||||
- Status banner is strict and mutually exclusive:
|
||||
**LIVE** (real `source: "esp32"`) / **SIMULATED — not real** (any other source)
|
||||
/ **NO-CSI-SERVER**. The page never invents frames.
|
||||
|
||||
## How to run
|
||||
|
||||
### 1. Start the real sensing-server (provides the CSI WebSocket on :8765)
|
||||
|
||||
```bash
|
||||
cd v2
|
||||
cargo build -p wifi-densepose-sensing-server
|
||||
./target/debug/sensing-server.exe --ws-port 8765 --udp-port 5005
|
||||
```
|
||||
|
||||
A real ESP32-S3 must be provisioned and streaming for `source` to read `esp32`
|
||||
(see `CLAUDE.local.md` for the firmware build/provision steps). The page expects
|
||||
the verified live endpoint **`ws://localhost:8765/ws/sensing`** with
|
||||
`source:"esp32"`, nodes `[9, 13]`, `features.*`, `node_features[].features.*`,
|
||||
and `signal_field.values` (400 floats).
|
||||
|
||||
### 2. Serve this page over localhost (camera + WebGPU need a localhost/secure origin)
|
||||
|
||||
Any static localhost server works. For example:
|
||||
|
||||
```bash
|
||||
python -m http.server 8099
|
||||
# then open: http://localhost:8099/examples/through-wall/wiflow_browser.html
|
||||
```
|
||||
|
||||
(8099 is just the static file server — 8765 is a separate process, the CSI
|
||||
WebSocket.) Allow camera access when the browser prompts.
|
||||
|
||||
Point at a CSI server on another host with `?ws=`:
|
||||
|
||||
```
|
||||
http://localhost:8099/examples/through-wall/wiflow_browser.html?ws=ws://192.168.1.20:8765/ws/sensing
|
||||
```
|
||||
|
||||
### 3. Use it
|
||||
|
||||
1. **CAPTURE** tab → *enable laptop camera* → *start recording*. Follow the guided
|
||||
routine (stand / turn / walk / arms / crouch / sit). A pair is stored only when
|
||||
a confident pose AND a fresh live `esp32` CSI frame coexist. Aim for a few
|
||||
thousand samples. Samples persist in IndexedDB across refreshes.
|
||||
2. **TRAIN** tab → *train model*. Watch the live loss curve, held-out PCK, and the
|
||||
baseline verdict. The model saves to IndexedDB.
|
||||
3. **INFER** tab → the green skeleton is now driven by WiFi CSI only, aligned over
|
||||
your camera. Toggle *hide camera* to see the CSI-only skeleton on black.
|
||||
|
||||
## The 410-d CSI vector (matches the Python pipeline exactly)
|
||||
|
||||
```
|
||||
[ mean_rssi, variance, motion_band_power, breathing_band_power ] # 4 (features.*)
|
||||
+ for node 9 then node 13: [ mean_rssi, variance, motion_band_power ] # 6 (node_features[].features.*)
|
||||
+ signal_field.values, padded / truncated to 400 # 400
|
||||
= 410-d
|
||||
```
|
||||
|
||||
Verified against a real live frame: the in-browser `csiVector()` produces the
|
||||
identical 410 vector as `wiflow_capture.py`'s `csi_vector()` (node 9 first, then
|
||||
node 13; field zero-padded).
|
||||
|
||||
## Libraries (CDN only, no bundler)
|
||||
|
||||
| Library | CDN |
|
||||
|---|---|
|
||||
| TensorFlow.js core | `@tensorflow/tfjs@4.22.0/dist/tf.min.js` |
|
||||
| TF.js WebGPU backend | `@tensorflow/tfjs-backend-webgpu@4.22.0/dist/tf-backend-webgpu.min.js` |
|
||||
| TF.js WASM backend | `@tensorflow/tfjs-backend-wasm@4.22.0/dist/tf-backend-wasm.min.js` |
|
||||
| MediaPipe Pose 0.5 (legacy solutions) | `@mediapipe/pose@0.5/pose.js` |
|
||||
|
||||
## Scope / honesty caveats
|
||||
|
||||
Same person, same room, same session. **Not** validated cross-day, cross-room, or
|
||||
through-wall. The inferred pose is coarse (PCK@0.05 is typically weak). If the
|
||||
model does not beat the mean-pose baseline, the page says so — that is a feature.
|
||||
@@ -0,0 +1,644 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>RuView · Through-Wall WiFi Sensing · LIVE CSI (no skeleton, no simulation)</title>
|
||||
<!--
|
||||
THROUGH-WALL WiFi-CSI SENSING DEMO — honest, real-data-only.
|
||||
|
||||
Renders ONLY what the running sensing-server actually streams over
|
||||
ws://localhost:8765/ws/sensing :
|
||||
- the 20x20 `signal_field` floor heatmap (real values)
|
||||
- a coarse RF-localization puck from persons[0].position (NOT pose)
|
||||
- live motion / presence / rssi / confidence meters
|
||||
- the real `source` ("esp32" = LIVE) verbatim in the banner
|
||||
|
||||
It deliberately does NOT draw a skeleton. The server's
|
||||
persons[].keypoints carry confidence:0.0 (image-pixel garbage, not
|
||||
real 3D joints) so we never render them. WiFi CSI gives
|
||||
motion/presence/coarse-position — that is the honest wow, and it
|
||||
penetrates drywall. See README.md.
|
||||
-->
|
||||
<style>
|
||||
:root {
|
||||
--bg: #050507; --bg-panel: rgba(8,10,14,0.80);
|
||||
--amber: #ffb840; --amber-hot: #ffe09f;
|
||||
--cyan: #4cf; --magenta: #ff4cc8;
|
||||
--text: #d8c69a; --text-mute: #6b6155;
|
||||
--green: #4f4; --red: #f64;
|
||||
--border: rgba(255,184,64,0.18);
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0; background: var(--bg); color: var(--text); overflow: hidden;
|
||||
font-family: 'SF Mono', 'Cascadia Code', Consolas, monospace;
|
||||
-webkit-font-smoothing: antialiased; font-size: 12px;
|
||||
}
|
||||
canvas { display: block; }
|
||||
.overlay-frame {
|
||||
position: fixed; inset: 0; pointer-events: none; z-index: 5;
|
||||
background:
|
||||
radial-gradient(ellipse at center, transparent 55%, rgba(0,0,0,0.55) 100%),
|
||||
linear-gradient(180deg, rgba(0,0,0,0.32) 0%, transparent 18%, transparent 82%, rgba(0,0,0,0.38) 100%);
|
||||
}
|
||||
.scanlines {
|
||||
position: fixed; inset: 0; pointer-events: none; z-index: 6;
|
||||
background: repeating-linear-gradient(0deg, rgba(0,0,0,0.04) 0px, rgba(0,0,0,0.04) 1px, transparent 1px, transparent 3px);
|
||||
mix-blend-mode: overlay; opacity: 0.5;
|
||||
}
|
||||
.panel {
|
||||
position: absolute; background: var(--bg-panel); border: 1px solid var(--border);
|
||||
border-radius: 4px; padding: 12px 14px; backdrop-filter: blur(8px);
|
||||
box-shadow: 0 1px 0 rgba(255,184,64,0.04), 0 8px 32px rgba(0,0,0,0.55); z-index: 10;
|
||||
}
|
||||
.panel h2 {
|
||||
margin: 0 0 8px 0; font-size: 10px; text-transform: uppercase; letter-spacing: 2px;
|
||||
color: var(--amber); font-weight: 600; border-bottom: 1px solid var(--border); padding-bottom: 6px;
|
||||
}
|
||||
|
||||
/* ---- Honest status banner (top-center, mutually exclusive states) ---- */
|
||||
#banner {
|
||||
position: fixed; top: 0; left: 0; right: 0; z-index: 30;
|
||||
text-align: center; padding: 7px 12px; font-size: 12px; letter-spacing: 1px;
|
||||
font-weight: 600; border-bottom: 1px solid rgba(0,0,0,0.4);
|
||||
transition: background 0.3s, color 0.3s;
|
||||
}
|
||||
#banner.live { background: rgba(40,255,80,0.12); color: var(--green); border-bottom-color: rgba(80,255,120,0.4); }
|
||||
#banner.sim { background: rgba(255,120,40,0.16); color: #ffae5a; border-bottom-color: rgba(255,140,60,0.5); }
|
||||
#banner.noserver { background: rgba(255,80,80,0.16); color: var(--red); border-bottom-color: rgba(255,90,90,0.5); }
|
||||
#banner .src { opacity: 0.8; font-weight: 400; }
|
||||
#banner-caption {
|
||||
position: fixed; top: 30px; left: 0; right: 0; z-index: 29;
|
||||
text-align: center; font-size: 10px; color: var(--text-mute); letter-spacing: 0.5px;
|
||||
pointer-events: none; padding-top: 2px;
|
||||
}
|
||||
|
||||
#info { top: 64px; left: 20px; min-width: 270px; }
|
||||
#info h1 { margin: 0 0 1px 0; font-size: 13px; letter-spacing: 1px; color: var(--amber-hot); font-weight: 600; }
|
||||
#info .sub { font-size: 10px; color: var(--text-mute); letter-spacing: 0.5px; margin-bottom: 10px; padding-bottom: 8px; border-bottom: 1px solid var(--border); }
|
||||
#info .row { display: flex; justify-content: space-between; gap: 12px; padding: 2px 0; }
|
||||
#info .row .k { color: var(--text-mute); font-size: 11px; }
|
||||
#info .row .v { color: var(--text); font-variant-numeric: tabular-nums; font-size: 11px; }
|
||||
#info .row .v.amber { color: var(--amber); }
|
||||
#info .row .v.cyan { color: var(--cyan); }
|
||||
#info .row .v.green { color: var(--green); }
|
||||
#info .row .v.red { color: var(--red); }
|
||||
#info .row .v.mag { color: var(--magenta); }
|
||||
#info .row .v.mute { color: var(--text-mute); }
|
||||
|
||||
#csi { top: 64px; right: 20px; min-width: 270px; }
|
||||
#csi .bar-row { display: flex; align-items: center; gap: 8px; padding: 3px 0; font-size: 10px; }
|
||||
#csi .bar-row .label { width: 86px; color: var(--text-mute); }
|
||||
#csi .bar-row .bar-track { flex: 1; height: 6px; background: rgba(255,184,64,0.08); border-radius: 2px; overflow: hidden; }
|
||||
#csi .bar-row .bar-fill {
|
||||
height: 100%; background: linear-gradient(90deg, var(--amber-hot), var(--amber));
|
||||
box-shadow: 0 0 6px var(--amber); transition: width 0.1s linear;
|
||||
}
|
||||
#csi .bar-row .val { width: 44px; text-align: right; color: var(--amber); font-variant-numeric: tabular-nums; }
|
||||
#csi .spark { margin-top: 8px; }
|
||||
#csi canvas { width: 100%; height: 38px; display: block; border: 1px solid var(--border); border-radius: 3px; background: rgba(0,0,0,0.3); }
|
||||
#csi .legend { margin-top: 8px; padding-top: 8px; border-top: 1px solid var(--border); font-size: 10px; color: var(--text-mute); line-height: 1.5; }
|
||||
|
||||
/* ---- waiting / no-server overlay ---- */
|
||||
#waiting {
|
||||
position: fixed; inset: 0; z-index: 25; display: none;
|
||||
flex-direction: column; align-items: center; justify-content: center;
|
||||
background: rgba(5,5,7,0.94); color: var(--amber); text-align: center; padding: 24px;
|
||||
}
|
||||
#waiting.show { display: flex; }
|
||||
#waiting .big { font-size: 22px; letter-spacing: 2px; color: var(--red); margin-bottom: 16px; text-transform: uppercase; }
|
||||
#waiting code {
|
||||
display: block; text-align: left; max-width: 640px; margin: 8px auto;
|
||||
background: rgba(255,184,64,0.06); border: 1px solid var(--border); border-radius: 4px;
|
||||
padding: 10px 14px; color: var(--amber-hot); font-size: 12px; white-space: pre-wrap;
|
||||
}
|
||||
#waiting .pulse { animation: pulse 1.4s ease-in-out infinite; }
|
||||
@keyframes pulse { 0%,100% { opacity: 0.55; } 50% { opacity: 1; } }
|
||||
|
||||
/* ---- optional webcam ground-truth tile ---- */
|
||||
#cam-tile {
|
||||
position: absolute; bottom: 20px; right: 20px; width: 240px; z-index: 12;
|
||||
background: var(--bg-panel); border: 1px solid var(--border); border-radius: 4px;
|
||||
padding: 8px; backdrop-filter: blur(8px);
|
||||
}
|
||||
#cam-tile h2 { margin: 0 0 6px 0; font-size: 9px; text-transform: uppercase; letter-spacing: 1.5px;
|
||||
color: var(--cyan); font-weight: 600; }
|
||||
#cam-tile .gt-note { font-size: 9px; color: var(--text-mute); margin-top: 4px; line-height: 1.4; }
|
||||
#cam-video { width: 100%; border-radius: 3px; display: none; background: #000; }
|
||||
#cam-tile button {
|
||||
width: 100%; margin-top: 6px; padding: 5px 8px; font-family: inherit; font-size: 11px;
|
||||
background: transparent; color: var(--cyan); border: 1px solid var(--cyan); border-radius: 3px; cursor: pointer;
|
||||
}
|
||||
#cam-tile button:hover { background: rgba(68,204,255,0.12); }
|
||||
#cam-tile button:disabled { opacity: 0.5; cursor: not-allowed; }
|
||||
|
||||
#legend-nodes {
|
||||
position: absolute; bottom: 20px; left: 20px; min-width: 220px;
|
||||
background: var(--bg-panel); border: 1px solid var(--border); border-radius: 4px;
|
||||
padding: 12px 14px; backdrop-filter: blur(8px); z-index: 10;
|
||||
}
|
||||
#legend-nodes h2 { margin: 0 0 8px 0; font-size: 10px; text-transform: uppercase; letter-spacing: 2px;
|
||||
color: var(--amber); font-weight: 600; border-bottom: 1px solid var(--border); padding-bottom: 6px; }
|
||||
#legend-nodes .lr { display: flex; align-items: center; gap: 8px; padding: 2px 0; font-size: 11px; }
|
||||
#legend-nodes .dot { width: 9px; height: 9px; border-radius: 50%; box-shadow: 0 0 6px currentColor; flex: 0 0 auto; }
|
||||
#legend-nodes .muted { color: var(--text-mute); }
|
||||
</style>
|
||||
|
||||
<!-- three.js r128 + addons (same CDN set as examples/three.js/demos/05) -->
|
||||
<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/controls/OrbitControls.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/postprocessing/EffectComposer.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/postprocessing/RenderPass.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/postprocessing/ShaderPass.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/postprocessing/UnrealBloomPass.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/shaders/CopyShader.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/shaders/LuminosityHighPassShader.js"></script>
|
||||
</head>
|
||||
<body>
|
||||
<div id="banner" class="noserver">NO SERVER — start the sensing-server <span class="src"></span></div>
|
||||
<div id="banner-caption">Real WiFi CSI motion / presence / coarse-localization — penetrates drywall. Not skeletal pose.</div>
|
||||
|
||||
<div class="overlay-frame"></div>
|
||||
<div class="scanlines"></div>
|
||||
|
||||
<div class="panel" id="info">
|
||||
<h1>THROUGH-WALL WiFi SENSING</h1>
|
||||
<div class="sub">Live CSI · ws://localhost:8765/ws/sensing</div>
|
||||
<div class="row"><span class="k">source</span><span class="v amber" id="m-source">—</span></div>
|
||||
<div class="row"><span class="k">presence</span><span class="v" id="m-presence">—</span></div>
|
||||
<div class="row"><span class="k">motion level</span><span class="v" id="m-motion">—</span></div>
|
||||
<div class="row"><span class="k">confidence</span><span class="v cyan" id="m-conf">—</span></div>
|
||||
<div class="row"><span class="k">est. persons</span><span class="v amber" id="m-persons">—</span></div>
|
||||
<div class="row"><span class="k">active nodes</span><span class="v" id="m-nodes">—</span></div>
|
||||
<div class="row"><span class="k">tick</span><span class="v" id="m-tick">—</span></div>
|
||||
<div class="row"><span class="k">update rate</span><span class="v cyan" id="m-fps">—</span></div>
|
||||
</div>
|
||||
|
||||
<div class="panel" id="csi">
|
||||
<h2>Live RF features</h2>
|
||||
<div class="bar-row"><span class="label">motion</span><div class="bar-track"><div class="bar-fill" id="bar-motion"></div></div><span class="val" id="v-motion">—</span></div>
|
||||
<div class="bar-row"><span class="label">breathing</span><div class="bar-track"><div class="bar-fill" id="bar-breath"></div></div><span class="val" id="v-breath">—</span></div>
|
||||
<div class="bar-row"><span class="label">variance</span><div class="bar-track"><div class="bar-fill" id="bar-var"></div></div><span class="val" id="v-var">—</span></div>
|
||||
<div class="bar-row"><span class="label">mean rssi</span><div class="bar-track"><div class="bar-fill" id="bar-rssi"></div></div><span class="val" id="v-rssi">—</span></div>
|
||||
<div class="spark"><canvas id="spark" width="252" height="38"></canvas></div>
|
||||
<div class="legend">motion sparkline (last ~6s of real motion_band_power)</div>
|
||||
</div>
|
||||
|
||||
<div id="legend-nodes">
|
||||
<h2>Sensor nodes</h2>
|
||||
<div class="lr"><span class="dot" style="color:#4cf"></span><span>ESP32-S3 office <span class="muted">(node 9)</span></span></div>
|
||||
<div class="lr"><span class="dot" style="color:#ff4cc8"></span><span>ESP32-S3 hallway <span class="muted">(node 13)</span></span></div>
|
||||
<div class="lr" style="margin-top:6px"><span class="dot" style="color:#4f4"></span><span>RF localization <span class="muted">(coarse)</span></span></div>
|
||||
<div class="lr"><span class="muted" style="font-size:10px;line-height:1.4">Office & hallway split by a wall + doorway. WiFi motion still shows through drywall.</span></div>
|
||||
</div>
|
||||
|
||||
<div id="cam-tile">
|
||||
<h2>camera — ground truth when visible</h2>
|
||||
<video id="cam-video" autoplay muted playsinline></video>
|
||||
<button id="cam-btn">▶ enable webcam (optional)</button>
|
||||
<div class="gt-note">Independent of the CSI sensing. The WiFi works in the dark and through walls; the camera does not.</div>
|
||||
</div>
|
||||
|
||||
<div id="waiting" class="show">
|
||||
<div class="big pulse">Waiting for live sensing-server</div>
|
||||
<div>No connection to <b>ws://localhost:8765/ws/sensing</b>. Start the real server, then this page connects automatically.</div>
|
||||
<code>cd v2
|
||||
cargo build -p wifi-densepose-sensing-server
|
||||
./target/debug/sensing-server.exe --ws-port 8765 --udp-port 5005</code>
|
||||
<div style="margin-top:10px; color:var(--text-mute); font-size:11px;">This demo renders ONLY real data. It never invents frames.</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
"use strict";
|
||||
// =====================================================================
|
||||
// Config + WS endpoint (allow ?ws= override)
|
||||
// =====================================================================
|
||||
const params = new URLSearchParams(location.search);
|
||||
const WS_URL = params.get('ws') || 'ws://localhost:8765/ws/sensing';
|
||||
const ROOM_HALF = 5; // half-extent of the floor plane in metres
|
||||
const GRID_N = 20; // signal_field is 20 x 20
|
||||
|
||||
// Known node anchor positions (server sends node 9 @ [2,0,1.5]; node 13
|
||||
// joins later from the hallway side once its firmware is flashed). These
|
||||
// are anchors for the room model + labels, NOT fabricated sensing data.
|
||||
const NODE_ANCHORS = {
|
||||
9: { pos: [ 2.0, 0.0, 1.5], color: 0x44ccff, label: 'office (node 9)' },
|
||||
13: { pos: [-2.0, 0.0, -3.0], color: 0xff4cc8, label: 'hallway (node 13)' },
|
||||
};
|
||||
|
||||
// =====================================================================
|
||||
// Three.js scene (reused pattern from demos/05-skinned-realtime.html)
|
||||
// =====================================================================
|
||||
const scene = new THREE.Scene();
|
||||
scene.background = new THREE.Color(0x050507);
|
||||
scene.fog = new THREE.FogExp2(0x050507, 0.045);
|
||||
|
||||
const camera = new THREE.PerspectiveCamera(50, window.innerWidth/window.innerHeight, 0.05, 100);
|
||||
camera.position.set(4.5, 4.2, 6.0);
|
||||
|
||||
const renderer = new THREE.WebGLRenderer({ antialias: true, powerPreference: 'high-performance' });
|
||||
renderer.setPixelRatio(Math.min(2, window.devicePixelRatio));
|
||||
renderer.setSize(window.innerWidth, window.innerHeight);
|
||||
renderer.toneMapping = THREE.ACESFilmicToneMapping;
|
||||
renderer.toneMappingExposure = 0.85;
|
||||
renderer.outputEncoding = THREE.sRGBEncoding;
|
||||
document.body.appendChild(renderer.domElement);
|
||||
|
||||
const controls = new THREE.OrbitControls(camera, renderer.domElement);
|
||||
controls.target.set(0, 0.4, -0.5);
|
||||
controls.enableDamping = true; controls.dampingFactor = 0.06;
|
||||
controls.minDistance = 3; controls.maxDistance = 18;
|
||||
controls.maxPolarAngle = Math.PI * 0.49;
|
||||
|
||||
scene.add(new THREE.HemisphereLight(0x553a18, 0x080606, 0.7));
|
||||
const keyLight = new THREE.DirectionalLight(0xffc070, 0.9);
|
||||
keyLight.position.set(3, 6, 4);
|
||||
scene.add(keyLight);
|
||||
|
||||
// Post-processing — gentle bloom so the heatmap + puck glow.
|
||||
const composer = new THREE.EffectComposer(renderer);
|
||||
composer.addPass(new THREE.RenderPass(scene, camera));
|
||||
const bloom = new THREE.UnrealBloomPass(
|
||||
new THREE.Vector2(window.innerWidth, window.innerHeight), 0.55, 0.45, 0.82);
|
||||
composer.addPass(bloom);
|
||||
|
||||
// =====================================================================
|
||||
// Room: floor grid + wall + doorway dividing office / hallway
|
||||
// =====================================================================
|
||||
const gridHelper = new THREE.GridHelper(2*ROOM_HALF, GRID_N, 0x554a32, 0x2a2418);
|
||||
gridHelper.position.y = 0.002;
|
||||
scene.add(gridHelper);
|
||||
|
||||
// Dividing wall runs along world X near z = -1 (office z>-1, hallway z<-1),
|
||||
// with a doorway gap. Two wall segments leave a gap in the middle.
|
||||
const wallMat = new THREE.MeshStandardMaterial({
|
||||
color: 0x1b2330, transparent: true, opacity: 0.55, roughness: 0.9,
|
||||
side: THREE.DoubleSide,
|
||||
});
|
||||
const wallH = 1.4, wallZ = -1.0;
|
||||
function addWallSeg(cx, w) {
|
||||
const m = new THREE.Mesh(new THREE.BoxGeometry(w, wallH, 0.08), wallMat);
|
||||
m.position.set(cx, wallH/2, wallZ);
|
||||
scene.add(m);
|
||||
// top edge highlight
|
||||
const edge = new THREE.Mesh(new THREE.BoxGeometry(w, 0.02, 0.10),
|
||||
new THREE.MeshBasicMaterial({ color: 0x4cf, transparent: true, opacity: 0.5 }));
|
||||
edge.position.set(cx, wallH, wallZ);
|
||||
scene.add(edge);
|
||||
}
|
||||
// left segment, doorway gap (-0.7..0.7), right segment
|
||||
addWallSeg(-3.15, 3.7);
|
||||
addWallSeg( 3.15, 3.7);
|
||||
|
||||
// Room labels (sprite text) for OFFICE / HALLWAY
|
||||
function makeLabel(text, color) {
|
||||
const c = document.createElement('canvas'); c.width = 256; c.height = 64;
|
||||
const ctx = c.getContext('2d');
|
||||
ctx.fillStyle = color; ctx.font = 'bold 30px Consolas, monospace';
|
||||
ctx.textAlign = 'center'; ctx.textBaseline = 'middle';
|
||||
ctx.fillText(text, 128, 34);
|
||||
const tex = new THREE.CanvasTexture(c);
|
||||
const spr = new THREE.Sprite(new THREE.SpriteMaterial({ map: tex, transparent: true, depthTest: false }));
|
||||
spr.scale.set(2.0, 0.5, 1);
|
||||
return spr;
|
||||
}
|
||||
const officeLbl = makeLabel('OFFICE', '#ffb840'); officeLbl.position.set(2.6, 0.06, 2.6); scene.add(officeLbl);
|
||||
const hallLbl = makeLabel('HALLWAY', '#ff4cc8'); hallLbl.position.set(-2.6, 0.06, -3.2); scene.add(hallLbl);
|
||||
|
||||
// =====================================================================
|
||||
// Node markers (office / hallway). The hallway node is dimmed until it
|
||||
// actually appears in the live `nodes` list.
|
||||
// =====================================================================
|
||||
const nodeMeshes = {};
|
||||
function buildNode(id) {
|
||||
const a = NODE_ANCHORS[id];
|
||||
const g = new THREE.Group();
|
||||
const post = new THREE.Mesh(
|
||||
new THREE.CylinderGeometry(0.05, 0.07, 0.9, 12),
|
||||
new THREE.MeshStandardMaterial({ color: a.color, emissive: a.color, emissiveIntensity: 0.4, roughness: 0.4 }));
|
||||
post.position.y = 0.45; g.add(post);
|
||||
const orb = new THREE.Mesh(
|
||||
new THREE.SphereGeometry(0.12, 20, 16),
|
||||
new THREE.MeshBasicMaterial({ color: a.color }));
|
||||
orb.position.y = 0.95; g.add(orb);
|
||||
const ring = new THREE.Mesh(
|
||||
new THREE.RingGeometry(0.18, 0.24, 32),
|
||||
new THREE.MeshBasicMaterial({ color: a.color, transparent: true, opacity: 0.6, side: THREE.DoubleSide }));
|
||||
ring.rotation.x = -Math.PI/2; ring.position.y = 0.01; g.add(ring);
|
||||
const lbl = makeLabel('ESP32-S3 ' + a.label, '#' + a.color.toString(16).padStart(6,'0'));
|
||||
lbl.scale.set(2.6, 0.65, 1); lbl.position.set(0, 1.25, 0); g.add(lbl);
|
||||
g.position.set(a.pos[0], 0, a.pos[2]);
|
||||
g.userData.parts = { post, orb, ring };
|
||||
scene.add(g);
|
||||
return g;
|
||||
}
|
||||
Object.keys(NODE_ANCHORS).forEach(id => { nodeMeshes[id] = buildNode(+id); });
|
||||
function setNodeActive(id, active) {
|
||||
const g = nodeMeshes[id]; if (!g) return;
|
||||
const o = active ? 1.0 : 0.22;
|
||||
const parts = g.userData.parts;
|
||||
parts.orb.material.opacity = o; parts.orb.material.transparent = true;
|
||||
parts.ring.material.opacity = 0.6 * o;
|
||||
parts.post.material.emissiveIntensity = active ? 0.5 : 0.12;
|
||||
}
|
||||
setNodeActive(9, false); setNodeActive(13, false);
|
||||
|
||||
// =====================================================================
|
||||
// signal_field 20x20 floor heatmap — instanced colored tiles.
|
||||
// Driven ONLY by real `signal_field.values` (400 floats ~0..1).
|
||||
// =====================================================================
|
||||
const TILE = (2*ROOM_HALF) / GRID_N;
|
||||
const heatGeo = new THREE.PlaneGeometry(TILE * 0.96, TILE * 0.96);
|
||||
const heatMat = new THREE.MeshBasicMaterial({ vertexColors: true, transparent: true, opacity: 0.85, side: THREE.DoubleSide });
|
||||
const heatMesh = new THREE.InstancedMesh(heatGeo, heatMat, GRID_N * GRID_N);
|
||||
heatMesh.instanceMatrix.setUsage(THREE.DynamicDrawUsage);
|
||||
const heatColor = new THREE.InstancedBufferAttribute(new Float32Array(GRID_N * GRID_N * 3), 3);
|
||||
heatMesh.instanceColor = heatColor;
|
||||
const _m = new THREE.Matrix4();
|
||||
const _q = new THREE.Quaternion().setFromAxisAngle(new THREE.Vector3(1,0,0), -Math.PI/2);
|
||||
const _s = new THREE.Vector3(1,1,1);
|
||||
const _p = new THREE.Vector3();
|
||||
// gridCell (gx,gz) -> world (x,z). gx,gz in [0,GRID_N).
|
||||
function cellToWorld(gx, gz) {
|
||||
return [ (gx + 0.5) * TILE - ROOM_HALF, (gz + 0.5) * TILE - ROOM_HALF ];
|
||||
}
|
||||
for (let gz = 0; gz < GRID_N; gz++) {
|
||||
for (let gx = 0; gx < GRID_N; gx++) {
|
||||
const i = gz * GRID_N + gx;
|
||||
const [wx, wz] = cellToWorld(gx, gz);
|
||||
_p.set(wx, 0.012, wz);
|
||||
_m.compose(_p, _q, _s);
|
||||
heatMesh.setMatrixAt(i, _m);
|
||||
heatColor.setXYZ(i, 0.02, 0.02, 0.03);
|
||||
}
|
||||
}
|
||||
heatMesh.instanceMatrix.needsUpdate = true;
|
||||
scene.add(heatMesh);
|
||||
|
||||
// amber→white heat ramp for a value in [0,1]
|
||||
function heatRamp(v, out) {
|
||||
v = Math.max(0, Math.min(1, v));
|
||||
// dark -> amber -> hot white
|
||||
const r = Math.min(1, 0.05 + 1.6 * v);
|
||||
const g = Math.min(1, 0.02 + 1.1 * v * v);
|
||||
const b = Math.min(1, 0.04 + 0.9 * Math.pow(v, 3));
|
||||
out.set(r, g, b);
|
||||
return out;
|
||||
}
|
||||
const _c = new THREE.Color();
|
||||
let lastFieldPeak = { gx: GRID_N/2|0, gz: GRID_N/2|0, v: 0 };
|
||||
function updateHeatmap(field) {
|
||||
if (!field || !Array.isArray(field.values)) return;
|
||||
const vals = field.values;
|
||||
// grid_size is [20,1,20]; values are row-major 400 floats.
|
||||
let peakV = -1, peakGx = lastFieldPeak.gx, peakGz = lastFieldPeak.gz;
|
||||
const n = Math.min(vals.length, GRID_N * GRID_N);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const v = vals[i];
|
||||
heatRamp(v, _c);
|
||||
heatColor.setXYZ(i, _c.r, _c.g, _c.b);
|
||||
if (v > peakV) { peakV = v; peakGx = i % GRID_N; peakGz = (i / GRID_N) | 0; }
|
||||
}
|
||||
heatColor.needsUpdate = true;
|
||||
lastFieldPeak = { gx: peakGx, gz: peakGz, v: peakV };
|
||||
}
|
||||
|
||||
// =====================================================================
|
||||
// RF-localization puck — from persons[0].position (coarse, NOT pose).
|
||||
// Falls back to the signal_field peak cell when no person is present.
|
||||
// =====================================================================
|
||||
const puck = new THREE.Group();
|
||||
const puckCore = new THREE.Mesh(
|
||||
new THREE.SphereGeometry(0.16, 24, 18),
|
||||
new THREE.MeshBasicMaterial({ color: 0x66ff88 }));
|
||||
puckCore.position.y = 0.16; puck.add(puckCore);
|
||||
const puckRing = new THREE.Mesh(
|
||||
new THREE.RingGeometry(0.28, 0.36, 40),
|
||||
new THREE.MeshBasicMaterial({ color: 0x66ff88, transparent: true, opacity: 0.7, side: THREE.DoubleSide }));
|
||||
puckRing.rotation.x = -Math.PI/2; puckRing.position.y = 0.02; puck.add(puckRing);
|
||||
const puckBeam = new THREE.Mesh(
|
||||
new THREE.CylinderGeometry(0.03, 0.03, 1.2, 8),
|
||||
new THREE.MeshBasicMaterial({ color: 0x66ff88, transparent: true, opacity: 0.35 }));
|
||||
puckBeam.position.y = 0.6; puck.add(puckBeam);
|
||||
puck.visible = false;
|
||||
scene.add(puck);
|
||||
const puckTarget = new THREE.Vector3(0, 0, 0);
|
||||
|
||||
function updatePuck(frame) {
|
||||
let wx = null, wz = null, present = false;
|
||||
const persons = frame.persons || [];
|
||||
if (persons.length && Array.isArray(persons[0].position)) {
|
||||
// server position is [x, 0, z] in metres, origin at room centre
|
||||
wx = persons[0].position[0];
|
||||
wz = persons[0].position[2];
|
||||
present = true;
|
||||
}
|
||||
// If no person but the field has clear energy, show the peak cell
|
||||
// (coarse) so the puck honestly tracks "where the RF energy is".
|
||||
if (!present && lastFieldPeak.v > 0.55) {
|
||||
const peak = cellToWorld(lastFieldPeak.gx, lastFieldPeak.gz);
|
||||
wx = peak[0]; wz = peak[1]; present = true;
|
||||
}
|
||||
if (present && wx !== null) {
|
||||
// clamp into the room so it never flies off the floor
|
||||
wx = Math.max(-ROOM_HALF+0.3, Math.min(ROOM_HALF-0.3, wx));
|
||||
wz = Math.max(-ROOM_HALF+0.3, Math.min(ROOM_HALF-0.3, wz));
|
||||
puckTarget.set(wx, 0, wz);
|
||||
puck.visible = true;
|
||||
} else {
|
||||
puck.visible = false;
|
||||
}
|
||||
}
|
||||
|
||||
// =====================================================================
|
||||
// HUD updates
|
||||
// =====================================================================
|
||||
const $ = id => document.getElementById(id);
|
||||
function clamp01(x){ return Math.max(0, Math.min(1, x)); }
|
||||
function setBar(barId, valId, frac, text) {
|
||||
$(barId).style.width = (clamp01(frac) * 100).toFixed(0) + '%';
|
||||
$(valId).textContent = text;
|
||||
}
|
||||
|
||||
// motion sparkline ring buffer
|
||||
const sparkCtx = $('spark').getContext('2d');
|
||||
const SPARK_N = 120;
|
||||
const sparkBuf = new Array(SPARK_N).fill(0);
|
||||
function pushSpark(v) {
|
||||
sparkBuf.push(v); if (sparkBuf.length > SPARK_N) sparkBuf.shift();
|
||||
const w = sparkCtx.canvas.width, h = sparkCtx.canvas.height;
|
||||
sparkCtx.clearRect(0,0,w,h);
|
||||
let maxV = 40; for (const x of sparkBuf) if (x > maxV) maxV = x;
|
||||
sparkCtx.strokeStyle = '#ffb840'; sparkCtx.lineWidth = 1.5; sparkCtx.beginPath();
|
||||
for (let i = 0; i < sparkBuf.length; i++) {
|
||||
const x = (i / (SPARK_N-1)) * w;
|
||||
const y = h - (sparkBuf[i] / maxV) * (h - 3) - 1.5;
|
||||
i === 0 ? sparkCtx.moveTo(x, y) : sparkCtx.lineTo(x, y);
|
||||
}
|
||||
sparkCtx.stroke();
|
||||
}
|
||||
|
||||
// =====================================================================
|
||||
// Honest status banner (strict, mutually exclusive)
|
||||
// =====================================================================
|
||||
const banner = $('banner');
|
||||
function setBannerLive(source, nodeCount) {
|
||||
if (source === 'esp32') {
|
||||
banner.className = 'live';
|
||||
banner.innerHTML = 'LIVE — real ESP32 CSI <span class="src">(source=' + source + ', ' + nodeCount + ' node' + (nodeCount === 1 ? '' : 's') + ')</span>';
|
||||
} else {
|
||||
// anything not esp32 = explicitly NOT real, badged
|
||||
banner.className = 'sim';
|
||||
banner.innerHTML = 'SIMULATED — not real <span class="src">(source=' + source + ' — start an ESP32 for live CSI)</span>';
|
||||
}
|
||||
}
|
||||
function setBannerNoServer() {
|
||||
banner.className = 'noserver';
|
||||
banner.innerHTML = 'NO SERVER — start the sensing-server <span class="src">(ws://localhost:8765/ws/sensing unreachable)</span>';
|
||||
}
|
||||
|
||||
// =====================================================================
|
||||
// WebSocket — render ONLY real frames. Reconnect; never fabricate.
|
||||
// =====================================================================
|
||||
let ws = null, gotFrame = false;
|
||||
let frameTimes = []; // for measured update rate (fps)
|
||||
let lastFrame = null; // most recent real frame (render loop interpolates puck)
|
||||
|
||||
function connect() {
|
||||
setBannerNoServer();
|
||||
try { ws = new WebSocket(WS_URL); }
|
||||
catch (e) { scheduleReconnect(); return; }
|
||||
|
||||
ws.onopen = () => { /* wait for first frame before claiming LIVE */ };
|
||||
ws.onmessage = (ev) => {
|
||||
let d; try { d = JSON.parse(ev.data); } catch (e) { return; }
|
||||
if (!d || d.type !== 'sensing_update') return;
|
||||
onFrame(d);
|
||||
};
|
||||
ws.onclose = () => { gotFrame = false; $('waiting').classList.add('show'); setBannerNoServer(); scheduleReconnect(); };
|
||||
ws.onerror = () => { try { ws.close(); } catch (e) {} };
|
||||
}
|
||||
let reconnectT = null;
|
||||
function scheduleReconnect() {
|
||||
if (reconnectT) return;
|
||||
reconnectT = setTimeout(() => { reconnectT = null; connect(); }, 1500);
|
||||
}
|
||||
|
||||
function onFrame(d) {
|
||||
gotFrame = true;
|
||||
lastFrame = d;
|
||||
$('waiting').classList.remove('show');
|
||||
|
||||
const source = d.source || 'unknown';
|
||||
const nodes = Array.isArray(d.nodes) ? d.nodes : [];
|
||||
setBannerLive(source, nodes.length);
|
||||
|
||||
// measured update rate
|
||||
const now = performance.now();
|
||||
frameTimes.push(now);
|
||||
while (frameTimes.length && now - frameTimes[0] > 2000) frameTimes.shift();
|
||||
const fps = frameTimes.length > 1 ? (frameTimes.length - 1) / ((frameTimes[frameTimes.length-1] - frameTimes[0]) / 1000) : 0;
|
||||
|
||||
const cls = d.classification || {};
|
||||
const feat = d.features || {};
|
||||
|
||||
// info panel
|
||||
$('m-source').textContent = source.toUpperCase();
|
||||
$('m-source').className = 'v ' + (source === 'esp32' ? 'green' : 'red');
|
||||
const presence = !!cls.presence;
|
||||
$('m-presence').textContent = presence ? (cls.motion_level === 'present_moving' ? 'PRESENT · MOVING' : 'PRESENT') : 'CLEAR';
|
||||
$('m-presence').className = 'v ' + (presence ? 'green' : 'mute');
|
||||
$('m-motion').textContent = cls.motion_level || '—';
|
||||
$('m-conf').textContent = (cls.confidence != null) ? cls.confidence.toFixed(2) : '—';
|
||||
$('m-persons').textContent = (d.estimated_persons != null) ? d.estimated_persons : '—';
|
||||
$('m-nodes').textContent = nodes.length + ' (' + nodes.map(n => n.node_id).join(', ') + ')';
|
||||
$('m-tick').textContent = (d.tick != null) ? d.tick : '—';
|
||||
$('m-fps').textContent = fps ? fps.toFixed(1) + ' Hz' : '—';
|
||||
|
||||
// feature bars (real values, scaled into 0..1 for the bar width only)
|
||||
const motion = feat.motion_band_power || 0;
|
||||
const breath = feat.breathing_band_power || 0;
|
||||
const variance = feat.variance || 0;
|
||||
const rssi = feat.mean_rssi != null ? feat.mean_rssi : -100;
|
||||
setBar('bar-motion', 'v-motion', motion / 100, motion.toFixed(1));
|
||||
setBar('bar-breath', 'v-breath', breath / 100, breath.toFixed(1));
|
||||
setBar('bar-var', 'v-var', variance / 80, variance.toFixed(1));
|
||||
// rssi: map -90..-30 dBm -> 0..1
|
||||
setBar('bar-rssi', 'v-rssi', (rssi + 90) / 60, rssi.toFixed(0));
|
||||
pushSpark(motion);
|
||||
|
||||
// node activity
|
||||
const activeIds = new Set(nodes.map(n => n.node_id));
|
||||
[9, 13].forEach(id => setNodeActive(id, activeIds.has(id)));
|
||||
|
||||
// heatmap + puck
|
||||
updateHeatmap(d.signal_field);
|
||||
updatePuck(d);
|
||||
}
|
||||
|
||||
// =====================================================================
|
||||
// Optional webcam ground-truth tile (reused from demos/05). Camera is
|
||||
// separate from CSI sensing — labeled "ground truth when visible".
|
||||
// =====================================================================
|
||||
let camStream = null;
|
||||
$('cam-btn').addEventListener('click', async () => {
|
||||
const btn = $('cam-btn');
|
||||
if (camStream) { // toggle off
|
||||
camStream.getTracks().forEach(t => t.stop());
|
||||
$('cam-video').style.display = 'none'; camStream = null;
|
||||
btn.textContent = '▶ enable webcam (optional)';
|
||||
return;
|
||||
}
|
||||
btn.disabled = true; btn.textContent = 'requesting camera…';
|
||||
try {
|
||||
camStream = await navigator.mediaDevices.getUserMedia({
|
||||
video: { width: { ideal: 640 }, height: { ideal: 480 }, facingMode: 'user' }, audio: false,
|
||||
});
|
||||
const v = $('cam-video'); v.srcObject = camStream; v.style.display = 'block';
|
||||
btn.textContent = '■ stop webcam'; btn.disabled = false;
|
||||
} catch (e) {
|
||||
btn.textContent = '✗ camera unavailable'; btn.disabled = false; console.error(e);
|
||||
setTimeout(() => { if (!camStream) btn.textContent = '▶ enable webcam (optional)'; }, 2000);
|
||||
}
|
||||
});
|
||||
|
||||
// =====================================================================
|
||||
// Render loop — smooth the puck toward its real target; pulse rings.
|
||||
// =====================================================================
|
||||
const clock = new THREE.Clock();
|
||||
function animate() {
|
||||
requestAnimationFrame(animate);
|
||||
const t = clock.getElapsedTime();
|
||||
controls.update();
|
||||
|
||||
if (puck.visible) {
|
||||
puck.position.lerp(puckTarget, 0.18);
|
||||
const pulse = 0.8 + 0.25 * Math.sin(t * 3.0);
|
||||
puckRing.scale.set(pulse, pulse, pulse);
|
||||
puckRing.material.opacity = 0.5 + 0.25 * Math.sin(t * 3.0);
|
||||
}
|
||||
// node rings breathe when active
|
||||
[9,13].forEach(id => {
|
||||
const g = nodeMeshes[id]; if (!g) return;
|
||||
const r = g.userData.parts.ring;
|
||||
const s = 1 + 0.08 * Math.sin(t * 2 + id);
|
||||
r.scale.set(s, s, s);
|
||||
});
|
||||
|
||||
composer.render();
|
||||
}
|
||||
animate();
|
||||
|
||||
window.addEventListener('resize', () => {
|
||||
camera.aspect = window.innerWidth / window.innerHeight;
|
||||
camera.updateProjectionMatrix();
|
||||
renderer.setSize(window.innerWidth, window.innerHeight);
|
||||
composer.setSize(window.innerWidth, window.innerHeight);
|
||||
});
|
||||
|
||||
// kick off
|
||||
connect();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Tiny threaded static server for the through-wall WiFi-CSI sensing demo.
|
||||
|
||||
Adapted from examples/three.js/server/serve-demo.py. Serves the
|
||||
`examples/through-wall/` page so a browser can fetch index.html, then the
|
||||
page connects directly to the LIVE sensing-server WebSocket at
|
||||
ws://localhost:8765/ws/sensing (NOT proxied through here).
|
||||
|
||||
Why a threaded server (not `python -m http.server`)?
|
||||
The stdlib SimpleHTTPServer is single-threaded; a browser opens several
|
||||
parallel connections (HTML + the three.js CDN tags fetch in parallel),
|
||||
the first eats the worker, the rest can stall. ThreadingHTTPServer fixes it.
|
||||
|
||||
IMPORTANT: this serves on port 8080 — port 8765 is taken by the
|
||||
sensing-server's WebSocket. They are two different processes.
|
||||
|
||||
Usage:
|
||||
# 1) start the REAL sensing-server (separate terminal):
|
||||
# cd v2
|
||||
# cargo build -p wifi-densepose-sensing-server
|
||||
# ./target/debug/sensing-server.exe --ws-port 8765 --udp-port 5005
|
||||
# 2) start this static server:
|
||||
python examples/through-wall/serve.py
|
||||
# 3) open:
|
||||
# http://localhost:8080/examples/through-wall/index.html
|
||||
|
||||
Override the WS endpoint with a query param, e.g.:
|
||||
http://localhost:8080/examples/through-wall/index.html?ws=ws://192.168.1.20:8765/ws/sensing
|
||||
"""
|
||||
from http.server import ThreadingHTTPServer, SimpleHTTPRequestHandler
|
||||
import os
|
||||
import sys
|
||||
|
||||
PORT = int(os.environ.get("PORT", 8080))
|
||||
|
||||
# Serve from the repo root regardless of where this script is launched.
|
||||
# This file lives at examples/through-wall/serve.py — two levels deep.
|
||||
os.chdir(os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")))
|
||||
|
||||
|
||||
class NoCacheHandler(SimpleHTTPRequestHandler):
|
||||
def end_headers(self):
|
||||
# Aggressive no-cache so the browser ALWAYS fetches the latest
|
||||
# index.html after edits, even on a soft refresh.
|
||||
self.send_header("Cache-Control", "no-store, no-cache, must-revalidate, max-age=0")
|
||||
self.send_header("Pragma", "no-cache")
|
||||
self.send_header("Expires", "0")
|
||||
super().end_headers()
|
||||
|
||||
def log_message(self, fmt, *args): # quieter logs
|
||||
sys.stderr.write("[serve] " + (fmt % args) + "\n")
|
||||
|
||||
|
||||
PAGE = "examples/through-wall/index.html"
|
||||
|
||||
with ThreadingHTTPServer(("127.0.0.1", PORT), NoCacheHandler) as srv:
|
||||
print(f"serving {os.getcwd()} on http://127.0.0.1:{PORT}/")
|
||||
print(f" open http://localhost:{PORT}/{PAGE}")
|
||||
print("")
|
||||
print(" The page connects to the LIVE sensing-server at")
|
||||
print(" ws://localhost:8765/ws/sensing (start it first — see README.md).")
|
||||
print(" Override with ?ws=ws://HOST:PORT/ws/sensing")
|
||||
try:
|
||||
srv.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
sys.exit(0)
|
||||
@@ -0,0 +1,926 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8"/>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
|
||||
<title>WiFlow Browser Trainer · calibrate → capture → train → infer, in your camera's frame</title>
|
||||
<!--
|
||||
WiFlow in-browser trainer (ADR-079 / ADR-180 + ADR-151 empty-room baseline).
|
||||
A 4-STAGE GATED FLOW, all in the LAPTOP camera's coordinate frame:
|
||||
0. CALIBRATE — empty-room baseline (Welford mean+std over the 410-d CSI vector).
|
||||
Every CSI vector afterwards is expressed as deviation-from-baseline,
|
||||
so a body's perturbation stands out from the static channel.
|
||||
1. CAPTURE — MediaPipe Pose on the laptop camera = 17 COCO keypoints (the LABEL),
|
||||
paired with the baseline-normalized live ESP32 CSI vector (the INPUT).
|
||||
Guided, balanced routine with a per-pose coverage meter.
|
||||
2. TRAIN — a TF.js MLP (WebGPU/WASM/WebGL) learns CSI -> pose in-browser. Honest
|
||||
held-out PCK + a mean-pose baseline it must beat.
|
||||
3. INFER — the trained model drives a skeleton FROM WiFi CSI ONLY, drawn over the
|
||||
same camera frame, so it ALIGNS (the whole point of doing it in-browser).
|
||||
Self-contained. CDN libs only. No bundler. Real data only — CSI source must read "esp32".
|
||||
-->
|
||||
<style>
|
||||
:root{--bg:#0a0c10;--panel:#11151c;--panel2:#0d1117;--amber:#ffb840;--green:#46e08a;
|
||||
--red:#ff5a5a;--blue:#5aa9ff;--mute:#7d8796;--line:#1d2430;--txt:#dfe6ee}
|
||||
*{box-sizing:border-box}
|
||||
body{margin:0;background:var(--bg);color:var(--txt);
|
||||
font:14px/1.5 'JetBrains Mono',ui-monospace,Menlo,Consolas,monospace}
|
||||
header{padding:14px 18px;border-bottom:1px solid var(--line);display:flex;align-items:center;gap:14px;flex-wrap:wrap}
|
||||
h1{font-size:15px;margin:0;letter-spacing:1px;text-transform:uppercase;font-weight:600}
|
||||
h1 span{color:var(--amber)}
|
||||
#compute{padding:4px 10px;border-radius:5px;font-weight:600;font-size:11px;letter-spacing:.5px;
|
||||
background:rgba(90,169,255,.12);color:var(--blue);border:1px solid var(--blue)}
|
||||
#banner{margin-left:auto;padding:5px 12px;border-radius:5px;font-weight:600;font-size:12px;letter-spacing:.5px}
|
||||
.live{background:rgba(70,224,138,.15);color:var(--green);border:1px solid var(--green)}
|
||||
.sim{background:rgba(255,184,64,.15);color:var(--amber);border:1px solid var(--amber)}
|
||||
.down{background:rgba(255,90,90,.15);color:var(--red);border:1px solid var(--red)}
|
||||
/* progress stepper */
|
||||
.steps{display:flex;gap:6px;padding:14px 18px 0;flex-wrap:wrap;align-items:center}
|
||||
.step{display:flex;align-items:center;gap:8px;background:var(--panel);color:var(--mute);
|
||||
border:1px solid var(--line);border-radius:8px;padding:8px 16px;cursor:pointer;font-weight:600;letter-spacing:.5px}
|
||||
.step .num{display:inline-flex;width:20px;height:20px;border-radius:50%;background:var(--line);color:var(--txt);
|
||||
align-items:center;justify-content:center;font-size:11px}
|
||||
.step.on{color:var(--amber);border-color:var(--amber)}
|
||||
.step.on .num{background:var(--amber);color:#0a0c10}
|
||||
.step.done .num{background:var(--green);color:#0a0c10}
|
||||
.step.locked{opacity:.45;cursor:not-allowed}
|
||||
.arrow{color:var(--mute)}
|
||||
main{padding:14px 18px 24px}
|
||||
.panel{display:none;background:var(--panel2);border:1px solid var(--line);border-radius:10px;padding:18px}
|
||||
.panel.on{display:block}
|
||||
.cols{display:flex;gap:18px;flex-wrap:wrap}
|
||||
.card{background:var(--panel);border:1px solid var(--line);border-radius:10px;padding:14px}
|
||||
canvas{background:#070a0e;border-radius:8px;display:block}
|
||||
.label{font-size:11px;text-transform:uppercase;letter-spacing:1.5px;color:var(--mute);margin-bottom:8px}
|
||||
.stats{min-width:260px;flex:1}
|
||||
.row{display:flex;justify-content:space-between;padding:3px 0;border-bottom:1px dashed var(--line)}
|
||||
.row .k{color:var(--mute)} .row .v{color:var(--amber);font-variant-numeric:tabular-nums;text-align:right}
|
||||
.v.green{color:var(--green)} .v.red{color:var(--red)} .v.blue{color:var(--blue)}
|
||||
.note{margin-top:12px;font-size:11px;color:var(--mute);line-height:1.6}
|
||||
.note b{color:var(--txt)}
|
||||
button.btn{background:var(--amber);color:#0a0c10;border:0;border-radius:6px;padding:8px 16px;
|
||||
font:inherit;font-weight:600;cursor:pointer}
|
||||
button.btn:disabled{opacity:.4;cursor:not-allowed}
|
||||
button.ghost{background:transparent;color:var(--txt);border:1px solid var(--line)}
|
||||
select,input{background:var(--panel);color:var(--txt);border:1px solid var(--line);border-radius:6px;
|
||||
padding:7px;font:inherit;max-width:260px}
|
||||
.bar{height:8px;background:var(--line);border-radius:5px;overflow:hidden;margin-top:4px}
|
||||
.bar>i{display:block;height:100%;background:var(--green);width:0%}
|
||||
.verdict{padding:10px 14px;border-radius:8px;margin-top:12px;font-weight:600;font-size:13px}
|
||||
.verdict.good{background:rgba(70,224,138,.12);color:var(--green);border:1px solid var(--green)}
|
||||
.verdict.bad{background:rgba(255,90,90,.12);color:var(--red);border:1px solid var(--red)}
|
||||
.verdict.idle{background:rgba(125,135,150,.1);color:var(--mute);border:1px solid var(--line)}
|
||||
.pill{display:inline-block;padding:2px 8px;border-radius:10px;font-size:11px;font-weight:600;margin-left:6px}
|
||||
.pill.gt{background:rgba(90,169,255,.15);color:var(--blue);border:1px solid var(--blue)}
|
||||
.pill.csi{background:rgba(70,224,138,.15);color:var(--green);border:1px solid var(--green)}
|
||||
code{background:#0a0c10;border:1px solid var(--line);border-radius:4px;padding:1px 5px;color:var(--amber)}
|
||||
a{color:var(--blue)}
|
||||
/* big guided prompt */
|
||||
#prompt{font-size:30px;font-weight:700;color:var(--amber);letter-spacing:1px;text-align:center;margin:6px 0}
|
||||
#countdown{font-size:13px;color:var(--mute);text-align:center}
|
||||
/* coverage meter */
|
||||
.cov{display:flex;flex-direction:column;gap:5px;margin-top:8px}
|
||||
.covrow{display:flex;align-items:center;gap:8px;font-size:11px}
|
||||
.covrow .nm{width:90px;color:var(--mute);text-transform:capitalize}
|
||||
.covrow .bar{flex:1;margin:0}
|
||||
.covrow .ct{width:42px;text-align:right;color:var(--txt);font-variant-numeric:tabular-nums}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>WiFlow <span>Browser Trainer</span> — calibrate · capture · train · infer</h1>
|
||||
<div id="compute">compute: …</div>
|
||||
<div id="banner" class="down">CONNECTING…</div>
|
||||
</header>
|
||||
|
||||
<!-- progress stepper, each gated on the previous -->
|
||||
<div class="steps">
|
||||
<div class="step on" data-stage="calibrate"><span class="num">0</span> CALIBRATE</div>
|
||||
<span class="arrow">→</span>
|
||||
<div class="step locked" data-stage="capture"><span class="num">1</span> CAPTURE</div>
|
||||
<span class="arrow">→</span>
|
||||
<div class="step locked" data-stage="train"><span class="num">2</span> TRAIN</div>
|
||||
<span class="arrow">→</span>
|
||||
<div class="step locked" data-stage="infer"><span class="num">3</span> INFER</div>
|
||||
</div>
|
||||
|
||||
<main>
|
||||
<!-- ============================ STAGE 0 · CALIBRATE ============================ -->
|
||||
<section id="stage-calibrate" class="panel on">
|
||||
<div class="cols">
|
||||
<div class="card">
|
||||
<div class="label">empty-room baseline (ADR-151) — step OUT of the space</div>
|
||||
<canvas id="calCv" width="420" height="300"></canvas>
|
||||
<div style="margin-top:10px;display:flex;gap:8px;align-items:center;flex-wrap:wrap">
|
||||
<button id="calBtn" class="btn">calibrate baseline (10 s)</button>
|
||||
<button id="recalBtn" class="ghost btn">recalibrate</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card stats">
|
||||
<div class="label">baseline</div>
|
||||
<div class="row"><span class="k">CSI source</span><span class="v" id="calSrc">—</span></div>
|
||||
<div class="row"><span class="k">status</span><span class="v" id="calStatus">NOT CALIBRATED</span></div>
|
||||
<div class="row"><span class="k">frames in baseline</span><span class="v" id="calN">0</span></div>
|
||||
<div class="row"><span class="k">age</span><span class="v" id="calAge">—</span></div>
|
||||
<div style="margin-top:8px"><div class="bar"><i id="calBar"></i></div></div>
|
||||
<div class="note">
|
||||
The room's static WiFi channel is mostly constant. We capture ~10 s of the
|
||||
<b>quiescent</b> field (you OUT of the space) and compute a per-feature running
|
||||
<b>mean + std</b> (Welford) over the 410-d CSI vector. Afterwards every CSI vector
|
||||
is expressed as <b>deviation from baseline</b>:
|
||||
<code>x_norm = (x − base_mean) / (base_std + ε)</code> — applied consistently in
|
||||
capture, train, and infer. This makes a <b>body's perturbation</b> stand out from
|
||||
the static channel. You must calibrate before capturing.
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============================ STAGE 1 · CAPTURE ============================ -->
|
||||
<section id="stage-capture" class="panel">
|
||||
<div class="cols">
|
||||
<div class="card">
|
||||
<div class="label">laptop camera <span class="pill gt">MediaPipe skeleton = GROUND TRUTH (the label)</span></div>
|
||||
<canvas id="capCv" width="420" height="480"></canvas>
|
||||
<div id="prompt">stand still</div>
|
||||
<div id="countdown">—</div>
|
||||
<div style="margin-top:8px;display:flex;gap:8px;align-items:center;flex-wrap:wrap">
|
||||
<button id="camBtn" class="btn">enable laptop camera</button>
|
||||
<select id="camSel" style="display:none"></select>
|
||||
</div>
|
||||
<div id="camStatus" class="note" style="margin-top:6px">camera: off</div>
|
||||
</div>
|
||||
<div class="card stats">
|
||||
<div class="label">guided capture</div>
|
||||
<div class="row"><span class="k">CSI source</span><span class="v" id="capSrc">—</span></div>
|
||||
<div class="row"><span class="k">CSI nodes</span><span class="v" id="capNodes">—</span></div>
|
||||
<div class="row"><span class="k">pose visibility</span><span class="v" id="capVis">—</span></div>
|
||||
<div class="row"><span class="k">total samples</span><span class="v green" id="capN">0</span></div>
|
||||
<div class="row"><span class="k">last skip reason</span><span class="v" id="capSkip">—</span></div>
|
||||
<div style="margin-top:12px;display:flex;gap:8px;flex-wrap:wrap">
|
||||
<button id="recBtn" class="btn" disabled>● start guided recording</button>
|
||||
<button id="clrBtn" class="ghost btn">clear dataset</button>
|
||||
</div>
|
||||
<div class="label" style="margin-top:16px">per-pose coverage (balance the dataset)</div>
|
||||
<div id="cov" class="cov"></div>
|
||||
<div class="note">
|
||||
A pair is recorded <b>only</b> when BOTH (a) a confident MediaPipe pose
|
||||
(mean visibility > 0.5) AND (b) a fresh <b>live</b> CSI frame (<code>source==esp32</code>)
|
||||
exist. We store the <b>baseline-normalized</b> CSI + the 17 keypoints, mirrored to
|
||||
IndexedDB so a refresh keeps them. Follow the prompt so every pose bucket fills up —
|
||||
a balanced set beats 2 000 frames of standing.
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============================ STAGE 2 · TRAIN ============================ -->
|
||||
<section id="stage-train" class="panel">
|
||||
<div class="cols">
|
||||
<div class="card stats">
|
||||
<div class="label">train (TensorFlow.js)</div>
|
||||
<div class="row"><span class="k">total samples</span><span class="v" id="trN">0</span></div>
|
||||
<div class="row"><span class="k">train / val split</span><span class="v" id="trSplit">— / — (chronological 80/20)</span></div>
|
||||
<div class="row"><span class="k">epoch</span><span class="v" id="trEpoch">0</span></div>
|
||||
<div class="row"><span class="k">train MSE</span><span class="v" id="trLoss">—</span></div>
|
||||
<div class="row"><span class="k">val MSE</span><span class="v" id="trVal">—</span></div>
|
||||
<div class="row"><span class="k">held-out PCK@0.10</span><span class="v green" id="trP10">—</span></div>
|
||||
<div class="row"><span class="k">held-out PCK@0.05</span><span class="v" id="trP05">—</span></div>
|
||||
<div class="row"><span class="k">held-out MPJPE</span><span class="v" id="trMpj">—</span></div>
|
||||
<div class="row"><span class="k">mean-pose baseline PCK@0.10</span><span class="v red" id="trBase">—</span></div>
|
||||
<div style="margin-top:8px"><div class="bar"><i id="trBar"></i></div></div>
|
||||
<div style="margin-top:12px;display:flex;gap:8px;flex-wrap:wrap;align-items:center">
|
||||
<label class="note" style="margin:0">epochs <input id="trEpochs" type="number" value="200" min="20" max="600" style="width:80px"></label>
|
||||
<button id="trainBtn" class="btn" disabled>train model</button>
|
||||
<button id="trStop" class="ghost btn" disabled>stop</button>
|
||||
</div>
|
||||
<div id="verdict" class="verdict idle">no model yet — calibrate, capture, then train.</div>
|
||||
<div class="note">
|
||||
<b>The bar to beat</b> is the mean-pose baseline (predict the train-mean pose for
|
||||
everything). A model that doesn't clear it has learned <b>no usable CSI→pose signal</b> —
|
||||
this page says so plainly. Inputs are standardized on the <b>train split only</b>
|
||||
(after baseline-normalization); the val split is the chronological last 20%, never trained on.
|
||||
</div>
|
||||
</div>
|
||||
<div class="card">
|
||||
<div class="label">loss curve — train (amber) vs val (blue)</div>
|
||||
<canvas id="lossCv" width="460" height="300"></canvas>
|
||||
<div class="note" id="trMsg">Idle.</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ============================ STAGE 3 · INFER ============================ -->
|
||||
<section id="stage-infer" class="panel">
|
||||
<div class="cols">
|
||||
<div class="card">
|
||||
<div class="label">WiFi-inferred pose <span class="pill csi">CSI ONLY — no camera in the loop</span></div>
|
||||
<canvas id="infCv" width="420" height="560"></canvas>
|
||||
<div style="margin-top:10px;display:flex;gap:8px;align-items:center;flex-wrap:wrap">
|
||||
<label class="note" style="margin:0"><input type="checkbox" id="hideCam"> hide camera (skeleton on black)</label>
|
||||
<span class="note" id="infModelState" style="margin:0">no model loaded</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="card stats">
|
||||
<div class="label">live inference</div>
|
||||
<div class="row"><span class="k">CSI source</span><span class="v" id="infSrc">—</span></div>
|
||||
<div class="row"><span class="k">CSI nodes</span><span class="v" id="infNodes">—</span></div>
|
||||
<div class="row"><span class="k">presence</span><span class="v" id="infPres">—</span></div>
|
||||
<div class="row"><span class="k">infer fps</span><span class="v" id="infFps">—</span></div>
|
||||
<div class="row"><span class="k">measured held-out PCK@0.10</span><span class="v green" id="infPck">—</span></div>
|
||||
<div class="note">
|
||||
This skeleton is inferred <b>from WiFi CSI only</b> (baseline-normalized, then through
|
||||
the model). It is <b>coarse</b> — the held-out PCK above is the real number. It is drawn
|
||||
over the <b>same</b> laptop-camera frame it trained in, so it <b>aligns</b> with the image.
|
||||
Same person / room / session — not validated cross-day or through-wall.
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<!-- TensorFlow.js core + WebGPU/WASM backends (WebGL ships inside core as the final fallback) -->
|
||||
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@4.22.0/dist/tf.min.js" crossorigin="anonymous"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-backend-webgpu@4.22.0/dist/tf-backend-webgpu.min.js" crossorigin="anonymous"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-backend-wasm@4.22.0/dist/tf-backend-wasm.min.js" crossorigin="anonymous"></script>
|
||||
<!-- MediaPipe Pose 0.5 (legacy solutions API — same CDN the 05-skinned-realtime demo uses) -->
|
||||
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/pose@0.5/pose.js" crossorigin="anonymous"></script>
|
||||
|
||||
<script>
|
||||
"use strict";
|
||||
// ============================================================================
|
||||
// Constants & shared state
|
||||
// ============================================================================
|
||||
const CSI_WS = (new URLSearchParams(location.search)).get('ws')
|
||||
|| `ws://${location.hostname || 'localhost'}:8765/ws/sensing`;
|
||||
const NODE_IDS = [9, 13]; // per-node features in this fixed order (matches Python pipeline)
|
||||
const FIELD_LEN = 400; // signal_field.values padded/truncated to 400
|
||||
const CSI_DIM = 4 + NODE_IDS.length * 3 + FIELD_LEN; // 4 + 6 + 400 = 410
|
||||
const N_KP = 17, OUT_DIM = N_KP * 2; // 17 COCO keypoints -> 34 coords
|
||||
const BASELINE_SECONDS = 10; // empty-room calibration window
|
||||
const EPS = 1e-6;
|
||||
|
||||
// MediaPipe BlazePose (33) -> 17 COCO keypoints (identical to wiflow_capture.py / ADR-079)
|
||||
const COCO_FROM_MP = [0, 2, 5, 7, 8, 11, 12, 13, 14, 15, 16, 23, 24, 25, 26, 27, 28];
|
||||
const EDGES = [[5,7],[7,9],[6,8],[8,10],[5,6],[11,12],[5,11],[6,12],
|
||||
[11,13],[13,15],[12,14],[14,16],[0,1],[0,2],[1,3],[2,4],[0,5],[0,6]];
|
||||
|
||||
const $ = id => document.getElementById(id);
|
||||
function banner(state, txt){ const b=$('banner'); b.className=state; b.textContent=txt; }
|
||||
|
||||
// In-memory dataset of {csi:Float32Array(410, baseline-normalized), kps:Float32Array(34), bucket:int}
|
||||
let SAMPLES = [];
|
||||
// Latest live CSI frame + RAW 410-vector (baseline-normalization applied at use sites)
|
||||
let latestCSI = { t: 0, frame: null, vec: null, source: null, nodes: [] };
|
||||
// Empty-room baseline: per-feature mean + std (ADR-151)
|
||||
let baseline = null; // { mean:Float32Array(410), std:Float32Array(410), n:int, ts:number }
|
||||
|
||||
// ============================================================================
|
||||
// TF.js backend selection — WebGPU primary, WASM-SIMD fallback, WebGL last.
|
||||
// ============================================================================
|
||||
const BACKEND_LABEL = { webgpu:'WebGPU', wasm:'WASM-SIMD', webgl:'WebGL', cpu:'CPU (slow)' };
|
||||
let activeBackend = null;
|
||||
async function selectBackend(){
|
||||
try{ if (tf.wasm && tf.wasm.setWasmPaths)
|
||||
tf.wasm.setWasmPaths('https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-backend-wasm@4.22.0/dist/'); }catch(e){}
|
||||
const tryBackend = async (name)=>{
|
||||
try{ const ok = await tf.setBackend(name); if (!ok) return false; await tf.ready();
|
||||
return tf.getBackend() === name; }
|
||||
catch(e){ console.warn('backend '+name+' unavailable:', e.message); return false; }
|
||||
};
|
||||
if (await tryBackend('webgpu')) activeBackend = 'webgpu';
|
||||
else if (await tryBackend('wasm')) activeBackend = 'wasm';
|
||||
else if (await tryBackend('webgl')) activeBackend = 'webgl';
|
||||
else { await tf.ready(); activeBackend = tf.getBackend(); }
|
||||
const badge = $('compute');
|
||||
badge.textContent = 'compute: ' + (BACKEND_LABEL[activeBackend] || activeBackend);
|
||||
badge.title = 'TensorFlow.js backend actually running (WebGPU → WASM-SIMD → WebGL)';
|
||||
return activeBackend;
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// CSI vector construction — MUST match wiflow_capture.py csi_vector() exactly.
|
||||
// [mean_rssi, variance, motion_band_power, breathing_band_power] (4 global)
|
||||
// + for node 9 then node 13: [mean_rssi, variance, motion_band_power] (6 per-node)
|
||||
// + signal_field.values padded/truncated to 400 (400 field)
|
||||
// = 410-d (RAW — baseline-normalization applied separately, see baselineNorm)
|
||||
// ============================================================================
|
||||
function csiVector(frame){
|
||||
const f = frame.features || {};
|
||||
const out = new Float32Array(CSI_DIM);
|
||||
let o = 0;
|
||||
out[o++] = +f.mean_rssi || 0;
|
||||
out[o++] = +f.variance || 0;
|
||||
out[o++] = +f.motion_band_power || 0;
|
||||
out[o++] = +f.breathing_band_power || 0;
|
||||
const perNode = {};
|
||||
for (const nf of (frame.node_features || [])) perNode[nf.node_id] = (nf.features || {});
|
||||
for (const nid of NODE_IDS){
|
||||
const nf = perNode[nid] || {};
|
||||
out[o++] = +nf.mean_rssi || 0;
|
||||
out[o++] = +nf.variance || 0;
|
||||
out[o++] = +nf.motion_band_power || 0;
|
||||
}
|
||||
const field = ((frame.signal_field || {}).values) || [];
|
||||
for (let i = 0; i < FIELD_LEN; i++) out[o++] = +field[i] || 0;
|
||||
return out;
|
||||
}
|
||||
|
||||
// ADR-151 baseline-deviation normalization: x_norm = (x - base_mean) / (base_std + eps).
|
||||
// Applied BEFORE the model's own input standardization, consistently everywhere.
|
||||
function baselineNorm(vecRaw){
|
||||
if (!baseline) return null;
|
||||
const out = new Float32Array(CSI_DIM);
|
||||
for (let j = 0; j < CSI_DIM; j++)
|
||||
out[j] = (vecRaw[j] - baseline.mean[j]) / (baseline.std[j] + EPS);
|
||||
return out;
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// CSI WebSocket
|
||||
// ============================================================================
|
||||
function connectCSI(){
|
||||
banner('down','CONNECTING…');
|
||||
let ws;
|
||||
try { ws = new WebSocket(CSI_WS); }
|
||||
catch(e){ banner('down','NO-CSI-SERVER — start sensing-server :8765'); setTimeout(connectCSI, 1500); return; }
|
||||
ws.onopen = ()=> banner('sim','WAITING FOR CSI…');
|
||||
ws.onmessage = ev => {
|
||||
let d; try { d = JSON.parse(ev.data); } catch(e){ return; }
|
||||
if (!d.features && !d.signal_field) return;
|
||||
const src = d.source || 'unknown';
|
||||
latestCSI = {
|
||||
t: performance.now(),
|
||||
frame: d,
|
||||
vec: csiVector(d), // RAW
|
||||
source: src,
|
||||
nodes: (d.nodes || []).map(n => n.node_id).filter(x => x != null).sort((a,b)=>a-b)
|
||||
};
|
||||
if (src === 'esp32') banner('live','LIVE — real ESP32 CSI');
|
||||
else banner('sim',`SIMULATED — not real (source=${src})`);
|
||||
};
|
||||
ws.onerror = ()=>{ try{ws.close();}catch(e){} };
|
||||
ws.onclose = ()=>{ banner('down','NO-CSI-SERVER — start sensing-server :8765'); setTimeout(connectCSI, 1500); };
|
||||
}
|
||||
function freshLiveCSI(){
|
||||
return latestCSI.frame && latestCSI.source === 'esp32' && (performance.now() - latestCSI.t) < 400;
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Camera + MediaPipe Pose
|
||||
// ============================================================================
|
||||
let camStream = null;
|
||||
const camEl = document.createElement('video');
|
||||
camEl.autoplay = true; camEl.muted = true; camEl.playsInline = true;
|
||||
let mpPose = null, mpReady = false, mpBusy = false;
|
||||
let latestKps = null, latestVis = 0;
|
||||
|
||||
function initPose(){
|
||||
if (mpPose || typeof Pose === 'undefined') return;
|
||||
mpPose = new Pose({ locateFile: f => `https://cdn.jsdelivr.net/npm/@mediapipe/pose@0.5/${f}` });
|
||||
mpPose.setOptions({ modelComplexity:1, smoothLandmarks:true, enableSegmentation:false,
|
||||
minDetectionConfidence:0.5, minTrackingConfidence:0.5 });
|
||||
mpPose.onResults(onPoseResults);
|
||||
mpReady = true;
|
||||
}
|
||||
function onPoseResults(res){
|
||||
mpBusy = false;
|
||||
if (!res.poseLandmarks){ latestKps = null; latestVis = 0; return; }
|
||||
const lm = res.poseLandmarks;
|
||||
const kps = new Float32Array(OUT_DIM);
|
||||
let visSum = 0;
|
||||
for (let i = 0; i < N_KP; i++){
|
||||
const p = lm[COCO_FROM_MP[i]];
|
||||
kps[i*2] = p.x; kps[i*2+1] = p.y;
|
||||
visSum += (p.visibility != null ? p.visibility : 0);
|
||||
}
|
||||
latestKps = kps; latestVis = visSum / N_KP;
|
||||
}
|
||||
|
||||
async function startCam(deviceId){
|
||||
if (camStream) camStream.getTracks().forEach(t => t.stop());
|
||||
const constraints = deviceId ? { video:{ deviceId:{ exact:deviceId } } } : { video:true };
|
||||
const st = $('camStatus');
|
||||
try{
|
||||
st.textContent = 'camera: requesting…';
|
||||
camStream = await navigator.mediaDevices.getUserMedia(constraints);
|
||||
camEl.srcObject = camStream;
|
||||
await camEl.play().catch(()=>{});
|
||||
const tr = camStream.getVideoTracks()[0];
|
||||
const tick = ()=>{ st.textContent =
|
||||
`camera: "${tr.label}" ${camEl.videoWidth}x${camEl.videoHeight} ${tr.readyState} ${camEl.paused?'PAUSED':'playing'}`; };
|
||||
tick(); setInterval(tick, 1000);
|
||||
$('camBtn').textContent = 'switch camera ↻';
|
||||
$('recBtn').disabled = !baseline; // still gated on a baseline
|
||||
const devs = (await navigator.mediaDevices.enumerateDevices()).filter(d => d.kind === 'videoinput');
|
||||
const sel = $('camSel'); sel.style.display = devs.length > 1 ? 'inline-block' : 'none';
|
||||
sel.innerHTML = devs.map((d,i)=>`<option value="${d.deviceId}">${d.label || ('camera '+(i+1))}</option>`).join('');
|
||||
const cur = tr.getSettings().deviceId; if (cur) sel.value = cur;
|
||||
initPose();
|
||||
}catch(e){
|
||||
$('camBtn').textContent = 'camera error: ' + e.name +
|
||||
(e.name === 'NotReadableError' ? ' (in use by Zoom/Teams?)' : '');
|
||||
console.error('getUserMedia', e);
|
||||
}
|
||||
}
|
||||
$('camBtn').addEventListener('click', ()=> startCam());
|
||||
$('camSel').addEventListener('change', e => startCam(e.target.value));
|
||||
|
||||
// ============================================================================
|
||||
// Drawing helpers
|
||||
// ============================================================================
|
||||
function drawCameraFrame(ctx, W, H, alpha){
|
||||
if (camEl && camEl.videoWidth > 0){
|
||||
ctx.save(); ctx.globalAlpha = alpha;
|
||||
const vr = camEl.videoWidth / camEl.videoHeight, cr = W / H;
|
||||
let dw=W, dh=H, dx=0, dy=0;
|
||||
if (vr > cr){ dh=H; dw=H*vr; dx=(W-dw)/2; } else { dw=W; dh=W/vr; dy=(H-dh)/2; }
|
||||
ctx.drawImage(camEl, dx, dy, dw, dh); ctx.restore();
|
||||
return true;
|
||||
}
|
||||
ctx.fillStyle = '#070a0e'; ctx.fillRect(0,0,W,H);
|
||||
return false;
|
||||
}
|
||||
function drawSkeleton(ctx, kps, W, H, color, glow){
|
||||
const k = [];
|
||||
for (let i = 0; i < N_KP; i++) k.push([kps[i*2]*W, kps[i*2+1]*H]);
|
||||
ctx.lineWidth = 5; ctx.strokeStyle = color; ctx.lineCap = 'round';
|
||||
ctx.shadowColor = glow; ctx.shadowBlur = 8;
|
||||
for (const [a,b] of EDGES){ ctx.beginPath(); ctx.moveTo(k[a][0],k[a][1]); ctx.lineTo(k[b][0],k[b][1]); ctx.stroke(); }
|
||||
ctx.shadowBlur = 0;
|
||||
for (const [x,y] of k){ ctx.beginPath(); ctx.arc(x,y,5,0,7); ctx.fillStyle = color; ctx.fill(); }
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Stage navigation + gating
|
||||
// ============================================================================
|
||||
const STAGES = ['calibrate','capture','train','infer'];
|
||||
let stageDone = { calibrate:false, capture:false, train:false };
|
||||
function stageUnlocked(name){
|
||||
if (name === 'calibrate') return true;
|
||||
if (name === 'capture') return stageDone.calibrate;
|
||||
if (name === 'train') return stageDone.calibrate && SAMPLES.length >= 200;
|
||||
if (name === 'infer') return !!model;
|
||||
return false;
|
||||
}
|
||||
function gotoStage(name){
|
||||
if (!stageUnlocked(name)) return;
|
||||
document.querySelectorAll('.step').forEach(s => s.classList.remove('on'));
|
||||
document.querySelectorAll('.panel').forEach(p => p.classList.remove('on'));
|
||||
document.querySelector(`.step[data-stage="${name}"]`).classList.add('on');
|
||||
$('stage-' + name).classList.add('on');
|
||||
}
|
||||
function refreshGates(){
|
||||
document.querySelectorAll('.step').forEach(s=>{
|
||||
const name = s.dataset.stage;
|
||||
s.classList.toggle('locked', !stageUnlocked(name));
|
||||
s.classList.toggle('done', !!stageDone[name]);
|
||||
});
|
||||
$('recBtn').disabled = !(baseline && camStream);
|
||||
refreshTrainAvail();
|
||||
}
|
||||
document.querySelectorAll('.step').forEach(s => s.addEventListener('click', ()=> gotoStage(s.dataset.stage)));
|
||||
|
||||
// ============================================================================
|
||||
// STAGE 0 · CALIBRATE — Welford running mean+std over the 410-d CSI vector
|
||||
// ============================================================================
|
||||
const calCtx = $('calCv').getContext('2d');
|
||||
let calibrating = false;
|
||||
let cw = null; // welford accumulators { n, mean:Float64Array, m2:Float64Array, t0 }
|
||||
|
||||
function startCalibration(){
|
||||
if (calibrating) return;
|
||||
cw = { n:0, mean:new Float64Array(CSI_DIM), m2:new Float64Array(CSI_DIM), t0:performance.now() };
|
||||
calibrating = true;
|
||||
$('calStatus').textContent = 'CALIBRATING…'; $('calStatus').className = 'v';
|
||||
$('calBtn').disabled = true;
|
||||
}
|
||||
function welfordUpdate(vec){
|
||||
cw.n++;
|
||||
for (let j = 0; j < CSI_DIM; j++){
|
||||
const d = vec[j] - cw.mean[j];
|
||||
cw.mean[j] += d / cw.n;
|
||||
cw.m2[j] += d * (vec[j] - cw.mean[j]);
|
||||
}
|
||||
}
|
||||
function finishCalibration(){
|
||||
calibrating = false;
|
||||
const mean = new Float32Array(CSI_DIM), std = new Float32Array(CSI_DIM);
|
||||
for (let j = 0; j < CSI_DIM; j++){
|
||||
mean[j] = cw.mean[j];
|
||||
std[j] = cw.n > 1 ? Math.sqrt(cw.m2[j] / (cw.n - 1)) : 0;
|
||||
}
|
||||
baseline = { mean, std, n: cw.n, ts: Date.now() };
|
||||
stageDone.calibrate = true;
|
||||
$('calStatus').textContent = 'CALIBRATED'; $('calStatus').className = 'v green';
|
||||
$('calN').textContent = cw.n; $('calBtn').disabled = false;
|
||||
$('calBar').style.width = '100%';
|
||||
saveBaseline();
|
||||
refreshGates();
|
||||
}
|
||||
$('calBtn').addEventListener('click', startCalibration);
|
||||
$('recalBtn').addEventListener('click', ()=>{ baseline = null; stageDone.calibrate = false;
|
||||
$('calStatus').textContent = 'NOT CALIBRATED'; $('calStatus').className = 'v';
|
||||
$('calBar').style.width = '0%'; $('calN').textContent = '0'; idbDel('baseline'); refreshGates(); startCalibration(); });
|
||||
|
||||
function calibrateLoop(){
|
||||
const W = $('calCv').width, H = $('calCv').height;
|
||||
calCtx.fillStyle = '#070a0e'; calCtx.fillRect(0,0,W,H);
|
||||
// little live trace of motion_band_power to show the channel is quiescent
|
||||
$('calSrc').textContent = latestCSI.source || '—';
|
||||
$('calSrc').className = latestCSI.source === 'esp32' ? 'v green' : 'v';
|
||||
if (baseline){
|
||||
const ageS = Math.round((Date.now() - baseline.ts)/1000);
|
||||
$('calAge').textContent = ageS < 60 ? ageS+' s ago' : Math.round(ageS/60)+' min ago';
|
||||
}
|
||||
if (calibrating){
|
||||
const el = (performance.now() - cw.t0) / 1000;
|
||||
$('calBar').style.width = Math.min(100, 100*el/BASELINE_SECONDS) + '%';
|
||||
// accumulate only fresh live frames; ignore sim so the baseline is real
|
||||
if (freshLiveCSI() && latestCSI.vec){ welfordUpdate(latestCSI.vec); $('calN').textContent = cw.n; }
|
||||
// draw a centered "STEP OUT" reminder + countdown
|
||||
calCtx.fillStyle = '#ffb840'; calCtx.font = 'bold 22px monospace'; calCtx.textAlign='center';
|
||||
calCtx.fillText('STEP OUT OF THE SPACE', W/2, H/2-10);
|
||||
calCtx.fillStyle = '#7d8796'; calCtx.font = '14px monospace';
|
||||
calCtx.fillText('baseline: '+Math.max(0,Math.ceil(BASELINE_SECONDS-el))+' s · '+cw.n+' frames', W/2, H/2+18);
|
||||
calCtx.textAlign='start';
|
||||
if (el >= BASELINE_SECONDS && cw.n > 0) finishCalibration();
|
||||
else if (el >= BASELINE_SECONDS*2){ // safety: timed out with no live frames
|
||||
calibrating = false; $('calStatus').textContent = 'NO LIVE CSI — check esp32'; $('calStatus').className='v red';
|
||||
$('calBtn').disabled = false;
|
||||
}
|
||||
} else {
|
||||
calCtx.fillStyle = baseline ? '#46e08a' : '#7d8796'; calCtx.font = '14px monospace'; calCtx.textAlign='center';
|
||||
calCtx.fillText(baseline ? 'baseline ready ('+baseline.n+' frames)' : 'click “calibrate baseline”', W/2, H/2);
|
||||
calCtx.textAlign='start';
|
||||
}
|
||||
requestAnimationFrame(calibrateLoop);
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// STAGE 1 · GUIDED CAPTURE — balanced buckets + coverage meter
|
||||
// ============================================================================
|
||||
const capCtx = $('capCv').getContext('2d');
|
||||
let recording = false;
|
||||
// pose buckets (the guided routine cycles through these)
|
||||
const BUCKETS = ['stand still','turn left','turn right','walk left','walk right',
|
||||
'arms up','arms down','crouch','sit','reach'];
|
||||
const SECS_PER_BUCKET = 12;
|
||||
let bucketIx = 0, bucketT0 = performance.now();
|
||||
let covCounts = new Array(BUCKETS.length).fill(0);
|
||||
|
||||
function renderCoverage(){
|
||||
const max = Math.max(1, ...covCounts);
|
||||
$('cov').innerHTML = BUCKETS.map((b,i)=>
|
||||
`<div class="covrow"><span class="nm">${b}</span>`+
|
||||
`<span class="bar"><i style="width:${Math.round(100*covCounts[i]/max)}%"></i></span>`+
|
||||
`<span class="ct">${covCounts[i]}</span></div>`).join('');
|
||||
}
|
||||
$('recBtn').addEventListener('click', ()=>{
|
||||
if (!baseline || !camStream) return;
|
||||
recording = !recording;
|
||||
$('recBtn').textContent = recording ? '◼ stop recording' : '● start guided recording';
|
||||
$('recBtn').classList.toggle('ghost', recording);
|
||||
if (recording){ bucketT0 = performance.now(); }
|
||||
});
|
||||
$('clrBtn').addEventListener('click', async ()=>{
|
||||
SAMPLES = []; covCounts = new Array(BUCKETS.length).fill(0);
|
||||
await idbPut('samples', []);
|
||||
$('capN').textContent = '0'; $('trN').textContent = '0'; renderCoverage(); refreshGates();
|
||||
});
|
||||
|
||||
function captureLoop(){
|
||||
const W = $('capCv').width, H = $('capCv').height;
|
||||
drawCameraFrame(capCtx, W, H, 0.9);
|
||||
if (mpReady && !mpBusy && camEl.videoWidth > 0){
|
||||
mpBusy = true; mpPose.send({ image: camEl }).catch(()=>{ mpBusy = false; });
|
||||
}
|
||||
if (latestKps) drawSkeleton(capCtx, latestKps, W, H, 'rgba(90,169,255,.95)', 'rgba(90,169,255,.6)');
|
||||
|
||||
$('capSrc').textContent = latestCSI.source || '—';
|
||||
$('capSrc').className = latestCSI.source === 'esp32' ? 'v green' : 'v';
|
||||
$('capNodes').textContent = latestCSI.nodes.length ? latestCSI.nodes.join(', ') : '—';
|
||||
$('capVis').textContent = latestKps ? latestVis.toFixed(2) : '—';
|
||||
|
||||
if (recording){
|
||||
// advance the guided bucket
|
||||
const el = (performance.now() - bucketT0)/1000;
|
||||
if (el >= SECS_PER_BUCKET){ bucketIx = (bucketIx+1) % BUCKETS.length; bucketT0 = performance.now(); }
|
||||
$('prompt').textContent = BUCKETS[bucketIx];
|
||||
$('countdown').textContent = `${Math.max(0,Math.ceil(SECS_PER_BUCKET - el))} s · bucket ${bucketIx+1}/${BUCKETS.length}`;
|
||||
|
||||
let skip = null;
|
||||
if (!latestKps || latestVis <= 0.5) skip = 'no confident pose';
|
||||
else if (!freshLiveCSI()) skip = (latestCSI.source && latestCSI.source!=='esp32') ? 'CSI not esp32 (sim)' : 'no fresh CSI';
|
||||
else if (!baseline) skip = 'no baseline';
|
||||
if (skip){ $('capSkip').textContent = skip; }
|
||||
else {
|
||||
const norm = baselineNorm(latestCSI.vec); // baseline-deviation normalized
|
||||
SAMPLES.push({ csi: norm, kps: latestKps.slice(), bucket: bucketIx });
|
||||
covCounts[bucketIx]++;
|
||||
$('capSkip').textContent = '—';
|
||||
const n = SAMPLES.length; $('capN').textContent = n; $('trN').textContent = n;
|
||||
if (n % 20 === 0){ renderCoverage(); idbSave(); refreshGates(); }
|
||||
}
|
||||
} else {
|
||||
$('prompt').textContent = baseline ? 'ready — press start' : 'calibrate baseline first';
|
||||
$('countdown').textContent = '—';
|
||||
}
|
||||
requestAnimationFrame(captureLoop);
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// IndexedDB persistence
|
||||
// ============================================================================
|
||||
const IDB_NAME = 'wiflow-browser', IDB_STORE = 'kv';
|
||||
function idbOpen(){
|
||||
return new Promise((res, rej)=>{
|
||||
const r = indexedDB.open(IDB_NAME, 1);
|
||||
r.onupgradeneeded = ()=> r.result.createObjectStore(IDB_STORE);
|
||||
r.onsuccess = ()=> res(r.result); r.onerror = ()=> rej(r.error);
|
||||
});
|
||||
}
|
||||
async function idbPut(key, val){
|
||||
const db = await idbOpen();
|
||||
return new Promise((res, rej)=>{
|
||||
const tx = db.transaction(IDB_STORE, 'readwrite');
|
||||
tx.objectStore(IDB_STORE).put(val, key); tx.oncomplete = res; tx.onerror = ()=> rej(tx.error);
|
||||
});
|
||||
}
|
||||
async function idbGet(key){
|
||||
const db = await idbOpen();
|
||||
return new Promise((res, rej)=>{
|
||||
const tx = db.transaction(IDB_STORE, 'readonly');
|
||||
const r = tx.objectStore(IDB_STORE).get(key);
|
||||
r.onsuccess = ()=> res(r.result); r.onerror = ()=> rej(r.error);
|
||||
});
|
||||
}
|
||||
async function idbDel(key){
|
||||
const db = await idbOpen();
|
||||
return new Promise((res, rej)=>{
|
||||
const tx = db.transaction(IDB_STORE, 'readwrite');
|
||||
tx.objectStore(IDB_STORE).delete(key); tx.oncomplete = res; tx.onerror = ()=> rej(tx.error);
|
||||
});
|
||||
}
|
||||
async function idbSave(){
|
||||
try{
|
||||
const flat = SAMPLES.map(s => ({ csi: Array.from(s.csi), kps: Array.from(s.kps), bucket: s.bucket }));
|
||||
await idbPut('samples', flat);
|
||||
}catch(e){ console.warn('idbSave', e); }
|
||||
}
|
||||
async function idbLoad(){
|
||||
try{
|
||||
const flat = await idbGet('samples');
|
||||
if (Array.isArray(flat) && flat.length){
|
||||
SAMPLES = flat.map(s => ({ csi: Float32Array.from(s.csi), kps: Float32Array.from(s.kps), bucket: s.bucket||0 }));
|
||||
covCounts = new Array(BUCKETS.length).fill(0);
|
||||
for (const s of SAMPLES) if (s.bucket < BUCKETS.length) covCounts[s.bucket]++;
|
||||
$('capN').textContent = SAMPLES.length; $('trN').textContent = SAMPLES.length;
|
||||
}
|
||||
}catch(e){ console.warn('idbLoad', e); }
|
||||
}
|
||||
async function saveBaseline(){
|
||||
try{ await idbPut('baseline', { mean: Array.from(baseline.mean), std: Array.from(baseline.std), n: baseline.n, ts: baseline.ts }); }
|
||||
catch(e){ console.warn('saveBaseline', e); }
|
||||
}
|
||||
async function loadBaseline(){
|
||||
try{
|
||||
const b = await idbGet('baseline');
|
||||
if (b && b.mean){
|
||||
baseline = { mean: Float32Array.from(b.mean), std: Float32Array.from(b.std), n: b.n, ts: b.ts };
|
||||
stageDone.calibrate = true;
|
||||
$('calStatus').textContent = 'CALIBRATED (restored)'; $('calStatus').className = 'v green';
|
||||
$('calN').textContent = baseline.n; $('calBar').style.width = '100%';
|
||||
}
|
||||
}catch(e){ /* none yet */ }
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// STAGE 2 · TRAIN (TensorFlow.js)
|
||||
// ============================================================================
|
||||
let model = null, normMu = null, normSd = null, trainedPck10 = null, trainStop = false;
|
||||
const lossCtx = $('lossCv').getContext('2d');
|
||||
let lossHist = [];
|
||||
|
||||
function refreshTrainAvail(){
|
||||
const ok = !!baseline && SAMPLES.length >= 200;
|
||||
$('trainBtn').disabled = !ok;
|
||||
if (!baseline) $('trMsg').innerHTML = 'Calibrate a baseline first (stage 0).';
|
||||
else $('trMsg').innerHTML = SAMPLES.length >= 200
|
||||
? `Ready: ${SAMPLES.length} samples. Click <b>train model</b>.`
|
||||
: `Need ≥200 samples to train (have ${SAMPLES.length}). Capture more in stage 1.`;
|
||||
}
|
||||
|
||||
function buildMatrices(){
|
||||
const n = SAMPLES.length;
|
||||
const X = new Float32Array(n * CSI_DIM), Y = new Float32Array(n * OUT_DIM);
|
||||
for (let i = 0; i < n; i++){ X.set(SAMPLES[i].csi, i*CSI_DIM); Y.set(SAMPLES[i].kps, i*OUT_DIM); }
|
||||
return { X, Y, n };
|
||||
}
|
||||
function pckMpjpe(predArr, gtArr, m, thr){
|
||||
let hit = 0, tot = 0, dsum = 0;
|
||||
for (let i = 0; i < m; i++) for (let j = 0; j < N_KP; j++){
|
||||
const dx = predArr[i*OUT_DIM+j*2]-gtArr[i*OUT_DIM+j*2];
|
||||
const dy = predArr[i*OUT_DIM+j*2+1]-gtArr[i*OUT_DIM+j*2+1];
|
||||
const d = Math.hypot(dx, dy);
|
||||
if (d < thr) hit++; dsum += d; tot++;
|
||||
}
|
||||
return { pck: tot?hit/tot:0, mpjpe: tot?dsum/tot:NaN };
|
||||
}
|
||||
function drawLoss(){
|
||||
const W = $('lossCv').width, H = $('lossCv').height;
|
||||
lossCtx.fillStyle = '#070a0e'; lossCtx.fillRect(0,0,W,H);
|
||||
if (lossHist.length < 2) return;
|
||||
let mx = 0; for (const p of lossHist) mx = Math.max(mx, p.tr, p.va||0); mx = mx||1;
|
||||
const X = i => 8 + (W-16)*i/(lossHist.length-1);
|
||||
const Yv = v => H-8 - (H-16)*Math.min(v/mx,1);
|
||||
const line = (key,color)=>{
|
||||
lossCtx.strokeStyle=color; lossCtx.lineWidth=2; lossCtx.beginPath(); let st=false;
|
||||
lossHist.forEach((p,i)=>{ const v=p[key]; if(v==null) return;
|
||||
const x=X(i), y=Yv(v); st?lossCtx.lineTo(x,y):lossCtx.moveTo(x,y); st=true; });
|
||||
lossCtx.stroke();
|
||||
};
|
||||
line('tr','#ffb840'); line('va','#5aa9ff');
|
||||
}
|
||||
|
||||
async function trainModel(){
|
||||
if (!baseline || SAMPLES.length < 200) return;
|
||||
trainStop = false;
|
||||
$('trainBtn').disabled = true; $('trStop').disabled = false; lossHist = [];
|
||||
const epochs = Math.max(20, Math.min(600, parseInt($('trEpochs').value)||200));
|
||||
const { X, Y, n } = buildMatrices(); // X is already baseline-normalized
|
||||
const cut = Math.floor(n*0.8);
|
||||
$('trSplit').textContent = `${cut} / ${n-cut} (chronological 80/20)`;
|
||||
|
||||
// input standardization on TRAIN split only (on top of baseline-normalization)
|
||||
normMu = new Float32Array(CSI_DIM); normSd = new Float32Array(CSI_DIM);
|
||||
for (let j = 0; j < CSI_DIM; j++){
|
||||
let s=0; for (let i=0;i<cut;i++) s += X[i*CSI_DIM+j];
|
||||
const mu=s/cut; normMu[j]=mu;
|
||||
let v=0; for (let i=0;i<cut;i++){ const d=X[i*CSI_DIM+j]-mu; v+=d*d; }
|
||||
normSd[j]=Math.sqrt(v/cut)+EPS;
|
||||
}
|
||||
const Xn = new Float32Array(n*CSI_DIM);
|
||||
for (let i=0;i<n;i++) for (let j=0;j<CSI_DIM;j++) Xn[i*CSI_DIM+j]=(X[i*CSI_DIM+j]-normMu[j])/normSd[j];
|
||||
|
||||
// mean-pose baseline — the bar to beat
|
||||
const meanPose = new Float32Array(OUT_DIM);
|
||||
for (let i=0;i<cut;i++) for (let j=0;j<OUT_DIM;j++) meanPose[j]+=Y[i*OUT_DIM+j];
|
||||
for (let j=0;j<OUT_DIM;j++) meanPose[j]/=cut;
|
||||
const mVal = n-cut;
|
||||
const basePred = new Float32Array(mVal*OUT_DIM);
|
||||
for (let i=0;i<mVal;i++) basePred.set(meanPose, i*OUT_DIM);
|
||||
const gtVal = Y.slice(cut*OUT_DIM);
|
||||
const base = pckMpjpe(basePred, gtVal, mVal, 0.10);
|
||||
$('trBase').textContent = (base.pck*100).toFixed(1)+'%';
|
||||
|
||||
const xtr = tf.tensor2d(Xn.slice(0,cut*CSI_DIM),[cut,CSI_DIM]);
|
||||
const ytr = tf.tensor2d(Y.slice(0,cut*OUT_DIM),[cut,OUT_DIM]);
|
||||
const xva = tf.tensor2d(Xn.slice(cut*CSI_DIM),[mVal,CSI_DIM]);
|
||||
|
||||
if (model){ model.dispose(); }
|
||||
model = tf.sequential();
|
||||
model.add(tf.layers.dense({ inputShape:[CSI_DIM], units:512, activation:'relu' }));
|
||||
model.add(tf.layers.dropout({ rate:0.3 }));
|
||||
model.add(tf.layers.dense({ units:256, activation:'relu' }));
|
||||
model.add(tf.layers.dropout({ rate:0.3 }));
|
||||
model.add(tf.layers.dense({ units:128, activation:'relu' }));
|
||||
model.add(tf.layers.dense({ units:OUT_DIM, activation:'sigmoid' }));
|
||||
model.compile({ optimizer: tf.train.adam(1e-3), loss:'meanSquaredError' });
|
||||
|
||||
let bestP10 = 0, bestVal = 1e9;
|
||||
$('trMsg').innerHTML = 'Training… on <code>'+(BACKEND_LABEL[tf.getBackend()]||tf.getBackend())+'</code>';
|
||||
await model.fit(xtr, ytr, {
|
||||
epochs, batchSize:64, shuffle:true, verbose:0,
|
||||
callbacks:{ onEpochEnd: async (ep, logs)=>{
|
||||
let va=null,p10=null,p05=null,mpj=null;
|
||||
if (ep % 5 === 0 || ep === epochs-1){
|
||||
const pv = model.predict(xva); const pvArr = await pv.data(); pv.dispose();
|
||||
let vsum=0; for (let i=0;i<pvArr.length;i++){ const d=pvArr[i]-gtVal[i]; vsum+=d*d; }
|
||||
va = vsum/pvArr.length;
|
||||
const r10=pckMpjpe(pvArr,gtVal,mVal,0.10), r05=pckMpjpe(pvArr,gtVal,mVal,0.05);
|
||||
p10=r10.pck; p05=r05.pck; mpj=r10.mpjpe;
|
||||
$('trP10').textContent=(p10*100).toFixed(1)+'%'; $('trP05').textContent=(p05*100).toFixed(1)+'%';
|
||||
$('trMpj').textContent=mpj.toFixed(4); $('trVal').textContent=va.toFixed(4);
|
||||
if (va<bestVal){ bestVal=va; bestP10=p10; }
|
||||
}
|
||||
$('trEpoch').textContent=(ep+1); $('trLoss').textContent=logs.loss.toFixed(4);
|
||||
$('trBar').style.width=(100*(ep+1)/epochs)+'%';
|
||||
lossHist.push({ ep, tr:logs.loss, va }); drawLoss();
|
||||
if (trainStop) model.stopTraining = true;
|
||||
await tf.nextFrame();
|
||||
}}
|
||||
});
|
||||
|
||||
const pvF = model.predict(xva); const pvFArr = await pvF.data(); pvF.dispose();
|
||||
const fin10 = pckMpjpe(pvFArr,gtVal,mVal,0.10), fin05 = pckMpjpe(pvFArr,gtVal,mVal,0.05);
|
||||
const finPck = Math.max(bestP10, fin10.pck); trainedPck10 = finPck;
|
||||
$('trP10').textContent=(fin10.pck*100).toFixed(1)+'%'; $('trP05').textContent=(fin05.pck*100).toFixed(1)+'%';
|
||||
$('trMpj').textContent=fin10.mpjpe.toFixed(4); $('infPck').textContent=(finPck*100).toFixed(1)+'%';
|
||||
|
||||
const delta = (finPck - base.pck)*100;
|
||||
const v = $('verdict');
|
||||
if (delta > 1){
|
||||
v.className='verdict good';
|
||||
v.innerHTML = `model <b>BEATS</b> mean-pose baseline by <b>+${delta.toFixed(1)} pp</b> → real CSI→pose signal.`;
|
||||
} else {
|
||||
v.className='verdict bad';
|
||||
v.innerHTML = `model does <b>NOT</b> beat baseline (Δ ${delta.toFixed(1)} pp) → <b>no usable signal (honest)</b>. Capture more / more varied data.`;
|
||||
}
|
||||
stageDone.train = true;
|
||||
$('infModelState').textContent = `model ready · held-out PCK@0.10 ${(finPck*100).toFixed(1)}%`;
|
||||
$('trMsg').innerHTML = 'Done. Saving model to IndexedDB…';
|
||||
xtr.dispose(); ytr.dispose(); xva.dispose();
|
||||
await saveModel();
|
||||
$('trMsg').innerHTML = 'Saved. Go to <b>3 · INFER</b> to see WiFi drive the skeleton.';
|
||||
$('trainBtn').disabled = false; $('trStop').disabled = true;
|
||||
refreshGates();
|
||||
}
|
||||
$('trainBtn').addEventListener('click', trainModel);
|
||||
$('trStop').addEventListener('click', ()=>{ trainStop = true; });
|
||||
|
||||
async function saveModel(){
|
||||
if (!model) return;
|
||||
try{
|
||||
await model.save('indexeddb://wiflow-model');
|
||||
await idbPut('norm', { mu:Array.from(normMu), sd:Array.from(normSd), pck10:trainedPck10 });
|
||||
}catch(e){ console.warn('saveModel', e); }
|
||||
}
|
||||
async function loadModel(){
|
||||
try{
|
||||
const m = await tf.loadLayersModel('indexeddb://wiflow-model');
|
||||
const norm = await idbGet('norm');
|
||||
if (m && norm){
|
||||
model = m; normMu = Float32Array.from(norm.mu); normSd = Float32Array.from(norm.sd);
|
||||
trainedPck10 = norm.pck10; stageDone.train = true;
|
||||
$('infPck').textContent = trainedPck10!=null ? (trainedPck10*100).toFixed(1)+'%' : '—';
|
||||
$('infModelState').textContent = `model loaded · held-out PCK@0.10 ${trainedPck10!=null?(trainedPck10*100).toFixed(1)+'%':'?'}`;
|
||||
}
|
||||
}catch(e){ /* none yet */ }
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// STAGE 3 · INFER — live CSI → baseline-normalize → standardize → model
|
||||
// ============================================================================
|
||||
const infCtx = $('infCv').getContext('2d');
|
||||
let infSm = null, infFrames = 0, infT0 = performance.now();
|
||||
function inferSmooth(kps){
|
||||
if (!infSm){ infSm = Float32Array.from(kps); return infSm; }
|
||||
const a = 0.35; for (let i=0;i<kps.length;i++) infSm[i]+=a*(kps[i]-infSm[i]);
|
||||
return infSm;
|
||||
}
|
||||
function inferLoop(){
|
||||
const W = $('infCv').width, H = $('infCv').height;
|
||||
const showCam = !$('hideCam').checked;
|
||||
if (showCam) drawCameraFrame(infCtx, W, H, 0.85);
|
||||
else { infCtx.fillStyle='#070a0e'; infCtx.fillRect(0,0,W,H); }
|
||||
|
||||
$('infSrc').textContent = latestCSI.source || '—';
|
||||
$('infSrc').className = latestCSI.source === 'esp32' ? 'v green' : 'v';
|
||||
$('infNodes').textContent = latestCSI.nodes.length ? latestCSI.nodes.join(', ') : '—';
|
||||
const cls = (latestCSI.frame && latestCSI.frame.classification) || {};
|
||||
$('infPres').textContent = cls.presence ? 'PRESENT' : '—';
|
||||
|
||||
if (model && normMu && baseline && latestCSI.vec){
|
||||
const out = tf.tidy(()=>{
|
||||
const xn = new Float32Array(CSI_DIM);
|
||||
for (let j=0;j<CSI_DIM;j++){
|
||||
const bn = (latestCSI.vec[j]-baseline.mean[j])/(baseline.std[j]+EPS); // baseline-normalize
|
||||
xn[j] = (bn - normMu[j])/normSd[j]; // then standardize
|
||||
}
|
||||
return model.predict(tf.tensor2d(xn,[1,CSI_DIM]));
|
||||
});
|
||||
out.data().then(arr=>{
|
||||
const sm = inferSmooth(arr); const present = !!cls.presence;
|
||||
drawSkeleton(infCtx, sm, W, H,
|
||||
present?'rgba(70,224,138,.95)':'rgba(125,135,150,.85)','rgba(70,224,138,.6)');
|
||||
out.dispose();
|
||||
}).catch(()=> out.dispose());
|
||||
infFrames++;
|
||||
} else {
|
||||
infCtx.fillStyle='#7d8796'; infCtx.font='13px monospace';
|
||||
infCtx.fillText(model?'waiting for CSI…':'train a model first (stage 2)', 20, 30);
|
||||
}
|
||||
const now = performance.now();
|
||||
if (now-infT0 > 1000){ $('infFps').textContent = infFrames; infFrames = 0; infT0 = now; }
|
||||
requestAnimationFrame(inferLoop);
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Boot
|
||||
// ============================================================================
|
||||
(async function boot(){
|
||||
connectCSI();
|
||||
await selectBackend();
|
||||
await loadBaseline();
|
||||
await idbLoad();
|
||||
await loadModel();
|
||||
renderCoverage();
|
||||
refreshGates();
|
||||
requestAnimationFrame(calibrateLoop);
|
||||
requestAnimationFrame(captureLoop);
|
||||
requestAnimationFrame(inferLoop);
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user