mirror of
https://github.com/ruvnet/RuView
synced 2026-06-18 11:43:19 +00:00
rUv
27a6edba8b
* feat(examples/three.js): cinematic skinned realtime pose demo + ESP32 CSI bridge
Five-stage example progression exploring three.js helpers (ADR-097 surface) as
a viewer for live RuView sensor data:
1. helpers-demo.html — clean ADR-097 helper reference (GridHelper,
PolarGridHelper, BoxHelper, AxesHelper),
file://-safe, no backend
2. helpers-cinematic.html — same scene + UnrealBloomPass + pseudo-CSI
sonar pings + tomography sweep + procedural
cyber floor + ambient drift particles
3. helpers-skinned.html — replaces sphere skeleton with Mixamo X Bot
via GLTFLoader from threejs.org CDN, plays
bundled animations with additive blending
4. helpers-skinned-fbx.html — same but loads a local Mixamo FBX (needs
serve-demo.py — file:// can't fetch local
siblings). Drop X Bot.fbx alongside.
5. helpers-skinned-realtime.html — webcam → MediaPipe Pose Heavy →
poseWorldLandmarks → direct quaternion
retargeting onto the Mixamo skeleton.
Real ESP32-S3 CSI streamed over WebSocket
from ruvultra (Tailscale, port 8766).
Supporting:
- serve-demo.py threaded HTTP server with no-cache headers
(fixes net::ERR_EMPTY_RESPONSE on the FBX path)
- ruvultra-csi-bridge.py ESP32 RuView firmware tick → WebSocket bridge,
runs as systemd-run unit on ruvultra
Bugs found + fixed along the way (all documented in code comments):
- FBX exports yield TWO parallel Bone trees with identical names; only the
SkinnedMesh.skeleton.bones one drives visible deformation. model.traverse
finds orphans.
- Mixamo FBX nests a zero-length wrapper bone above the real bone, same name.
bone.children[0].getWorldPosition == bone.getWorldPosition → restDir is
(0,0,0) → setFromUnitVectors collapses to identity. Walk past same-named
same-position wrappers when computing tail.
- AnimationMixer.update() with a "stopped" action still mutates bones unless
enabled=false is set.
Retargeting layer in helpers-skinned-realtime.html:
- 12 bones direct quaternion retarget (arms × 2, legs × 2, spine × 3, neck)
- Hips root rotation from shoulder/hip line basis (torso twist + lean)
- Neck aims at ear-midpoint (kp 7+8), not nose (kp 0), to remove the
forward bias of the protruding-nose anchor
- One Euro Filter per landmark per axis (Casiez 2012) — adaptive low-pass
- Visibility-weighted per-bone slerp gain — occluded limbs relax to rest
- URL toggles: ?mirror= ?yflip= ?zflip= ?cnn=0/1/2 ?csi=ws://...
Live CSI integration:
- Bridge parses adaptive_ctrl tick lines (motion/presence/rssi/yield)
- Browser fans single ESP32 reading across 4 UI nodes with phase-shifted
wobble (0.88–1.00 × sin(t·0.55 + offsetᵢ))
- EMA α=0.06 (~3 sec time constant), HUD update throttled 3 Hz
Co-Authored-By: claude-flow <ruv@ruv.net>
* refactor(examples/three.js): organize into demos/screenshots/server/assets + add README
Flatten the 13-file flat layout into purposeful subfolders so the demo
collection has a clean top-level entry point (README.md) and the file roles
are obvious from a directory listing.
Layout:
demos/ 01..05 — numbered for the progression (helpers → cinematic →
skinned → skinned-fbx → skinned-realtime)
screenshots/ one PNG per demo, matching the demo's filename prefix
server/ serve-demo.py + ruvultra-csi-bridge.py
assets/ X Bot.fbx (gitignored, used by demos 04 and 05)
Touched files (beyond the renames):
- 04-skinned-fbx.html, 05-skinned-realtime.html: MODEL_URL now resolves
'../assets/X%20Bot.fbx' instead of './X%20Bot.fbx'
- server/serve-demo.py: chdir() walks 3 levels up to repo root (was 2), and
the URL banner now lists all 5 demos
- .gitignore: comment refresh — points at assets/ and screenshots/
- 05-skinned-realtime.html also picks up in-flight fps-tune work from this
branch (Holistic script, SMOOTH_K URL param, slerp gain scaling) since
those edits and the rename hit the same file
Verified end-to-end:
- python examples/three.js/server/serve-demo.py
- all 5 demos return 200, X Bot.fbx returns 200 from new asset/ path
- demos 04 + 05 render the X Bot mesh; 0 JS errors via browser eval
- screenshots reproduced match the originals
Co-Authored-By: claude-flow <ruv@ruv.net>
78 lines
3.1 KiB
Markdown
78 lines
3.1 KiB
Markdown
# three.js demos
|
|
|
|
Five progressively richer browser demos of the ADR-097 sensing-helpers scene,
|
|
ending with a live MediaPipe-Pose → Mixamo X Bot retargeting pipeline driven
|
|
by a real ESP32 CSI feed.
|
|
|
|
## Run them
|
|
|
|
```bash
|
|
python examples/three.js/server/serve-demo.py
|
|
# then open one of the URLs the script prints
|
|
```
|
|
|
|
`server/serve-demo.py` is a tiny `ThreadingHTTPServer` with aggressive
|
|
no-cache headers — the stdlib `http.server` is single-threaded and times out
|
|
on the parallel script + FBX fetches the demos make.
|
|
|
|
## Demos
|
|
|
|
| # | File | What it shows |
|
|
|---|------|---------------|
|
|
| 01 | [`demos/01-helpers.html`](demos/01-helpers.html) | Plain ADR-097 helpers in the point-cloud viewer |
|
|
| 02 | [`demos/02-cinematic.html`](demos/02-cinematic.html) | Cinematic camera + pseudo-CSI visualization on top of #01 |
|
|
| 03 | [`demos/03-skinned.html`](demos/03-skinned.html) | GLTF skinned mesh + additive animation blending |
|
|
| 04 | [`demos/04-skinned-fbx.html`](demos/04-skinned-fbx.html) | Mixamo X Bot loaded from FBX in the ADR-097 scene |
|
|
| 05 | [`demos/05-skinned-realtime.html`](demos/05-skinned-realtime.html) | Webcam → MediaPipe Pose Heavy → Mixamo IK retarget, live ESP32 CSI overlay |
|
|
|
|
| Screenshot | |
|
|
|---|---|
|
|
|  |  |
|
|
|  |  |
|
|
|  | |
|
|
|
|
## Layout
|
|
|
|
```
|
|
examples/three.js/
|
|
├── README.md
|
|
├── .gitignore
|
|
├── demos/ # 5 self-contained HTML demos
|
|
│ ├── 01-helpers.html
|
|
│ ├── 02-cinematic.html
|
|
│ ├── 03-skinned.html
|
|
│ ├── 04-skinned-fbx.html
|
|
│ └── 05-skinned-realtime.html
|
|
├── screenshots/ # one PNG per demo
|
|
│ └── 0N-*.png
|
|
├── server/
|
|
│ ├── serve-demo.py # local HTTP server with no-cache headers
|
|
│ └── ruvultra-csi-bridge.py # ESP32 CSI WebSocket bridge (ruvultra:8766)
|
|
└── assets/
|
|
└── X Bot.fbx # gitignored — get your own from mixamo.com
|
|
# (FBX Binary, T-Pose, Without Skin)
|
|
# used by demos 04 and 05
|
|
```
|
|
|
|
## Mixamo X Bot
|
|
|
|
Demos 04 and 05 expect `assets/X Bot.fbx`. It's gitignored (size + license
|
|
boundary). Download yours from [mixamo.com](https://mixamo.com): pick the
|
|
"X Bot" character, export as **FBX Binary**, **T-Pose**, **Without Skin**,
|
|
and drop it into `assets/`.
|
|
|
|
## Live ESP32 CSI overlay (demo 05 only)
|
|
|
|
`server/ruvultra-csi-bridge.py` is the systemd-deployable bridge that runs on
|
|
the `ruvultra` host (over Tailscale). It listens for ESP32-S3 CSI on UDP and
|
|
re-broadcasts it as WebSocket frames at `ws://ruvultra:8766/csi`. Demo 05
|
|
auto-connects; if the socket is down, it falls back to the bundled idle clip
|
|
plus a synthetic CSI driver.
|
|
|
|
## Open issues
|
|
|
|
- [#583](https://github.com/ruvnet/RuView/issues/583) — head/face tracking
|
|
fidelity in `05-skinned-realtime.html`. Recommended fix: swap MediaPipe
|
|
Pose Heavy for MediaPipe Holistic (same API, adds 468-point face mesh +
|
|
hand landmarks for proper PnP head pose and finger curl tracking).
|