Files
rUv 27a6edba8b feat(examples/three.js): cinematic skinned realtime pose demo + folder reorg (#584)
* feat(examples/three.js): cinematic skinned realtime pose demo + ESP32 CSI bridge

Five-stage example progression exploring three.js helpers (ADR-097 surface) as
a viewer for live RuView sensor data:

1. helpers-demo.html              — clean ADR-097 helper reference (GridHelper,
                                    PolarGridHelper, BoxHelper, AxesHelper),
                                    file://-safe, no backend
2. helpers-cinematic.html         — same scene + UnrealBloomPass + pseudo-CSI
                                    sonar pings + tomography sweep + procedural
                                    cyber floor + ambient drift particles
3. helpers-skinned.html           — replaces sphere skeleton with Mixamo X Bot
                                    via GLTFLoader from threejs.org CDN, plays
                                    bundled animations with additive blending
4. helpers-skinned-fbx.html       — same but loads a local Mixamo FBX (needs
                                    serve-demo.py — file:// can't fetch local
                                    siblings). Drop X Bot.fbx alongside.
5. helpers-skinned-realtime.html  — webcam → MediaPipe Pose Heavy →
                                    poseWorldLandmarks → direct quaternion
                                    retargeting onto the Mixamo skeleton.
                                    Real ESP32-S3 CSI streamed over WebSocket
                                    from ruvultra (Tailscale, port 8766).

Supporting:
  - serve-demo.py             threaded HTTP server with no-cache headers
                               (fixes net::ERR_EMPTY_RESPONSE on the FBX path)
  - ruvultra-csi-bridge.py    ESP32 RuView firmware tick → WebSocket bridge,
                               runs as systemd-run unit on ruvultra

Bugs found + fixed along the way (all documented in code comments):
  - FBX exports yield TWO parallel Bone trees with identical names; only the
    SkinnedMesh.skeleton.bones one drives visible deformation. model.traverse
    finds orphans.
  - Mixamo FBX nests a zero-length wrapper bone above the real bone, same name.
    bone.children[0].getWorldPosition == bone.getWorldPosition → restDir is
    (0,0,0) → setFromUnitVectors collapses to identity. Walk past same-named
    same-position wrappers when computing tail.
  - AnimationMixer.update() with a "stopped" action still mutates bones unless
    enabled=false is set.

Retargeting layer in helpers-skinned-realtime.html:
  - 12 bones direct quaternion retarget (arms × 2, legs × 2, spine × 3, neck)
  - Hips root rotation from shoulder/hip line basis (torso twist + lean)
  - Neck aims at ear-midpoint (kp 7+8), not nose (kp 0), to remove the
    forward bias of the protruding-nose anchor
  - One Euro Filter per landmark per axis (Casiez 2012) — adaptive low-pass
  - Visibility-weighted per-bone slerp gain — occluded limbs relax to rest
  - URL toggles: ?mirror= ?yflip= ?zflip= ?cnn=0/1/2 ?csi=ws://...

Live CSI integration:
  - Bridge parses adaptive_ctrl tick lines (motion/presence/rssi/yield)
  - Browser fans single ESP32 reading across 4 UI nodes with phase-shifted
    wobble (0.88–1.00 × sin(t·0.55 + offsetᵢ))
  - EMA α=0.06 (~3 sec time constant), HUD update throttled 3 Hz

Co-Authored-By: claude-flow <ruv@ruv.net>

* refactor(examples/three.js): organize into demos/screenshots/server/assets + add README

Flatten the 13-file flat layout into purposeful subfolders so the demo
collection has a clean top-level entry point (README.md) and the file roles
are obvious from a directory listing.

Layout:
  demos/         01..05 — numbered for the progression (helpers → cinematic →
                          skinned → skinned-fbx → skinned-realtime)
  screenshots/   one PNG per demo, matching the demo's filename prefix
  server/        serve-demo.py + ruvultra-csi-bridge.py
  assets/        X Bot.fbx (gitignored, used by demos 04 and 05)

Touched files (beyond the renames):
- 04-skinned-fbx.html, 05-skinned-realtime.html: MODEL_URL now resolves
  '../assets/X%20Bot.fbx' instead of './X%20Bot.fbx'
- server/serve-demo.py: chdir() walks 3 levels up to repo root (was 2), and
  the URL banner now lists all 5 demos
- .gitignore: comment refresh — points at assets/ and screenshots/
- 05-skinned-realtime.html also picks up in-flight fps-tune work from this
  branch (Holistic script, SMOOTH_K URL param, slerp gain scaling) since
  those edits and the rename hit the same file

Verified end-to-end:
- python examples/three.js/server/serve-demo.py
- all 5 demos return 200, X Bot.fbx returns 200 from new asset/ path
- demos 04 + 05 render the X Bot mesh; 0 JS errors via browser eval
- screenshots reproduced match the originals

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-05-17 17:01:02 -04:00

154 lines
4.8 KiB
Python

#!/usr/bin/env python3
"""ruvultra → browser CSI bridge.
Reads adaptive_ctrl tick lines from the ESP32-S3 RuView firmware on
/dev/ttyACM0 and forwards normalized per-node metrics over a WebSocket
that the helpers-skinned-realtime demo can subscribe to via Tailscale.
Sample serial line (1 Hz cadence from firmware):
I (22890561) adaptive_ctrl: medium tick: state=6 yield=15pps motion=1.00 presence=5.35 rssi=-33
Output JSON (per tick):
{
"ts": 1716830400.123,
"node": 0, # always 0 (single node), client expands to 4
"motion": 1.00, # raw firmware metric
"presence": 5.35,
"rssi": -33,
"yield_pps": 15,
"amp": 0.78 # synthesized CSI amplitude in [0..1] for the bar
}
Run on ruvultra:
python3 -u ruvultra-csi-bridge.py
"""
import asyncio
import builtins
import json
import re
import sys
import time
from contextlib import suppress
# Force every print to flush — we're often piped to a log file
_orig_print = builtins.print
def _print(*a, **kw):
kw.setdefault("flush", True)
return _orig_print(*a, **kw)
builtins.print = _print
import serial
import websockets
PORT = "/dev/ttyACM0"
BAUD = 115200
WS_HOST = "0.0.0.0"
WS_PORT = 8766
TICK_RE = re.compile(
r"adaptive_ctrl:\s*\w+\s+tick:\s*"
r"state=(?P<state>\d+)\s+"
r"yield=(?P<yield>\d+)pps\s+"
r"motion=(?P<motion>[\d.]+)\s+"
r"presence=(?P<presence>[\d.]+)\s+"
r"rssi=(?P<rssi>-?\d+)"
)
clients = set()
last_payload = None
def amp_from_metrics(motion, presence, rssi):
"""Map firmware metrics to a [0..1] CSI-style amplitude."""
rssi_norm = max(0.0, min(1.0, (rssi + 80) / 50)) # -80..-30 → 0..1
presence_norm = max(0.0, min(1.0, presence / 8.0)) # cap at 8
motion_norm = max(0.0, min(1.0, motion)) # already 0..1ish
return 0.40 * rssi_norm + 0.35 * presence_norm + 0.25 * motion_norm
async def serial_reader_loop():
global last_payload
print(f"[bridge] opening {PORT} @ {BAUD}")
while True:
try:
ser = serial.Serial(PORT, BAUD, timeout=1)
except (serial.SerialException, OSError) as e:
print(f"[bridge] serial open failed ({e}); retry in 3s")
await asyncio.sleep(3)
continue
print(f"[bridge] connected to {PORT}")
loop = asyncio.get_event_loop()
try:
while True:
line = await loop.run_in_executor(None, ser.readline)
if not line:
continue
try:
text = line.decode(errors="replace").strip()
except Exception:
continue
m = TICK_RE.search(text)
if not m:
continue
motion = float(m["motion"])
presence = float(m["presence"])
rssi = int(m["rssi"])
payload = {
"ts": time.time(),
"node": 0,
"state": int(m["state"]),
"yield_pps": int(m["yield"]),
"motion": motion,
"presence": presence,
"rssi": rssi,
"amp": amp_from_metrics(motion, presence, rssi),
}
last_payload = payload
msg = json.dumps(payload)
if clients:
dead = []
for ws in list(clients):
try:
await ws.send(msg)
except websockets.ConnectionClosed:
dead.append(ws)
for d in dead:
clients.discard(d)
print(
f"[tick] motion={motion:.2f} presence={presence:5.2f} "
f"rssi={rssi:+d} yield={int(m['yield']):3d}pps "
f"amp={payload['amp']:.2f} clients={len(clients)}"
)
except (serial.SerialException, OSError) as e:
print(f"[bridge] serial error ({e}); reopen in 1s")
with suppress(Exception):
ser.close()
await asyncio.sleep(1)
async def ws_handler(ws):
addr = ws.remote_address
clients.add(ws)
print(f"[ws] client connected: {addr} total={len(clients)}")
try:
if last_payload is not None:
await ws.send(json.dumps(last_payload))
await ws.wait_closed()
finally:
clients.discard(ws)
print(f"[ws] client gone: {addr} total={len(clients)}")
async def main():
print(f"[bridge] websocket on ws://{WS_HOST}:{WS_PORT}")
async with websockets.serve(ws_handler, WS_HOST, WS_PORT):
await serial_reader_loop()
if __name__ == "__main__":
try:
asyncio.run(main())
except KeyboardInterrupt:
pass