Compare commits

..

6 Commits

Author SHA1 Message Date
Dragan Spiridonov ee11a5a766 fix(firmware): address PR #397 review feedback
Applies @ruvnet's five review requests on PR #397 (RuView#397 comment
4289417527):

1. **Inline comment on `provision.py` `write_flash`** — ESP-IDF v5.4
   bundles esptool 4.10.0 (underscore-only). #391's hyphen swap broke
   the documented venv flow; kept the underscore form and added a
   three-line comment warning future maintainers not to "re-fix" it.

2. **Correct `edge_processing.c` sample_rate** (blocking) — changed
   hard-coded `20.0f` → `10.0f` at line 718 so
   `estimate_bpm_zero_crossing()` matches the MGMT-only CSI rate.
   Without this, breathing and heart-rate reports were 2× the true
   value. Added a comment tying the constant to the callback rate gate.

3. **Removed disabled probe-injection infrastructure** — dropped the
   forward declaration, the `CSI_PROBE_INTERVAL_MS` define, six static
   variables (`s_probe_timer`, `s_probe_tx_count`, `s_probe_tx_fail`,
   `s_ap_bssid`, `s_ap_bssid_known`), and three functions
   (`csi_send_probe_request`, `probe_timer_cb`,
   `csi_collector_start_probe_timer`). None were reachable.
   `csi_inject_ndp_frame()` reverted to the original ADR-029 stub.
   Can be revived from this commit's parent if needed.

4. **Cleaned `sdkconfig.defaults`** — removed the SPIRAM prose and
   commented-out `# CONFIG_SPIRAM is not set` line. Kept only the live
   `CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y` with a concise rationale.

5. **Bumped firmware version 0.6.1 → 0.6.2** and added four
   `[Unreleased]` CHANGELOG entries covering the SPI cache crash fix,
   the `filter_mac` / `node_id` clobber defense, the sample-rate
   correction, and the `write_flash` command-form revert.

Net: +39 / -128 across six files.

Validation in this devcontainer:
- Static sanity on modified C files: braces balance (csi_collector.c
  59/59; edge_processing.c 96/96), zero dangling references to removed
  probe-injection symbols.
- Rust workspace tests and Python proof not executed here — cargo not
  installed and pip blocked by PEP 668. Deferring hardware build +
  flash + miniterm verification to @ruvnet's COM7 per his offer in
  the review comment.

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-28 08:41:00 -04:00
Dragan Spiridonov a14835bb91 fix(firmware): 50 Hz callback rate gate + sdkconfig extra IRAM opt
- Add early rate gate in wifi_csi_callback at 50 Hz (defense-in-depth,
  does not prevent crash alone but reduces callback execution time)
- Add null-data injection timer infrastructure (disabled — TX adds
  interrupt pressure that triggers the SPI cache crash, RuView#396)
- sdkconfig.defaults: add CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y
- sdkconfig.defaults: document SPIRAM XIP attempt (crashes differently)

Co-Authored-By: Ruflo & AQE
2026-04-28 08:41:00 -04:00
Dragan Spiridonov ace61696b1 fix(provision): write-flash → write_flash for esptool v5 compat
esptool v5+ rejects hyphenated subcommands. The provision script
used 'write-flash' which fails with "invalid choice". Changed to
'write_flash' (underscore) which works with both old and new esptool.

Co-Authored-By: Ruflo & AQE
2026-04-28 08:39:51 -04:00
Dragan Spiridonov c442669ea8 fix(firmware): MGMT-only promiscuous filter to prevent SPI cache crash
The WiFi driver's wDev_ProcessFiq interrupt handler crashes with
LoadProhibited in cache_ll_l1_resume_icache when promiscuous mode
captures MGMT+DATA frames (100-500 interrupts/sec). The high interrupt
rate races with SPI flash cache operations, corrupting cache state.

Changes:
- Promiscuous filter: MGMT+DATA → MGMT-only (~10 Hz beacons)
- CSI config: disable htltf_en and stbc_htltf2_en (LLTF-only)

LLTF provides 64 subcarriers (HT20) — sufficient for presence,
breathing, and fall detection. The 10 Hz beacon rate eliminates
the SPI flash cache contention that caused the crash.

Verified on device 80:b5:4e:c1:be:b8:
- Before: LoadProhibited crash at ~1600-2400 callbacks (every ~70s)
- After: 2700+ callbacks over 4.7 minutes, zero crashes

Backtrace decode confirmed crash in ESP-IDF closed-source WiFi blob:
  _xt_lowint1 → wDev_ProcessFiq → spi_flash_restore_cache
  → cache_ll_l1_resume_icache → EXCVADDR=0x00000004 (NULL deref)

Co-Authored-By: Ruflo & AQE
2026-04-28 08:39:51 -04:00
Dragan Spiridonov b8e332cd2a fix(firmware): defensive copy of filter_mac to prevent callback crash
The CSI callback reads g_nvs_config.filter_mac_set and filter_mac on
every invocation (100-500 Hz). If wifi_init_sta() corrupts g_nvs_config
(same root cause as the node_id clobber), the callback reads garbage
from the struct, leading to Core 0 LoadProhibited panic after ~2400
callbacks (~70 seconds of operation).

Extends the early-capture pattern from the node_id fix to also copy
filter_mac_set and filter_mac into module-local statics before WiFi
init runs. Adds canary logging to detect filter_mac corruption.

Observed on device 80:b5:4e:c1:be:b8 via serial:
  CSI cb #2400 → Guru Meditation Error: Core 0 panic'ed (LoadProhibited)
  → TG0WDT_SYS_RST → reboot → crash again at ~2900 callbacks

Refs #232 #375 #385 #386 #390

Co-Authored-By: Ruflo & AQE
2026-04-28 08:39:50 -04:00
Dragan Spiridonov 415cb5606d fix(firmware): move defensive node_id capture before wifi_init_sta()
The original defensive copy in csi_collector_init() (line 172 of main.c)
runs AFTER wifi_init_sta() (line 147), which on some ESP32-S3 devices
corrupts g_nvs_config.node_id back to the Kconfig default of 1.

Reproduced on device 80:b5:4e:c1:be:b8 (ESP32-S3 QFN56 rev v0.2):
  - NVS provisioned with node_id=5
  - Release firmware (no fix): seed receives node_id=1 (clobbered)
  - This patch: seed receives node_id=5 (correct)

Changes:
  - Add csi_collector_set_node_id() called from main.c immediately
    after nvs_config_load(), before wifi_init_sta() runs
  - csi_collector_init() now detects and logs the clobber if early
    capture disagrees with current g_nvs_config value
  - Fallback path preserved: if set_node_id() is never called,
    init() still captures from g_nvs_config (backwards compatible)

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-28 08:39:50 -04:00
15 changed files with 1944 additions and 2906 deletions
-58
View File
@@ -1,58 +0,0 @@
version: 2
updates:
# Keep all third-party GitHub Actions on verified, pinned commit SHAs.
# Pairs with the SHA pinning in security-scan.yml and ci.yml so that
# future bumps stay automated and reviewable rather than drifting back
# to mutable @master / @main refs. See issue #442.
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
open-pull-requests-limit: 5
labels:
- dependencies
- github-actions
# Mobile app npm deps. Includes the @xmldom/xmldom, node-forge, and
# picomatch advisories from #442 plus axios and any future surface.
- package-ecosystem: npm
directory: /ui/mobile
schedule:
interval: weekly
open-pull-requests-limit: 10
labels:
- dependencies
- mobile
# Desktop UI npm deps. Direct vite devDep currently has a HIGH advisory
# (dev-server-only path traversal); track future bumps automatically.
- package-ecosystem: npm
directory: /v2/crates/wifi-densepose-desktop/ui
schedule:
interval: weekly
open-pull-requests-limit: 5
labels:
- dependencies
- desktop
# Python deps used by v1/ and the FastAPI service. requirements.txt is
# only loosely pinned; let Dependabot surface upstream CVE bumps.
- package-ecosystem: pip
directory: /
schedule:
interval: weekly
open-pull-requests-limit: 10
labels:
- dependencies
- python
# Rust workspace (15+ crates). cargo audit is not currently wired into
# any workflow, so Dependabot is the primary automated bump path.
- package-ecosystem: cargo
directory: /v2
schedule:
interval: weekly
open-pull-requests-limit: 10
labels:
- dependencies
- rust
+1 -1
View File
@@ -255,7 +255,7 @@ jobs:
docker stop test-container
- name: Run container security scan
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
uses: aquasecurity/trivy-action@master
with:
image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
format: 'sarif'
-26
View File
@@ -98,32 +98,6 @@ jobs:
echo "Flash image integrity verified"
fi
- name: Verify embedded version string matches version.txt (fixes #505)
working-directory: firmware/esp32-csi-node
run: |
EXPECTED=$(cat version.txt | tr -d '[:space:]')
BIN=build/esp32-csi-node.bin
# Extract version from ESP-IDF app_desc: magic 0xABCD5432 at offset 0
# followed by version string at offset 16, null-terminated, max 32 chars.
EMBEDDED=$(python3 -c "
import struct, sys
data = open('$BIN','rb').read()
magic = struct.pack('<I', 0xABCD5432)
i = data.find(magic)
if i < 0:
sys.exit('app_desc magic not found')
ver = data[i+16:i+48].split(b'\\x00',1)[0].decode('ascii','replace')
print(ver)
" 2>&1)
echo "Expected version: $EXPECTED"
echo "Embedded version: $EMBEDDED"
if [ "$EMBEDDED" != "$EXPECTED" ]; then
echo "::error::Version string mismatch! version.txt='$EXPECTED' but binary reports '$EMBEDDED'."
echo "::error::Ensure version.txt is updated before building and tagging."
exit 1
fi
echo "Version string verified: $EMBEDDED"
- name: Stage release binaries with variant-specific names
working-directory: firmware/esp32-csi-node
run: |
-74
View File
@@ -1,74 +0,0 @@
name: Point Cloud Viewer → GitHub Pages
# Publishes the live 3D point cloud viewer to gh-pages/pointcloud/.
# The viewer defaults to a synthetic in-browser demo; users can append
# ?backend=<url> or ?backend=auto to point it at a real ruview-pointcloud
# server (CORS-permitting host required). See ADR-094.
#
# Uses keep_files: true to preserve the existing observatory/, pose-fusion/,
# nvsim/, and root index.html demos already on gh-pages.
on:
push:
branches: [main]
paths:
- 'v2/crates/wifi-densepose-pointcloud/src/viewer.html'
- '.github/workflows/pointcloud-pages.yml'
workflow_dispatch:
permissions:
contents: write
concurrency:
group: pointcloud-pages
cancel-in-progress: true
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout main
uses: actions/checkout@v4
- name: Stage viewer for Pages
run: |
mkdir -p _site/pointcloud
cp v2/crates/wifi-densepose-pointcloud/src/viewer.html _site/pointcloud/index.html
# Drop a tiny README so direct browsers of the directory get context.
cat > _site/pointcloud/README.md <<'EOF'
# RuView — Live 3D Point Cloud Viewer
Hosted at: https://ruvnet.github.io/RuView/pointcloud/
## Modes
- Default — synthetic in-browser demo (no backend, no network calls).
- `?backend=auto` — fetch from `/api/splats` on the same origin
(only works when the viewer is served by `ruview-pointcloud serve`).
- `?backend=<url>` — fetch from `<url>/api/splats`. The intended
local-ESP32 use is `?backend=http://127.0.0.1:9880`: run
`ruview-pointcloud serve --bind 127.0.0.1:9880` on the same
machine with your ESP32 streaming CSI to UDP port 3333, then
visit the URL above. The local server's CorsLayer permits
requests from `https://ruvnet.github.io`, and modern browsers
permit HTTPS→127.0.0.1 mixed-content as a trustworthy origin.
The "📡 Connect ESP32" button in the viewer prompts for this
URL and persists it in localStorage.
- `?live=1` — require a live backend; show an offline message instead
of falling back to the synthetic demo.
See ADR-094 for the deployment design.
EOF
- name: Deploy to gh-pages/pointcloud/
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./_site/pointcloud
destination_dir: pointcloud
# CRITICAL: preserves observatory/, pose-fusion/, nvsim/, and root
# index.html already on gh-pages.
keep_files: true
commit_message: 'deploy(pointcloud): ${{ github.sha }}'
user_name: 'github-actions[bot]'
user_email: 'github-actions[bot]@users.noreply.github.com'
+6 -6
View File
@@ -111,7 +111,7 @@ jobs:
continue-on-error: true
- name: Run Snyk vulnerability scan
uses: snyk/actions/python@9adf32b1121593767fc3c057af55b55db032dc04 # v1.0.0
uses: snyk/actions/python@master
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
@@ -163,7 +163,7 @@ jobs:
cache-to: type=gha,mode=max
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
uses: aquasecurity/trivy-action@master
with:
image-ref: 'wifi-densepose:scan'
format: 'sarif'
@@ -221,7 +221,7 @@ jobs:
uses: actions/checkout@v4
- name: Run Checkov IaC scan
uses: bridgecrewio/checkov-action@99bb2caf247dfd9f03cf984373bc6043d4e32ebf # v12.1347.0
uses: bridgecrewio/checkov-action@master
with:
directory: .
framework: kubernetes,dockerfile,terraform,ansible
@@ -238,7 +238,7 @@ jobs:
category: checkov
- name: Run Terrascan IaC scan
uses: tenable/terrascan-action@3a6e87da8e244513bd77b631e624552643f794c6 # v1.4.1
uses: tenable/terrascan-action@main
with:
iac_type: 'k8s'
iac_version: 'v1'
@@ -247,7 +247,7 @@ jobs:
sarif_upload: true
- name: Run KICS IaC scan
uses: checkmarx/kics-github-action@05aa5eb70eede1355220f4ca5238d96b397e30a6 # v2.1.20
uses: checkmarx/kics-github-action@master
with:
path: '.'
output_path: kics-results
@@ -277,7 +277,7 @@ jobs:
fetch-depth: 0
- name: Run TruffleHog secret scan
uses: trufflesecurity/trufflehog@17456f8c7d042d8c82c9a8ca9e937231f9f42e26 # v3.95.2
uses: trufflesecurity/trufflehog@main
with:
path: ./
base: main
+1875 -27
View File
File diff suppressed because it is too large Load Diff
@@ -1,203 +0,0 @@
# ADR-094: Live 3D Point Cloud Viewer — GitHub Pages Deployment with Optional Real-Data Stream
| Field | Value |
|---|---|
| **Status** | Proposed (2026-04-29) |
| **Date** | 2026-04-29 |
| **Authors** | ruv |
| **Related** | ADR-092 (nvsim dashboard Pages deployment), ADR-059 (live ESP32 CSI pipeline), ADR-079 (camera ground-truth training) |
| **Branch** | `feat/pointcloud-pages-demo` |
---
## 1. Context
The `wifi-densepose-pointcloud` crate ships a Three.js-based viewer
(`v2/crates/wifi-densepose-pointcloud/src/viewer.html`) that renders the
fused camera-depth + WiFi CSI + mmWave point cloud produced by the
`ruview-pointcloud serve` binary. Today the viewer is local-only:
- It is served by the Axum binary on `127.0.0.1:9880`.
- It polls `/api/splats` every 500 ms expecting a backend on the same
origin.
- There is no GitHub Pages deployment, so the README's
"▶ Live 3D Point Cloud" link points at the moved-content section in
`docs/readme-details.md`, not at a hosted demo. The two sibling demos
(Live Observatory, Dual-Modal Pose Fusion) are already hosted at
`https://ruvnet.github.io/RuView/` and `…/pose-fusion.html`.
This is an asymmetry: a first-time visitor can preview the WiFi pose
demo and the Observatory in one click, but cannot preview the point
cloud without cloning the repo, building Rust, plugging in an ESP32,
and pointing a webcam at themselves. That gap suppresses the most
visually compelling demonstration of the v0.7+ sensor-fusion work.
A naive fix — drop the static HTML at `gh-pages/pointcloud/` — does
not work because the viewer's `fetch("/api/splats")` will 404 on Pages
and the canvas will hang at "Loading…". A second naive fix — bake in a
fixed sample dataset — solves the loading state but loses the live-data
story entirely, and forks the viewer into a "demo build" and a "real
build" that drift apart.
## 2. Decision
Ship **one** viewer that auto-selects its transport from URL parameters,
and publish it to `gh-pages/pointcloud/` alongside the other demos:
1. **Default mode** — when the viewer is opened with no query parameters
on `https://ruvnet.github.io/RuView/pointcloud/`, present a "▶ Enable
camera" CTA. On click the viewer requests webcam access, runs
**MediaPipe Face Mesh** in-browser (~30 fps, 478 refined landmarks),
and renders the visitor's own face as a point cloud — the closest
browser equivalent of the local pipeline's depth-backprojected face
geometry that motivated this ADR (`I could see the outline of my face
in points`). The viewer mirrors x to match selfie convention and
maps Face Mesh's relative-z to the same world-coordinate range the
live `/api/splats` payload uses, so a single render path drives both.
Badge reads `● DEMO Your Face (MediaPipe)`. If the user denies
camera permission, dismisses the prompt, or visits on a device
without a webcam, the viewer falls back automatically to a
procedural scaffold (floor grid, walls, breathing figure, 17-keypoint
skeleton). All processing is client-side; no frames leave the
browser. ~480-500 splats from the face plus ~110 floor/wall context
splats.
2. **Auto mode** (`?backend=auto`) — fetch from `/api/splats` on the same
origin. This is the local-development case (`ruview-pointcloud serve`
serves the viewer and the API together). On any failure (404, network
error, CORS), fall back silently to synthetic-demo rendering so the
tab never dies.
3. **Remote mode** (`?backend=<url>`) — fetch from `<url>/api/splats`.
This is the **integrated-ESP32** path: the user runs
`ruview-pointcloud serve --bind 127.0.0.1:9880` locally with an
ESP32-S3 streaming CSI to UDP port 3333, then opens
`https://ruvnet.github.io/RuView/pointcloud/?backend=http://127.0.0.1:9880`.
The hosted Pages viewer becomes a thin client for the local Rust
fusion pipeline (camera depth + WiFi CSI + mmWave) without a clone
or rebuild. The viewer also exposes a "📡 Connect ESP32" button that
prompts for the URL, persists it in `localStorage`, and reloads
with the query param.
For this to work the local server must answer the browser's CORS
preflight. `stream.rs` therefore installs a `tower_http` `CorsLayer`
that allows three origin classes:
- `https://ruvnet.github.io` — the published Pages demo.
- `http://localhost:*` and `http://127.0.0.1:*` — developer running
the bundled `viewer.html` directly.
- `null``file://` origins.
Mixed-content (HTTPS Pages → HTTP loopback) is permitted because
modern browsers (Chrome 94+, Firefox 116+, Safari 16.4+) classify
`127.0.0.1` and `localhost` as "potentially trustworthy" origins.
Any other origin (a public hostname, etc.) is denied — this is not
a wildcard CORS posture. Badge reads `● REMOTE <url>`. Same silent
demo fallback on failure.
4. **Strict-live mode** (`?live=1`) — disable the demo fallback. If the
chosen transport fails, replace the info panel with an explicit offline
message (`● OFFLINE — Live backend required but unreachable`). Useful
for embedding the viewer in a status page or kiosk.
The synthetic frame returned by the in-browser generator matches the
JSON shape of the live `/api/splats` payload exactly (`splats`, `count`,
`frame`, `live`, `pipeline.{skeleton,vitals,…}`), so a single render path
drives both modes. There is no demo build vs real build — only one HTML
file, one render path, and one set of bugs.
A new GitHub Actions workflow (`.github/workflows/pointcloud-pages.yml`)
copies the viewer to `gh-pages/pointcloud/index.html` on every push to
`main` that touches the viewer, using `peaceiris/actions-gh-pages@v4`
with `keep_files: true` to preserve the existing observatory, pose-fusion,
and nvsim deployments.
## 3. Consequences
### Positive
- **First-click demo.** Visitors clicking the README's
"▶ Live 3D Point Cloud" link land on a working Three.js scene in <1 s,
no toolchain required. Matches the parity of the other two demos.
- **Real-data on demand.** Users with their own `ruview-pointcloud serve`
host can use the same hosted viewer URL with
`?backend=https://their-host.example.com` — no clone, no rebuild. The
hosted demo doubles as a thin client for self-hosted backends.
- **Single render path.** Synthetic frames flow through the same
`handleData → updateSplats → drawSkeleton` pipeline as live frames, so
visual regressions surface in the demo and the live build at the same
time. This is the same dual-transport pattern ADR-092 chose for nvsim.
- **No backend deploy required.** Pages serves static HTML; the demo
works without standing up an Axum host on the public internet, and
there is no per-visitor CSI/camera plumbing to provision.
- **Preserves existing deployments.** `keep_files: true` plus the
`pointcloud/` destination means observatory/, pose-fusion/, nvsim/,
and the root index.html on gh-pages are untouched.
### Negative / tradeoffs
- **Face mesh ≠ CSI.** Browser webcam + MediaPipe gives real face
geometry but does not produce CSI-derived pose. Visitors who want to
see the *WiFi-driven* path still need `?backend=<their-host>`. The
procedural fallback is not WiFi-driven either; it is purely visual
scaffolding. We accept this — the goal of the hosted demo is to
convey the *shape* of what the local pipeline produces (a point
cloud of the user) rather than reproduce the WiFi physics in the
browser. The latter is a future ADR (WASM port of the fusion crate).
- **CORS burden on remote mode.** Users who want to share their backend
must add `Access-Control-Allow-Origin: https://ruvnet.github.io` (or
`*`) to their `ruview-pointcloud serve` config. We document this in the
workflow's generated README; we do **not** add a public proxy.
- **Synthetic generator lives in the viewer.** ~80 LOC of procedural JS
is now part of `viewer.html`. Acceptable: the file is already the
client-side render bundle, and the generator is bounded and inert
(deterministic, no I/O, no eval).
- **No replay-from-recording in this ADR.** A future ADR may add a
`?recording=<url>.jsonl` mode that replays captured frames at native
rate; that is out of scope here.
### Neutral
- The local-dev experience is unchanged. `ruview-pointcloud serve` still
serves `viewer.html` from the bundled asset and the viewer still hits
`/api/splats` because `?backend` defaults to `auto`. Nothing in the
Rust crate changes — this is HTML + workflow only.
## 4. Implementation
| File | Change |
|---|---|
| `v2/crates/wifi-densepose-pointcloud/src/viewer.html` | Add URL-param transport selector (`backend`, `live`), synthetic frame generator, demo-fallback path, transport-aware mode badge. ~120 LOC added, no removed behavior. |
| `.github/workflows/pointcloud-pages.yml` | New workflow: stage viewer to `_site/pointcloud/index.html`, deploy to `gh-pages/pointcloud/` with `keep_files: true`. Triggers on viewer changes and on manual dispatch. |
| `README.md` | Already updated — `▶ Live 3D Point Cloud` link will be retargeted to `https://ruvnet.github.io/RuView/pointcloud/` once the first deploy succeeds. (Tracked separately, not blocking this ADR.) |
| `docs/adr/README.md` | ADR index — add ADR-094 row. |
## 5. Acceptance Gates
This ADR is **Implemented** when all of the following hold:
1. Pushing to `main` with a viewer change triggers
`pointcloud-pages.yml`, which deploys to `gh-pages/pointcloud/` in
under 60 seconds.
2. `https://ruvnet.github.io/RuView/pointcloud/` loads, shows the
"Enable camera" CTA, and on accept renders the visitor's face as a
point cloud with badge `● DEMO Your Face (MediaPipe)` and non-zero
splat + frame counts. On camera denial, falls back to the
procedural scene with badge `● DEMO Synthetic`.
3. Existing demos at `https://ruvnet.github.io/RuView/` and
`…/pose-fusion.html` and `…/nvsim/` are still reachable after the
first deploy (smoke-tested manually).
4. `https://ruvnet.github.io/RuView/pointcloud/?live=1` shows the
`● OFFLINE` panel (because no same-origin backend exists on Pages).
5. `https://ruvnet.github.io/RuView/pointcloud/?backend=https://example.invalid`
falls back to demo within one poll interval (~500 ms) without
throwing in the console.
6. Running `./target/release/ruview-pointcloud serve` locally and
opening `http://127.0.0.1:9880/` (which serves the same HTML) still
shows live-mode rendering with the `● LIVE Local Backend` badge.
## 6. Out of Scope
- Replaying recorded JSONL frames in the browser (future ADR).
- WASM-side execution of the fusion pipeline in the browser (would
require porting the camera + mmWave path; deferred).
- Authentication / signed splats payloads — backend-side concern,
unaffected by this client-side change.
- Hosting a public CORS proxy for users without their own backend.
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -1 +1 @@
0.6.4
0.6.2
+46 -10
View File
@@ -5127,9 +5127,9 @@
"license": "BSD-3-Clause"
},
"node_modules/@xmldom/xmldom": {
"version": "0.8.13",
"resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.13.tgz",
"integrity": "sha512-KRYzxepc14G/CEpEGc3Yn+JKaAeT63smlDr+vjB8jRfgTBBI9wRj/nkQEO+ucV8p8I9bfKLWp37uHgFrbntPvw==",
"version": "0.8.11",
"resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.11.tgz",
"integrity": "sha512-cQzWCtO6C8TQiYl1ruKNn2U6Ao4o4WBBcbL61yJl84x+j5sOWWFU9X7DpND8XZG3daDppSsigMdfAIl2upQBRw==",
"license": "MIT",
"engines": {
"node": ">=10.0.0"
@@ -5310,6 +5310,18 @@
"node": ">= 8"
}
},
"node_modules/anymatch/node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"license": "MIT",
"engines": {
"node": ">=8.6"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/arg": {
"version": "5.0.2",
"resolved": "https://registry.npmjs.org/arg/-/arg-5.0.2.tgz",
@@ -11923,6 +11935,18 @@
"node": ">=8"
}
},
"node_modules/jest-util/node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"license": "MIT",
"engines": {
"node": ">=8.6"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/jest-validate": {
"version": "29.7.0",
"resolved": "https://registry.npmjs.org/jest-validate/-/jest-validate-29.7.0.tgz",
@@ -13365,6 +13389,18 @@
"node": ">=8.6"
}
},
"node_modules/micromatch/node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"license": "MIT",
"engines": {
"node": ">=8.6"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/mime": {
"version": "1.6.0",
"resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz",
@@ -13558,9 +13594,9 @@
}
},
"node_modules/node-forge": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/node-forge/-/node-forge-1.4.0.tgz",
"integrity": "sha512-LarFH0+6VfriEhqMMcLX2F7SwSXeWwnEAJEsYm5QKWchiVYVvJyV9v7UDvUv+w5HO23ZpQTXDv/GxdDdMyOuoQ==",
"version": "1.3.3",
"resolved": "https://registry.npmjs.org/node-forge/-/node-forge-1.3.3.tgz",
"integrity": "sha512-rLvcdSyRCyouf6jcOIPe/BgwG/d7hKjzMKOas33/pHEr6gbq18IK9zV7DiPvzsz0oBJPme6qr6H6kGZuI9/DZg==",
"license": "(BSD-3-Clause OR GPL-2.0)",
"engines": {
"node": ">= 6.13.0"
@@ -14020,12 +14056,12 @@
"license": "ISC"
},
"node_modules/picomatch": {
"version": "2.3.2",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz",
"integrity": "sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==",
"version": "4.0.3",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.3.tgz",
"integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==",
"license": "MIT",
"engines": {
"node": ">=8.6"
"node": ">=12"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
-5
View File
@@ -49,10 +49,5 @@
"react-native-worklets": "^0.7.4",
"typescript": "~5.9.2"
},
"overrides": {
"@xmldom/xmldom": "0.8.13",
"node-forge": "^1.4.0",
"picomatch": "^2.3.2"
},
"private": true
}
@@ -14,7 +14,6 @@ serde_json = { workspace = true }
tokio = { workspace = true }
anyhow = { workspace = true }
axum = { workspace = true }
tower-http = { workspace = true }
clap = { version = "4", features = ["derive"] }
chrono = "0.4"
dirs = "5"
@@ -65,9 +65,6 @@ pub struct CsiPipelineState {
pub current_location: Option<(String, f32)>,
/// Night mode — true when camera luminance is below threshold
pub is_dark: bool,
/// Wall-clock instant the last real ESP32 UDP CSI frame was received.
/// `None` if no frame has arrived since startup.
pub last_csi_received: Option<std::time::Instant>,
/// Metadata from the on-disk WiFlow JSON, if one is present. NOTE: the
/// weights themselves are NOT loaded or executed in this crate — this
/// flag merely enables the amplitude-energy heuristic pose code path.
@@ -94,7 +91,6 @@ impl Default for CsiPipelineState {
fingerprints: Vec::new(),
current_location: None,
is_dark: false,
last_csi_received: None,
pose_model_present: detect_pose_model_metadata(),
}
}
@@ -137,7 +133,6 @@ impl CsiPipelineState {
pub fn process_frame(&mut self, frame: CsiFrame) {
let node_id = frame.node_id;
self.total_frames += 1;
self.last_csi_received = Some(std::time::Instant::now());
// Once every 500 frames log a one-line node stats summary. This keeps
// us honest about the CSI shape we are actually receiving and also
@@ -589,9 +584,6 @@ pub fn get_pipeline_output(state: &Arc<Mutex<CsiPipelineState>>) -> PipelineOutp
num_nodes: st.node_frames.len(),
current_location: st.current_location.clone(),
is_dark: st.is_dark,
csi_live: st.last_csi_received
.map(|t| t.elapsed() < std::time::Duration::from_secs(5))
.unwrap_or(false),
}
}
@@ -606,10 +598,6 @@ pub struct PipelineOutput {
pub num_nodes: usize,
pub current_location: Option<(String, f32)>,
pub is_dark: bool,
/// True when a real ESP32 CSI frame was received in the last 5 seconds.
/// False means the pipeline is running on stale data — show a NO SIGNAL
/// indicator in the UI rather than presenting stale skeletons as live.
pub csi_live: bool,
}
// Serialize implementations
@@ -8,13 +8,11 @@ use crate::fusion;
use crate::pointcloud;
use axum::{
extract::State,
http::{HeaderValue, Method},
response::Html,
routing::get,
Json, Router,
};
use std::sync::{Arc, Mutex};
use tower_http::cors::{AllowOrigin, CorsLayer};
struct AppState {
latest_cloud: Mutex<pointcloud::PointCloud>,
@@ -110,36 +108,12 @@ pub async fn serve(bind: &str, _brain: Option<&str>) -> anyhow::Result<()> {
if has_camera { eprintln!(" Camera: LIVE (/dev/video0)"); }
else { eprintln!(" Camera: DEMO"); }
// CORS — allow the hosted GitHub Pages viewer to fetch /api/splats from a
// locally-running instance of this server. Modern browsers treat
// 127.0.0.1/localhost as a "potentially trustworthy" origin so the HTTPS
// page can reach a plain-HTTP loopback backend without mixed-content
// blocking. Origins permitted:
// - https://ruvnet.github.io (the published RuView Pages demo)
// - http://localhost:* / http://127.0.0.1:* (developer running the
// viewer.html bundled with this binary)
// Anything else is denied, so this is not a "wildcard" CORS.
let cors = CorsLayer::new()
.allow_origin(AllowOrigin::predicate(|origin: &HeaderValue, _req| {
let s = match origin.to_str() {
Ok(v) => v,
Err(_) => return false,
};
s == "https://ruvnet.github.io"
|| s.starts_with("http://localhost")
|| s.starts_with("http://127.0.0.1")
|| s == "null" // file:// origins
}))
.allow_methods([Method::GET, Method::OPTIONS])
.allow_headers([axum::http::header::CONTENT_TYPE]);
let app = Router::new()
.route("/", get(index))
.route("/api/cloud", get(api_cloud))
.route("/api/splats", get(api_splats))
.route("/api/status", get(api_status))
.route("/health", get(api_health))
.layer(cors)
.with_state(state);
println!("╔══════════════════════════════════════════════╗");
@@ -219,12 +193,10 @@ async fn api_splats(State(state): State<Arc<AppState>>) -> Json<serde_json::Valu
let splats = state.latest_splats.lock().unwrap();
let frames = *state.frame_count.lock().unwrap();
let pipeline = state.latest_pipeline.lock().unwrap();
let csi_live = pipeline.as_ref().map(|p| p.csi_live).unwrap_or(false);
Json(serde_json::json!({
"splats": &*splats,
"count": splats.len(),
"live": state.use_camera,
"csi_live": csi_live,
"frame": frames,
"pipeline": &*pipeline,
"timestamp": chrono::Utc::now().timestamp_millis(),
@@ -2,60 +2,27 @@
<html>
<head>
<title>RuView — Camera + WiFi CSI Point Cloud</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate">
<meta name="ruview-viewer-version" content="0.2.0-face-mesh">
<!-- Inline amber-dot favicon avoids a stray /favicon.ico 404 in the console. -->
<link rel="icon" type="image/svg+xml" href="data:image/svg+xml;utf8,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><circle cx='16' cy='16' r='10' fill='%23e8a634'/></svg>">
<style>
body { margin: 0; background: #0a0a0a; color: #e8a634; font-family: monospace; }
canvas { display: block; }
#info { position: absolute; top: 10px; left: 10px; padding: 12px; background: rgba(0,0,0,0.85); border: 1px solid #e8a634; border-radius: 6px; min-width: 240px; font-size: 13px; line-height: 1.5; z-index: 10; }
#cam-cta { position: absolute; bottom: 16px; left: 50%; transform: translateX(-50%); padding: 10px 18px; background: #e8a634; color: #0a0a0a; border: none; border-radius: 4px; font-family: monospace; font-size: 14px; font-weight: bold; cursor: pointer; z-index: 10; }
#cam-cta:hover { background: #ffc04d; }
#cam-cta.hidden { display: none; }
#esp-cta { position: absolute; bottom: 16px; right: 16px; padding: 8px 14px; background: transparent; color: #e8a634; border: 1px solid #e8a634; border-radius: 4px; font-family: monospace; font-size: 12px; cursor: pointer; z-index: 10; }
#esp-cta:hover { background: rgba(232, 166, 52, 0.12); }
#esp-cta.connected { background: #4f4; color: #0a0a0a; border-color: #4f4; }
#info { position: absolute; top: 10px; left: 10px; padding: 12px; background: rgba(0,0,0,0.85); border: 1px solid #e8a634; border-radius: 6px; min-width: 240px; font-size: 13px; line-height: 1.5; }
.live { color: #4f4; } .demo { color: #f44; }
.face { color: #4cf; }
.section { margin-top: 6px; padding-top: 6px; border-top: 1px solid #333; }
.label { color: #888; }
#no-signal {
display: none;
position: absolute; top: 50%; left: 50%;
transform: translate(-50%, -50%);
background: rgba(160,0,0,0.93); color: #fff;
font-family: monospace; font-size: 18px; font-weight: bold;
padding: 18px 32px; border-radius: 8px;
border: 2px solid #f44; text-align: center;
pointer-events: none; z-index: 20;
}
#no-signal .sub { font-size: 12px; font-weight: normal; margin-top: 6px; color: #fbb; }
</style>
<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/controls/OrbitControls.js"></script>
<!-- MediaPipe Face Mesh — runs in demo mode so each visitor sees their own face as a point cloud -->
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh@0.4/face_mesh.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/camera_utils@0.3/camera_utils.js"></script>
</head>
<body>
<div id="no-signal">
&#x25CF; NO CSI SIGNAL
<div class="sub">No ESP32 frames received for &gt;5s.<br>Check that your node is powered and provisioned.</div>
</div>
<div id="info">
<h3 style="margin:0 0 4px 0">RuView · Seldon Vault</h3>
<div style="font-size: 11px; color: #888; margin-bottom: 8px; max-width: 240px; line-height: 1.4; font-style: italic;">"Psychohistory deals with reactions of human conglomerates to fixed social and economic stimuli." — Hari Seldon</div>
<h3 style="margin:0 0 8px 0">RuView Point Cloud</h3>
<div id="stats">Loading...</div>
</div>
<button id="cam-cta">▶ Project Subject — render your face into the Vault</button>
<button id="esp-cta" title="Stream live CSI from a local ruview-pointcloud serve instance (e.g. http://127.0.0.1:9880)">📡 Connect ESP32…</button>
<script>
var scene = new THREE.Scene();
scene.background = new THREE.Color(0x0a0a0a);
var camera = new THREE.PerspectiveCamera(72, window.innerWidth/window.innerHeight, 0.1, 200);
camera.position.set(0, 0.2, -3.5);
var camera = new THREE.PerspectiveCamera(75, window.innerWidth/window.innerHeight, 0.1, 100);
camera.position.set(0, 2, -4);
camera.lookAt(0, 0, 2);
var renderer = new THREE.WebGLRenderer({ antialias: true });
@@ -71,11 +38,6 @@
var skeletonGroup = null;
var prevTimestamp = 0;
var frameRateVal = 0;
// No-signal detection: track server-reported csi_live flag
var noSignalBanner = document.getElementById("no-signal");
function setNoSignal(isNoSignal) {
noSignalBanner.style.display = isNoSignal ? "block" : "none";
}
// COCO skeleton connections: pairs of keypoint indices
// 0=nose 1=leftEye 2=rightEye 3=leftEar 4=rightEar
@@ -142,436 +104,10 @@
scene.add(skeletonGroup);
}
// ----- Transport configuration -----
// ?backend=<url> → fetch splats from <url>/api/splats (CORS-permitting host)
// ?backend=auto → try /api/splats, fall back to synthetic demo on failure (default)
// ?backend=demo → always render synthetic demo (no network)
// ?live=1 → require live; show error instead of demo fallback
var urlParams = new URLSearchParams(window.location.search);
var backendArg = urlParams.get("backend") || "auto";
var requireLive = urlParams.get("live") === "1";
var transportMode = "demo"; // resolved at first fetch: "live" | "remote" | "demo"
var demoStartMs = Date.now();
var demoFrameNum = 0;
var latestFaceLandmarks = null; // populated by MediaPipe when camera enabled
var faceMeshState = "idle"; // "idle" | "starting" | "running" | "denied" | "unavailable"
// ----- MediaPipe Face Mesh (browser equivalent of camera-depth backprojection) -----
// Locally, ruview-pointcloud serve fuses real camera depth + WiFi CSI. In the
// browser we don't have depth from a webcam, but Face Mesh produces 468
// 3D landmarks (x,y in [0,1], z roughly in [-0.5,0.5]) at ~30 fps — enough to
// reproduce the "I can see the outline of my face in points" experience. The
// landmarks feed into the same splat render path as live /api/splats data.
async function startFaceMesh() {
if (faceMeshState !== "idle") return;
if (!window.FaceMesh || !window.Camera) {
faceMeshState = "unavailable";
return;
}
faceMeshState = "starting";
try {
var videoEl = document.createElement("video");
videoEl.style.display = "none";
videoEl.autoplay = true;
videoEl.playsInline = true;
videoEl.muted = true;
document.body.appendChild(videoEl);
var fm = new FaceMesh({
locateFile: function(file) {
return "https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh@0.4/" + file;
}
});
fm.setOptions({
maxNumFaces: 1,
refineLandmarks: true,
minDetectionConfidence: 0.5,
minTrackingConfidence: 0.5
});
fm.onResults(function(results) {
if (results.multiFaceLandmarks && results.multiFaceLandmarks[0]) {
latestFaceLandmarks = results.multiFaceLandmarks[0];
}
});
var mpCamera = new Camera(videoEl, {
onFrame: async function() { await fm.send({ image: videoEl }); },
width: 640,
height: 480
});
await mpCamera.start();
faceMeshState = "running";
var btn = document.getElementById("cam-cta");
if (btn) btn.classList.add("hidden");
} catch (err) {
faceMeshState = "denied";
console.warn("Face mesh unavailable:", err);
}
}
// ---- Foundation-inspired galactic context (Asimov / Trantor / Seldon) ----
// Shared between face-mesh and synthetic-fallback paths. The subject (face
// or procedural figure) is the foreground; this function paints the Seldon
// time-vault around it: holographic surveyor grid underfoot, slow galactic
// spiral receding into the distance, distant starfield, and a halo ring.
function pushFoundationContext(splats) {
var t = (Date.now() - demoStartMs) / 1000.0;
// 1. Holographic surveyor grid — amber lattice at y=+1.4 (renders below
// the subject because the renderer flips y to Three.js Y-up).
var gx, gz;
for (gx = -10; gx <= 10; gx++) {
for (gz = 0; gz <= 30; gz++) {
var alpha = 0.35 + 0.15 * Math.sin(t * 0.5 + gz * 0.2);
splats.push({
center: [gx * 0.5, 1.4, gz * 0.4],
color: [0.40 * alpha, 0.28 * alpha, 0.10 * alpha],
opacity: 1.0,
scale: [0.018, 0.018, 0.018]
});
}
}
for (gz = 0; gz <= 30; gz += 2) {
for (gx = -20; gx <= 20; gx++) {
splats.push({
center: [gx * 0.25, 1.4, gz * 0.4 + 0.1],
color: [0.30, 0.22, 0.08],
opacity: 1.0,
scale: [0.014, 0.014, 0.014]
});
}
}
// 2. Galactic spiral — Trantor recedes behind the subject. ~640 stars
// across two logarithmic arms, slowly rotating. Warmer at the core,
// cooler at the edges (Hertzsprung-Russell-ish).
var arm, k, theta_arm, r_arm, sx, sz, twist, arm_color;
for (arm = 0; arm < 2; arm++) {
for (k = 0; k < 320; k++) {
var prog = k / 320;
theta_arm = arm * Math.PI + prog * 6.0 + t * 0.05;
r_arm = 4.0 + prog * 14.0;
twist = Math.sin(prog * 8.0) * 0.4;
sx = Math.cos(theta_arm) * r_arm + twist;
sz = Math.sin(theta_arm) * r_arm + 12.0;
var coreFade = Math.max(0.15, 1.0 - prog);
arm_color = [
coreFade * 0.85 + 0.15 * (1 - prog),
coreFade * 0.70 + 0.20,
coreFade * 0.55 + 0.45 * prog
];
splats.push({
center: [sx, -2.5 + Math.sin(prog * 12) * 0.3, sz],
color: arm_color,
opacity: 1.0,
scale: [0.025, 0.025, 0.025]
});
}
}
// 3. Distant starfield — 800 deterministic stars on a spherical shell.
// Fixed LCG seed so visitors don't see noise flicker between frames.
var seed = 42;
function nextRand() {
seed = (seed * 1664525 + 1013904223) >>> 0;
return seed / 4294967296;
}
var s, r_s, phi, costheta, sinphi;
for (s = 0; s < 800; s++) {
phi = nextRand() * Math.PI * 2;
costheta = nextRand() * 2 - 1;
sinphi = Math.sqrt(1 - costheta * costheta);
r_s = 22 + nextRand() * 8;
var brightness = 0.4 + nextRand() * 0.6;
var hue = nextRand();
splats.push({
center: [
Math.cos(phi) * sinphi * r_s,
costheta * r_s * 0.5 - 1.0,
Math.sin(phi) * sinphi * r_s + 5.0
],
color: hue > 0.85
? [brightness, brightness * 0.85, brightness * 0.6]
: (hue > 0.3
? [brightness * 0.9, brightness * 0.95, brightness]
: [brightness * 0.5, brightness * 0.7, brightness]),
opacity: 1.0,
scale: [0.020, 0.020, 0.020]
});
}
// 4. Holographic projection halo around the subject — Seldon vault
// projections always had a faint encircling ring of particles.
var ring;
for (ring = 0; ring < 60; ring++) {
var rt = ring / 60 * Math.PI * 2 + t * 0.3;
splats.push({
center: [
Math.cos(rt) * 1.6,
Math.sin(rt) * 1.2 - 0.2,
2.0 + Math.sin(rt * 3 + t * 0.5) * 0.3
],
color: [0.95, 0.55, 0.15],
opacity: 1.0,
scale: [0.014, 0.014, 0.014]
});
}
}
// Map a single landmark to world coords. Coord conventions:
// x: 0.5 - lm.x → mirror so left-of-screen = your left side (selfie)
// y: lm.y - 0.5 → keep MediaPipe's y-DOWN convention; the renderer's
// existing -y flip in updateSplats does the single flip
// to Three.js Y-up. Pre-flipping here would double-flip
// and the face would render upside down.
// z: 2.0 + lm.z*8 → amplify lm.z (~[-0.1,+0.1]) so the nose/eye-socket
// depth is visible from an oblique camera angle.
function lmToCenter(lm) {
return [
(0.5 - lm.x) * 4.0,
(lm.y - 0.5) * 3.0,
2.0 + lm.z * 8.0
];
}
function pushFaceSplat(splats, center, alpha) {
splats.push({
center: center,
color: [0.95 * alpha, 0.65 * alpha, 0.20 * alpha],
opacity: 1.0,
scale: [0.006, 0.006, 0.006]
});
}
// FACEMESH_TESSELATION is a flat array [a0,b0, a1,b1, ...] of vertex indices
// forming edges of the triangulated face mesh. ~1300 edges × 2 = 2600 entries.
// We interpolate 6 splats per edge → ~8000 splats per face vs 478 vertices.
var FACE_EDGES = (typeof FACEMESH_TESSELATION !== "undefined") ? FACEMESH_TESSELATION : null;
// Push the user's face mesh point cloud into `splats` (no Foundation
// context — that is the demo path's responsibility). Used both as the
// demo subject AND as an overlay on top of live/remote backend data
// when the camera is enabled. Returns true if any splats were pushed.
function pushFaceSplats(splats) {
if (faceMeshState !== "running" || !latestFaceLandmarks) return false;
var lms = latestFaceLandmarks;
var i;
// 1. Original 478 vertices — bright anchor points for features.
for (i = 0; i < lms.length; i++) {
splats.push({
center: lmToCenter(lms[i]),
color: [1.0, 0.72, 0.25],
opacity: 1.0,
scale: [0.010, 0.010, 0.010]
});
}
// 2. Edge interpolation — 6 splats per FACEMESH_TESSELATION edge.
if (FACE_EDGES) {
var edgeCount = FACE_EDGES.length;
var SAMPLES = 6;
var e, a, b, t, f;
for (e = 0; e < edgeCount; e += 2) {
a = lms[FACE_EDGES[e]];
b = lms[FACE_EDGES[e + 1]];
if (!a || !b) continue;
var aPos = lmToCenter(a);
var bPos = lmToCenter(b);
var ax = aPos[0], ay = aPos[1], az = aPos[2];
var bx = bPos[0], by = bPos[1], bz = bPos[2];
for (t = 1; t <= SAMPLES; t++) {
f = t / (SAMPLES + 1);
pushFaceSplat(splats, [
ax * (1 - f) + bx * f,
ay * (1 - f) + by * f,
az * (1 - f) + bz * f
], 0.85);
}
}
}
return true;
}
function faceMeshFrame() {
if (faceMeshState !== "running" || !latestFaceLandmarks) return null;
var splats = [];
pushFaceSplats(splats);
pushFoundationContext(splats);
demoFrameNum += 1;
return {
splats: splats,
count: splats.length,
frame: demoFrameNum,
live: false,
source: "face-mesh",
pipeline: {
skeleton: null,
vitals: { breathing_rate: 14, motion_score: 0.15 }
}
};
}
function buildSplatsUrl() {
if (backendArg === "demo") return null;
if (backendArg === "auto") return "/api/splats";
// User-supplied URL — strip trailing slash and append /api/splats.
var base = backendArg.replace(/\/+$/, "");
return base + "/api/splats";
}
function syntheticFrame() {
// Used when camera permission is denied / unavailable. Renders a
// procedural standing figure inside the Seldon vault context.
// y-down convention: head at small/negative y, feet at large/positive y;
// the renderer flips y so head ends up at the top of the screen.
var t = (Date.now() - demoStartMs) / 1000.0;
var sway = Math.sin(t * 0.8) * 0.05;
var breath = Math.sin(t * 1.2) * 0.015;
var splats = [];
// Standing figure — 240 points in a vertical cylinder, denser than
// before to feel like a holographic projection.
var ring, k_ring, theta, r, py;
for (ring = 0; ring < 30; ring++) {
py = -1.0 + (ring / 30) * 2.2; // head (-1.0) → feet (+1.2) in y-down
r = 0.20 + breath * (py < 0 ? 1.5 : 0); // chest expands more on inhale
for (k_ring = 0; k_ring < 16; k_ring++) {
theta = (k_ring / 16) * Math.PI * 2;
splats.push({
center: [
sway + Math.cos(theta) * r,
py,
2.3 + Math.sin(theta) * r
],
color: [0.91, 0.65, 0.20],
opacity: 1.0,
scale: [0.018, 0.018, 0.018]
});
}
}
// 17 COCO keypoints in normalized [0,1] image coords (matches live shape)
var headY = 0.18;
var keypoints = [
[0.50 + sway * 0.05, headY, 0.95], // 0 nose
[0.52 + sway * 0.05, headY - 0.01, 0.92], // 1 leftEye
[0.48 + sway * 0.05, headY - 0.01, 0.92], // 2 rightEye
[0.54 + sway * 0.05, headY, 0.85], // 3 leftEar
[0.46 + sway * 0.05, headY, 0.85], // 4 rightEar
[0.60 + sway * 0.04, 0.32, 0.93], // 5 leftShoulder
[0.40 + sway * 0.04, 0.32, 0.93], // 6 rightShoulder
[0.65 + sway * 0.03, 0.46, 0.90], // 7 leftElbow
[0.35 + sway * 0.03, 0.46, 0.90], // 8 rightElbow
[0.68, 0.60 + Math.sin(t * 1.4) * 0.02, 0.86], // 9 leftWrist
[0.32, 0.60 - Math.sin(t * 1.4) * 0.02, 0.86], // 10 rightWrist
[0.57, 0.58, 0.94], // 11 leftHip
[0.43, 0.58, 0.94], // 12 rightHip
[0.58, 0.74, 0.90], // 13 leftKnee
[0.42, 0.74, 0.90], // 14 rightKnee
[0.59, 0.92, 0.88], // 15 leftAnkle
[0.41, 0.92, 0.88] // 16 rightAnkle
];
// Wrap the figure in the Seldon-vault context (grid, spiral, starfield, halo)
pushFoundationContext(splats);
demoFrameNum += 1;
return {
splats: splats,
count: splats.length,
frame: demoFrameNum,
live: false,
pipeline: {
skeleton: { keypoints: keypoints, confidence: 0.86 },
vitals: {
breathing_rate: 14 + Math.round(Math.sin(t * 0.05) * 2),
motion_score: 0.18 + Math.abs(sway) * 2
}
}
};
}
function pickDemoFrame() {
// Prefer real face-mesh data when the camera is running; else procedural.
return faceMeshFrame() || syntheticFrame();
}
// Once auto mode confirms there is no /api/splats backend on this origin,
// set this flag so we stop hammering the network with 404 fetches every
// tick. Console stays clean; demo renders locally.
var networkDisabled = false;
// Exponential backoff state for explicit ?backend=<url>. The user's
// local server may be down (ERR_CONNECTION_REFUSED) and we shouldn't
// hammer it 10 Hz indefinitely. After each failure we lengthen the
// delay; on success we snap back to the normal cadence.
var BASE_INTERVAL_MS = 250;
var MAX_INTERVAL_MS = 30000;
var currentIntervalMs = BASE_INTERVAL_MS;
var consecutiveFailures = 0;
var fetchTimer = null;
var lastBackendError = null;
function scheduleNextFetch(delayMs) {
if (fetchTimer) clearTimeout(fetchTimer);
fetchTimer = setTimeout(fetchCloud, delayMs);
}
async function fetchCloud() {
// Demo-only mode: never hit the network. Use the normal cadence.
if (backendArg === "demo" || networkDisabled) {
transportMode = "demo";
handleData(pickDemoFrame());
scheduleNextFetch(BASE_INTERVAL_MS);
return;
}
try {
var resp = await fetch(buildSplatsUrl(), { cache: "no-store" });
if (!resp.ok) throw new Error("HTTP " + resp.status);
var resp = await fetch("/api/splats");
var data = await resp.json();
transportMode = (backendArg === "auto") ? "live" : "remote";
consecutiveFailures = 0;
currentIntervalMs = BASE_INTERVAL_MS;
lastBackendError = null;
handleData(data);
scheduleNextFetch(BASE_INTERVAL_MS);
} catch (err) {
consecutiveFailures += 1;
lastBackendError = err && err.message ? err.message : String(err);
if (requireLive) {
document.getElementById("stats").innerHTML =
'<span class="demo">&#9679; OFFLINE</span><br>Live backend required (?live=1) but unreachable.<br><span class="label">' + lastBackendError + '</span>';
// Even strict-live: back off so we don't spam.
currentIntervalMs = Math.min(currentIntervalMs * 2, MAX_INTERVAL_MS);
scheduleNextFetch(currentIntervalMs);
return;
}
// Auto mode + first failure → assume static host (Pages), disable
// network entirely so the console stays clean.
if (backendArg === "auto") {
networkDisabled = true;
transportMode = "demo";
handleData(pickDemoFrame());
scheduleNextFetch(BASE_INTERVAL_MS);
return;
}
// Explicit backend (?backend=<url>) — keep trying with
// exponential backoff: 250 ms → 500 ms → 1 s → 2 s … up to 30 s.
// Render the demo while we wait so the scene stays alive, and
// surface the failure so the user knows the server is down.
currentIntervalMs = Math.min(Math.max(BASE_INTERVAL_MS * Math.pow(2, consecutiveFailures - 1), 1000), MAX_INTERVAL_MS);
transportMode = "demo";
var demoFrame = pickDemoFrame();
demoFrame._backendUnreachable = true;
demoFrame._backendUrl = backendArg;
demoFrame._backendError = lastBackendError;
demoFrame._retryInMs = currentIntervalMs;
handleData(demoFrame);
scheduleNextFetch(currentIntervalMs);
}
}
function handleData(data) {
try {
if (data.splats && data.frame !== lastFrame) {
// Compute CSI frame rate
var now = Date.now();
@@ -581,75 +117,24 @@
}
prevTimestamp = now;
lastFrame = data.frame;
updateSplats(data.splats);
// Overlay browser face mesh on top of backend splats when both
// are active — lets visitors see their own face *plus* the
// ESP32-driven point cloud in the same scene. Demo mode (where
// data.source === "face-mesh") already includes the face, so
// we skip this branch there to avoid double-counting.
var rendered = data.splats;
var faceOverlay = false;
if (data.source !== "face-mesh"
&& faceMeshState === "running"
&& latestFaceLandmarks) {
rendered = data.splats.slice();
pushFaceSplats(rendered);
faceOverlay = true;
}
data._faceOverlay = faceOverlay;
updateSplats(rendered);
// No-signal detection: hide skeleton and show banner when
// the server reports no live CSI frames in the last 5s.
// Draw skeleton if available
var pipe = data.pipeline;
var csiLive = data.csi_live || (pipe && pipe.csi_live);
// Only show no-signal when connected to a real backend
// (not demo/face-mesh mode where csi_live is always false).
var showNoSignal = (transportMode === "live" || transportMode === "remote")
&& csiLive === false;
setNoSignal(showNoSignal);
if (showNoSignal) {
clearSkeleton();
} else if (pipe && pipe.skeleton && pipe.skeleton.keypoints) {
if (pipe && pipe.skeleton && pipe.skeleton.keypoints) {
drawSkeleton(pipe.skeleton.keypoints);
} else {
clearSkeleton();
}
// Build info panel — badge reflects active transport
var mode;
if (transportMode === "live") {
mode = '<span class="live">&#9679; LIVE</span> Local Backend';
} else if (transportMode === "remote") {
mode = '<span class="live">&#9679; REMOTE</span> ' + backendArg;
} else if (data.source === "face-mesh") {
mode = '<span class="face">&#9679; DEMO</span> Your Face (MediaPipe)';
} else {
mode = '<span class="demo">&#9679; DEMO</span> Synthetic';
}
if (data._faceOverlay) {
mode += ' <span class="face">+ face overlay</span>';
}
var splatCount = rendered ? rendered.length : data.count;
var html = mode + "<br>"
+ "Splats: " + splatCount + "<br>"
// Build info panel
var mode = data.live
? '<span class="live">&#9679; LIVE</span>'
: '<span class="demo">&#9679; DEMO</span>';
var html = mode + " Camera + CSI<br>"
+ "Splats: " + data.count + "<br>"
+ "Frame: " + data.frame;
// Unreachable backend banner — explicit ?backend=<url> failed
// to connect. Show actionable guidance instead of leaving the
// user staring at a "demo" badge wondering why their ESP32
// feed isn't visible.
if (data._backendUnreachable) {
var nextSec = Math.round((data._retryInMs || 1000) / 1000);
html += '<div class="section">'
+ '<span class="demo">&#9679; ' + data._backendUrl + '</span> unreachable'
+ '<br><span class="label">' + (data._backendError || "connection failed") + '</span>'
+ '<br><span class="label">retry in ' + nextSec + 's</span>'
+ '<br><br><span class="label">start the server:</span>'
+ '<br><code style="color:#e8a634">cargo run -p wifi-densepose-pointcloud --release \\<br>&nbsp;&nbsp;-- serve --bind 127.0.0.1:9880</code>'
+ '</div>';
}
// CSI frame rate
html += '<div class="section">'
+ '<span class="label">CSI Rate:</span> '
@@ -702,69 +187,8 @@
}
} catch(e) {}
}
// Wire the camera CTA. The camera is now overlay-able on every
// transport mode: in demo it IS the subject; in live/remote it
// overlays the backend splats so the visitor sees their face
// alongside the ESP32-driven point cloud.
(function wireCamCta() {
var btn = document.getElementById("cam-cta");
if (!btn) return;
if (requireLive) {
// Strict-live mode shows the offline panel — no camera UI.
btn.classList.add("hidden");
return;
}
// In remote mode, label the button as an overlay action.
if (backendArg.startsWith("http")) {
btn.textContent = "▶ Add face overlay";
}
btn.addEventListener("click", function() {
btn.textContent = backendArg.startsWith("http")
? "Starting overlay…"
: "Initializing the Vault…";
startFaceMesh();
});
})();
// Wire the ESP32 backend CTA: prompts for a ruview-pointcloud serve URL,
// persists last-used value in localStorage, and reloads with the
// ?backend=<url> query so the existing remote-mode path takes over.
// Disconnect by clicking again when already connected.
(function wireEspCta() {
var btn = document.getElementById("esp-cta");
if (!btn) return;
var connected = backendArg.startsWith("http");
if (connected) {
btn.classList.add("connected");
btn.textContent = "📡 ESP32 connected · disconnect";
}
btn.addEventListener("click", function() {
if (connected) {
// Strip ?backend= from current URL and reload — return to demo.
var u = new URL(window.location.href);
u.searchParams.delete("backend");
window.location.href = u.toString();
return;
}
var stored;
try { stored = localStorage.getItem("ruview.backendUrl"); } catch (_) { stored = null; }
var def = stored || "http://127.0.0.1:9880";
var url = window.prompt(
"Enter the ruview-pointcloud serve URL (run `ruview-pointcloud serve` locally with your ESP32 streaming CSI to UDP port 3333):",
def
);
if (!url) return;
url = url.replace(/\/+$/, "");
try { localStorage.setItem("ruview.backendUrl", url); } catch (_) {}
var u2 = new URL(window.location.href);
u2.searchParams.set("backend", url);
window.location.href = u2.toString();
});
})();
// fetchCloud self-schedules via setTimeout — no setInterval to avoid
// overlapping calls on slow networks and to support exponential backoff.
fetchCloud();
setInterval(fetchCloud, 500);
function updateSplats(splats) {
if (pointsMesh) scene.remove(pointsMesh);