Compare commits

..

6 Commits

Author SHA1 Message Date
Dragan Spiridonov ee11a5a766 fix(firmware): address PR #397 review feedback
Applies @ruvnet's five review requests on PR #397 (RuView#397 comment
4289417527):

1. **Inline comment on `provision.py` `write_flash`** — ESP-IDF v5.4
   bundles esptool 4.10.0 (underscore-only). #391's hyphen swap broke
   the documented venv flow; kept the underscore form and added a
   three-line comment warning future maintainers not to "re-fix" it.

2. **Correct `edge_processing.c` sample_rate** (blocking) — changed
   hard-coded `20.0f` → `10.0f` at line 718 so
   `estimate_bpm_zero_crossing()` matches the MGMT-only CSI rate.
   Without this, breathing and heart-rate reports were 2× the true
   value. Added a comment tying the constant to the callback rate gate.

3. **Removed disabled probe-injection infrastructure** — dropped the
   forward declaration, the `CSI_PROBE_INTERVAL_MS` define, six static
   variables (`s_probe_timer`, `s_probe_tx_count`, `s_probe_tx_fail`,
   `s_ap_bssid`, `s_ap_bssid_known`), and three functions
   (`csi_send_probe_request`, `probe_timer_cb`,
   `csi_collector_start_probe_timer`). None were reachable.
   `csi_inject_ndp_frame()` reverted to the original ADR-029 stub.
   Can be revived from this commit's parent if needed.

4. **Cleaned `sdkconfig.defaults`** — removed the SPIRAM prose and
   commented-out `# CONFIG_SPIRAM is not set` line. Kept only the live
   `CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y` with a concise rationale.

5. **Bumped firmware version 0.6.1 → 0.6.2** and added four
   `[Unreleased]` CHANGELOG entries covering the SPI cache crash fix,
   the `filter_mac` / `node_id` clobber defense, the sample-rate
   correction, and the `write_flash` command-form revert.

Net: +39 / -128 across six files.

Validation in this devcontainer:
- Static sanity on modified C files: braces balance (csi_collector.c
  59/59; edge_processing.c 96/96), zero dangling references to removed
  probe-injection symbols.
- Rust workspace tests and Python proof not executed here — cargo not
  installed and pip blocked by PEP 668. Deferring hardware build +
  flash + miniterm verification to @ruvnet's COM7 per his offer in
  the review comment.

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-28 08:41:00 -04:00
Dragan Spiridonov a14835bb91 fix(firmware): 50 Hz callback rate gate + sdkconfig extra IRAM opt
- Add early rate gate in wifi_csi_callback at 50 Hz (defense-in-depth,
  does not prevent crash alone but reduces callback execution time)
- Add null-data injection timer infrastructure (disabled — TX adds
  interrupt pressure that triggers the SPI cache crash, RuView#396)
- sdkconfig.defaults: add CONFIG_ESP_WIFI_EXTRA_IRAM_OPT=y
- sdkconfig.defaults: document SPIRAM XIP attempt (crashes differently)

Co-Authored-By: Ruflo & AQE
2026-04-28 08:41:00 -04:00
Dragan Spiridonov ace61696b1 fix(provision): write-flash → write_flash for esptool v5 compat
esptool v5+ rejects hyphenated subcommands. The provision script
used 'write-flash' which fails with "invalid choice". Changed to
'write_flash' (underscore) which works with both old and new esptool.

Co-Authored-By: Ruflo & AQE
2026-04-28 08:39:51 -04:00
Dragan Spiridonov c442669ea8 fix(firmware): MGMT-only promiscuous filter to prevent SPI cache crash
The WiFi driver's wDev_ProcessFiq interrupt handler crashes with
LoadProhibited in cache_ll_l1_resume_icache when promiscuous mode
captures MGMT+DATA frames (100-500 interrupts/sec). The high interrupt
rate races with SPI flash cache operations, corrupting cache state.

Changes:
- Promiscuous filter: MGMT+DATA → MGMT-only (~10 Hz beacons)
- CSI config: disable htltf_en and stbc_htltf2_en (LLTF-only)

LLTF provides 64 subcarriers (HT20) — sufficient for presence,
breathing, and fall detection. The 10 Hz beacon rate eliminates
the SPI flash cache contention that caused the crash.

Verified on device 80:b5:4e:c1:be:b8:
- Before: LoadProhibited crash at ~1600-2400 callbacks (every ~70s)
- After: 2700+ callbacks over 4.7 minutes, zero crashes

Backtrace decode confirmed crash in ESP-IDF closed-source WiFi blob:
  _xt_lowint1 → wDev_ProcessFiq → spi_flash_restore_cache
  → cache_ll_l1_resume_icache → EXCVADDR=0x00000004 (NULL deref)

Co-Authored-By: Ruflo & AQE
2026-04-28 08:39:51 -04:00
Dragan Spiridonov b8e332cd2a fix(firmware): defensive copy of filter_mac to prevent callback crash
The CSI callback reads g_nvs_config.filter_mac_set and filter_mac on
every invocation (100-500 Hz). If wifi_init_sta() corrupts g_nvs_config
(same root cause as the node_id clobber), the callback reads garbage
from the struct, leading to Core 0 LoadProhibited panic after ~2400
callbacks (~70 seconds of operation).

Extends the early-capture pattern from the node_id fix to also copy
filter_mac_set and filter_mac into module-local statics before WiFi
init runs. Adds canary logging to detect filter_mac corruption.

Observed on device 80:b5:4e:c1:be:b8 via serial:
  CSI cb #2400 → Guru Meditation Error: Core 0 panic'ed (LoadProhibited)
  → TG0WDT_SYS_RST → reboot → crash again at ~2900 callbacks

Refs #232 #375 #385 #386 #390

Co-Authored-By: Ruflo & AQE
2026-04-28 08:39:50 -04:00
Dragan Spiridonov 415cb5606d fix(firmware): move defensive node_id capture before wifi_init_sta()
The original defensive copy in csi_collector_init() (line 172 of main.c)
runs AFTER wifi_init_sta() (line 147), which on some ESP32-S3 devices
corrupts g_nvs_config.node_id back to the Kconfig default of 1.

Reproduced on device 80:b5:4e:c1:be:b8 (ESP32-S3 QFN56 rev v0.2):
  - NVS provisioned with node_id=5
  - Release firmware (no fix): seed receives node_id=1 (clobbered)
  - This patch: seed receives node_id=5 (correct)

Changes:
  - Add csi_collector_set_node_id() called from main.c immediately
    after nvs_config_load(), before wifi_init_sta() runs
  - csi_collector_init() now detects and logs the clobber if early
    capture disagrees with current g_nvs_config value
  - Fallback path preserved: if set_node_id() is never called,
    init() still captures from g_nvs_config (backwards compatible)

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-28 08:39:50 -04:00
18 changed files with 1952 additions and 2895 deletions
-58
View File
@@ -1,58 +0,0 @@
version: 2
updates:
# Keep all third-party GitHub Actions on verified, pinned commit SHAs.
# Pairs with the SHA pinning in security-scan.yml and ci.yml so that
# future bumps stay automated and reviewable rather than drifting back
# to mutable @master / @main refs. See issue #442.
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
open-pull-requests-limit: 5
labels:
- dependencies
- github-actions
# Mobile app npm deps. Includes the @xmldom/xmldom, node-forge, and
# picomatch advisories from #442 plus axios and any future surface.
- package-ecosystem: npm
directory: /ui/mobile
schedule:
interval: weekly
open-pull-requests-limit: 10
labels:
- dependencies
- mobile
# Desktop UI npm deps. Direct vite devDep currently has a HIGH advisory
# (dev-server-only path traversal); track future bumps automatically.
- package-ecosystem: npm
directory: /v2/crates/wifi-densepose-desktop/ui
schedule:
interval: weekly
open-pull-requests-limit: 5
labels:
- dependencies
- desktop
# Python deps used by v1/ and the FastAPI service. requirements.txt is
# only loosely pinned; let Dependabot surface upstream CVE bumps.
- package-ecosystem: pip
directory: /
schedule:
interval: weekly
open-pull-requests-limit: 10
labels:
- dependencies
- python
# Rust workspace (15+ crates). cargo audit is not currently wired into
# any workflow, so Dependabot is the primary automated bump path.
- package-ecosystem: cargo
directory: /v2
schedule:
interval: weekly
open-pull-requests-limit: 10
labels:
- dependencies
- rust
+1 -20
View File
@@ -18,11 +18,6 @@ jobs:
code-quality:
name: Code Quality & Security
runs-on: ubuntu-latest
# Skip when there's no Python source at the root `src/` to lint. The
# active codebase is the Rust workspace under `v2/`; legacy Python
# lives at `archive/v1/src/` and is not part of CI gating. If `src/`
# is reintroduced this job will run automatically.
if: hashFiles('src/**/*.py') != ''
steps:
- name: Checkout code
uses: actions/checkout@v4
@@ -75,16 +70,6 @@ jobs:
- name: Checkout code
uses: actions/checkout@v4
- name: Install system dependencies
# glib-sys (transitive via gstreamer/gtk crates in the workspace)
# needs glib-2.0 + pkg-config at build time. Without these the
# workspace build fails: "failed to run custom build command for
# `glib-sys vN.M.K`".
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
pkg-config libglib2.0-dev
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
@@ -107,10 +92,6 @@ jobs:
test:
name: Tests
runs-on: ubuntu-latest
# Skip when there's no Python test suite under `tests/unit/` /
# `tests/integration/` to run. The legacy Python tests have been
# archived under `archive/v1/tests/` and are not part of CI gating.
if: hashFiles('tests/unit/**/*.py') != '' || hashFiles('tests/integration/**/*.py') != ''
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12']
@@ -274,7 +255,7 @@ jobs:
docker stop test-container
- name: Run container security scan
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
uses: aquasecurity/trivy-action@master
with:
image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
format: 'sarif'
-74
View File
@@ -1,74 +0,0 @@
name: Point Cloud Viewer → GitHub Pages
# Publishes the live 3D point cloud viewer to gh-pages/pointcloud/.
# The viewer defaults to a synthetic in-browser demo; users can append
# ?backend=<url> or ?backend=auto to point it at a real ruview-pointcloud
# server (CORS-permitting host required). See ADR-094.
#
# Uses keep_files: true to preserve the existing observatory/, pose-fusion/,
# nvsim/, and root index.html demos already on gh-pages.
on:
push:
branches: [main]
paths:
- 'v2/crates/wifi-densepose-pointcloud/src/viewer.html'
- '.github/workflows/pointcloud-pages.yml'
workflow_dispatch:
permissions:
contents: write
concurrency:
group: pointcloud-pages
cancel-in-progress: true
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout main
uses: actions/checkout@v4
- name: Stage viewer for Pages
run: |
mkdir -p _site/pointcloud
cp v2/crates/wifi-densepose-pointcloud/src/viewer.html _site/pointcloud/index.html
# Drop a tiny README so direct browsers of the directory get context.
cat > _site/pointcloud/README.md <<'EOF'
# RuView — Live 3D Point Cloud Viewer
Hosted at: https://ruvnet.github.io/RuView/pointcloud/
## Modes
- Default — synthetic in-browser demo (no backend, no network calls).
- `?backend=auto` — fetch from `/api/splats` on the same origin
(only works when the viewer is served by `ruview-pointcloud serve`).
- `?backend=<url>` — fetch from `<url>/api/splats`. The intended
local-ESP32 use is `?backend=http://127.0.0.1:9880`: run
`ruview-pointcloud serve --bind 127.0.0.1:9880` on the same
machine with your ESP32 streaming CSI to UDP port 3333, then
visit the URL above. The local server's CorsLayer permits
requests from `https://ruvnet.github.io`, and modern browsers
permit HTTPS→127.0.0.1 mixed-content as a trustworthy origin.
The "📡 Connect ESP32" button in the viewer prompts for this
URL and persists it in localStorage.
- `?live=1` — require a live backend; show an offline message instead
of falling back to the synthetic demo.
See ADR-094 for the deployment design.
EOF
- name: Deploy to gh-pages/pointcloud/
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./_site/pointcloud
destination_dir: pointcloud
# CRITICAL: preserves observatory/, pose-fusion/, nvsim/, and root
# index.html already on gh-pages.
keep_files: true
commit_message: 'deploy(pointcloud): ${{ github.sha }}'
user_name: 'github-actions[bot]'
user_email: 'github-actions[bot]@users.noreply.github.com'
+13 -40
View File
@@ -18,11 +18,6 @@ jobs:
sast:
name: Static Application Security Testing
runs-on: ubuntu-latest
# Skip when there's no Python source at the root `src/` to scan.
# The Bandit + Semgrep targets in this job are hard-coded to `src/`;
# the active codebase is the Rust workspace under `v2/` (covered by
# `cargo audit` in the dependency-scan job below).
if: hashFiles('src/**/*.py') != ''
permissions:
security-events: write
actions: read
@@ -116,7 +111,7 @@ jobs:
continue-on-error: true
- name: Run Snyk vulnerability scan
uses: snyk/actions/python@9adf32b1121593767fc3c057af55b55db032dc04 # v1.0.0
uses: snyk/actions/python@master
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
@@ -124,12 +119,8 @@ jobs:
continue-on-error: true
- name: Upload Snyk results to GitHub Security
# Skip when Snyk had no token / produced no SARIF (e.g. on PRs from
# forks without secrets). Without this guard the upload step fails
# the whole job whenever the optional Snyk scan was effectively a
# no-op.
if: ${{ always() && hashFiles('snyk-results.sarif') != '' }}
uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: snyk-results.sarif
category: snyk
@@ -142,11 +133,6 @@ jobs:
path: |
safety-report.json
pip-audit-report.json
# Both upstream scans use `continue-on-error: true` and may
# produce no JSON when their dependencies break or a PR runs
# without registry access; treat a missing report as a warning
# instead of failing the whole upload step.
if-no-files-found: ignore
snyk-results.sarif
# Container security scanning
@@ -177,7 +163,7 @@ jobs:
cache-to: type=gha,mode=max
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
uses: aquasecurity/trivy-action@master
with:
image-ref: 'wifi-densepose:scan'
format: 'sarif'
@@ -235,7 +221,7 @@ jobs:
uses: actions/checkout@v4
- name: Run Checkov IaC scan
uses: bridgecrewio/checkov-action@99bb2caf247dfd9f03cf984373bc6043d4e32ebf # v12.1347.0
uses: bridgecrewio/checkov-action@master
with:
directory: .
framework: kubernetes,dockerfile,terraform,ansible
@@ -252,7 +238,7 @@ jobs:
category: checkov
- name: Run Terrascan IaC scan
uses: tenable/terrascan-action@3a6e87da8e244513bd77b631e624552643f794c6 # v1.4.1
uses: tenable/terrascan-action@main
with:
iac_type: 'k8s'
iac_version: 'v1'
@@ -261,7 +247,7 @@ jobs:
sarif_upload: true
- name: Run KICS IaC scan
uses: checkmarx/kics-github-action@05aa5eb70eede1355220f4ca5238d96b397e30a6 # v2.1.20
uses: checkmarx/kics-github-action@master
with:
path: '.'
output_path: kics-results
@@ -270,11 +256,8 @@ jobs:
exclude_queries: 'a7ef1e8c-fbf8-4ac1-b8c7-2c3b0e6c6c6c'
- name: Upload KICS results to GitHub Security
# KICS does not always produce a SARIF (e.g. when no IaC files are
# present in the repo); guard the upload so a missing file does
# not fail the iac-scan job.
if: ${{ always() && hashFiles('kics-results/results.sarif') != '' }}
uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: kics-results/results.sarif
category: kics
@@ -294,7 +277,7 @@ jobs:
fetch-depth: 0
- name: Run TruffleHog secret scan
uses: trufflesecurity/trufflehog@17456f8c7d042d8c82c9a8ca9e937231f9f42e26 # v3.95.2
uses: trufflesecurity/trufflehog@main
with:
path: ./
base: main
@@ -355,10 +338,7 @@ jobs:
- name: Check security policy files
run: |
# Check for required security files. Missing policy is reported
# as a warning rather than a hard failure so the broader
# compliance job stays informational; tracked separately for
# follow-up.
# Check for required security files
files=("SECURITY.md" ".github/SECURITY.md" "docs/SECURITY.md")
found=false
for file in "${files[@]}"; do
@@ -369,21 +349,14 @@ jobs:
fi
done
if [[ "$found" == false ]]; then
echo "::warning::No security policy found. Please create SECURITY.md"
echo "No security policy found. Please create SECURITY.md"
exit 1
fi
- name: Check for security headers in code
run: |
# Check for security-related configurations. Skip cleanly when
# `src/` does not exist (Rust-first repo layout); a missing
# directory makes `grep -r` exit with status 2, which would
# fail the step despite the trailing `||`.
if [[ -d src ]]; then
grep -r "X-Frame-Options\|X-Content-Type-Options\|X-XSS-Protection\|Content-Security-Policy" src/ \
|| echo "⚠️ Consider adding security headers"
else
echo "️ No src/ directory at repo root — skipping web security headers grep"
fi
# Check for security-related configurations
grep -r "X-Frame-Options\|X-Content-Type-Options\|X-XSS-Protection\|Content-Security-Policy" src/ || echo "⚠️ Consider adding security headers"
- name: Validate Kubernetes security contexts
run: |
+1874 -26
View File
File diff suppressed because it is too large Load Diff
@@ -1,203 +0,0 @@
# ADR-094: Live 3D Point Cloud Viewer — GitHub Pages Deployment with Optional Real-Data Stream
| Field | Value |
|---|---|
| **Status** | Proposed (2026-04-29) |
| **Date** | 2026-04-29 |
| **Authors** | ruv |
| **Related** | ADR-092 (nvsim dashboard Pages deployment), ADR-059 (live ESP32 CSI pipeline), ADR-079 (camera ground-truth training) |
| **Branch** | `feat/pointcloud-pages-demo` |
---
## 1. Context
The `wifi-densepose-pointcloud` crate ships a Three.js-based viewer
(`v2/crates/wifi-densepose-pointcloud/src/viewer.html`) that renders the
fused camera-depth + WiFi CSI + mmWave point cloud produced by the
`ruview-pointcloud serve` binary. Today the viewer is local-only:
- It is served by the Axum binary on `127.0.0.1:9880`.
- It polls `/api/splats` every 500 ms expecting a backend on the same
origin.
- There is no GitHub Pages deployment, so the README's
"▶ Live 3D Point Cloud" link points at the moved-content section in
`docs/readme-details.md`, not at a hosted demo. The two sibling demos
(Live Observatory, Dual-Modal Pose Fusion) are already hosted at
`https://ruvnet.github.io/RuView/` and `…/pose-fusion.html`.
This is an asymmetry: a first-time visitor can preview the WiFi pose
demo and the Observatory in one click, but cannot preview the point
cloud without cloning the repo, building Rust, plugging in an ESP32,
and pointing a webcam at themselves. That gap suppresses the most
visually compelling demonstration of the v0.7+ sensor-fusion work.
A naive fix — drop the static HTML at `gh-pages/pointcloud/` — does
not work because the viewer's `fetch("/api/splats")` will 404 on Pages
and the canvas will hang at "Loading…". A second naive fix — bake in a
fixed sample dataset — solves the loading state but loses the live-data
story entirely, and forks the viewer into a "demo build" and a "real
build" that drift apart.
## 2. Decision
Ship **one** viewer that auto-selects its transport from URL parameters,
and publish it to `gh-pages/pointcloud/` alongside the other demos:
1. **Default mode** — when the viewer is opened with no query parameters
on `https://ruvnet.github.io/RuView/pointcloud/`, present a "▶ Enable
camera" CTA. On click the viewer requests webcam access, runs
**MediaPipe Face Mesh** in-browser (~30 fps, 478 refined landmarks),
and renders the visitor's own face as a point cloud — the closest
browser equivalent of the local pipeline's depth-backprojected face
geometry that motivated this ADR (`I could see the outline of my face
in points`). The viewer mirrors x to match selfie convention and
maps Face Mesh's relative-z to the same world-coordinate range the
live `/api/splats` payload uses, so a single render path drives both.
Badge reads `● DEMO Your Face (MediaPipe)`. If the user denies
camera permission, dismisses the prompt, or visits on a device
without a webcam, the viewer falls back automatically to a
procedural scaffold (floor grid, walls, breathing figure, 17-keypoint
skeleton). All processing is client-side; no frames leave the
browser. ~480-500 splats from the face plus ~110 floor/wall context
splats.
2. **Auto mode** (`?backend=auto`) — fetch from `/api/splats` on the same
origin. This is the local-development case (`ruview-pointcloud serve`
serves the viewer and the API together). On any failure (404, network
error, CORS), fall back silently to synthetic-demo rendering so the
tab never dies.
3. **Remote mode** (`?backend=<url>`) — fetch from `<url>/api/splats`.
This is the **integrated-ESP32** path: the user runs
`ruview-pointcloud serve --bind 127.0.0.1:9880` locally with an
ESP32-S3 streaming CSI to UDP port 3333, then opens
`https://ruvnet.github.io/RuView/pointcloud/?backend=http://127.0.0.1:9880`.
The hosted Pages viewer becomes a thin client for the local Rust
fusion pipeline (camera depth + WiFi CSI + mmWave) without a clone
or rebuild. The viewer also exposes a "📡 Connect ESP32" button that
prompts for the URL, persists it in `localStorage`, and reloads
with the query param.
For this to work the local server must answer the browser's CORS
preflight. `stream.rs` therefore installs a `tower_http` `CorsLayer`
that allows three origin classes:
- `https://ruvnet.github.io` — the published Pages demo.
- `http://localhost:*` and `http://127.0.0.1:*` — developer running
the bundled `viewer.html` directly.
- `null``file://` origins.
Mixed-content (HTTPS Pages → HTTP loopback) is permitted because
modern browsers (Chrome 94+, Firefox 116+, Safari 16.4+) classify
`127.0.0.1` and `localhost` as "potentially trustworthy" origins.
Any other origin (a public hostname, etc.) is denied — this is not
a wildcard CORS posture. Badge reads `● REMOTE <url>`. Same silent
demo fallback on failure.
4. **Strict-live mode** (`?live=1`) — disable the demo fallback. If the
chosen transport fails, replace the info panel with an explicit offline
message (`● OFFLINE — Live backend required but unreachable`). Useful
for embedding the viewer in a status page or kiosk.
The synthetic frame returned by the in-browser generator matches the
JSON shape of the live `/api/splats` payload exactly (`splats`, `count`,
`frame`, `live`, `pipeline.{skeleton,vitals,…}`), so a single render path
drives both modes. There is no demo build vs real build — only one HTML
file, one render path, and one set of bugs.
A new GitHub Actions workflow (`.github/workflows/pointcloud-pages.yml`)
copies the viewer to `gh-pages/pointcloud/index.html` on every push to
`main` that touches the viewer, using `peaceiris/actions-gh-pages@v4`
with `keep_files: true` to preserve the existing observatory, pose-fusion,
and nvsim deployments.
## 3. Consequences
### Positive
- **First-click demo.** Visitors clicking the README's
"▶ Live 3D Point Cloud" link land on a working Three.js scene in <1 s,
no toolchain required. Matches the parity of the other two demos.
- **Real-data on demand.** Users with their own `ruview-pointcloud serve`
host can use the same hosted viewer URL with
`?backend=https://their-host.example.com` — no clone, no rebuild. The
hosted demo doubles as a thin client for self-hosted backends.
- **Single render path.** Synthetic frames flow through the same
`handleData → updateSplats → drawSkeleton` pipeline as live frames, so
visual regressions surface in the demo and the live build at the same
time. This is the same dual-transport pattern ADR-092 chose for nvsim.
- **No backend deploy required.** Pages serves static HTML; the demo
works without standing up an Axum host on the public internet, and
there is no per-visitor CSI/camera plumbing to provision.
- **Preserves existing deployments.** `keep_files: true` plus the
`pointcloud/` destination means observatory/, pose-fusion/, nvsim/,
and the root index.html on gh-pages are untouched.
### Negative / tradeoffs
- **Face mesh ≠ CSI.** Browser webcam + MediaPipe gives real face
geometry but does not produce CSI-derived pose. Visitors who want to
see the *WiFi-driven* path still need `?backend=<their-host>`. The
procedural fallback is not WiFi-driven either; it is purely visual
scaffolding. We accept this — the goal of the hosted demo is to
convey the *shape* of what the local pipeline produces (a point
cloud of the user) rather than reproduce the WiFi physics in the
browser. The latter is a future ADR (WASM port of the fusion crate).
- **CORS burden on remote mode.** Users who want to share their backend
must add `Access-Control-Allow-Origin: https://ruvnet.github.io` (or
`*`) to their `ruview-pointcloud serve` config. We document this in the
workflow's generated README; we do **not** add a public proxy.
- **Synthetic generator lives in the viewer.** ~80 LOC of procedural JS
is now part of `viewer.html`. Acceptable: the file is already the
client-side render bundle, and the generator is bounded and inert
(deterministic, no I/O, no eval).
- **No replay-from-recording in this ADR.** A future ADR may add a
`?recording=<url>.jsonl` mode that replays captured frames at native
rate; that is out of scope here.
### Neutral
- The local-dev experience is unchanged. `ruview-pointcloud serve` still
serves `viewer.html` from the bundled asset and the viewer still hits
`/api/splats` because `?backend` defaults to `auto`. Nothing in the
Rust crate changes — this is HTML + workflow only.
## 4. Implementation
| File | Change |
|---|---|
| `v2/crates/wifi-densepose-pointcloud/src/viewer.html` | Add URL-param transport selector (`backend`, `live`), synthetic frame generator, demo-fallback path, transport-aware mode badge. ~120 LOC added, no removed behavior. |
| `.github/workflows/pointcloud-pages.yml` | New workflow: stage viewer to `_site/pointcloud/index.html`, deploy to `gh-pages/pointcloud/` with `keep_files: true`. Triggers on viewer changes and on manual dispatch. |
| `README.md` | Already updated — `▶ Live 3D Point Cloud` link will be retargeted to `https://ruvnet.github.io/RuView/pointcloud/` once the first deploy succeeds. (Tracked separately, not blocking this ADR.) |
| `docs/adr/README.md` | ADR index — add ADR-094 row. |
## 5. Acceptance Gates
This ADR is **Implemented** when all of the following hold:
1. Pushing to `main` with a viewer change triggers
`pointcloud-pages.yml`, which deploys to `gh-pages/pointcloud/` in
under 60 seconds.
2. `https://ruvnet.github.io/RuView/pointcloud/` loads, shows the
"Enable camera" CTA, and on accept renders the visitor's face as a
point cloud with badge `● DEMO Your Face (MediaPipe)` and non-zero
splat + frame counts. On camera denial, falls back to the
procedural scene with badge `● DEMO Synthetic`.
3. Existing demos at `https://ruvnet.github.io/RuView/` and
`…/pose-fusion.html` and `…/nvsim/` are still reachable after the
first deploy (smoke-tested manually).
4. `https://ruvnet.github.io/RuView/pointcloud/?live=1` shows the
`● OFFLINE` panel (because no same-origin backend exists on Pages).
5. `https://ruvnet.github.io/RuView/pointcloud/?backend=https://example.invalid`
falls back to demo within one poll interval (~500 ms) without
throwing in the console.
6. Running `./target/release/ruview-pointcloud serve` locally and
opening `http://127.0.0.1:9880/` (which serves the same HTML) still
shows live-mode rendering with the `● LIVE Local Backend` badge.
## 6. Out of Scope
- Replaying recorded JSONL frames in the browser (future ADR).
- WASM-side execution of the fusion pipeline in the browser (would
require porting the camera + mmWave path; deferred).
- Authentication / signed splats payloads — backend-side concern,
unaffected by this client-side change.
- Hosting a public CORS proxy for users without their own backend.
File diff suppressed because it is too large Load Diff
Binary file not shown.
@@ -29,7 +29,5 @@ CONFIG_LWIP_SO_RCVBUF=y
CONFIG_ESP_MAIN_TASK_STACK_SIZE=8192
# ADR-081: adaptive_controller runs emit_feature_state + stream_sender
# network I/O inside Timer Svc callbacks. 8 KiB was insufficient under
# lwIP sendto + state-transition rv_mesh anomaly emit (issue #505 4MB
# reset loop on Tmr Svc); 16 KiB absorbs the worst-case path.
CONFIG_FREERTOS_TIMER_TASK_STACK_DEPTH=16384
# network I/O inside Timer Svc callbacks, exceeding the 2 KiB default.
CONFIG_FREERTOS_TIMER_TASK_STACK_DEPTH=8192
@@ -33,7 +33,5 @@ CONFIG_LWIP_SO_RCVBUF=y
CONFIG_ESP_MAIN_TASK_STACK_SIZE=8192
# ADR-081: adaptive_controller runs emit_feature_state + stream_sender
# network I/O inside Timer Svc callbacks. 8 KiB was insufficient under
# lwIP sendto + state-transition rv_mesh anomaly emit (issue #505 4MB
# reset loop on Tmr Svc); 16 KiB absorbs the worst-case path.
CONFIG_FREERTOS_TIMER_TASK_STACK_DEPTH=16384
# network I/O inside Timer Svc callbacks, exceeding the 2 KiB default.
CONFIG_FREERTOS_TIMER_TASK_STACK_DEPTH=8192
+1 -1
View File
@@ -1 +1 @@
0.6.4
0.6.2
+46 -10
View File
@@ -5127,9 +5127,9 @@
"license": "BSD-3-Clause"
},
"node_modules/@xmldom/xmldom": {
"version": "0.8.13",
"resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.13.tgz",
"integrity": "sha512-KRYzxepc14G/CEpEGc3Yn+JKaAeT63smlDr+vjB8jRfgTBBI9wRj/nkQEO+ucV8p8I9bfKLWp37uHgFrbntPvw==",
"version": "0.8.11",
"resolved": "https://registry.npmjs.org/@xmldom/xmldom/-/xmldom-0.8.11.tgz",
"integrity": "sha512-cQzWCtO6C8TQiYl1ruKNn2U6Ao4o4WBBcbL61yJl84x+j5sOWWFU9X7DpND8XZG3daDppSsigMdfAIl2upQBRw==",
"license": "MIT",
"engines": {
"node": ">=10.0.0"
@@ -5310,6 +5310,18 @@
"node": ">= 8"
}
},
"node_modules/anymatch/node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"license": "MIT",
"engines": {
"node": ">=8.6"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/arg": {
"version": "5.0.2",
"resolved": "https://registry.npmjs.org/arg/-/arg-5.0.2.tgz",
@@ -11923,6 +11935,18 @@
"node": ">=8"
}
},
"node_modules/jest-util/node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"license": "MIT",
"engines": {
"node": ">=8.6"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/jest-validate": {
"version": "29.7.0",
"resolved": "https://registry.npmjs.org/jest-validate/-/jest-validate-29.7.0.tgz",
@@ -13365,6 +13389,18 @@
"node": ">=8.6"
}
},
"node_modules/micromatch/node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"license": "MIT",
"engines": {
"node": ">=8.6"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
}
},
"node_modules/mime": {
"version": "1.6.0",
"resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz",
@@ -13558,9 +13594,9 @@
}
},
"node_modules/node-forge": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/node-forge/-/node-forge-1.4.0.tgz",
"integrity": "sha512-LarFH0+6VfriEhqMMcLX2F7SwSXeWwnEAJEsYm5QKWchiVYVvJyV9v7UDvUv+w5HO23ZpQTXDv/GxdDdMyOuoQ==",
"version": "1.3.3",
"resolved": "https://registry.npmjs.org/node-forge/-/node-forge-1.3.3.tgz",
"integrity": "sha512-rLvcdSyRCyouf6jcOIPe/BgwG/d7hKjzMKOas33/pHEr6gbq18IK9zV7DiPvzsz0oBJPme6qr6H6kGZuI9/DZg==",
"license": "(BSD-3-Clause OR GPL-2.0)",
"engines": {
"node": ">= 6.13.0"
@@ -14020,12 +14056,12 @@
"license": "ISC"
},
"node_modules/picomatch": {
"version": "2.3.2",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz",
"integrity": "sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==",
"version": "4.0.3",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.3.tgz",
"integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==",
"license": "MIT",
"engines": {
"node": ">=8.6"
"node": ">=12"
},
"funding": {
"url": "https://github.com/sponsors/jonschlinkert"
-5
View File
@@ -49,10 +49,5 @@
"react-native-worklets": "^0.7.4",
"typescript": "~5.9.2"
},
"overrides": {
"@xmldom/xmldom": "0.8.13",
"node-forge": "^1.4.0",
"picomatch": "^2.3.2"
},
"private": true
}
@@ -14,7 +14,6 @@ serde_json = { workspace = true }
tokio = { workspace = true }
anyhow = { workspace = true }
axum = { workspace = true }
tower-http = { workspace = true }
clap = { version = "4", features = ["derive"] }
chrono = "0.4"
dirs = "5"
@@ -8,13 +8,11 @@ use crate::fusion;
use crate::pointcloud;
use axum::{
extract::State,
http::{HeaderValue, Method},
response::Html,
routing::get,
Json, Router,
};
use std::sync::{Arc, Mutex};
use tower_http::cors::{AllowOrigin, CorsLayer};
struct AppState {
latest_cloud: Mutex<pointcloud::PointCloud>,
@@ -110,36 +108,12 @@ pub async fn serve(bind: &str, _brain: Option<&str>) -> anyhow::Result<()> {
if has_camera { eprintln!(" Camera: LIVE (/dev/video0)"); }
else { eprintln!(" Camera: DEMO"); }
// CORS — allow the hosted GitHub Pages viewer to fetch /api/splats from a
// locally-running instance of this server. Modern browsers treat
// 127.0.0.1/localhost as a "potentially trustworthy" origin so the HTTPS
// page can reach a plain-HTTP loopback backend without mixed-content
// blocking. Origins permitted:
// - https://ruvnet.github.io (the published RuView Pages demo)
// - http://localhost:* / http://127.0.0.1:* (developer running the
// viewer.html bundled with this binary)
// Anything else is denied, so this is not a "wildcard" CORS.
let cors = CorsLayer::new()
.allow_origin(AllowOrigin::predicate(|origin: &HeaderValue, _req| {
let s = match origin.to_str() {
Ok(v) => v,
Err(_) => return false,
};
s == "https://ruvnet.github.io"
|| s.starts_with("http://localhost")
|| s.starts_with("http://127.0.0.1")
|| s == "null" // file:// origins
}))
.allow_methods([Method::GET, Method::OPTIONS])
.allow_headers([axum::http::header::CONTENT_TYPE]);
let app = Router::new()
.route("/", get(index))
.route("/api/cloud", get(api_cloud))
.route("/api/splats", get(api_splats))
.route("/api/status", get(api_status))
.route("/health", get(api_health))
.layer(cors)
.with_state(state);
println!("╔══════════════════════════════════════════════╗");
@@ -2,45 +2,27 @@
<html>
<head>
<title>RuView — Camera + WiFi CSI Point Cloud</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate">
<meta name="ruview-viewer-version" content="0.2.0-face-mesh">
<!-- Inline amber-dot favicon avoids a stray /favicon.ico 404 in the console. -->
<link rel="icon" type="image/svg+xml" href="data:image/svg+xml;utf8,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><circle cx='16' cy='16' r='10' fill='%23e8a634'/></svg>">
<style>
body { margin: 0; background: #0a0a0a; color: #e8a634; font-family: monospace; }
canvas { display: block; }
#info { position: absolute; top: 10px; left: 10px; padding: 12px; background: rgba(0,0,0,0.85); border: 1px solid #e8a634; border-radius: 6px; min-width: 240px; font-size: 13px; line-height: 1.5; z-index: 10; }
#cam-cta { position: absolute; bottom: 16px; left: 50%; transform: translateX(-50%); padding: 10px 18px; background: #e8a634; color: #0a0a0a; border: none; border-radius: 4px; font-family: monospace; font-size: 14px; font-weight: bold; cursor: pointer; z-index: 10; }
#cam-cta:hover { background: #ffc04d; }
#cam-cta.hidden { display: none; }
#esp-cta { position: absolute; bottom: 16px; right: 16px; padding: 8px 14px; background: transparent; color: #e8a634; border: 1px solid #e8a634; border-radius: 4px; font-family: monospace; font-size: 12px; cursor: pointer; z-index: 10; }
#esp-cta:hover { background: rgba(232, 166, 52, 0.12); }
#esp-cta.connected { background: #4f4; color: #0a0a0a; border-color: #4f4; }
#info { position: absolute; top: 10px; left: 10px; padding: 12px; background: rgba(0,0,0,0.85); border: 1px solid #e8a634; border-radius: 6px; min-width: 240px; font-size: 13px; line-height: 1.5; }
.live { color: #4f4; } .demo { color: #f44; }
.face { color: #4cf; }
.section { margin-top: 6px; padding-top: 6px; border-top: 1px solid #333; }
.label { color: #888; }
</style>
<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/controls/OrbitControls.js"></script>
<!-- MediaPipe Face Mesh — runs in demo mode so each visitor sees their own face as a point cloud -->
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh@0.4/face_mesh.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/camera_utils@0.3/camera_utils.js"></script>
</head>
<body>
<div id="info">
<h3 style="margin:0 0 4px 0">RuView · Seldon Vault</h3>
<div style="font-size: 11px; color: #888; margin-bottom: 8px; max-width: 240px; line-height: 1.4; font-style: italic;">"Psychohistory deals with reactions of human conglomerates to fixed social and economic stimuli." — Hari Seldon</div>
<h3 style="margin:0 0 8px 0">RuView Point Cloud</h3>
<div id="stats">Loading...</div>
</div>
<button id="cam-cta">▶ Project Subject — render your face into the Vault</button>
<button id="esp-cta" title="Stream live CSI from a local ruview-pointcloud serve instance (e.g. http://127.0.0.1:9880)">📡 Connect ESP32…</button>
<script>
var scene = new THREE.Scene();
scene.background = new THREE.Color(0x0a0a0a);
var camera = new THREE.PerspectiveCamera(72, window.innerWidth/window.innerHeight, 0.1, 200);
camera.position.set(0, 0.2, -3.5);
var camera = new THREE.PerspectiveCamera(75, window.innerWidth/window.innerHeight, 0.1, 100);
camera.position.set(0, 2, -4);
camera.lookAt(0, 0, 2);
var renderer = new THREE.WebGLRenderer({ antialias: true });
@@ -122,436 +104,10 @@
scene.add(skeletonGroup);
}
// ----- Transport configuration -----
// ?backend=<url> → fetch splats from <url>/api/splats (CORS-permitting host)
// ?backend=auto → try /api/splats, fall back to synthetic demo on failure (default)
// ?backend=demo → always render synthetic demo (no network)
// ?live=1 → require live; show error instead of demo fallback
var urlParams = new URLSearchParams(window.location.search);
var backendArg = urlParams.get("backend") || "auto";
var requireLive = urlParams.get("live") === "1";
var transportMode = "demo"; // resolved at first fetch: "live" | "remote" | "demo"
var demoStartMs = Date.now();
var demoFrameNum = 0;
var latestFaceLandmarks = null; // populated by MediaPipe when camera enabled
var faceMeshState = "idle"; // "idle" | "starting" | "running" | "denied" | "unavailable"
// ----- MediaPipe Face Mesh (browser equivalent of camera-depth backprojection) -----
// Locally, ruview-pointcloud serve fuses real camera depth + WiFi CSI. In the
// browser we don't have depth from a webcam, but Face Mesh produces 468
// 3D landmarks (x,y in [0,1], z roughly in [-0.5,0.5]) at ~30 fps — enough to
// reproduce the "I can see the outline of my face in points" experience. The
// landmarks feed into the same splat render path as live /api/splats data.
async function startFaceMesh() {
if (faceMeshState !== "idle") return;
if (!window.FaceMesh || !window.Camera) {
faceMeshState = "unavailable";
return;
}
faceMeshState = "starting";
try {
var videoEl = document.createElement("video");
videoEl.style.display = "none";
videoEl.autoplay = true;
videoEl.playsInline = true;
videoEl.muted = true;
document.body.appendChild(videoEl);
var fm = new FaceMesh({
locateFile: function(file) {
return "https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh@0.4/" + file;
}
});
fm.setOptions({
maxNumFaces: 1,
refineLandmarks: true,
minDetectionConfidence: 0.5,
minTrackingConfidence: 0.5
});
fm.onResults(function(results) {
if (results.multiFaceLandmarks && results.multiFaceLandmarks[0]) {
latestFaceLandmarks = results.multiFaceLandmarks[0];
}
});
var mpCamera = new Camera(videoEl, {
onFrame: async function() { await fm.send({ image: videoEl }); },
width: 640,
height: 480
});
await mpCamera.start();
faceMeshState = "running";
var btn = document.getElementById("cam-cta");
if (btn) btn.classList.add("hidden");
} catch (err) {
faceMeshState = "denied";
console.warn("Face mesh unavailable:", err);
}
}
// ---- Foundation-inspired galactic context (Asimov / Trantor / Seldon) ----
// Shared between face-mesh and synthetic-fallback paths. The subject (face
// or procedural figure) is the foreground; this function paints the Seldon
// time-vault around it: holographic surveyor grid underfoot, slow galactic
// spiral receding into the distance, distant starfield, and a halo ring.
function pushFoundationContext(splats) {
var t = (Date.now() - demoStartMs) / 1000.0;
// 1. Holographic surveyor grid — amber lattice at y=+1.4 (renders below
// the subject because the renderer flips y to Three.js Y-up).
var gx, gz;
for (gx = -10; gx <= 10; gx++) {
for (gz = 0; gz <= 30; gz++) {
var alpha = 0.35 + 0.15 * Math.sin(t * 0.5 + gz * 0.2);
splats.push({
center: [gx * 0.5, 1.4, gz * 0.4],
color: [0.40 * alpha, 0.28 * alpha, 0.10 * alpha],
opacity: 1.0,
scale: [0.018, 0.018, 0.018]
});
}
}
for (gz = 0; gz <= 30; gz += 2) {
for (gx = -20; gx <= 20; gx++) {
splats.push({
center: [gx * 0.25, 1.4, gz * 0.4 + 0.1],
color: [0.30, 0.22, 0.08],
opacity: 1.0,
scale: [0.014, 0.014, 0.014]
});
}
}
// 2. Galactic spiral — Trantor recedes behind the subject. ~640 stars
// across two logarithmic arms, slowly rotating. Warmer at the core,
// cooler at the edges (Hertzsprung-Russell-ish).
var arm, k, theta_arm, r_arm, sx, sz, twist, arm_color;
for (arm = 0; arm < 2; arm++) {
for (k = 0; k < 320; k++) {
var prog = k / 320;
theta_arm = arm * Math.PI + prog * 6.0 + t * 0.05;
r_arm = 4.0 + prog * 14.0;
twist = Math.sin(prog * 8.0) * 0.4;
sx = Math.cos(theta_arm) * r_arm + twist;
sz = Math.sin(theta_arm) * r_arm + 12.0;
var coreFade = Math.max(0.15, 1.0 - prog);
arm_color = [
coreFade * 0.85 + 0.15 * (1 - prog),
coreFade * 0.70 + 0.20,
coreFade * 0.55 + 0.45 * prog
];
splats.push({
center: [sx, -2.5 + Math.sin(prog * 12) * 0.3, sz],
color: arm_color,
opacity: 1.0,
scale: [0.025, 0.025, 0.025]
});
}
}
// 3. Distant starfield — 800 deterministic stars on a spherical shell.
// Fixed LCG seed so visitors don't see noise flicker between frames.
var seed = 42;
function nextRand() {
seed = (seed * 1664525 + 1013904223) >>> 0;
return seed / 4294967296;
}
var s, r_s, phi, costheta, sinphi;
for (s = 0; s < 800; s++) {
phi = nextRand() * Math.PI * 2;
costheta = nextRand() * 2 - 1;
sinphi = Math.sqrt(1 - costheta * costheta);
r_s = 22 + nextRand() * 8;
var brightness = 0.4 + nextRand() * 0.6;
var hue = nextRand();
splats.push({
center: [
Math.cos(phi) * sinphi * r_s,
costheta * r_s * 0.5 - 1.0,
Math.sin(phi) * sinphi * r_s + 5.0
],
color: hue > 0.85
? [brightness, brightness * 0.85, brightness * 0.6]
: (hue > 0.3
? [brightness * 0.9, brightness * 0.95, brightness]
: [brightness * 0.5, brightness * 0.7, brightness]),
opacity: 1.0,
scale: [0.020, 0.020, 0.020]
});
}
// 4. Holographic projection halo around the subject — Seldon vault
// projections always had a faint encircling ring of particles.
var ring;
for (ring = 0; ring < 60; ring++) {
var rt = ring / 60 * Math.PI * 2 + t * 0.3;
splats.push({
center: [
Math.cos(rt) * 1.6,
Math.sin(rt) * 1.2 - 0.2,
2.0 + Math.sin(rt * 3 + t * 0.5) * 0.3
],
color: [0.95, 0.55, 0.15],
opacity: 1.0,
scale: [0.014, 0.014, 0.014]
});
}
}
// Map a single landmark to world coords. Coord conventions:
// x: 0.5 - lm.x → mirror so left-of-screen = your left side (selfie)
// y: lm.y - 0.5 → keep MediaPipe's y-DOWN convention; the renderer's
// existing -y flip in updateSplats does the single flip
// to Three.js Y-up. Pre-flipping here would double-flip
// and the face would render upside down.
// z: 2.0 + lm.z*8 → amplify lm.z (~[-0.1,+0.1]) so the nose/eye-socket
// depth is visible from an oblique camera angle.
function lmToCenter(lm) {
return [
(0.5 - lm.x) * 4.0,
(lm.y - 0.5) * 3.0,
2.0 + lm.z * 8.0
];
}
function pushFaceSplat(splats, center, alpha) {
splats.push({
center: center,
color: [0.95 * alpha, 0.65 * alpha, 0.20 * alpha],
opacity: 1.0,
scale: [0.006, 0.006, 0.006]
});
}
// FACEMESH_TESSELATION is a flat array [a0,b0, a1,b1, ...] of vertex indices
// forming edges of the triangulated face mesh. ~1300 edges × 2 = 2600 entries.
// We interpolate 6 splats per edge → ~8000 splats per face vs 478 vertices.
var FACE_EDGES = (typeof FACEMESH_TESSELATION !== "undefined") ? FACEMESH_TESSELATION : null;
// Push the user's face mesh point cloud into `splats` (no Foundation
// context — that is the demo path's responsibility). Used both as the
// demo subject AND as an overlay on top of live/remote backend data
// when the camera is enabled. Returns true if any splats were pushed.
function pushFaceSplats(splats) {
if (faceMeshState !== "running" || !latestFaceLandmarks) return false;
var lms = latestFaceLandmarks;
var i;
// 1. Original 478 vertices — bright anchor points for features.
for (i = 0; i < lms.length; i++) {
splats.push({
center: lmToCenter(lms[i]),
color: [1.0, 0.72, 0.25],
opacity: 1.0,
scale: [0.010, 0.010, 0.010]
});
}
// 2. Edge interpolation — 6 splats per FACEMESH_TESSELATION edge.
if (FACE_EDGES) {
var edgeCount = FACE_EDGES.length;
var SAMPLES = 6;
var e, a, b, t, f;
for (e = 0; e < edgeCount; e += 2) {
a = lms[FACE_EDGES[e]];
b = lms[FACE_EDGES[e + 1]];
if (!a || !b) continue;
var aPos = lmToCenter(a);
var bPos = lmToCenter(b);
var ax = aPos[0], ay = aPos[1], az = aPos[2];
var bx = bPos[0], by = bPos[1], bz = bPos[2];
for (t = 1; t <= SAMPLES; t++) {
f = t / (SAMPLES + 1);
pushFaceSplat(splats, [
ax * (1 - f) + bx * f,
ay * (1 - f) + by * f,
az * (1 - f) + bz * f
], 0.85);
}
}
}
return true;
}
function faceMeshFrame() {
if (faceMeshState !== "running" || !latestFaceLandmarks) return null;
var splats = [];
pushFaceSplats(splats);
pushFoundationContext(splats);
demoFrameNum += 1;
return {
splats: splats,
count: splats.length,
frame: demoFrameNum,
live: false,
source: "face-mesh",
pipeline: {
skeleton: null,
vitals: { breathing_rate: 14, motion_score: 0.15 }
}
};
}
function buildSplatsUrl() {
if (backendArg === "demo") return null;
if (backendArg === "auto") return "/api/splats";
// User-supplied URL — strip trailing slash and append /api/splats.
var base = backendArg.replace(/\/+$/, "");
return base + "/api/splats";
}
function syntheticFrame() {
// Used when camera permission is denied / unavailable. Renders a
// procedural standing figure inside the Seldon vault context.
// y-down convention: head at small/negative y, feet at large/positive y;
// the renderer flips y so head ends up at the top of the screen.
var t = (Date.now() - demoStartMs) / 1000.0;
var sway = Math.sin(t * 0.8) * 0.05;
var breath = Math.sin(t * 1.2) * 0.015;
var splats = [];
// Standing figure — 240 points in a vertical cylinder, denser than
// before to feel like a holographic projection.
var ring, k_ring, theta, r, py;
for (ring = 0; ring < 30; ring++) {
py = -1.0 + (ring / 30) * 2.2; // head (-1.0) → feet (+1.2) in y-down
r = 0.20 + breath * (py < 0 ? 1.5 : 0); // chest expands more on inhale
for (k_ring = 0; k_ring < 16; k_ring++) {
theta = (k_ring / 16) * Math.PI * 2;
splats.push({
center: [
sway + Math.cos(theta) * r,
py,
2.3 + Math.sin(theta) * r
],
color: [0.91, 0.65, 0.20],
opacity: 1.0,
scale: [0.018, 0.018, 0.018]
});
}
}
// 17 COCO keypoints in normalized [0,1] image coords (matches live shape)
var headY = 0.18;
var keypoints = [
[0.50 + sway * 0.05, headY, 0.95], // 0 nose
[0.52 + sway * 0.05, headY - 0.01, 0.92], // 1 leftEye
[0.48 + sway * 0.05, headY - 0.01, 0.92], // 2 rightEye
[0.54 + sway * 0.05, headY, 0.85], // 3 leftEar
[0.46 + sway * 0.05, headY, 0.85], // 4 rightEar
[0.60 + sway * 0.04, 0.32, 0.93], // 5 leftShoulder
[0.40 + sway * 0.04, 0.32, 0.93], // 6 rightShoulder
[0.65 + sway * 0.03, 0.46, 0.90], // 7 leftElbow
[0.35 + sway * 0.03, 0.46, 0.90], // 8 rightElbow
[0.68, 0.60 + Math.sin(t * 1.4) * 0.02, 0.86], // 9 leftWrist
[0.32, 0.60 - Math.sin(t * 1.4) * 0.02, 0.86], // 10 rightWrist
[0.57, 0.58, 0.94], // 11 leftHip
[0.43, 0.58, 0.94], // 12 rightHip
[0.58, 0.74, 0.90], // 13 leftKnee
[0.42, 0.74, 0.90], // 14 rightKnee
[0.59, 0.92, 0.88], // 15 leftAnkle
[0.41, 0.92, 0.88] // 16 rightAnkle
];
// Wrap the figure in the Seldon-vault context (grid, spiral, starfield, halo)
pushFoundationContext(splats);
demoFrameNum += 1;
return {
splats: splats,
count: splats.length,
frame: demoFrameNum,
live: false,
pipeline: {
skeleton: { keypoints: keypoints, confidence: 0.86 },
vitals: {
breathing_rate: 14 + Math.round(Math.sin(t * 0.05) * 2),
motion_score: 0.18 + Math.abs(sway) * 2
}
}
};
}
function pickDemoFrame() {
// Prefer real face-mesh data when the camera is running; else procedural.
return faceMeshFrame() || syntheticFrame();
}
// Once auto mode confirms there is no /api/splats backend on this origin,
// set this flag so we stop hammering the network with 404 fetches every
// tick. Console stays clean; demo renders locally.
var networkDisabled = false;
// Exponential backoff state for explicit ?backend=<url>. The user's
// local server may be down (ERR_CONNECTION_REFUSED) and we shouldn't
// hammer it 10 Hz indefinitely. After each failure we lengthen the
// delay; on success we snap back to the normal cadence.
var BASE_INTERVAL_MS = 250;
var MAX_INTERVAL_MS = 30000;
var currentIntervalMs = BASE_INTERVAL_MS;
var consecutiveFailures = 0;
var fetchTimer = null;
var lastBackendError = null;
function scheduleNextFetch(delayMs) {
if (fetchTimer) clearTimeout(fetchTimer);
fetchTimer = setTimeout(fetchCloud, delayMs);
}
async function fetchCloud() {
// Demo-only mode: never hit the network. Use the normal cadence.
if (backendArg === "demo" || networkDisabled) {
transportMode = "demo";
handleData(pickDemoFrame());
scheduleNextFetch(BASE_INTERVAL_MS);
return;
}
try {
var resp = await fetch(buildSplatsUrl(), { cache: "no-store" });
if (!resp.ok) throw new Error("HTTP " + resp.status);
var resp = await fetch("/api/splats");
var data = await resp.json();
transportMode = (backendArg === "auto") ? "live" : "remote";
consecutiveFailures = 0;
currentIntervalMs = BASE_INTERVAL_MS;
lastBackendError = null;
handleData(data);
scheduleNextFetch(BASE_INTERVAL_MS);
} catch (err) {
consecutiveFailures += 1;
lastBackendError = err && err.message ? err.message : String(err);
if (requireLive) {
document.getElementById("stats").innerHTML =
'<span class="demo">&#9679; OFFLINE</span><br>Live backend required (?live=1) but unreachable.<br><span class="label">' + lastBackendError + '</span>';
// Even strict-live: back off so we don't spam.
currentIntervalMs = Math.min(currentIntervalMs * 2, MAX_INTERVAL_MS);
scheduleNextFetch(currentIntervalMs);
return;
}
// Auto mode + first failure → assume static host (Pages), disable
// network entirely so the console stays clean.
if (backendArg === "auto") {
networkDisabled = true;
transportMode = "demo";
handleData(pickDemoFrame());
scheduleNextFetch(BASE_INTERVAL_MS);
return;
}
// Explicit backend (?backend=<url>) — keep trying with
// exponential backoff: 250 ms → 500 ms → 1 s → 2 s … up to 30 s.
// Render the demo while we wait so the scene stays alive, and
// surface the failure so the user knows the server is down.
currentIntervalMs = Math.min(Math.max(BASE_INTERVAL_MS * Math.pow(2, consecutiveFailures - 1), 1000), MAX_INTERVAL_MS);
transportMode = "demo";
var demoFrame = pickDemoFrame();
demoFrame._backendUnreachable = true;
demoFrame._backendUrl = backendArg;
demoFrame._backendError = lastBackendError;
demoFrame._retryInMs = currentIntervalMs;
handleData(demoFrame);
scheduleNextFetch(currentIntervalMs);
}
}
function handleData(data) {
try {
if (data.splats && data.frame !== lastFrame) {
// Compute CSI frame rate
var now = Date.now();
@@ -561,23 +117,7 @@
}
prevTimestamp = now;
lastFrame = data.frame;
// Overlay browser face mesh on top of backend splats when both
// are active — lets visitors see their own face *plus* the
// ESP32-driven point cloud in the same scene. Demo mode (where
// data.source === "face-mesh") already includes the face, so
// we skip this branch there to avoid double-counting.
var rendered = data.splats;
var faceOverlay = false;
if (data.source !== "face-mesh"
&& faceMeshState === "running"
&& latestFaceLandmarks) {
rendered = data.splats.slice();
pushFaceSplats(rendered);
faceOverlay = true;
}
data._faceOverlay = faceOverlay;
updateSplats(rendered);
updateSplats(data.splats);
// Draw skeleton if available
var pipe = data.pipeline;
@@ -587,40 +127,14 @@
clearSkeleton();
}
// Build info panel — badge reflects active transport
var mode;
if (transportMode === "live") {
mode = '<span class="live">&#9679; LIVE</span> Local Backend';
} else if (transportMode === "remote") {
mode = '<span class="live">&#9679; REMOTE</span> ' + backendArg;
} else if (data.source === "face-mesh") {
mode = '<span class="face">&#9679; DEMO</span> Your Face (MediaPipe)';
} else {
mode = '<span class="demo">&#9679; DEMO</span> Synthetic';
}
if (data._faceOverlay) {
mode += ' <span class="face">+ face overlay</span>';
}
var splatCount = rendered ? rendered.length : data.count;
var html = mode + "<br>"
+ "Splats: " + splatCount + "<br>"
// Build info panel
var mode = data.live
? '<span class="live">&#9679; LIVE</span>'
: '<span class="demo">&#9679; DEMO</span>';
var html = mode + " Camera + CSI<br>"
+ "Splats: " + data.count + "<br>"
+ "Frame: " + data.frame;
// Unreachable backend banner — explicit ?backend=<url> failed
// to connect. Show actionable guidance instead of leaving the
// user staring at a "demo" badge wondering why their ESP32
// feed isn't visible.
if (data._backendUnreachable) {
var nextSec = Math.round((data._retryInMs || 1000) / 1000);
html += '<div class="section">'
+ '<span class="demo">&#9679; ' + data._backendUrl + '</span> unreachable'
+ '<br><span class="label">' + (data._backendError || "connection failed") + '</span>'
+ '<br><span class="label">retry in ' + nextSec + 's</span>'
+ '<br><br><span class="label">start the server:</span>'
+ '<br><code style="color:#e8a634">cargo run -p wifi-densepose-pointcloud --release \\<br>&nbsp;&nbsp;-- serve --bind 127.0.0.1:9880</code>'
+ '</div>';
}
// CSI frame rate
html += '<div class="section">'
+ '<span class="label">CSI Rate:</span> '
@@ -673,69 +187,8 @@
}
} catch(e) {}
}
// Wire the camera CTA. The camera is now overlay-able on every
// transport mode: in demo it IS the subject; in live/remote it
// overlays the backend splats so the visitor sees their face
// alongside the ESP32-driven point cloud.
(function wireCamCta() {
var btn = document.getElementById("cam-cta");
if (!btn) return;
if (requireLive) {
// Strict-live mode shows the offline panel — no camera UI.
btn.classList.add("hidden");
return;
}
// In remote mode, label the button as an overlay action.
if (backendArg.startsWith("http")) {
btn.textContent = "▶ Add face overlay";
}
btn.addEventListener("click", function() {
btn.textContent = backendArg.startsWith("http")
? "Starting overlay…"
: "Initializing the Vault…";
startFaceMesh();
});
})();
// Wire the ESP32 backend CTA: prompts for a ruview-pointcloud serve URL,
// persists last-used value in localStorage, and reloads with the
// ?backend=<url> query so the existing remote-mode path takes over.
// Disconnect by clicking again when already connected.
(function wireEspCta() {
var btn = document.getElementById("esp-cta");
if (!btn) return;
var connected = backendArg.startsWith("http");
if (connected) {
btn.classList.add("connected");
btn.textContent = "📡 ESP32 connected · disconnect";
}
btn.addEventListener("click", function() {
if (connected) {
// Strip ?backend= from current URL and reload — return to demo.
var u = new URL(window.location.href);
u.searchParams.delete("backend");
window.location.href = u.toString();
return;
}
var stored;
try { stored = localStorage.getItem("ruview.backendUrl"); } catch (_) { stored = null; }
var def = stored || "http://127.0.0.1:9880";
var url = window.prompt(
"Enter the ruview-pointcloud serve URL (run `ruview-pointcloud serve` locally with your ESP32 streaming CSI to UDP port 3333):",
def
);
if (!url) return;
url = url.replace(/\/+$/, "");
try { localStorage.setItem("ruview.backendUrl", url); } catch (_) {}
var u2 = new URL(window.location.href);
u2.searchParams.set("backend", url);
window.location.href = u2.toString();
});
})();
// fetchCloud self-schedules via setTimeout — no setInterval to avoid
// overlapping calls on slow networks and to support exponential backoff.
fetchCloud();
setInterval(fetchCloud, 500);
function updateSplats(splats) {
if (pointsMesh) scene.remove(pointsMesh);