Compare commits

...

13 Commits

Author SHA1 Message Date
ruv 0223ef6d2e docs: add ADR-059 live ESP32 CSI pipeline + update README with demo links
- ADR-059: Documents end-to-end ESP32 → sensing server → browser pipeline
- README: Add dual-modal pose fusion demo link, update ADR count to 49
- References issue #245

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 17:40:16 -04:00
ruv 2f5e7ffb41 feat: live ESP32 CSI pipeline + auto-connect WebSocket
- Add auto-connect to local sensing server WebSocket (ws://localhost:8765)
- Demo shows "Live ESP32" when connected to real CSI data
- Add build_firmware.ps1 for native Windows ESP-IDF builds (no Docker)
- Add read_serial.ps1 for ESP32 serial monitor

Pipeline: ESP32 → UDP:5005 → sensing-server → WS:8765 → browser demo

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 17:37:27 -04:00
ruv 4ce8ffc465 fix: video fills available space + correct WASM path resolution
- Remove fixed aspect-ratio and max-height from video panel so it
  fills the available viewport space without scrolling
- Grid uses 1fr row for content area, overflow:hidden on main grid
- Fix WASM path: resolve relative to JS module file using import.meta.url
  instead of hardcoded ./pkg/ which resolved incorrectly on gh-pages
- Responsive: mobile still gets aspect-ratio constraint

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 16:16:07 -04:00
ruv 3be63a7589 fix: motion-responsive skeleton + through-wall CSI tracking
- Pose decoder now uses per-cell motion grid to track actual arm/head
  positions — raising arms moves the skeleton's arms, head follows
  lateral movement
- Motion grid (10x8 cells) tracks intensity per body zone: head,
  left/right arm upper/mid, legs
- Through-wall mode: when person exits frame, CSI maintains presence
  with slow decay (~10s) and skeleton drifts in exit direction
- CSI simulator persists sensing after video loss, ghost pose renders
  with decreasing confidence
- Reduced temporal smoothing (0.45) for faster response to movement

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 16:10:29 -04:00
ruv c4e640c812 feat: dual-modal WASM browser pose estimation demo (ADR-058)
Live webcam video + WiFi CSI fusion for real-time pose estimation.
Two parallel CNN pipelines (ruvector-cnn-wasm) with attention-weighted
fusion and dynamic confidence gating. Three modes: Dual, Video-only,
CSI-only. Includes pre-built WASM package (~52KB) for browser deployment.

- ADR-058: Dual-modal architecture design
- ui/pose-fusion.html: Main demo page with dark theme UI
- 7 JS modules: video-capture, csi-simulator, cnn-embedder, fusion-engine,
  pose-decoder, canvas-renderer, main orchestrator
- Pre-built ruvector-cnn-wasm WASM package for browser
- CSI heatmap, embedding space visualization, latency metrics
- WebSocket support for live ESP32 CSI data
- Navigation link added to main dashboard

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 14:26:39 -04:00
ruv 6e03a47867 docs: update user guide with v0.4.1 firmware release and CSI troubleshooting
- Add v0.4.1 to firmware release table as recommended stable release
- Update flash command with correct partition offsets (8MB, OTA)
- Add "CSI not enabled" troubleshooting entry
- Add warning about pre-v0.4.1 firmware CSI bug

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 13:49:20 -04:00
ruv 9d1140de2d docs: update README firmware release table with v0.4.1
Add v0.4.1-esp32 as the recommended stable release and update the
flash command to match the current partition layout.

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 13:49:20 -04:00
ruv 952f27a1ce fix(firmware): enable CSI in sdkconfig and add build guard (ADR-057)
The committed sdkconfig had CONFIG_ESP_WIFI_CSI_ENABLED disabled, causing
all builds to crash at runtime with "CSI not enabled in menuconfig".
Root cause: sdkconfig.defaults.template existed but ESP-IDF only reads
sdkconfig.defaults (no .template suffix).

Fixes:
- Add sdkconfig.defaults with CONFIG_ESP_WIFI_CSI_ENABLED=y
- Add #error compile guard in csi_collector.c to prevent recurrence
- Fix NVS encryption default (requires eFuse, breaks clean builds)

Verified: Docker build + flash to ESP32-S3 + CSI callbacks confirmed.

Closes #241
Relates to #223, #238, #234, #210, #190

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 13:49:20 -04:00
Reuven f7d043d727 docs: fix Docker commands to use CSI_SOURCE environment variable
The Docker image uses CSI_SOURCE env var to select the data source,
not command-line arguments appended after the image name.

Fixed:
- ESP32 mode examples now use -e CSI_SOURCE=esp32
- Training mode example now uses --entrypoint override
- Added CSI_SOURCE value table in Docker section

Fixes #226

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-10 12:16:06 -04:00
Reuven ff91d4e8cf fix(desktop): remove bundled sensing-server resource for CI build
The sensing-server binary was referenced in tauri.conf.json but doesn't
exist in CI environment. Removed the resources section to fix the build.

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-10 10:56:31 -04:00
Reuven fc92436f52 chore: add build artifacts and session state
- NVS config binaries for ESP32 WiFi provisioning
- macOS Tauri schema
- package-lock.json update
- Claude Flow session state

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-10 10:36:16 -04:00
Reuven 285bb0ad37 feat(desktop): v0.4.4 - WiFi configuration via serial port
## New Features
- WiFi Configuration Modal: Configure ESP32 WiFi credentials directly from the desktop app
- Serial port WiFi commands: Sends wifi_config/wifi/set ssid commands via serial
- Improved feedback UI with status indicators (Success/Commands Sent/Error)

## API Improvements
- New Tauri command: configure_esp32_wifi(port, ssid, password)
- 21 new integration tests covering all API functionality
- ESP32 VID/PID detection for CP210x, CH340, FTDI, and native USB

## UI Enhancements
- WiFi button in Serial Ports table for ESP32-compatible devices
- Modal with SSID/password inputs and clear status feedback
- "Done" button after configuration with "Try Again" option

## Testing
- 18 unit tests + 21 integration tests = 39 total tests passing
- Tests cover: discovery, settings, server, flash, OTA, provision, WASM, state, domain models

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-10 10:35:30 -04:00
Reuven b5ec4ef043 chore: update Cargo.lock
Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-10 10:02:02 -04:00
53 changed files with 7953 additions and 68 deletions
+13 -13
View File
@@ -1,6 +1,6 @@
{
"running": true,
"startedAt": "2026-02-28T15:54:19.353Z",
"startedAt": "2026-03-09T15:26:00.921Z",
"workers": {
"map": {
"runCount": 49,
@@ -8,16 +8,16 @@
"failureCount": 0,
"averageDurationMs": 1.2857142857142858,
"lastRun": "2026-02-28T16:13:19.194Z",
"nextRun": "2026-02-28T16:28:19.195Z",
"nextRun": "2026-03-09T15:56:00.928Z",
"isRunning": false
},
"audit": {
"runCount": 44,
"runCount": 45,
"successCount": 0,
"failureCount": 44,
"failureCount": 45,
"averageDurationMs": 0,
"lastRun": "2026-02-28T16:20:19.184Z",
"nextRun": "2026-02-28T16:30:19.185Z",
"lastRun": "2026-03-09T15:43:00.933Z",
"nextRun": "2026-03-09T15:38:00.914Z",
"isRunning": false
},
"optimize": {
@@ -26,7 +26,7 @@
"failureCount": 34,
"averageDurationMs": 0,
"lastRun": "2026-02-28T16:23:19.387Z",
"nextRun": "2026-02-28T16:18:19.361Z",
"nextRun": "2026-03-09T15:45:00.915Z",
"isRunning": false
},
"consolidate": {
@@ -35,7 +35,7 @@
"failureCount": 0,
"averageDurationMs": 0.6521739130434783,
"lastRun": "2026-02-28T16:05:19.091Z",
"nextRun": "2026-02-28T16:35:19.054Z",
"nextRun": "2026-03-09T16:02:00.918Z",
"isRunning": false
},
"testgaps": {
@@ -44,8 +44,8 @@
"failureCount": 27,
"averageDurationMs": 0,
"lastRun": "2026-02-28T16:08:19.369Z",
"nextRun": "2026-02-28T16:22:19.355Z",
"isRunning": true
"nextRun": "2026-03-09T15:54:00.920Z",
"isRunning": false
},
"predict": {
"runCount": 0,
@@ -64,8 +64,8 @@
},
"config": {
"autoStart": false,
"logDir": "/home/user/wifi-densepose/.claude-flow/logs",
"stateFile": "/home/user/wifi-densepose/.claude-flow/daemon-state.json",
"logDir": "/Users/cohen/GitHub/ruvnet/RuView/.claude-flow/logs",
"stateFile": "/Users/cohen/GitHub/ruvnet/RuView/.claude-flow/daemon-state.json",
"maxConcurrent": 2,
"workerTimeoutMs": 300000,
"resourceThresholds": {
@@ -131,5 +131,5 @@
}
]
},
"savedAt": "2026-02-28T16:23:19.387Z"
"savedAt": "2026-03-09T15:43:00.933Z"
}
+1 -1
View File
@@ -1 +1 @@
54612
31273
+13 -13
View File
@@ -6,7 +6,7 @@
"hooks": [
{
"type": "command",
"command": "node .claude/helpers/hook-handler.cjs pre-bash",
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/hook-handler.cjs\" pre-bash",
"timeout": 5000
}
]
@@ -18,7 +18,7 @@
"hooks": [
{
"type": "command",
"command": "node .claude/helpers/hook-handler.cjs post-edit",
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/hook-handler.cjs\" post-edit",
"timeout": 10000
}
]
@@ -29,7 +29,7 @@
"hooks": [
{
"type": "command",
"command": "node .claude/helpers/hook-handler.cjs route",
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/hook-handler.cjs\" route",
"timeout": 10000
}
]
@@ -40,12 +40,12 @@
"hooks": [
{
"type": "command",
"command": "node .claude/helpers/hook-handler.cjs session-restore",
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/hook-handler.cjs\" session-restore",
"timeout": 15000
},
{
"type": "command",
"command": "node .claude/helpers/auto-memory-hook.mjs import",
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/auto-memory-hook.mjs\" import",
"timeout": 8000
}
]
@@ -56,7 +56,7 @@
"hooks": [
{
"type": "command",
"command": "node .claude/helpers/hook-handler.cjs session-end",
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/hook-handler.cjs\" session-end",
"timeout": 10000
}
]
@@ -67,7 +67,7 @@
"hooks": [
{
"type": "command",
"command": "node .claude/helpers/auto-memory-hook.mjs sync",
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/auto-memory-hook.mjs\" sync",
"timeout": 10000
}
]
@@ -79,11 +79,11 @@
"hooks": [
{
"type": "command",
"command": "node .claude/helpers/hook-handler.cjs compact-manual"
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/hook-handler.cjs\" compact-manual"
},
{
"type": "command",
"command": "node .claude/helpers/hook-handler.cjs session-end",
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/hook-handler.cjs\" session-end",
"timeout": 5000
}
]
@@ -93,11 +93,11 @@
"hooks": [
{
"type": "command",
"command": "node .claude/helpers/hook-handler.cjs compact-auto"
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/hook-handler.cjs\" compact-auto"
},
{
"type": "command",
"command": "node .claude/helpers/hook-handler.cjs session-end",
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/hook-handler.cjs\" session-end",
"timeout": 6000
}
]
@@ -108,7 +108,7 @@
"hooks": [
{
"type": "command",
"command": "node .claude/helpers/hook-handler.cjs status",
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/hook-handler.cjs\" status",
"timeout": 3000
}
]
@@ -117,7 +117,7 @@
},
"statusLine": {
"type": "command",
"command": "node .claude/helpers/statusline.cjs"
"command": "node \"$CLAUDE_PROJECT_DIR/.claude/helpers/statusline.cjs\""
},
"permissions": {
"allow": [
+10 -4
View File
@@ -75,7 +75,7 @@ docker run -p 3000:3000 ruvnet/wifi-densepose:latest
|----------|-------------|
| [User Guide](docs/user-guide.md) | Step-by-step guide: installation, first run, API usage, hardware setup, training |
| [Build Guide](docs/build-guide.md) | Building from source (Rust and Python) |
| [Architecture Decisions](docs/adr/README.md) | 48 ADRs — why each technical choice was made, organized by domain (hardware, signal processing, ML, platform, infrastructure) |
| [Architecture Decisions](docs/adr/README.md) | 49 ADRs — why each technical choice was made, organized by domain (hardware, signal processing, ML, platform, infrastructure) |
| [Domain Models](docs/ddd/README.md) | 7 DDD models (RuvSense, Signal Processing, Training Pipeline, Hardware Platform, Sensing Server, WiFi-Mat, CHCI) — bounded contexts, aggregates, domain events, and ubiquitous language |
| [Desktop App](rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop/README.md) | **WIP** — Tauri v2 desktop app for node management, OTA updates, WASM deployment, and mesh visualization |
@@ -89,8 +89,12 @@ docker run -p 3000:3000 ruvnet/wifi-densepose:latest
<em>Real-time pose skeleton from WiFi CSI signals — no cameras, no wearables</em>
<br>
<a href="https://ruvnet.github.io/RuView/"><strong>▶ Live Observatory Demo</strong></a>
&nbsp;|&nbsp;
<a href="https://ruvnet.github.io/RuView/pose-fusion.html"><strong>▶ Dual-Modal Pose Fusion Demo</strong></a>
> The [server](#-quick-start) is optional for visualization and aggregation — the ESP32 [runs independently](#esp32-s3-hardware-pipeline) for presence detection, vital signs, and fall alerts.
>
> **Live ESP32 pipeline**: Connect an ESP32-S3 node → run the [sensing server](#sensing-server) → open the [pose fusion demo](https://ruvnet.github.io/RuView/pose-fusion.html) for real-time dual-modal pose estimation (webcam + WiFi CSI). See [ADR-059](docs/adr/ADR-059-live-esp32-csi-pipeline.md).
## 🚀 Key Features
@@ -1043,14 +1047,16 @@ Download a pre-built binary — no build toolchain needed:
| Release | What's included | Tag |
|---------|-----------------|-----|
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Stable — raw CSI streaming, multi-node TDM, channel hopping | `v0.2.0-esp32` |
| [v0.4.1](https://github.com/ruvnet/RuView/releases/tag/v0.4.1-esp32) | **Stable** — CSI build fix, compile guard, AMOLED display, edge intelligence ([ADR-057](docs/adr/ADR-057-firmware-csi-build-guard.md)) | `v0.4.1-esp32` |
| [v0.3.0-alpha](https://github.com/ruvnet/RuView/releases/tag/v0.3.0-alpha-esp32) | Alpha — adds on-device edge intelligence and WASM modules ([ADR-039](docs/adr/ADR-039-esp32-edge-intelligence.md), [ADR-040](docs/adr/ADR-040-wasm-programmable-sensing.md)) | `v0.3.0-alpha-esp32` |
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Raw CSI streaming, multi-node TDM, channel hopping | `v0.2.0-esp32` |
```bash
# 1. Flash the firmware to your ESP32-S3
python -m esptool --chip esp32s3 --port COM7 --baud 460800 \
write_flash --flash_mode dio --flash_size 8MB \
0x0 bootloader.bin 0x8000 partition-table.bin 0x10000 esp32-csi-node.bin
write_flash --flash-mode dio --flash-size 8MB --flash-freq 80m \
0x0 bootloader.bin 0x8000 partition-table.bin \
0xf000 ota_data_initial.bin 0x20000 esp32-csi-node.bin
# 2. Set WiFi credentials and server address (stored in flash, survives reboots)
python firmware/esp32-csi-node/provision.py --port COM7 \
@@ -0,0 +1,82 @@
# ADR-057: Firmware CSI Build Guard and sdkconfig.defaults
| Field | Value |
|-------------|---------------------------------------------|
| **Status** | Accepted |
| **Date** | 2026-03-12 |
| **Authors** | ruv |
| **Issues** | #223, #238, #234, #210, #190 |
## Context
Multiple GitHub issues (#223, #238, #234, #210, #190) report firmware problems
that fall into two categories:
1. **CSI not enabled at runtime** — The committed `sdkconfig` had
`# CONFIG_ESP_WIFI_CSI_ENABLED is not set` (line 1135), meaning users who
built from source or used pre-built binaries got the runtime error:
`E (6700) wifi:CSI not enabled in menuconfig!`
Root cause: `sdkconfig.defaults.template` existed with the correct setting
(`CONFIG_ESP_WIFI_CSI_ENABLED=y`) but ESP-IDF only reads
`sdkconfig.defaults` — not `.template` suffixed files. No `sdkconfig.defaults`
file was committed.
2. **Unsupported ESP32 variants** — Users attempting to use original ESP32
(D0WD) and ESP32-C3 boards. The firmware targets ESP32-S3 only
(`CONFIG_IDF_TARGET="esp32s3"`, Xtensa architecture) and this was not
surfaced clearly enough in documentation or build errors.
## Decision
### Fix 1: Commit `sdkconfig.defaults` (not just template)
Copy `sdkconfig.defaults.template``sdkconfig.defaults` so that ESP-IDF
applies the correct defaults (including `CONFIG_ESP_WIFI_CSI_ENABLED=y`)
automatically when `sdkconfig` is regenerated.
### Fix 2: `#error` compile-time guard in `csi_collector.c`
Add a preprocessor guard:
```c
#ifndef CONFIG_ESP_WIFI_CSI_ENABLED
#error "CONFIG_ESP_WIFI_CSI_ENABLED must be set in sdkconfig."
#endif
```
This turns a confusing runtime crash into a clear compile-time error with
instructions on how to fix it.
### Fix 3: Fix committed `sdkconfig`
Change line 1135 from `# CONFIG_ESP_WIFI_CSI_ENABLED is not set` to
`CONFIG_ESP_WIFI_CSI_ENABLED=y`.
## Consequences
- **Positive**: New builds will always have CSI enabled. Users building from
source will get a clear compile error if CSI is somehow disabled.
- **Positive**: Pre-built release binaries will include CSI support.
- **Neutral**: Original ESP32 and ESP32-C3 remain unsupported. This is by
design — only ESP32-S3 has the CSI API surface we depend on. Future ADRs
may address multi-target support if demand warrants it.
- **Negative**: None identified.
## Hardware Support Matrix
| Variant | CSI Support | Firmware Target | Status |
|--------------|-------------|-----------------|---------------|
| ESP32-S3 | Yes | Yes | Supported |
| ESP32 (orig) | Partial | No | Unsupported |
| ESP32-C3 | Yes (IDF 5.1+) | No | Unsupported |
| ESP32-C6 | Yes | No | Unsupported |
## Notes
- ESP32-C3 and C6 use RISC-V architecture; a separate build target
(`idf.py set-target esp32c3`) would be needed.
- Original ESP32 has limited CSI (no STBC HT-LTF2, fewer subcarriers).
- Users on unsupported hardware can still write custom firmware using the
ADR-018 binary frame format (magic `0xC5110001`) for interop with the
Rust aggregator.
@@ -0,0 +1,392 @@
# ADR-058: Dual-Modal WASM Browser Pose Estimation — Live Video + WiFi CSI Fusion
- **Status**: Proposed
- **Date**: 2026-03-12
- **Deciders**: ruv
- **Tags**: wasm, browser, cnn, pose-estimation, ruvector, video, multimodal, fusion
## Context
WiFi-DensePose estimates human poses from WiFi CSI (Channel State Information).
The `ruvector-cnn` crate provides a pure Rust CNN (MobileNet-V3) with WASM bindings.
Both modalities exist independently — what's missing is **fusing live webcam video
with WiFi CSI** in a single browser demo to achieve robust pose estimation that
works even when one modality degrades (occlusion, signal noise, poor lighting).
Existing assets:
1. **`wifi-densepose-wasm`** — CSI signal processing compiled to WASM
2. **`wifi-densepose-sensing-server`** — Axum server streaming live CSI via WebSocket
3. **`ruvector-cnn`** — Pure Rust CNN with MobileNet-V3 backbones, SIMD, contrastive learning
4. **`ruvector-cnn-wasm`** — wasm-bindgen bindings: `WasmCnnEmbedder`, `SimdOps`, `LayerOps`, contrastive losses
5. **`vendor/ruvector/examples/wasm-vanilla/`** — Reference vanilla JS WASM example
Research shows multi-modal fusion (camera + WiFi) significantly outperforms either alone:
- Camera fails under occlusion, poor lighting, privacy constraints
- WiFi CSI fails with signal noise, multipath, low spatial resolution
- Fusion compensates: WiFi provides through-wall coverage, camera provides fine-grained detail
## Decision
Build a **dual-modal browser demo** at `examples/wasm-browser-pose/` that:
1. Captures **live webcam video** via `getUserMedia` API
2. Receives **live WiFi CSI** via WebSocket from the sensing server
3. Processes **both streams** through separate CNN pipelines in `ruvector-cnn-wasm`
4. **Fuses embeddings** with learned attention weights for combined pose estimation
5. Renders **video overlay** with skeleton + WiFi confidence heatmap on Canvas
6. Runs entirely in the browser — all inference client-side via WASM
### Architecture
```
┌──────────────────────────────────────────────────────────────────┐
│ Browser │
│ │
│ ┌────────────┐ ┌────────────────┐ ┌───────────────────┐ │
│ │ getUserMedia│───▶│ Video Frame │───▶│ CNN WASM │ │
│ │ (Webcam) │ │ Capture │ │ (Visual Embedder) │ │
│ └────────────┘ │ 224×224 RGB │ │ → 512-dim │ │
│ └────────────────┘ └────────┬──────────┘ │
│ │ │
│ visual_embedding │
│ │ │
│ ┌──────▼──────┐ │
│ ┌────────────┐ ┌────────────────┐ │ │ │
│ │ WebSocket │───▶│ CSI WASM │ │ Attention │ │
│ │ Client │ │ (densepose- │ │ Fusion │ │
│ │ │ │ wasm) │ │ Module │ │
│ └────────────┘ └───────┬────────┘ │ │ │
│ │ └──────┬──────┘ │
│ ┌───────▼────────┐ │ │
│ │ CNN WASM │ fused_embedding │
│ │ (CSI Embedder) │ │ │
│ │ → 512-dim │ ┌──────▼──────┐ │
│ └───────┬────────┘ │ Pose │ │
│ │ │ Decoder │ │
│ csi_embedding │ → 17 kpts │ │
│ │ └──────┬──────┘ │
│ └──────────────────────┘ │
│ │ │
│ ┌──────────────┐ ┌─────▼──────┐ │
│ │ Video Canvas │◀────────│ Overlay │ │
│ │ + Skeleton │ │ Renderer │ │
│ │ + Heatmap │ └────────────┘ │
│ └──────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────┘
▲ ▲
│ getUserMedia │ WebSocket
│ (camera) │ (ws://host:3030/ws/csi)
│ │
┌────┴────┐ ┌───────┴─────────┐
│ Webcam │ │ Sensing Server │
└─────────┘ └─────────────────┘
```
### Dual Pipeline Design
Two parallel CNN pipelines run on each frame tick (~30 FPS):
| Pipeline | Input | Preprocessing | CNN Config | Output |
|----------|-------|---------------|------------|--------|
| **Visual** | Webcam frame (640×480) | Resize to 224×224 RGB, ImageNet normalize | MobileNet-V3 Small, 512-dim | Visual embedding |
| **CSI** | CSI frame (ADR-018 binary) | Amplitude/phase/delta → 224×224 pseudo-RGB | MobileNet-V3 Small, 512-dim | CSI embedding |
Both use the same `WasmCnnEmbedder` but with separate instances and weight sets.
### Fusion Strategy
**Learned attention-weighted fusion** combines the two 512-dim embeddings:
```javascript
// Attention fusion: learn which modality to trust per-dimension
// α ∈ [0,1]^512 — attention weights (shipped as JSON, trained offline)
// visual_emb, csi_emb ∈ R^512
function fuseEmbeddings(visual_emb, csi_emb, attention_weights) {
const fused = new Float32Array(512);
for (let i = 0; i < 512; i++) {
const α = attention_weights[i];
fused[i] = α * visual_emb[i] + (1 - α) * csi_emb[i];
}
return fused;
}
```
**Dynamic confidence gating** adjusts fusion based on signal quality:
| Condition | Behavior |
|-----------|----------|
| Good video + good CSI | Balanced fusion (α ≈ 0.5) |
| Poor lighting / occlusion | CSI-dominant (α → 0, WiFi takes over) |
| CSI noise / no ESP32 | Video-dominant (α → 1, camera only) |
| Video-only mode (no WiFi) | α = 1.0, pure visual CNN pose estimation |
| CSI-only mode (no camera) | α = 0.0, pure WiFi pose estimation |
Quality detection:
- **Video quality**: Frame brightness variance (dark = low quality), motion blur score
- **CSI quality**: Signal-to-noise ratio from `wifi-densepose-wasm`, coherence gate output
### CSI-to-Image Encoding
CSI data encoded as 3-channel pseudo-image for the CSI CNN pipeline:
| Channel | Data | Normalization |
|---------|------|---------------|
| R | CSI amplitude (subcarrier × time window) | Min-max to [0, 255] |
| G | CSI phase (unwrapped, subcarrier × time window) | Min-max to [0, 255] |
| B | Temporal difference (frame-to-frame Δ amplitude) | Abs, min-max to [0, 255] |
### Video Processing
Webcam frames processed through standard ImageNet pipeline:
```javascript
// Capture frame from video element
const frame = captureVideoFrame(videoElement, 224, 224); // Returns Uint8Array RGB
// ImageNet normalization happens inside WasmCnnEmbedder.extract()
const visual_embedding = visual_embedder.extract(frame, 224, 224);
```
### Pose Keypoint Mapping
17 COCO-format keypoints decoded from the fused 512-dim embedding:
```
0: nose 1: left_eye 2: right_eye
3: left_ear 4: right_ear 5: left_shoulder
6: right_shoulder 7: left_elbow 8: right_elbow
9: left_wrist 10: right_wrist 11: left_hip
12: right_hip 13: left_knee 14: right_knee
15: left_ankle 16: right_ankle
```
Each keypoint decoded as (x, y, confidence) = 51 values from the 512-dim embedding
via a learned linear projection.
### Operating Modes
The demo supports three modes, selectable in the UI:
| Mode | Video | CSI | Fusion | Use Case |
|------|-------|-----|--------|----------|
| **Dual (default)** | ✅ | ✅ | Attention-weighted | Best accuracy, full demo |
| **Video Only** | ✅ | ❌ | α = 1.0 | No ESP32 available, quick demo |
| **CSI Only** | ❌ | ✅ | α = 0.0 | Privacy mode, through-wall sensing |
**Video Only mode works without any hardware** — just a webcam — making the demo
instantly accessible for anyone wanting to try it.
### File Layout
```
examples/wasm-browser-pose/
├── index.html # Single-page app (vanilla JS, no bundler)
├── js/
│ ├── app.js # Main entry, mode selection, orchestration
│ ├── video-capture.js # getUserMedia, frame extraction, quality detection
│ ├── csi-processor.js # WebSocket CSI client, frame parsing, pseudo-image encoding
│ ├── fusion.js # Attention-weighted embedding fusion, confidence gating
│ ├── pose-decoder.js # Fused embedding → 17 keypoints
│ └── canvas-renderer.js # Video overlay, skeleton, CSI heatmap, confidence bars
├── data/
│ ├── visual-weights.json # Visual CNN → embedding projection (placeholder until trained)
│ ├── csi-weights.json # CSI CNN → embedding projection (placeholder until trained)
│ ├── fusion-weights.json # Attention fusion α weights (512 values)
│ └── pose-weights.json # Fused embedding → keypoint projection
├── css/
│ └── style.css # Dark theme UI styling
├── pkg/ # Built WASM packages (gitignored, built by script)
│ ├── wifi_densepose_wasm/
│ └── ruvector_cnn_wasm/
├── build.sh # wasm-pack build script for both packages
└── README.md # Setup and usage instructions
```
### Build Pipeline
```bash
#!/bin/bash
# build.sh — builds both WASM packages into pkg/
set -e
# Build wifi-densepose-wasm (CSI processing)
wasm-pack build ../../rust-port/wifi-densepose-rs/crates/wifi-densepose-wasm \
--target web --out-dir "$(pwd)/pkg/wifi_densepose_wasm" --no-typescript
# Build ruvector-cnn-wasm (CNN inference for both video and CSI)
wasm-pack build ../../vendor/ruvector/crates/ruvector-cnn-wasm \
--target web --out-dir "$(pwd)/pkg/ruvector_cnn_wasm" --no-typescript
echo "Build complete. Serve with: python3 -m http.server 8080"
```
### UI Layout
```
┌─────────────────────────────────────────────────────────┐
│ WiFi-DensePose — Live Dual-Modal Pose Estimation │
│ [Dual Mode ▼] [⚙ Settings] FPS: 28 ◉ Live │
├───────────────────────────┬─────────────────────────────┤
│ │ │
│ ┌───────────────────┐ │ ┌───────────────────┐ │
│ │ │ │ │ │ │
│ │ Video + Skeleton │ │ │ CSI Heatmap │ │
│ │ Overlay │ │ │ (amplitude × │ │
│ │ (main canvas) │ │ │ subcarrier) │ │
│ │ │ │ │ │ │
│ └───────────────────┘ │ └───────────────────┘ │
│ │ │
├───────────────────────────┴─────────────────────────────┤
│ Fusion Confidence: ████████░░ 78% │
│ Video: ██████████ 95% │ CSI: ██████░░░░ 61% │
├─────────────────────────────────────────────────────────┤
│ ┌─────────────────────────────────────────────────┐ │
│ │ Embedding Space (2D projection) │ │
│ │ · · · │ │
│ │ · · · · · · (color = pose cluster) │ │
│ │ · · · · │ │
│ └─────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────┤
│ Latency: Video 12ms │ CSI 8ms │ Fusion 1ms │ Total 21ms│
│ [▶ Record] [📷 Snapshot] [Confidence: ████ 0.6] │
└─────────────────────────────────────────────────────────┘
```
### WASM Module Structure
| Package | Source Crate | Provides | Size (est.) |
|---------|-------------|----------|-------------|
| `wifi_densepose_wasm` | `wifi-densepose-wasm` | CSI frame parsing, signal processing, feature extraction | ~200KB |
| `ruvector_cnn_wasm` | `ruvector-cnn-wasm` | `WasmCnnEmbedder` (×2 instances), `SimdOps`, `LayerOps`, contrastive losses | ~150KB |
Two `WasmCnnEmbedder` instances are created — one for video frames, one for CSI pseudo-images.
They share the same WASM module but have independent state.
### Browser API Requirements
| API | Purpose | Required | Fallback |
|-----|---------|----------|----------|
| `getUserMedia` | Webcam capture | For video mode | CSI-only mode |
| WebAssembly | CNN inference | Yes | None (hard requirement) |
| WASM SIMD128 | Accelerated inference | No | Scalar fallback (~2× slower) |
| WebSocket | CSI data stream | For CSI mode | Video-only mode |
| Canvas 2D | Rendering | Yes | None |
| `requestAnimationFrame` | Render loop | Yes | `setTimeout` fallback |
| ES Modules | Code organization | Yes | None |
Target: Chrome 89+, Firefox 89+, Safari 15+, Edge 89+
### Performance Budget
| Stage | Target Latency | Notes |
|-------|---------------|-------|
| Video frame capture + resize | <3ms | `drawImage` to offscreen canvas |
| Video CNN embedding | <15ms | 224×224 RGB → 512-dim |
| CSI receive + parse | <2ms | Binary WebSocket message |
| CSI pseudo-image encoding | <3ms | Amplitude/phase/delta channels |
| CSI CNN embedding | <15ms | 224×224 pseudo-RGB → 512-dim |
| Attention fusion | <1ms | Element-wise weighted sum |
| Pose decoding | <1ms | Linear projection |
| Canvas overlay render | <3ms | Video + skeleton + heatmap |
| **Total (dual mode)** | **<33ms** | **30 FPS capable** |
| **Total (video only)** | **<22ms** | **45 FPS capable** |
Note: Video and CSI CNN pipelines can run in parallel using Web Workers,
reducing dual-mode latency to ~max(15, 15) + 5 = ~20ms (50 FPS).
### Contrastive Learning Integration
The demo optionally shows real-time contrastive learning in the browser:
- **InfoNCE loss** (`WasmInfoNCELoss`): Compare video vs CSI embeddings for the same pose — trains cross-modal alignment
- **Triplet loss** (`WasmTripletLoss`): Push apart different poses, pull together same pose across modalities
- **SimdOps**: Accelerated dot products for real-time similarity computation
- **Embedding space panel**: Live 2D projection shows video and CSI embeddings converging when viewing the same person
### Relationship to Existing Crates
| Existing Crate | Role in This Demo |
|---------------|-------------------|
| `ruvector-cnn-wasm` | CNN inference for **both** video frames and CSI pseudo-images |
| `wifi-densepose-wasm` | CSI frame parsing and signal processing |
| `wifi-densepose-sensing-server` | WebSocket CSI data source |
| `wifi-densepose-core` | ADR-018 frame format definitions |
| `ruvector-cnn` | Underlying MobileNet-V3, layers, contrastive learning |
No new Rust crates are needed. The example is pure HTML/JS consuming existing WASM packages.
## Consequences
### Positive
- **Instant demo**: Video-only mode works with just a webcam — no ESP32 needed
- **Multi-modal showcase**: Demonstrates camera + WiFi fusion, the core innovation of the project
- **Graceful degradation**: Works with video-only, CSI-only, or both
- **Through-wall capability**: CSI mode shows pose estimation where cameras cannot reach
- **Zero-install**: Anyone with a browser can try it
- **Training data collection**: Can record paired (video, CSI) data for offline model training
- **Reusable**: JS modules embed directly in the Tauri desktop app's webview
### Negative
- **Model weights**: Requires offline-trained weights for visual CNN, CSI CNN, fusion, and pose decoder (~200KB total JSON)
- **WASM size**: Two WASM modules total ~350KB (acceptable)
- **No GPU**: CPU-only WASM inference; adequate at 224×224 but limits resolution scaling
- **Camera privacy**: Video mode requires camera permission (mitigated: CSI-only mode available)
- **Two CNN instances**: Memory footprint doubles vs single-modal (~10MB total, acceptable for desktop browsers)
### Risks
- **Cross-modal alignment**: Video and CSI embeddings must be trained jointly for fusion to work;
without proper training, fusion may be worse than either modality alone
- **Latency on mobile**: Dual CNN on mobile browsers may exceed 33ms; implement automatic quality reduction
- **WebSocket drops**: Network jitter → CSI frame gaps; buffer last 3 frames, interpolate missing data
## Implementation Plan
1. **Phase 1 — Scaffold**: File layout, build.sh, index.html shell, mode selector UI
2. **Phase 2 — Video pipeline**: getUserMedia → frame capture → CNN embedding → basic pose display
3. **Phase 3 — CSI pipeline**: WebSocket client → CSI parsing → pseudo-image → CNN embedding
4. **Phase 4 — Fusion**: Attention-weighted combination, confidence gating, mode switching
5. **Phase 5 — Pose decoder**: Linear projection with placeholder weights → 17 keypoints
6. **Phase 6 — Overlay renderer**: Video canvas with skeleton overlay, CSI heatmap panel
7. **Phase 7 — Training**: Use `wifi-densepose-train` to generate real weights for both CNNs + fusion + decoder
8. **Phase 8 — Contrastive demo**: Embedding space visualization, cross-modal similarity display
9. **Phase 9 — Web Workers**: Move CNN inference to workers for parallel video + CSI processing
10. **Phase 10 — Polish**: Recording, snapshots, adaptive quality, mobile optimization
## Alternatives Considered
### 1. CSI-Only (No Video)
Rejected: Misses the opportunity to show multi-modal fusion and makes the demo less
accessible (requires ESP32 hardware). Video-only mode as a fallback is strictly better.
### 2. Server-Side Video Inference
Rejected: Adds latency, requires webcam stream upload (privacy concern), and defeats
the WASM-first architecture. All inference must be client-side.
### 3. TensorFlow.js for Video, ruvector-cnn-wasm for CSI
Rejected: Would require two different ML frameworks. Using `ruvector-cnn-wasm` for both
keeps a single WASM module, unified embedding space, and simpler fusion.
### 4. Pre-recorded Video Demo
Rejected: Live webcam input is far more compelling for demonstrations.
Pre-recorded mode can be added as a secondary option.
### 5. React/Vue Framework
Rejected: Adds build tooling. Vanilla JS + ES modules keeps the demo self-contained.
## References
- [ADR-018: Binary CSI Frame Format](ADR-018-binary-csi-frame-format.md)
- [ADR-024: Contrastive CSI Embedding / AETHER](ADR-024-contrastive-csi-embedding.md)
- [ADR-055: Integrated Sensing Server](ADR-055-integrated-sensing-server.md)
- `vendor/ruvector/crates/ruvector-cnn/src/lib.rs` — CNN embedder implementation
- `vendor/ruvector/crates/ruvector-cnn-wasm/src/lib.rs` — WASM bindings
- `vendor/ruvector/examples/wasm-vanilla/index.html` — Reference vanilla JS WASM pattern
- Person-in-WiFi: Fine-grained Person Perception using WiFi (ICCV 2019) — camera+WiFi fusion precedent
- WiPose: Multi-Person WiFi Pose Estimation (TMC 2022) — cross-modal embedding approach
@@ -0,0 +1,83 @@
# ADR-059: Live ESP32 CSI Pipeline Integration
## Status
Accepted
## Date
2026-03-12
## Context
ADR-058 established a dual-modal browser demo combining webcam video and WiFi CSI for pose estimation. However, it used simulated CSI data. To demonstrate real-world capability, we need an end-to-end pipeline from physical ESP32 hardware through to the browser visualization.
The ESP32-S3 firmware (`firmware/esp32-csi-node/`) already supports CSI collection and UDP streaming (ADR-018). The sensing server (`wifi-densepose-sensing-server`) already supports UDP ingestion and WebSocket bridging. The missing piece was connecting these components and enabling the browser demo to consume live data.
## Decision
Implement a complete live CSI pipeline:
```
ESP32-S3 (CSI capture) → UDP:5005 → sensing-server (Rust/Axum) → WS:8765 → browser demo
```
### Components
1. **ESP32 Firmware** — Rebuilt with native Windows ESP-IDF v5.4.0 toolchain (no Docker). Configured for target network and PC IP via `sdkconfig`. Helper scripts added:
- `build_firmware.ps1` — Sets up IDF environment, cleans, builds, and flashes
- `read_serial.ps1` — Serial monitor with DTR/RTS reset capability
2. **Sensing Server**`wifi-densepose-sensing-server` started with:
- `--source esp32` — Expect real ESP32 UDP frames
- `--bind-addr 0.0.0.0` — Accept connections from any interface
- `--ui-path <path>` — Serve the demo UI via HTTP
3. **Browser Demo**`main.js` updated to auto-connect to `ws://localhost:8765/ws/sensing` on page load. Falls back to simulated CSI if the WebSocket is unavailable (GitHub Pages).
### Network Configuration
The ESP32 sends UDP packets to a configured target IP. If the PC's IP doesn't match the firmware's compiled target, a secondary IP alias can be added:
```powershell
# PowerShell (Admin)
New-NetIPAddress -IPAddress 192.168.1.100 -PrefixLength 24 -InterfaceAlias "Wi-Fi"
```
### Data Flow
| Stage | Protocol | Format | Rate |
|-------|----------|--------|------|
| ESP32 → Server | UDP | ADR-018 binary frame (magic `0xC5110001`, I/Q pairs) | ~100 Hz |
| Server → Browser | WebSocket | ADR-018 binary frame (forwarded) | ~10 Hz (tick-ms=100) |
| Browser decode | JavaScript | Float32 amplitude/phase arrays | Per frame |
### Build Environment (Windows)
ESP-IDF v5.4.0 on Windows requires:
- IDF_PATH pointing to the ESP-IDF framework
- IDF_TOOLS_PATH pointing to toolchain binaries
- MSYS/MinGW environment variables removed (ESP-IDF rejects them)
- Python venv from ESP-IDF tools for `idf.py` execution
The `build_firmware.ps1` script handles all of this automatically.
## Consequences
### Positive
- First end-to-end demonstration of real WiFi CSI → pose estimation in a browser
- No Docker required for firmware builds on Windows
- Demo gracefully degrades to simulated CSI when no server is available
- Same demo works on GitHub Pages (simulated) and locally (live ESP32)
### Negative
- ESP32 target IP is compiled into firmware; changing it requires a rebuild or NVS override
- Windows firewall may block UDP:5005; user must allow it
- Mixed content restrictions prevent HTTPS pages from connecting to ws:// (local only)
## Related
- [ADR-018](ADR-018-esp32-dev-implementation.md) — ESP32 CSI frame format and UDP streaming
- [ADR-058](ADR-058-ruvector-wasm-browser-pose-example.md) — Dual-modal WASM browser pose demo
- [ADR-039](ADR-039-edge-intelligence-framework.md) — Edge intelligence on ESP32
- Issue [#245](https://github.com/ruvnet/RuView/issues/245) — Tracking issue
+33 -11
View File
@@ -78,6 +78,17 @@ docker pull ruvnet/wifi-densepose:latest
Multi-architecture image (amd64 + arm64). Works on Intel/AMD and Apple Silicon Macs. Contains the Rust sensing server, Three.js UI, and all signal processing.
**Data source selection:** Use the `CSI_SOURCE` environment variable to select the sensing mode:
| Value | Description |
|-------|-------------|
| `auto` | (default) Probe for ESP32 on UDP 5005, fall back to simulation |
| `esp32` | Receive real CSI frames from ESP32 devices over UDP |
| `simulated` | Generate synthetic CSI frames (no hardware required) |
| `wifi` | Host Wi-Fi RSSI (not available inside containers) |
Example: `docker run -e CSI_SOURCE=esp32 -p 3000:3000 -p 5005:5005/udp ruvnet/wifi-densepose:latest`
### From Source (Rust)
```bash
@@ -267,8 +278,8 @@ Real Channel State Information at 20 Hz with 56-192 subcarriers. Required for po
# From source
./target/release/sensing-server --source esp32 --udp-port 5005 --http-port 3000 --ws-port 3001
# Docker
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp ruvnet/wifi-densepose:latest --source esp32
# Docker (use CSI_SOURCE environment variable)
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp -e CSI_SOURCE=esp32 ruvnet/wifi-densepose:latest
```
The ESP32 nodes stream binary CSI frames over UDP to port 5005. See [Hardware Setup](#esp32-s3-mesh) for flashing instructions.
@@ -679,9 +690,11 @@ Download the dataset files and place them in a `data/` directory.
./target/release/sensing-server --train --dataset data/ --dataset-type mmfi --epochs 100 --save-rvf model.rvf
# Via Docker (mount your data directory)
# Note: Training mode requires overriding the default entrypoint
docker run --rm \
-v $(pwd)/data:/data \
-v $(pwd)/output:/output \
--entrypoint /app/sensing-server \
ruvnet/wifi-densepose:latest \
--train --dataset /data --epochs 100 --export-rvf /output/model.rvf
```
@@ -797,14 +810,18 @@ Pre-built binaries are available at [Releases](https://github.com/ruvnet/RuView/
| Release | What It Includes | Tag |
|---------|-----------------|-----|
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Stable — raw CSI streaming, TDM, channel hopping, QUIC mesh | `v0.2.0-esp32` |
| [v0.4.1](https://github.com/ruvnet/RuView/releases/tag/v0.4.1-esp32) | **Stable** — CSI build fix, compile guard, AMOLED display, edge intelligence ([ADR-057](../docs/adr/ADR-057-firmware-csi-build-guard.md)) | `v0.4.1-esp32` |
| [v0.3.0-alpha](https://github.com/ruvnet/RuView/releases/tag/v0.3.0-alpha-esp32) | Alpha — adds on-device edge intelligence (ADR-039) | `v0.3.0-alpha-esp32` |
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Raw CSI streaming, TDM, channel hopping, QUIC mesh | `v0.2.0-esp32` |
> **Important:** Firmware versions prior to v0.4.1 had CSI **disabled** in the build config, causing a runtime error (`E wifi:CSI not enabled in menuconfig!`). Always use v0.4.1 or later.
```bash
# Flash an ESP32-S3 (requires esptool: pip install esptool)
python -m esptool --chip esp32s3 --port COM7 --baud 460800 \
write-flash --flash-mode dio --flash-size 4MB \
0x0 bootloader.bin 0x8000 partition-table.bin 0x10000 esp32-csi-node.bin
write-flash --flash-mode dio --flash-size 8MB --flash-freq 80m \
0x0 bootloader.bin 0x8000 partition-table.bin \
0xf000 ota_data_initial.bin 0x20000 esp32-csi-node.bin
```
**Provisioning:**
@@ -885,8 +902,8 @@ Binary size: 777 KB (24% free in the 1 MB app partition).
# From source
./target/release/sensing-server --source esp32 --udp-port 5005 --http-port 3000 --ws-port 3001
# Docker
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp ruvnet/wifi-densepose:latest --source esp32
# Docker (use CSI_SOURCE environment variable)
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp -e CSI_SOURCE=esp32 ruvnet/wifi-densepose:latest
```
See [ADR-018](../docs/adr/ADR-018-esp32-dev-implementation.md), [ADR-029](../docs/adr/ADR-029-ruvsense-multistatic-sensing-mode.md), and [Tutorial #34](https://github.com/ruvnet/RuView/issues/34).
@@ -953,12 +970,17 @@ Add the WebSocket port mapping:
docker run -p 3000:3000 -p 3001:3001 ruvnet/wifi-densepose:latest
```
### ESP32: "CSI not enabled in menuconfig"
Firmware versions prior to v0.4.1 had `CONFIG_ESP_WIFI_CSI_ENABLED` disabled in the build config. Upgrade to [v0.4.1](https://github.com/ruvnet/RuView/releases/tag/v0.4.1-esp32) or later. If building from source, ensure `sdkconfig.defaults` exists (not just `sdkconfig.defaults.template`). See [ADR-057](../docs/adr/ADR-057-firmware-csi-build-guard.md).
### ESP32: No data arriving
1. Verify the ESP32 is connected to the same WiFi network
2. Check the target IP matches the sensing server machine: `python firmware/esp32-csi-node/provision.py --port COM7 --target-ip <YOUR_IP>`
3. Verify UDP port 5005 is not blocked by firewall
4. Test with: `nc -lu 5005` (Linux) or similar UDP listener
1. Verify firmware is v0.4.1+ (older versions had CSI disabled — see above)
2. Verify the ESP32 is connected to the same WiFi network
3. Check the target IP matches the sensing server machine: `python firmware/esp32-csi-node/provision.py --port COM7 --target-ip <YOUR_IP>`
4. Verify UDP port 5005 is not blocked by firewall
5. Test with: `nc -lu 5005` (Linux) or similar UDP listener
### Build: Rust compilation errors
@@ -0,0 +1,130 @@
{
"running": true,
"startedAt": "2026-03-10T14:22:41.948Z",
"workers": {
"map": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T14:22:41.948Z"
},
"audit": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T14:24:41.948Z"
},
"optimize": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T14:26:41.948Z"
},
"consolidate": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T14:28:41.949Z"
},
"testgaps": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T14:30:41.949Z"
},
"predict": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false
},
"document": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false
}
},
"config": {
"autoStart": false,
"logDir": "/Users/cohen/GitHub/ruvnet/RuView/firmware/esp32-csi-node/.claude-flow/logs",
"stateFile": "/Users/cohen/GitHub/ruvnet/RuView/firmware/esp32-csi-node/.claude-flow/daemon-state.json",
"maxConcurrent": 2,
"workerTimeoutMs": 300000,
"resourceThresholds": {
"maxCpuLoad": 2,
"minFreeMemoryPercent": 20
},
"workers": [
{
"type": "map",
"intervalMs": 900000,
"offsetMs": 0,
"priority": "normal",
"description": "Codebase mapping",
"enabled": true
},
{
"type": "audit",
"intervalMs": 600000,
"offsetMs": 120000,
"priority": "critical",
"description": "Security analysis",
"enabled": true
},
{
"type": "optimize",
"intervalMs": 900000,
"offsetMs": 240000,
"priority": "high",
"description": "Performance optimization",
"enabled": true
},
{
"type": "consolidate",
"intervalMs": 1800000,
"offsetMs": 360000,
"priority": "low",
"description": "Memory consolidation",
"enabled": true
},
{
"type": "testgaps",
"intervalMs": 1200000,
"offsetMs": 480000,
"priority": "normal",
"description": "Test coverage analysis",
"enabled": true
},
{
"type": "predict",
"intervalMs": 600000,
"offsetMs": 0,
"priority": "low",
"description": "Predictive preloading",
"enabled": false
},
{
"type": "document",
"intervalMs": 3600000,
"offsetMs": 0,
"priority": "low",
"description": "Auto-documentation",
"enabled": false
}
]
},
"savedAt": "2026-03-10T14:22:41.949Z"
}
@@ -0,0 +1,7 @@
{"type":"edit","file":"unknown","timestamp":1773152422749,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773152444021,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773152460956,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773152493971,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773152501432,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773152510853,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773152596890,"sessionId":null}
@@ -0,0 +1,12 @@
{
"id": "session-1773152560779",
"startedAt": "2026-03-10T14:22:40.779Z",
"cwd": "/Users/cohen/GitHub/ruvnet/RuView/firmware/esp32-csi-node",
"context": {},
"metrics": {
"edits": 1,
"commands": 0,
"tasks": 0,
"errors": 0
}
}
@@ -0,0 +1,31 @@
# Remove MSYS environment variables that trigger ESP-IDF's MinGW rejection
Remove-Item env:MSYSTEM -ErrorAction SilentlyContinue
Remove-Item env:MSYSTEM_CARCH -ErrorAction SilentlyContinue
Remove-Item env:MSYSTEM_CHOST -ErrorAction SilentlyContinue
Remove-Item env:MSYSTEM_PREFIX -ErrorAction SilentlyContinue
Remove-Item env:MINGW_CHOST -ErrorAction SilentlyContinue
Remove-Item env:MINGW_PACKAGE_PREFIX -ErrorAction SilentlyContinue
Remove-Item env:MINGW_PREFIX -ErrorAction SilentlyContinue
$env:IDF_PATH = "C:\Users\ruv\esp\v5.4\esp-idf"
$env:IDF_TOOLS_PATH = "C:\Espressif\tools"
$env:IDF_PYTHON_ENV_PATH = "C:\Espressif\tools\python\v5.4\venv"
$env:PATH = "C:\Espressif\tools\xtensa-esp-elf\esp-14.2.0_20241119\xtensa-esp-elf\bin;C:\Espressif\tools\cmake\3.30.2\cmake-3.30.2-windows-x86_64\bin;C:\Espressif\tools\ninja\1.12.1;C:\Espressif\tools\ccache\4.10.2\ccache-4.10.2-windows-x86_64;C:\Espressif\tools\idf-exe\1.0.3;C:\Espressif\tools\python\v5.4\venv\Scripts;$env:PATH"
Set-Location "C:\Users\ruv\Projects\wifi-densepose\firmware\esp32-csi-node"
$python = "$env:IDF_PYTHON_ENV_PATH\Scripts\python.exe"
$idf = "$env:IDF_PATH\tools\idf.py"
Write-Host "=== Cleaning stale build cache ==="
& $python $idf fullclean
Write-Host "=== Building firmware (SSID=ruv.net, target=192.168.1.20:5005) ==="
& $python $idf build
if ($LASTEXITCODE -eq 0) {
Write-Host "=== Build succeeded! Flashing to COM7 ==="
& $python $idf -p COM7 flash
} else {
Write-Host "=== Build failed with exit code $LASTEXITCODE ==="
}
@@ -21,6 +21,16 @@
#include "esp_timer.h"
#include "sdkconfig.h"
/* ADR-057: Build-time guard — fail early if CSI is not enabled in sdkconfig.
* Without this, the firmware compiles but crashes at runtime with:
* "E (xxxx) wifi:CSI not enabled in menuconfig!"
* which is confusing for users flashing pre-built binaries. */
#ifndef CONFIG_ESP_WIFI_CSI_ENABLED
#error "CONFIG_ESP_WIFI_CSI_ENABLED must be set in sdkconfig. " \
"Run: idf.py menuconfig -> Component config -> Wi-Fi -> Enable WiFi CSI, " \
"or copy sdkconfig.defaults.template to sdkconfig.defaults before building."
#endif
static const char *TAG = "csi_collector";
static uint32_t s_sequence = 0;
File diff suppressed because one or more lines are too long
Binary file not shown.
+14
View File
@@ -0,0 +1,14 @@
$p = New-Object System.IO.Ports.SerialPort('COM7', 115200)
$p.ReadTimeout = 5000
$p.Open()
Start-Sleep -Milliseconds 200
for ($i = 0; $i -lt 60; $i++) {
try {
$line = $p.ReadLine()
Write-Host $line
} catch {
break
}
}
$p.Close()
@@ -0,0 +1,33 @@
# ESP32-S3 CSI Node — Default SDK Configuration
# This file is applied automatically by idf.py when no sdkconfig exists.
# Target: ESP32-S3
CONFIG_IDF_TARGET="esp32s3"
# Use custom partition table (8MB flash with OTA — ADR-045)
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions_display.csv"
# Flash configuration: 8MB (Quad SPI)
CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y
CONFIG_ESPTOOLPY_FLASHSIZE="8MB"
# Compiler optimization: optimize for size to reduce binary
CONFIG_COMPILER_OPTIMIZATION_SIZE=y
# Enable CSI (Channel State Information) in WiFi driver
CONFIG_ESP_WIFI_CSI_ENABLED=y
# NVS encryption disabled by default (requires eFuse provisioning).
# Enable only after burning HMAC key to eFuse block.
# CONFIG_NVS_ENCRYPTION is not set
# Disable unused features to reduce binary size
CONFIG_BOOTLOADER_LOG_LEVEL_WARN=y
CONFIG_LOG_DEFAULT_LEVEL_INFO=y
# LWIP: enable extended socket options for UDP multicast
CONFIG_LWIP_SO_RCVBUF=y
# FreeRTOS: increase task stack for CSI processing
CONFIG_ESP_MAIN_TASK_STACK_SIZE=8192
@@ -18,8 +18,9 @@ CONFIG_COMPILER_OPTIMIZATION_SIZE=y
# Enable CSI (Channel State Information) in WiFi driver
CONFIG_ESP_WIFI_CSI_ENABLED=y
# Enable NVS encryption for secure credential storage
CONFIG_NVS_ENCRYPTION=y
# NVS encryption disabled by default (requires eFuse provisioning).
# Enable only after burning HMAC key to eFuse block.
# CONFIG_NVS_ENCRYPTION is not set
# Disable unused features to reduce binary size
CONFIG_BOOTLOADER_LOG_LEVEL_WARN=y
@@ -3,3 +3,8 @@
{"type":"edit","file":"unknown","timestamp":1772820472219,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1772832571444,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1772832585997,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773099593107,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773115162931,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773115172336,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773147087836,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773149448951,"sessionId":null}
+22
View File
@@ -2870,6 +2870,26 @@ dependencies = [
"libc",
]
[[package]]
name = "libudev"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78b324152da65df7bb95acfcaab55e3097ceaab02fb19b228a9eb74d55f135e0"
dependencies = [
"libc",
"libudev-sys",
]
[[package]]
name = "libudev-sys"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3c8469b4a23b962c1396b9b451dda50ef5b283e8dd309d69033475fa9b334324"
dependencies = [
"libc",
"pkg-config",
]
[[package]]
name = "linux-raw-sys"
version = "0.12.1"
@@ -5614,6 +5634,7 @@ dependencies = [
"core-foundation",
"core-foundation-sys",
"io-kit-sys",
"libudev",
"mach2",
"nix 0.26.4",
"scopeguard",
@@ -7634,6 +7655,7 @@ dependencies = [
"reqwest 0.12.28",
"serde",
"serde_json",
"serialport",
"sha2",
"sysinfo",
"tauri",
@@ -0,0 +1,130 @@
{
"running": true,
"startedAt": "2026-03-09T23:56:03.574Z",
"workers": {
"map": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-09T23:56:03.574Z"
},
"audit": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-09T23:58:03.574Z"
},
"optimize": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T00:00:03.574Z"
},
"consolidate": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T00:02:03.574Z"
},
"testgaps": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T00:04:03.574Z"
},
"predict": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false
},
"document": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false
}
},
"config": {
"autoStart": false,
"logDir": "/Users/cohen/GitHub/ruvnet/RuView/rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop/.claude-flow/logs",
"stateFile": "/Users/cohen/GitHub/ruvnet/RuView/rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop/.claude-flow/daemon-state.json",
"maxConcurrent": 2,
"workerTimeoutMs": 300000,
"resourceThresholds": {
"maxCpuLoad": 2,
"minFreeMemoryPercent": 20
},
"workers": [
{
"type": "map",
"intervalMs": 900000,
"offsetMs": 0,
"priority": "normal",
"description": "Codebase mapping",
"enabled": true
},
{
"type": "audit",
"intervalMs": 600000,
"offsetMs": 120000,
"priority": "critical",
"description": "Security analysis",
"enabled": true
},
{
"type": "optimize",
"intervalMs": 900000,
"offsetMs": 240000,
"priority": "high",
"description": "Performance optimization",
"enabled": true
},
{
"type": "consolidate",
"intervalMs": 1800000,
"offsetMs": 360000,
"priority": "low",
"description": "Memory consolidation",
"enabled": true
},
{
"type": "testgaps",
"intervalMs": 1200000,
"offsetMs": 480000,
"priority": "normal",
"description": "Test coverage analysis",
"enabled": true
},
{
"type": "predict",
"intervalMs": 600000,
"offsetMs": 0,
"priority": "low",
"description": "Predictive preloading",
"enabled": false
},
{
"type": "document",
"intervalMs": 3600000,
"offsetMs": 0,
"priority": "low",
"description": "Auto-documentation",
"enabled": false
}
]
},
"savedAt": "2026-03-09T23:56:03.574Z"
}
@@ -0,0 +1,42 @@
{"type":"edit","file":"unknown","timestamp":1773100520674,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100630628,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100635269,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100648222,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100660593,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100670480,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100765961,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100793408,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100801110,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100806887,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100820942,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100857691,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100894224,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773100911798,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773101430507,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773101470221,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773101478246,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773103575668,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773103693989,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773115108388,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773115362485,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773115372676,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773115388605,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773115394377,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773115415015,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773115600459,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773146102258,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773146113449,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773146119695,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773146128174,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773146133721,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773146150082,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773146337071,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773150581963,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773150596765,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773152997925,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773153073387,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773153109436,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773153121443,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773153290476,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773153290781,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773153291056,"sessionId":null}
@@ -0,0 +1,12 @@
{
"id": "session-1773150558480",
"startedAt": "2026-03-10T13:49:18.480Z",
"cwd": "/Users/cohen/GitHub/ruvnet/RuView/rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop",
"context": {},
"metrics": {
"edits": 9,
"commands": 0,
"tasks": 0,
"errors": 0
}
}
@@ -0,0 +1,14 @@
{
"id": "session-1773100562538",
"startedAt": "2026-03-09T23:56:02.538Z",
"cwd": "/Users/cohen/GitHub/ruvnet/RuView/rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop",
"context": {},
"metrics": {
"edits": 13,
"commands": 0,
"tasks": 0,
"errors": 0
},
"endedAt": "2026-03-10T00:07:15.557Z",
"duration": 673020
}
@@ -0,0 +1,14 @@
{
"id": "session-1773101285009",
"startedAt": "2026-03-10T00:08:05.009Z",
"cwd": "/Users/cohen/GitHub/ruvnet/RuView/rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop",
"context": {},
"metrics": {
"edits": 19,
"commands": 0,
"tasks": 0,
"errors": 0
},
"endedAt": "2026-03-10T13:48:30.150Z",
"duration": 49225141
}
@@ -56,6 +56,11 @@ hex = "0.4"
# Regex for parsing espflash output
regex = "1.10"
# Serial port for WiFi configuration
serialport.workspace = true
# Unix signals for graceful process termination
[target.'cfg(unix)'.dependencies]
libc = "0.2"
[dev-dependencies]
@@ -411,6 +411,91 @@ fn is_esp32_compatible(vid: u16, pid: u16) -> bool {
false
}
/// Configure WiFi credentials on an ESP32 via serial port.
///
/// Sends WiFi credentials to the ESP32 using a simple serial protocol.
/// The ESP32 firmware should accept: `wifi_config <ssid> <password>\n`
#[tauri::command]
pub async fn configure_esp32_wifi(
port: String,
ssid: String,
password: String,
) -> Result<String, String> {
use std::io::{Read, Write};
use std::time::Duration;
tracing::info!("Configuring WiFi on port: {}", port);
// Open serial port
let mut serial = serialport::new(&port, 115200)
.timeout(Duration::from_secs(3))
.open()
.map_err(|e| format!("Failed to open port {}: {}", port, e))?;
// Wait for ESP32 to be ready
std::thread::sleep(Duration::from_millis(500));
// Try multiple command formats that different firmware versions might accept
let commands = [
format!("wifi_config {} {}\r\n", ssid, password),
format!("wifi {} {}\r\n", ssid, password),
format!("set ssid {}\r\n", ssid),
];
let mut response = String::new();
let mut buf = [0u8; 512];
for cmd in &commands {
// Clear any pending data
let _ = serial.read(&mut buf);
// Send command
serial.write_all(cmd.as_bytes())
.map_err(|e| format!("Failed to write: {}", e))?;
serial.flush().map_err(|e| format!("Failed to flush: {}", e))?;
// Wait and read response
std::thread::sleep(Duration::from_millis(500));
match serial.read(&mut buf) {
Ok(n) if n > 0 => {
let text = String::from_utf8_lossy(&buf[..n]).to_string();
response.push_str(&text);
// Check for success indicators
if text.to_lowercase().contains("ok")
|| text.to_lowercase().contains("saved")
|| text.to_lowercase().contains("configured") {
tracing::info!("WiFi config successful: {}", text.trim());
return Ok(format!("WiFi configured! Response: {}", text.trim()));
}
}
_ => {}
}
}
// Also try to send password separately if ssid command was sent
let pwd_cmd = format!("set password {}\r\n", password);
let _ = serial.write_all(pwd_cmd.as_bytes());
let _ = serial.flush();
std::thread::sleep(Duration::from_millis(300));
if let Ok(n) = serial.read(&mut buf) {
if n > 0 {
response.push_str(&String::from_utf8_lossy(&buf[..n]));
}
}
// Send reboot command
let _ = serial.write_all(b"reboot\r\n");
let _ = serial.flush();
if response.is_empty() {
Ok("Commands sent. ESP32 may need manual reboot to apply WiFi settings.".to_string())
} else {
Ok(format!("Commands sent. Response: {}", response.trim()))
}
}
#[derive(Debug, Clone, Serialize)]
pub struct SerialPortInfo {
pub name: String,
@@ -13,6 +13,7 @@ pub fn run() {
// Discovery
discovery::discover_nodes,
discovery::list_serial_ports,
discovery::configure_esp32_wifi,
// Flash
flash::flash_firmware,
flash::flash_progress,
@@ -1,7 +1,7 @@
{
"$schema": "https://raw.githubusercontent.com/tauri-apps/tauri/dev/crates/tauri-config-schema/schema.json",
"productName": "RuView Desktop",
"version": "0.4.3",
"version": "0.4.4",
"identifier": "net.ruv.ruview",
"build": {
"frontendDist": "ui/dist",
@@ -30,9 +30,6 @@
"icons/128x128@2x.png",
"icons/icon.icns",
"icons/icon.ico"
],
"resources": {
"../../target/release/sensing-server": "bin/sensing-server"
}
]
}
}
@@ -0,0 +1,420 @@
//! Integration tests for all Tauri API commands
//!
//! Tests the actual command implementations without the Tauri runtime.
// ============================================================================
// Discovery Tests
// ============================================================================
#[test]
fn test_serial_port_detection_logic() {
// Test ESP32 VID/PID detection
// CP210x (Silicon Labs)
assert!(is_esp32_vid_pid(0x10C4, 0xEA60), "CP2102 should be detected");
assert!(is_esp32_vid_pid(0x10C4, 0xEA70), "CP2104 should be detected");
// CH340/CH341 (QinHeng)
assert!(is_esp32_vid_pid(0x1A86, 0x7523), "CH340 should be detected");
assert!(is_esp32_vid_pid(0x1A86, 0x5523), "CH341 should be detected");
// FTDI
assert!(is_esp32_vid_pid(0x0403, 0x6001), "FTDI FT232 should be detected");
assert!(is_esp32_vid_pid(0x0403, 0x6010), "FTDI FT2232 should be detected");
// ESP32 native USB
assert!(is_esp32_vid_pid(0x303A, 0x1001), "ESP32-S2/S3 native should be detected");
// Unknown device
assert!(!is_esp32_vid_pid(0x0000, 0x0000), "Unknown VID/PID should not be detected");
assert!(!is_esp32_vid_pid(0x1234, 0x5678), "Random VID/PID should not be detected");
}
fn is_esp32_vid_pid(vid: u16, pid: u16) -> bool {
// CP210x (Silicon Labs)
if vid == 0x10C4 && (pid == 0xEA60 || pid == 0xEA70) {
return true;
}
// CH340/CH341 (QinHeng)
if vid == 0x1A86 && (pid == 0x7523 || pid == 0x5523) {
return true;
}
// FTDI
if vid == 0x0403 && (pid == 0x6001 || pid == 0x6010 || pid == 0x6011 || pid == 0x6014 || pid == 0x6015) {
return true;
}
// ESP32-S2/S3 native USB
if vid == 0x303A {
return true;
}
false
}
#[test]
fn test_beacon_parsing() {
let data = b"RUVIEW_BEACON|AA:BB:CC:DD:EE:FF|1|0.3.0|esp32s3|coordinator|0|4";
let text = std::str::from_utf8(data).unwrap();
let parts: Vec<&str> = text.split('|').collect();
assert_eq!(parts.len(), 8);
assert_eq!(parts[0], "RUVIEW_BEACON");
assert_eq!(parts[1], "AA:BB:CC:DD:EE:FF");
assert_eq!(parts[2], "1");
assert_eq!(parts[3], "0.3.0");
assert_eq!(parts[4], "esp32s3");
assert_eq!(parts[5], "coordinator");
assert_eq!(parts[6], "0");
assert_eq!(parts[7], "4");
}
// ============================================================================
// Settings Tests
// ============================================================================
#[test]
fn test_settings_structure() {
use wifi_densepose_desktop::commands::settings::AppSettings;
let settings = AppSettings::default();
// Check default values
assert!(!settings.theme.is_empty(), "Theme should have a default");
assert!(settings.discover_interval_ms > 0, "Discovery interval should be positive");
assert!(settings.auto_discover, "Auto-discover should default to true");
assert_eq!(settings.server_http_port, 8080);
}
#[test]
fn test_settings_serialization() {
use wifi_densepose_desktop::commands::settings::AppSettings;
let settings = AppSettings::default();
let json = serde_json::to_string(&settings).expect("Should serialize");
let restored: AppSettings = serde_json::from_str(&json).expect("Should deserialize");
assert_eq!(settings.theme, restored.theme);
assert_eq!(settings.server_http_port, restored.server_http_port);
assert_eq!(settings.discover_interval_ms, restored.discover_interval_ms);
}
// ============================================================================
// Server Tests
// ============================================================================
#[test]
fn test_server_state_default() {
use wifi_densepose_desktop::state::ServerState;
let server = ServerState::default();
assert!(!server.running, "Server should not be running by default");
assert!(server.pid.is_none());
assert!(server.http_port.is_none());
}
// ============================================================================
// Flash Tests
// ============================================================================
#[test]
fn test_chip_variants() {
use wifi_densepose_desktop::domain::node::Chip;
let chips = vec![
Chip::Esp32,
Chip::Esp32s2,
Chip::Esp32s3,
Chip::Esp32c3,
Chip::Esp32c6,
];
for chip in chips {
let name = format!("{:?}", chip).to_lowercase();
assert!(name.starts_with("esp32"), "All chips should be ESP32 variants");
}
}
#[test]
fn test_progress_parsing() {
// Test espflash progress output parsing
let output = "Flashing... [===> ] 35%";
let re = regex::Regex::new(r"(\d+)%").unwrap();
if let Some(caps) = re.captures(output) {
let pct: u8 = caps[1].parse().unwrap();
assert_eq!(pct, 35);
} else {
panic!("Should parse percentage");
}
}
// ============================================================================
// OTA Tests
// ============================================================================
#[test]
fn test_sha256_hash() {
use sha2::{Sha256, Digest};
let data = b"test firmware data";
let mut hasher = Sha256::new();
hasher.update(data);
let hash = hasher.finalize();
let hex = hex::encode(hash);
assert_eq!(hex.len(), 64, "SHA256 should produce 64 hex characters");
}
#[test]
fn test_hmac_signature() {
use hmac::{Hmac, Mac};
use sha2::Sha256;
type HmacSha256 = Hmac<Sha256>;
let key = b"test_psk_key";
let data = b"firmware_hash";
let mut mac = HmacSha256::new_from_slice(key).expect("HMAC can take key of any size");
mac.update(data);
let result = mac.finalize();
let signature = hex::encode(result.into_bytes());
assert_eq!(signature.len(), 64, "HMAC-SHA256 should produce 64 hex characters");
}
// ============================================================================
// Provision Tests
// ============================================================================
#[test]
fn test_nvs_config_format() {
// Test CSV format for NVS partition
let csv = "key,type,encoding,value\ncsi_cfg,namespace,,\nssid,data,string,TestNetwork\npassword,data,string,TestPass123\n";
let lines: Vec<&str> = csv.lines().collect();
assert_eq!(lines.len(), 4);
assert!(lines[0].starts_with("key,type"));
assert!(lines[1].contains("namespace"));
assert!(lines[2].contains("ssid"));
assert!(lines[3].contains("password"));
}
#[test]
fn test_mesh_config_generation() {
// Test that mesh configs have required fields
let config = serde_json::json!({
"node_id": 1,
"mesh_role": "node",
"tdm_slot": 0,
"tdm_total": 4,
"ssid": "TestNetwork",
"password": "TestPass",
"coordinator_ip": "192.168.1.100"
});
assert!(config.get("node_id").is_some());
assert!(config.get("mesh_role").is_some());
assert!(config.get("ssid").is_some());
}
// ============================================================================
// WASM Tests
// ============================================================================
#[test]
fn test_wasm_magic_bytes() {
// WebAssembly magic bytes: \0asm
let wasm_header: [u8; 4] = [0x00, 0x61, 0x73, 0x6D];
assert_eq!(wasm_header[0], 0x00);
assert_eq!(wasm_header[1], 0x61); // 'a'
assert_eq!(wasm_header[2], 0x73); // 's'
assert_eq!(wasm_header[3], 0x6D); // 'm'
}
#[test]
fn test_wasm_version() {
// WASM version 1
let wasm_version: [u8; 4] = [0x01, 0x00, 0x00, 0x00];
let version = u32::from_le_bytes(wasm_version);
assert_eq!(version, 1);
}
// ============================================================================
// State Tests
// ============================================================================
#[test]
fn test_app_state_initialization() {
use wifi_densepose_desktop::state::AppState;
let state = AppState::default();
// Check that all state components initialize correctly
let discovery = state.discovery.lock().unwrap();
assert!(discovery.nodes.is_empty(), "Should start with no nodes");
drop(discovery);
let flash = state.flash.lock().unwrap();
assert_eq!(flash.phase, "", "Should start with empty phase");
assert_eq!(flash.progress_pct, 0.0);
drop(flash);
let server = state.server.lock().unwrap();
assert!(!server.running, "Server should not be running initially");
}
// ============================================================================
// Domain Model Tests
// ============================================================================
#[test]
fn test_health_status_variants() {
use wifi_densepose_desktop::domain::node::HealthStatus;
let statuses = vec![
HealthStatus::Online,
HealthStatus::Degraded,
HealthStatus::Offline,
];
for status in statuses {
let json = serde_json::to_string(&status).expect("Should serialize");
assert!(!json.is_empty());
}
}
#[test]
fn test_discovery_method_variants() {
use wifi_densepose_desktop::domain::node::DiscoveryMethod;
let methods = vec![
DiscoveryMethod::Mdns,
DiscoveryMethod::UdpProbe,
DiscoveryMethod::Manual,
DiscoveryMethod::HttpSweep,
];
for method in methods {
let json = serde_json::to_string(&method).expect("Should serialize");
assert!(!json.is_empty());
}
}
#[test]
fn test_mesh_role_variants() {
use wifi_densepose_desktop::domain::node::MeshRole;
let roles = vec![
MeshRole::Coordinator,
MeshRole::Aggregator,
MeshRole::Node,
];
for role in roles {
let json = serde_json::to_string(&role).expect("Should serialize");
assert!(!json.is_empty());
}
}
// ============================================================================
// WiFi Config Tests (New Feature)
// ============================================================================
#[test]
fn test_wifi_config_command_format() {
let ssid = "TestNetwork";
let password = "TestPass123";
// Test all command formats
let cmd1 = format!("wifi_config {} {}\r\n", ssid, password);
let cmd2 = format!("wifi {} {}\r\n", ssid, password);
let cmd3 = format!("set ssid {}\r\n", ssid);
let cmd4 = format!("set password {}\r\n", password);
assert!(cmd1.contains("wifi_config"));
assert!(cmd1.contains(ssid));
assert!(cmd1.contains(password));
assert!(cmd1.ends_with("\r\n"));
assert!(cmd2.starts_with("wifi "));
assert!(cmd3.starts_with("set ssid "));
assert!(cmd4.starts_with("set password "));
}
#[test]
fn test_wifi_credentials_validation() {
// SSID: 1-32 characters
let valid_ssid = "MyNetwork";
let empty_ssid = "";
let long_ssid = "A".repeat(33);
assert!(!valid_ssid.is_empty() && valid_ssid.len() <= 32);
assert!(empty_ssid.is_empty());
assert!(long_ssid.len() > 32);
// Password: 8-63 characters for WPA2
let valid_pass = "password123";
let short_pass = "short";
let long_pass = "A".repeat(64);
assert!(valid_pass.len() >= 8 && valid_pass.len() <= 63);
assert!(short_pass.len() < 8);
assert!(long_pass.len() > 63);
}
// ============================================================================
// Node Registry Tests
// ============================================================================
#[test]
fn test_node_registry() {
use wifi_densepose_desktop::domain::node::{
DiscoveredNode, MacAddress, NodeRegistry, HealthStatus, Chip, MeshRole, DiscoveryMethod
};
let mut registry = NodeRegistry::new();
assert!(registry.is_empty());
let node = DiscoveredNode {
ip: "192.168.1.100".into(),
mac: Some("AA:BB:CC:DD:EE:FF".into()),
hostname: Some("csi-node-1".into()),
node_id: 1,
firmware_version: Some("0.3.0".into()),
health: HealthStatus::Online,
last_seen: "2024-01-01T00:00:00Z".into(),
chip: Chip::Esp32s3,
mesh_role: MeshRole::Node,
discovery_method: DiscoveryMethod::Mdns,
tdm_slot: Some(0),
tdm_total: Some(4),
edge_tier: None,
uptime_secs: Some(3600),
capabilities: None,
friendly_name: None,
notes: None,
};
registry.upsert(MacAddress::new("AA:BB:CC:DD:EE:FF"), node);
assert_eq!(registry.len(), 1);
let retrieved = registry.get(&MacAddress::new("AA:BB:CC:DD:EE:FF"));
assert!(retrieved.is_some());
assert_eq!(retrieved.unwrap().ip, "192.168.1.100");
}
// ============================================================================
// MAC Address Tests
// ============================================================================
#[test]
fn test_mac_address() {
use wifi_densepose_desktop::domain::node::MacAddress;
let mac = MacAddress::new("AA:BB:CC:DD:EE:FF");
assert_eq!(mac.to_string(), "AA:BB:CC:DD:EE:FF");
let mac2 = MacAddress::new("aa:bb:cc:dd:ee:ff");
assert_ne!(mac, mac2); // Case sensitive comparison
}
@@ -0,0 +1,130 @@
{
"running": true,
"startedAt": "2026-03-10T00:49:11.921Z",
"workers": {
"map": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T00:49:11.921Z"
},
"audit": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T00:51:11.921Z"
},
"optimize": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T00:53:11.921Z"
},
"consolidate": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T00:55:11.921Z"
},
"testgaps": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false,
"nextRun": "2026-03-10T00:57:11.921Z"
},
"predict": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false
},
"document": {
"runCount": 0,
"successCount": 0,
"failureCount": 0,
"averageDurationMs": 0,
"isRunning": false
}
},
"config": {
"autoStart": false,
"logDir": "/Users/cohen/GitHub/ruvnet/RuView/rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop/ui/.claude-flow/logs",
"stateFile": "/Users/cohen/GitHub/ruvnet/RuView/rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop/ui/.claude-flow/daemon-state.json",
"maxConcurrent": 2,
"workerTimeoutMs": 300000,
"resourceThresholds": {
"maxCpuLoad": 2,
"minFreeMemoryPercent": 20
},
"workers": [
{
"type": "map",
"intervalMs": 900000,
"offsetMs": 0,
"priority": "normal",
"description": "Codebase mapping",
"enabled": true
},
{
"type": "audit",
"intervalMs": 600000,
"offsetMs": 120000,
"priority": "critical",
"description": "Security analysis",
"enabled": true
},
{
"type": "optimize",
"intervalMs": 900000,
"offsetMs": 240000,
"priority": "high",
"description": "Performance optimization",
"enabled": true
},
{
"type": "consolidate",
"intervalMs": 1800000,
"offsetMs": 360000,
"priority": "low",
"description": "Memory consolidation",
"enabled": true
},
{
"type": "testgaps",
"intervalMs": 1200000,
"offsetMs": 480000,
"priority": "normal",
"description": "Test coverage analysis",
"enabled": true
},
{
"type": "predict",
"intervalMs": 600000,
"offsetMs": 0,
"priority": "low",
"description": "Predictive preloading",
"enabled": false
},
{
"type": "document",
"intervalMs": 3600000,
"offsetMs": 0,
"priority": "low",
"description": "Auto-documentation",
"enabled": false
}
]
},
"savedAt": "2026-03-10T00:49:11.921Z"
}
@@ -9,3 +9,20 @@
{"type":"edit","file":"unknown","timestamp":1772835930809,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1772835942468,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1772835952451,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773070971487,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773070977376,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773101503481,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773107530083,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773107530201,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773107530319,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773114830434,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773114834713,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773114838852,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773150617007,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773150621430,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773150628006,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773150640909,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773150672276,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773150677219,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773150683839,"sessionId":null}
{"type":"edit","file":"unknown","timestamp":1773150688912,"sessionId":null}
@@ -0,0 +1,12 @@
{
"id": "session-1773103750755",
"startedAt": "2026-03-10T00:49:10.755Z",
"cwd": "/Users/cohen/GitHub/ruvnet/RuView/rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop/ui",
"context": {},
"metrics": {
"edits": 14,
"commands": 0,
"tasks": 0,
"errors": 0
}
}
@@ -53,6 +53,7 @@
"integrity": "sha512-CGOfOJqWjg2qW/Mb6zNsDm+u5vFQ8DxXfbM09z69p5Z6+mE1ikP2jUXw+j42Pf1XTYED2Rni5f95npYeuwMDQA==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@babel/code-frame": "^7.29.0",
"@babel/generator": "^7.29.0",
@@ -1246,6 +1247,7 @@
"integrity": "sha512-z9VXpC7MWrhfWipitjNdgCauoMLRdIILQsAEV+ZesIzBq/oUlxk0m3ApZuMFCXdnS4U7KrI+l3WRUEGQ8K1QKw==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"@types/prop-types": "*",
"csstype": "^3.2.2"
@@ -1315,6 +1317,7 @@
}
],
"license": "MIT",
"peer": true,
"dependencies": {
"baseline-browser-mapping": "^2.9.0",
"caniuse-lite": "^1.0.30001759",
@@ -1584,6 +1587,7 @@
"integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==",
"dev": true,
"license": "MIT",
"peer": true,
"engines": {
"node": ">=12"
},
@@ -1625,6 +1629,7 @@
"resolved": "https://registry.npmjs.org/react/-/react-18.3.1.tgz",
"integrity": "sha512-wS+hAgJShR0KhEvPJArfuPVN1+Hz1t0Y6n5jLrGQbkb4urgPE/0Rve+1kMB1v/oWgHgm4WIcV+i7F2pTVj+2iQ==",
"license": "MIT",
"peer": true,
"dependencies": {
"loose-envify": "^1.1.0"
},
@@ -1797,6 +1802,7 @@
"integrity": "sha512-+Oxm7q9hDoLMyJOYfUYBuHQo+dkAloi33apOPP56pzj+vsdJDzr+j1NISE5pyaAuKL4A3UD34qd0lx5+kfKp2g==",
"dev": true,
"license": "MIT",
"peer": true,
"dependencies": {
"esbuild": "^0.25.0",
"fdir": "^6.4.4",
@@ -1,7 +1,7 @@
{
"name": "ruview-desktop-ui",
"private": true,
"version": "0.4.3",
"version": "0.4.4",
"type": "module",
"scripts": {
"dev": "vite",
@@ -49,6 +49,12 @@ const NetworkDiscovery: React.FC<NetworkDiscoveryProps> = ({ onNavigate }) => {
const [error, setError] = useState<string | null>(null);
const [selectedNode, setSelectedNode] = useState<DiscoveredNode | null>(null);
const [filterOnline, setFilterOnline] = useState(false);
// WiFi config state
const [wifiConfigPort, setWifiConfigPort] = useState<string | null>(null);
const [wifiSsid, setWifiSsid] = useState("");
const [wifiPassword, setWifiPassword] = useState("");
const [configuringWifi, setConfiguringWifi] = useState(false);
const [wifiResult, setWifiResult] = useState<string | null>(null);
const [searchQuery, setSearchQuery] = useState("");
// Manual add state
const [manualIp, setManualIp] = useState("");
@@ -83,6 +89,24 @@ const NetworkDiscovery: React.FC<NetworkDiscoveryProps> = ({ onNavigate }) => {
}
}, []);
const configureWifi = useCallback(async () => {
if (!wifiConfigPort || !wifiSsid) return;
setConfiguringWifi(true);
setWifiResult(null);
try {
const result = await invoke<string>("configure_esp32_wifi", {
port: wifiConfigPort,
ssid: wifiSsid,
password: wifiPassword,
});
setWifiResult(result);
} catch (err) {
setWifiResult(`Error: ${err instanceof Error ? err.message : String(err)}`);
} finally {
setConfiguringWifi(false);
}
}, [wifiConfigPort, wifiSsid, wifiPassword]);
const addManualNode = useCallback(async () => {
if (!manualIp.trim()) return;
setAddingManual(true);
@@ -471,23 +495,47 @@ const NetworkDiscovery: React.FC<NetworkDiscoveryProps> = ({ onNavigate }) => {
)}
</Td>
<Td>
{port.is_esp32_compatible && onNavigate && (
<button
onClick={() => onNavigate("flash")}
style={{
padding: "4px 12px",
background: "var(--accent)",
border: "none",
borderRadius: 4,
color: "#fff",
fontSize: 11,
fontWeight: 600,
cursor: "pointer",
}}
>
Flash
</button>
)}
<div style={{ display: "flex", gap: 6 }}>
{port.is_esp32_compatible && (
<button
onClick={() => {
setWifiConfigPort(port.name);
setWifiSsid("");
setWifiPassword("");
setWifiResult(null);
}}
style={{
padding: "4px 10px",
background: "rgba(56, 139, 253, 0.15)",
border: "1px solid rgba(56, 139, 253, 0.3)",
borderRadius: 4,
color: "var(--accent)",
fontSize: 11,
fontWeight: 600,
cursor: "pointer",
}}
>
WiFi
</button>
)}
{port.is_esp32_compatible && onNavigate && (
<button
onClick={() => onNavigate("flash")}
style={{
padding: "4px 10px",
background: "var(--accent)",
border: "none",
borderRadius: 4,
color: "#fff",
fontSize: 11,
fontWeight: 600,
cursor: "pointer",
}}
>
Flash
</button>
)}
</div>
</Td>
</tr>
))}
@@ -579,6 +627,224 @@ const NetworkDiscovery: React.FC<NetworkDiscoveryProps> = ({ onNavigate }) => {
{selectedNode && (
<NodeDetailModal node={selectedNode} onClose={() => setSelectedNode(null)} />
)}
{/* WiFi Configuration Modal */}
{wifiConfigPort && (
<div
style={{
position: "fixed",
inset: 0,
background: "rgba(0,0,0,0.6)",
display: "flex",
alignItems: "center",
justifyContent: "center",
zIndex: 1000,
padding: "var(--space-5)",
}}
onClick={(e) => {
if (e.target === e.currentTarget && !configuringWifi) {
setWifiConfigPort(null);
}
}}
>
<div
style={{
background: "var(--bg-surface)",
borderRadius: 12,
padding: "var(--space-5)",
maxWidth: 420,
width: "100%",
border: "1px solid var(--border)",
}}
>
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "start",
marginBottom: "var(--space-4)",
}}
>
<div>
<h2 className="heading-md" style={{ margin: 0 }}>
Configure WiFi
</h2>
<p className="mono" style={{ color: "var(--text-muted)", marginTop: 4, fontSize: 13 }}>
{wifiConfigPort}
</p>
</div>
<button
onClick={() => setWifiConfigPort(null)}
disabled={configuringWifi}
style={{
background: "none",
border: "none",
fontSize: 20,
cursor: configuringWifi ? "not-allowed" : "pointer",
color: "var(--text-muted)",
padding: 4,
opacity: configuringWifi ? 0.5 : 1,
}}
>
×
</button>
</div>
<div style={{ display: "flex", flexDirection: "column", gap: "var(--space-3)" }}>
<div>
<label
style={{
display: "block",
fontSize: 12,
fontWeight: 600,
color: "var(--text-secondary)",
marginBottom: 4,
}}
>
WiFi SSID *
</label>
<input
type="text"
placeholder="Your WiFi network name"
value={wifiSsid}
onChange={(e) => setWifiSsid(e.target.value)}
disabled={configuringWifi}
style={{
width: "100%",
padding: "10px 12px",
borderRadius: 6,
border: "1px solid var(--border)",
background: "var(--bg-base)",
color: "var(--text-primary)",
fontSize: 13,
}}
/>
</div>
<div>
<label
style={{
display: "block",
fontSize: 12,
fontWeight: 600,
color: "var(--text-secondary)",
marginBottom: 4,
}}
>
WiFi Password
</label>
<input
type="password"
placeholder="WiFi password"
value={wifiPassword}
onChange={(e) => setWifiPassword(e.target.value)}
disabled={configuringWifi}
style={{
width: "100%",
padding: "10px 12px",
borderRadius: 6,
border: "1px solid var(--border)",
background: "var(--bg-base)",
color: "var(--text-primary)",
fontSize: 13,
}}
/>
</div>
{wifiResult && (
<div
style={{
padding: "var(--space-3)",
borderRadius: 6,
fontSize: 12,
background: wifiResult.startsWith("Error")
? "rgba(248, 81, 73, 0.1)"
: wifiResult.includes("configured") || wifiResult.includes("saved")
? "rgba(63, 185, 80, 0.1)"
: "rgba(56, 139, 253, 0.1)",
border: wifiResult.startsWith("Error")
? "1px solid rgba(248, 81, 73, 0.3)"
: wifiResult.includes("configured") || wifiResult.includes("saved")
? "1px solid rgba(63, 185, 80, 0.3)"
: "1px solid rgba(56, 139, 253, 0.3)",
color: wifiResult.startsWith("Error")
? "var(--status-error)"
: wifiResult.includes("configured") || wifiResult.includes("saved")
? "var(--status-online)"
: "var(--accent)",
}}
>
<div style={{ fontWeight: 600, marginBottom: 6 }}>
{wifiResult.startsWith("Error") ? "Error" :
wifiResult.includes("configured") || wifiResult.includes("saved") ? "Success!" : "Commands Sent"}
</div>
<div style={{ fontFamily: "var(--font-mono)", whiteSpace: "pre-wrap", maxHeight: 100, overflow: "auto" }}>
{wifiResult}
</div>
{!wifiResult.startsWith("Error") && !wifiResult.includes("configured") && (
<div style={{ marginTop: 8, fontSize: 11, color: "var(--text-secondary)" }}>
If the ESP32 doesn't connect, try pressing its Reset button or re-flashing with WiFi credentials in the firmware.
</div>
)}
</div>
)}
<div style={{ display: "flex", gap: "var(--space-3)", marginTop: "var(--space-2)" }}>
<button
onClick={() => setWifiConfigPort(null)}
disabled={configuringWifi}
style={{
flex: 1,
padding: "10px 16px",
borderRadius: 6,
border: "1px solid var(--border)",
background: wifiResult ? "var(--accent)" : "transparent",
color: wifiResult ? "#fff" : "var(--text-secondary)",
fontSize: 13,
fontWeight: 600,
cursor: configuringWifi ? "not-allowed" : "pointer",
opacity: configuringWifi ? 0.5 : 1,
}}
>
{wifiResult ? "Done" : "Cancel"}
</button>
{!wifiResult && (
<button
onClick={configureWifi}
disabled={!wifiSsid.trim() || configuringWifi}
className="btn-gradient"
style={{
flex: 1,
opacity: !wifiSsid.trim() || configuringWifi ? 0.5 : 1,
}}
>
{configuringWifi ? "Configuring..." : "Configure WiFi"}
</button>
)}
{wifiResult && !wifiResult.startsWith("Error") && (
<button
onClick={() => {
setWifiResult(null);
}}
style={{
flex: 1,
padding: "10px 16px",
borderRadius: 6,
border: "1px solid var(--border)",
background: "transparent",
color: "var(--text-secondary)",
fontSize: 13,
fontWeight: 600,
cursor: "pointer",
}}
>
Try Again
</button>
)}
</div>
</div>
</div>
</div>
)}
</div>
);
};
@@ -1,2 +1,2 @@
// Application version - single source of truth
export const APP_VERSION = "0.4.3";
export const APP_VERSION = "0.4.4";
+1
View File
@@ -29,6 +29,7 @@
<button class="nav-tab" data-tab="applications">Applications</button>
<button class="nav-tab" data-tab="sensing">Sensing</button>
<button class="nav-tab" data-tab="training">Training</button>
<a href="pose-fusion.html" class="nav-tab" style="text-decoration:none">Pose Fusion</a>
<a href="observatory.html" class="nav-tab" style="text-decoration:none">Observatory</a>
</nav>
+160
View File
@@ -0,0 +1,160 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>WiFi-DensePose — Dual-Modal Pose Estimation</title>
<link rel="stylesheet" href="pose-fusion/css/style.css">
</head>
<body>
<!-- Header -->
<header class="header">
<div class="header-left">
<div class="logo"><span class="pi">&pi;</span> DensePose</div>
<div class="header-title">Dual-Modal Pose Estimation — Live Video + WiFi CSI Fusion</div>
</div>
<div class="header-right">
<select id="mode-select" class="mode-select">
<option value="dual">Dual Mode (Video + CSI)</option>
<option value="video">Video Only</option>
<option value="csi">CSI Only (WiFi)</option>
</select>
<div class="status-badge">
<span id="status-dot" class="status-dot offline"></span>
<span id="status-label">READY</span>
</div>
<span id="fps-display" class="fps-badge">-- FPS</span>
<a href="index.html" class="back-link">&larr; Dashboard</a>
<a href="observatory.html" class="back-link">Observatory &rarr;</a>
</div>
</header>
<!-- Main Grid -->
<div class="main-grid">
<!-- Video + Skeleton Panel -->
<div class="video-panel">
<video id="webcam" autoplay playsinline muted></video>
<canvas id="skeleton-canvas"></canvas>
<div class="video-overlay-label" id="mode-label">DUAL FUSION</div>
<div id="camera-prompt" class="camera-prompt">
<p>Enable your webcam for live video pose estimation.<br>
Or switch to <strong>CSI Only</strong> mode for WiFi-based sensing.</p>
<button id="start-camera-btn">Enable Camera</button>
</div>
</div>
<!-- Side Panels -->
<div class="side-panels">
<!-- Fusion Confidence -->
<div class="panel">
<div class="panel-title">&#9670; Fusion Confidence</div>
<div class="fusion-bars">
<div class="bar-row">
<span class="bar-label">Video</span>
<div class="bar-track"><div class="bar-fill video" id="video-bar" style="width:0%"></div></div>
<span class="bar-value" id="video-bar-val">0%</span>
</div>
<div class="bar-row">
<span class="bar-label">CSI</span>
<div class="bar-track"><div class="bar-fill csi" id="csi-bar" style="width:0%"></div></div>
<span class="bar-value" id="csi-bar-val">0%</span>
</div>
<div class="bar-row">
<span class="bar-label">Fused</span>
<div class="bar-track"><div class="bar-fill fused" id="fused-bar" style="width:0%"></div></div>
<span class="bar-value" id="fused-bar-val">0%</span>
</div>
</div>
<div style="margin-top:8px; font-size:10px; color:var(--text-label)">
Cross-modal: <span id="cross-modal-sim" style="color:var(--green-glow)">0.000</span>
</div>
</div>
<!-- CSI Heatmap -->
<div class="panel">
<div class="panel-title">&#9670; CSI Amplitude Heatmap</div>
<div class="csi-canvas-wrapper">
<canvas id="csi-canvas" width="320" height="120"></canvas>
</div>
</div>
<!-- Embedding Space -->
<div class="panel">
<div class="panel-title">&#9670; Embedding Space (2D Projection)</div>
<div class="embedding-canvas-wrapper">
<canvas id="embedding-canvas" width="320" height="140"></canvas>
</div>
</div>
<!-- Latency -->
<div class="panel">
<div class="panel-title">&#9670; Pipeline Latency</div>
<div class="latency-grid">
<div class="latency-item">
<div class="latency-value" id="lat-video">--</div>
<div class="latency-label">Video CNN</div>
</div>
<div class="latency-item">
<div class="latency-value" id="lat-csi">--</div>
<div class="latency-label">CSI CNN</div>
</div>
<div class="latency-item">
<div class="latency-value" id="lat-fusion">--</div>
<div class="latency-label">Fusion</div>
</div>
<div class="latency-item">
<div class="latency-value" id="lat-total">--</div>
<div class="latency-label">Total</div>
</div>
</div>
</div>
<!-- Controls -->
<div class="panel">
<div class="panel-title">&#9670; Controls</div>
<div class="controls-row">
<button class="btn" id="pause-btn">⏸ Pause</button>
</div>
<div class="slider-row">
<label>Confidence</label>
<input type="range" id="confidence-slider" min="0" max="1" step="0.05" value="0.3">
<span class="slider-val" id="confidence-value">0.30</span>
</div>
<div style="margin-top:10px">
<div class="panel-title" style="margin-bottom:6px">&#9670; Live CSI Source</div>
<div style="display:flex;gap:6px">
<input type="text" id="ws-url" placeholder="ws://localhost:3030/ws/csi"
style="flex:1;background:rgba(255,255,255,0.05);border:1px solid var(--bg-panel-border);
color:var(--text-primary);padding:5px 8px;border-radius:4px;font-size:11px;
font-family:'JetBrains Mono',monospace">
<button class="btn" id="connect-ws-btn">Connect</button>
</div>
</div>
</div>
</div><!-- /side-panels -->
<!-- Bottom Bar -->
<div class="bottom-bar">
<div>
WiFi-DensePose &middot; Dual-Modal Pose Estimation &middot;
Architecture: MobileNet-V3 &times; 2 &rarr; Attention Fusion &rarr; 17-Keypoint COCO
</div>
<div>
<a href="https://github.com/ruvnet/wifi-densepose">GitHub</a> &middot;
CNN: ruvector-cnn (JS fallback) &middot;
<a href="observatory.html">Observatory</a>
</div>
</div>
</div><!-- /main-grid -->
<script type="module" src="pose-fusion/js/main.js"></script>
</body>
</html>
+30
View File
@@ -0,0 +1,30 @@
#!/bin/bash
# Build WASM packages for the dual-modal pose estimation demo.
# Requires: wasm-pack (cargo install wasm-pack)
#
# Usage: ./build.sh
#
# Output: pkg/ruvector_cnn_wasm/ — WASM CNN embedder for browser
set -e
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
VENDOR_DIR="$SCRIPT_DIR/../../vendor/ruvector"
OUT_DIR="$SCRIPT_DIR/pkg/ruvector_cnn_wasm"
echo "Building ruvector-cnn-wasm..."
wasm-pack build "$VENDOR_DIR/crates/ruvector-cnn-wasm" \
--target web \
--out-dir "$OUT_DIR" \
--no-typescript
# Remove .gitignore so we can commit the build output for GitHub Pages
rm -f "$OUT_DIR/.gitignore"
echo ""
echo "Build complete!"
echo " WASM: $(du -sh "$OUT_DIR/ruvector_cnn_wasm_bg.wasm" | cut -f1)"
echo " JS: $(du -sh "$OUT_DIR/ruvector_cnn_wasm.js" | cut -f1)"
echo ""
echo "Serve the demo: cd $SCRIPT_DIR/.. && python3 -m http.server 8080"
echo "Open: http://localhost:8080/pose-fusion.html"
+405
View File
@@ -0,0 +1,405 @@
/* WiFi-DensePose — Dual-Modal Pose Fusion Demo
Dark theme matching Observatory */
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@300;400;600;700&family=JetBrains+Mono:wght@400;600&display=swap');
:root {
--bg-deep: #080c14;
--bg-panel: rgba(8, 16, 28, 0.92);
--bg-panel-border: rgba(0, 210, 120, 0.25);
--green-glow: #00d878;
--green-bright:#3eff8a;
--green-dim: #0a6b3a;
--amber: #ffb020;
--amber-dim: #a06800;
--blue-signal: #2090ff;
--blue-dim: #0a3060;
--red-alert: #ff3040;
--cyan: #00e5ff;
--text-primary: #e8ece0;
--text-secondary: rgba(232,236,224, 0.55);
--text-label: rgba(232,236,224, 0.35);
--radius: 8px;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: var(--bg-deep);
font-family: 'Inter', -apple-system, sans-serif;
color: var(--text-primary);
-webkit-font-smoothing: antialiased;
overflow-x: hidden;
min-height: 100vh;
}
/* === Header === */
.header {
display: flex;
align-items: center;
justify-content: space-between;
padding: 16px 24px;
border-bottom: 1px solid var(--bg-panel-border);
background: var(--bg-panel);
backdrop-filter: blur(12px);
}
.header-left {
display: flex;
align-items: center;
gap: 16px;
}
.logo {
font-weight: 700;
font-size: 24px;
color: var(--green-glow);
}
.logo .pi { font-style: normal; }
.header-title {
font-size: 14px;
color: var(--text-secondary);
font-weight: 300;
}
.header-right {
display: flex;
align-items: center;
gap: 16px;
}
.mode-select {
background: rgba(0,210,120,0.1);
border: 1px solid var(--bg-panel-border);
color: var(--text-primary);
padding: 6px 12px;
border-radius: var(--radius);
font-family: inherit;
font-size: 13px;
cursor: pointer;
}
.mode-select option { background: #0c1420; }
.status-badge {
display: flex;
align-items: center;
gap: 6px;
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
padding: 4px 10px;
border-radius: 12px;
background: rgba(0,210,120,0.1);
border: 1px solid var(--bg-panel-border);
}
.status-dot {
width: 8px; height: 8px;
border-radius: 50%;
background: var(--green-glow);
box-shadow: 0 0 8px var(--green-glow);
animation: pulse-dot 2s ease infinite;
}
.status-dot.offline { background: #555; box-shadow: none; animation: none; }
.status-dot.warning { background: var(--amber); box-shadow: 0 0 8px var(--amber); }
@keyframes pulse-dot {
0%, 100% { opacity: 1; }
50% { opacity: 0.5; }
}
.fps-badge {
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
color: var(--green-glow);
}
.back-link {
color: var(--text-secondary);
text-decoration: none;
font-size: 13px;
transition: color 0.2s;
}
.back-link:hover { color: var(--green-glow); }
/* === Main Layout === */
.main-grid {
display: grid;
grid-template-columns: 1fr 360px;
grid-template-rows: 1fr auto;
gap: 16px;
padding: 16px 24px;
height: calc(100vh - 72px);
overflow: hidden;
}
/* === Video Panel === */
.video-panel {
position: relative;
background: #000;
border-radius: var(--radius);
border: 1px solid var(--bg-panel-border);
overflow: hidden;
min-height: 0;
}
.video-panel video {
width: 100%;
height: 100%;
object-fit: cover;
transform: scaleX(-1);
}
.video-panel canvas {
position: absolute;
top: 0; left: 0;
width: 100%;
height: 100%;
transform: scaleX(-1);
}
.video-overlay-label {
position: absolute;
top: 12px; left: 12px;
font-family: 'JetBrains Mono', monospace;
font-size: 11px;
padding: 4px 8px;
background: rgba(0,0,0,0.7);
border-radius: 4px;
color: var(--green-glow);
z-index: 5;
transform: scaleX(-1);
}
.camera-prompt {
position: absolute;
top: 50%; left: 50%;
transform: translate(-50%, -50%);
text-align: center;
color: var(--text-secondary);
}
.camera-prompt button {
margin-top: 12px;
padding: 10px 24px;
background: var(--green-glow);
color: #000;
border: none;
border-radius: var(--radius);
font-family: inherit;
font-weight: 600;
font-size: 14px;
cursor: pointer;
transition: background 0.2s;
}
.camera-prompt button:hover { background: var(--green-bright); }
/* === Side Panels === */
.side-panels {
display: flex;
flex-direction: column;
gap: 12px;
overflow-y: auto;
min-height: 0;
}
.panel {
background: var(--bg-panel);
border: 1px solid var(--bg-panel-border);
border-radius: var(--radius);
padding: 14px;
}
.panel-title {
font-size: 11px;
text-transform: uppercase;
letter-spacing: 1.2px;
color: var(--text-label);
margin-bottom: 10px;
display: flex;
align-items: center;
gap: 6px;
}
/* === CSI Heatmap === */
.csi-canvas-wrapper {
position: relative;
border-radius: 4px;
overflow: hidden;
background: #000;
}
.csi-canvas-wrapper canvas {
width: 100%;
display: block;
}
/* === Fusion Bars === */
.fusion-bars {
display: flex;
flex-direction: column;
gap: 8px;
}
.bar-row {
display: flex;
align-items: center;
gap: 8px;
}
.bar-label {
font-family: 'JetBrains Mono', monospace;
font-size: 11px;
color: var(--text-secondary);
width: 55px;
text-align: right;
}
.bar-track {
flex: 1;
height: 6px;
background: rgba(255,255,255,0.06);
border-radius: 3px;
overflow: hidden;
}
.bar-fill {
height: 100%;
border-radius: 3px;
transition: width 0.3s ease;
}
.bar-fill.video { background: var(--cyan); }
.bar-fill.csi { background: var(--amber); }
.bar-fill.fused { background: var(--green-glow); box-shadow: 0 0 8px var(--green-glow); }
.bar-value {
font-family: 'JetBrains Mono', monospace;
font-size: 11px;
color: var(--text-primary);
width: 36px;
}
/* === Embedding Space === */
.embedding-canvas-wrapper {
position: relative;
background: #000;
border-radius: 4px;
overflow: hidden;
}
.embedding-canvas-wrapper canvas {
width: 100%;
display: block;
}
/* === Latency Panel === */
.latency-grid {
display: grid;
grid-template-columns: repeat(4, 1fr);
gap: 6px;
}
.latency-item {
text-align: center;
padding: 6px 0;
}
.latency-value {
font-family: 'JetBrains Mono', monospace;
font-size: 16px;
font-weight: 600;
color: var(--green-glow);
}
.latency-label {
font-size: 10px;
color: var(--text-label);
margin-top: 2px;
}
/* === Controls === */
.controls-row {
display: flex;
gap: 8px;
flex-wrap: wrap;
}
.btn {
padding: 6px 14px;
border: 1px solid var(--bg-panel-border);
background: rgba(0,210,120,0.08);
color: var(--text-primary);
border-radius: var(--radius);
font-family: inherit;
font-size: 12px;
cursor: pointer;
transition: all 0.2s;
}
.btn:hover { background: rgba(0,210,120,0.2); }
.btn.active { background: var(--green-glow); color: #000; font-weight: 600; }
.slider-row {
display: flex;
align-items: center;
gap: 8px;
margin-top: 8px;
}
.slider-row label {
font-size: 11px;
color: var(--text-secondary);
white-space: nowrap;
}
.slider-row input[type=range] {
flex: 1;
accent-color: var(--green-glow);
}
.slider-row .slider-val {
font-family: 'JetBrains Mono', monospace;
font-size: 11px;
width: 32px;
color: var(--green-glow);
}
/* === Bottom Bar === */
.bottom-bar {
grid-column: 1 / -1;
display: flex;
align-items: center;
justify-content: space-between;
padding: 10px 16px;
background: var(--bg-panel);
border: 1px solid var(--bg-panel-border);
border-radius: var(--radius);
font-family: 'JetBrains Mono', monospace;
font-size: 11px;
color: var(--text-secondary);
}
.bottom-bar a {
color: var(--green-glow);
text-decoration: none;
}
/* === Skeleton colors === */
.skeleton-joint { fill: var(--green-glow); }
.skeleton-limb { stroke: var(--green-bright); }
.skeleton-joint-csi { fill: var(--amber); }
.skeleton-limb-csi { stroke: var(--amber); }
/* === Responsive === */
@media (max-width: 900px) {
.main-grid {
grid-template-columns: 1fr;
height: auto;
overflow: auto;
}
.video-panel { aspect-ratio: 16/9; max-height: 50vh; }
.side-panels { max-height: none; overflow: visible; }
}
+247
View File
@@ -0,0 +1,247 @@
/**
* CanvasRenderer — Renders skeleton overlay on video, CSI heatmap,
* embedding space visualization, and fusion confidence bars.
*/
import { SKELETON_CONNECTIONS } from './pose-decoder.js';
export class CanvasRenderer {
constructor() {
this.colors = {
joint: '#00d878',
jointGlow: 'rgba(0, 216, 120, 0.4)',
limb: '#3eff8a',
limbGlow: 'rgba(62, 255, 138, 0.15)',
csiJoint: '#ffb020',
csiLimb: '#ffc850',
fused: '#00e5ff',
confidence: 'rgba(255,255,255,0.3)',
videoEmb: '#00e5ff',
csiEmb: '#ffb020',
fusedEmb: '#00d878',
};
}
/**
* Draw skeleton overlay on the video canvas
* @param {CanvasRenderingContext2D} ctx
* @param {Array<{x,y,confidence}>} keypoints - Normalized [0,1] coordinates
* @param {number} width - Canvas width
* @param {number} height - Canvas height
* @param {object} opts
*/
drawSkeleton(ctx, keypoints, width, height, opts = {}) {
const minConf = opts.minConfidence || 0.3;
const color = opts.color || 'green';
const jointColor = color === 'amber' ? this.colors.csiJoint : this.colors.joint;
const limbColor = color === 'amber' ? this.colors.csiLimb : this.colors.limb;
const glowColor = color === 'amber' ? 'rgba(255,176,32,0.4)' : this.colors.jointGlow;
ctx.clearRect(0, 0, width, height);
if (!keypoints || keypoints.length === 0) return;
// Draw limbs first (behind joints)
ctx.lineWidth = 3;
ctx.lineCap = 'round';
for (const [i, j] of SKELETON_CONNECTIONS) {
const kpA = keypoints[i];
const kpB = keypoints[j];
if (!kpA || !kpB || kpA.confidence < minConf || kpB.confidence < minConf) continue;
const ax = kpA.x * width, ay = kpA.y * height;
const bx = kpB.x * width, by = kpB.y * height;
const avgConf = (kpA.confidence + kpB.confidence) / 2;
// Glow
ctx.strokeStyle = this.colors.limbGlow;
ctx.lineWidth = 8;
ctx.globalAlpha = avgConf * 0.4;
ctx.beginPath();
ctx.moveTo(ax, ay);
ctx.lineTo(bx, by);
ctx.stroke();
// Main line
ctx.strokeStyle = limbColor;
ctx.lineWidth = 2.5;
ctx.globalAlpha = avgConf;
ctx.beginPath();
ctx.moveTo(ax, ay);
ctx.lineTo(bx, by);
ctx.stroke();
}
// Draw joints
ctx.globalAlpha = 1;
for (const kp of keypoints) {
if (!kp || kp.confidence < minConf) continue;
const x = kp.x * width;
const y = kp.y * height;
const r = 3 + kp.confidence * 3;
// Glow
ctx.beginPath();
ctx.arc(x, y, r + 4, 0, Math.PI * 2);
ctx.fillStyle = glowColor;
ctx.globalAlpha = kp.confidence * 0.6;
ctx.fill();
// Joint dot
ctx.beginPath();
ctx.arc(x, y, r, 0, Math.PI * 2);
ctx.fillStyle = jointColor;
ctx.globalAlpha = kp.confidence;
ctx.fill();
// White center
ctx.beginPath();
ctx.arc(x, y, r * 0.4, 0, Math.PI * 2);
ctx.fillStyle = '#fff';
ctx.globalAlpha = kp.confidence * 0.8;
ctx.fill();
}
ctx.globalAlpha = 1;
// Confidence label
if (opts.label) {
ctx.font = '11px "JetBrains Mono", monospace';
ctx.fillStyle = jointColor;
ctx.globalAlpha = 0.8;
ctx.fillText(opts.label, 8, height - 8);
ctx.globalAlpha = 1;
}
}
/**
* Draw CSI amplitude heatmap
* @param {CanvasRenderingContext2D} ctx
* @param {{ data: Float32Array, width: number, height: number }} heatmap
* @param {number} canvasW
* @param {number} canvasH
*/
drawCsiHeatmap(ctx, heatmap, canvasW, canvasH) {
ctx.clearRect(0, 0, canvasW, canvasH);
if (!heatmap || !heatmap.data || heatmap.height < 2) {
ctx.fillStyle = '#0a0e18';
ctx.fillRect(0, 0, canvasW, canvasH);
ctx.font = '11px "JetBrains Mono", monospace';
ctx.fillStyle = 'rgba(255,255,255,0.3)';
ctx.fillText('Waiting for CSI data...', 8, canvasH / 2);
return;
}
const { data, width: dw, height: dh } = heatmap;
const cellW = canvasW / dw;
const cellH = canvasH / dh;
for (let y = 0; y < dh; y++) {
for (let x = 0; x < dw; x++) {
const val = Math.min(1, Math.max(0, data[y * dw + x]));
ctx.fillStyle = this._heatmapColor(val);
ctx.fillRect(x * cellW, y * cellH, cellW + 0.5, cellH + 0.5);
}
}
// Axis labels
ctx.font = '9px "JetBrains Mono", monospace';
ctx.fillStyle = 'rgba(255,255,255,0.4)';
ctx.fillText('Subcarrier →', 4, canvasH - 4);
ctx.save();
ctx.translate(canvasW - 4, canvasH - 4);
ctx.rotate(-Math.PI / 2);
ctx.fillText('Time ↑', 0, 0);
ctx.restore();
}
/**
* Draw embedding space 2D projection
* @param {CanvasRenderingContext2D} ctx
* @param {{ video: Array, csi: Array, fused: Array }} points
* @param {number} w
* @param {number} h
*/
drawEmbeddingSpace(ctx, points, w, h) {
ctx.fillStyle = '#050810';
ctx.fillRect(0, 0, w, h);
// Grid
ctx.strokeStyle = 'rgba(255,255,255,0.05)';
ctx.lineWidth = 0.5;
for (let i = 0; i <= 4; i++) {
const x = (i / 4) * w;
ctx.beginPath(); ctx.moveTo(x, 0); ctx.lineTo(x, h); ctx.stroke();
const y = (i / 4) * h;
ctx.beginPath(); ctx.moveTo(0, y); ctx.lineTo(w, y); ctx.stroke();
}
// Axes
ctx.strokeStyle = 'rgba(255,255,255,0.1)';
ctx.lineWidth = 1;
ctx.beginPath(); ctx.moveTo(w / 2, 0); ctx.lineTo(w / 2, h); ctx.stroke();
ctx.beginPath(); ctx.moveTo(0, h / 2); ctx.lineTo(w, h / 2); ctx.stroke();
const drawPoints = (pts, color, size) => {
if (!pts || pts.length === 0) return;
const len = pts.length;
for (let i = 0; i < len; i++) {
const p = pts[i];
if (!p) continue;
const age = 1 - (i / len) * 0.7; // Fade older points
const px = w / 2 + p[0] * w * 0.35;
const py = h / 2 + p[1] * h * 0.35;
if (px < 0 || px > w || py < 0 || py > h) continue;
ctx.beginPath();
ctx.arc(px, py, size, 0, Math.PI * 2);
ctx.fillStyle = color;
ctx.globalAlpha = age * 0.7;
ctx.fill();
}
};
drawPoints(points.video, this.colors.videoEmb, 3);
drawPoints(points.csi, this.colors.csiEmb, 3);
drawPoints(points.fused, this.colors.fusedEmb, 4);
ctx.globalAlpha = 1;
// Legend
ctx.font = '9px "JetBrains Mono", monospace';
const legends = [
{ color: this.colors.videoEmb, label: 'Video' },
{ color: this.colors.csiEmb, label: 'CSI' },
{ color: this.colors.fusedEmb, label: 'Fused' },
];
legends.forEach((l, i) => {
const ly = 12 + i * 14;
ctx.fillStyle = l.color;
ctx.beginPath();
ctx.arc(10, ly - 3, 3, 0, Math.PI * 2);
ctx.fill();
ctx.fillStyle = 'rgba(255,255,255,0.5)';
ctx.fillText(l.label, 18, ly);
});
}
_heatmapColor(val) {
// Dark blue → cyan → green → yellow → red
if (val < 0.25) {
const t = val / 0.25;
return `rgb(${Math.floor(t * 20)}, ${Math.floor(20 + t * 60)}, ${Math.floor(60 + t * 100)})`;
} else if (val < 0.5) {
const t = (val - 0.25) / 0.25;
return `rgb(${Math.floor(20 + t * 20)}, ${Math.floor(80 + t * 100)}, ${Math.floor(160 - t * 60)})`;
} else if (val < 0.75) {
const t = (val - 0.5) / 0.25;
return `rgb(${Math.floor(40 + t * 180)}, ${Math.floor(180 + t * 75)}, ${Math.floor(100 - t * 80)})`;
} else {
const t = (val - 0.75) / 0.25;
return `rgb(${Math.floor(220 + t * 35)}, ${Math.floor(255 - t * 120)}, ${Math.floor(20 - t * 20)})`;
}
}
}
+226
View File
@@ -0,0 +1,226 @@
/**
* CNN Embedder — Lightweight MobileNet-V3-style feature extractor.
*
* Architecture mirrors ruvector-cnn: Conv2D → BatchNorm → ReLU → Pool → Project → L2 Normalize
* Uses pre-seeded random weights (deterministic). When ruvector-cnn-wasm is available,
* transparently delegates to the WASM implementation.
*
* Two instances are created: one for video frames, one for CSI pseudo-images.
*/
// Seeded PRNG for deterministic weight initialization
function mulberry32(seed) {
return function() {
let t = (seed += 0x6D2B79F5);
t = Math.imul(t ^ (t >>> 15), t | 1);
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
export class CnnEmbedder {
/**
* @param {object} opts
* @param {number} opts.inputSize - Square input dimension (default 56 for speed)
* @param {number} opts.embeddingDim - Output embedding dimension (default 128)
* @param {boolean} opts.normalize - L2 normalize output
* @param {number} opts.seed - PRNG seed for weight init
*/
constructor(opts = {}) {
this.inputSize = opts.inputSize || 56;
this.embeddingDim = opts.embeddingDim || 128;
this.normalize = opts.normalize !== false;
this.wasmEmbedder = null;
// Initialize weights with deterministic PRNG
const rng = mulberry32(opts.seed || 42);
const randRange = (lo, hi) => lo + rng() * (hi - lo);
// Conv 3x3: 3 input channels → 16 output channels
this.convWeights = new Float32Array(3 * 3 * 3 * 16);
for (let i = 0; i < this.convWeights.length; i++) {
this.convWeights[i] = randRange(-0.15, 0.15);
}
// BatchNorm params (16 channels)
this.bnGamma = new Float32Array(16).fill(1.0);
this.bnBeta = new Float32Array(16).fill(0.0);
this.bnMean = new Float32Array(16).fill(0.0);
this.bnVar = new Float32Array(16).fill(1.0);
// Projection: 16 → embeddingDim
this.projWeights = new Float32Array(16 * this.embeddingDim);
for (let i = 0; i < this.projWeights.length; i++) {
this.projWeights[i] = randRange(-0.1, 0.1);
}
}
/**
* Try to load WASM embedder from ruvector-cnn-wasm package
* @param {string} wasmPath - Path to the WASM package directory
*/
async tryLoadWasm(wasmPath) {
try {
const mod = await import(`${wasmPath}/ruvector_cnn_wasm.js`);
await mod.default();
const config = new mod.EmbedderConfig();
config.input_size = this.inputSize;
config.embedding_dim = this.embeddingDim;
config.normalize = this.normalize;
this.wasmEmbedder = new mod.WasmCnnEmbedder(config);
console.log('[CNN] WASM embedder loaded successfully');
return true;
} catch (e) {
console.log('[CNN] WASM not available, using JS fallback:', e.message);
return false;
}
}
/**
* Extract embedding from RGB image data
* @param {Uint8Array} rgbData - RGB pixel data (H*W*3)
* @param {number} width
* @param {number} height
* @returns {Float32Array} embedding vector
*/
extract(rgbData, width, height) {
if (this.wasmEmbedder) {
try {
const result = this.wasmEmbedder.extract(rgbData, width, height);
return new Float32Array(result);
} catch (_) { /* fallback to JS */ }
}
return this._extractJS(rgbData, width, height);
}
_extractJS(rgbData, width, height) {
// 1. Resize to inputSize × inputSize if needed
const sz = this.inputSize;
let input;
if (width === sz && height === sz) {
input = new Float32Array(rgbData.length);
for (let i = 0; i < rgbData.length; i++) input[i] = rgbData[i] / 255.0;
} else {
input = this._resize(rgbData, width, height, sz, sz);
}
// 2. ImageNet normalization
const mean = [0.485, 0.456, 0.406];
const std = [0.229, 0.224, 0.225];
const pixels = sz * sz;
for (let i = 0; i < pixels; i++) {
input[i * 3] = (input[i * 3] - mean[0]) / std[0];
input[i * 3 + 1] = (input[i * 3 + 1] - mean[1]) / std[1];
input[i * 3 + 2] = (input[i * 3 + 2] - mean[2]) / std[2];
}
// 3. Conv2D 3x3 (3 → 16 channels)
const convOut = this._conv2d3x3(input, sz, sz, 3, 16);
// 4. BatchNorm
this._batchNorm(convOut, 16);
// 5. ReLU
for (let i = 0; i < convOut.length; i++) {
if (convOut[i] < 0) convOut[i] = 0;
}
// 6. Global average pooling → 16-dim
const outH = sz - 2, outW = sz - 2;
const pooled = new Float32Array(16);
const spatial = outH * outW;
for (let i = 0; i < spatial; i++) {
for (let c = 0; c < 16; c++) {
pooled[c] += convOut[i * 16 + c];
}
}
for (let c = 0; c < 16; c++) pooled[c] /= spatial;
// 7. Linear projection → embeddingDim
const emb = new Float32Array(this.embeddingDim);
for (let o = 0; o < this.embeddingDim; o++) {
let sum = 0;
for (let i = 0; i < 16; i++) {
sum += pooled[i] * this.projWeights[i * this.embeddingDim + o];
}
emb[o] = sum;
}
// 8. L2 normalize
if (this.normalize) {
let norm = 0;
for (let i = 0; i < emb.length; i++) norm += emb[i] * emb[i];
norm = Math.sqrt(norm);
if (norm > 1e-8) {
for (let i = 0; i < emb.length; i++) emb[i] /= norm;
}
}
return emb;
}
_conv2d3x3(input, H, W, Cin, Cout) {
const outH = H - 2, outW = W - 2;
const output = new Float32Array(outH * outW * Cout);
for (let y = 0; y < outH; y++) {
for (let x = 0; x < outW; x++) {
for (let co = 0; co < Cout; co++) {
let sum = 0;
for (let ky = 0; ky < 3; ky++) {
for (let kx = 0; kx < 3; kx++) {
for (let ci = 0; ci < Cin; ci++) {
const px = ((y + ky) * W + (x + kx)) * Cin + ci;
const wt = (((ky * 3 + kx) * Cin) + ci) * Cout + co;
sum += input[px] * this.convWeights[wt];
}
}
}
output[(y * outW + x) * Cout + co] = sum;
}
}
}
return output;
}
_batchNorm(data, channels) {
const spatial = data.length / channels;
for (let i = 0; i < spatial; i++) {
for (let c = 0; c < channels; c++) {
const idx = i * channels + c;
data[idx] = this.bnGamma[c] * (data[idx] - this.bnMean[c]) / Math.sqrt(this.bnVar[c] + 1e-5) + this.bnBeta[c];
}
}
}
_resize(rgbData, srcW, srcH, dstW, dstH) {
const output = new Float32Array(dstW * dstH * 3);
const xRatio = srcW / dstW;
const yRatio = srcH / dstH;
for (let y = 0; y < dstH; y++) {
for (let x = 0; x < dstW; x++) {
const sx = Math.min(Math.floor(x * xRatio), srcW - 1);
const sy = Math.min(Math.floor(y * yRatio), srcH - 1);
const srcIdx = (sy * srcW + sx) * 3;
const dstIdx = (y * dstW + x) * 3;
output[dstIdx] = rgbData[srcIdx] / 255.0;
output[dstIdx + 1] = rgbData[srcIdx + 1] / 255.0;
output[dstIdx + 2] = rgbData[srcIdx + 2] / 255.0;
}
}
return output;
}
/** Cosine similarity between two embeddings */
static cosineSimilarity(a, b) {
let dot = 0, normA = 0, normB = 0;
for (let i = 0; i < a.length; i++) {
dot += a[i] * b[i];
normA += a[i] * a[i];
normB += b[i] * b[i];
}
normA = Math.sqrt(normA);
normB = Math.sqrt(normB);
if (normA < 1e-8 || normB < 1e-8) return 0;
return dot / (normA * normB);
}
}
+267
View File
@@ -0,0 +1,267 @@
/**
* CSI Simulator — Generates realistic WiFi Channel State Information data.
*
* In live mode, connects to the sensing server via WebSocket.
* In demo mode, generates synthetic CSI that correlates with detected motion.
*
* Outputs: 3-channel pseudo-image (amplitude, phase, temporal diff)
* matching the ADR-018 frame format expectations.
*/
export class CsiSimulator {
constructor(opts = {}) {
this.subcarriers = opts.subcarriers || 52; // 802.11n HT20
this.timeWindow = opts.timeWindow || 56; // frames in sliding window
this.mode = 'demo'; // 'demo' | 'live'
this.ws = null;
// Circular buffer for CSI frames
this.amplitudeBuffer = [];
this.phaseBuffer = [];
this.frameCount = 0;
// Noise parameters
this._rng = this._mulberry32(opts.seed || 7);
this._noiseState = new Float32Array(this.subcarriers);
this._baseAmplitude = new Float32Array(this.subcarriers);
this._basePhase = new Float32Array(this.subcarriers);
// Initialize base CSI profile (empty room)
for (let i = 0; i < this.subcarriers; i++) {
this._baseAmplitude[i] = 0.5 + 0.3 * Math.sin(i * 0.12);
this._basePhase[i] = (i / this.subcarriers) * Math.PI * 2;
}
// Person influence (updated from video motion)
this.personPresence = 0;
this.personX = 0.5;
this.personY = 0.5;
this.personMotion = 0;
}
/**
* Connect to live sensing server WebSocket
* @param {string} url - WebSocket URL (e.g. ws://localhost:3030/ws/csi)
*/
async connectLive(url) {
return new Promise((resolve) => {
try {
this.ws = new WebSocket(url);
this.ws.binaryType = 'arraybuffer';
this.ws.onmessage = (evt) => this._handleLiveFrame(evt.data);
this.ws.onopen = () => { this.mode = 'live'; resolve(true); };
this.ws.onerror = () => resolve(false);
this.ws.onclose = () => { this.mode = 'demo'; };
// Timeout after 3s
setTimeout(() => { if (this.mode !== 'live') resolve(false); }, 3000);
} catch {
resolve(false);
}
});
}
disconnect() {
if (this.ws) { this.ws.close(); this.ws = null; }
this.mode = 'demo';
}
get isLive() { return this.mode === 'live'; }
/**
* Update person state from video detection (for correlated demo data).
* When person exits frame, CSI maintains presence with slow decay
* (simulating through-wall sensing capability).
*/
updatePersonState(presence, x, y, motion) {
if (presence > 0.1) {
// Person detected in video — update CSI state directly
this.personPresence = presence;
this.personX = x;
this.personY = y;
this.personMotion = motion;
this._lastSeenTime = performance.now();
this._lastSeenX = x;
this._lastSeenY = y;
} else if (this._lastSeenTime) {
// Person NOT in video — CSI "through-wall" persistence
const elapsed = (performance.now() - this._lastSeenTime) / 1000;
// CSI can sense through walls for ~10 seconds with decaying confidence
const decayRate = 0.15; // Lose ~15% per second
this.personPresence = Math.max(0, 1.0 - elapsed * decayRate);
// Position slowly drifts (person walking behind wall)
this.personX = this._lastSeenX;
this.personY = this._lastSeenY;
this.personMotion = Math.max(0, motion * 0.5 + this.personPresence * 0.2);
if (this.personPresence < 0.05) {
this._lastSeenTime = null;
}
} else {
this.personPresence = 0;
this.personMotion = 0;
}
}
/**
* Generate next CSI frame (demo mode) or return latest live frame
* @param {number} elapsed - Time in seconds
* @returns {{ amplitude: Float32Array, phase: Float32Array, snr: number }}
*/
nextFrame(elapsed) {
const amp = new Float32Array(this.subcarriers);
const phase = new Float32Array(this.subcarriers);
if (this.mode === 'live' && this._liveAmplitude) {
amp.set(this._liveAmplitude);
phase.set(this._livePhase);
} else {
this._generateDemoFrame(amp, phase, elapsed);
}
// Push to circular buffer
this.amplitudeBuffer.push(new Float32Array(amp));
this.phaseBuffer.push(new Float32Array(phase));
if (this.amplitudeBuffer.length > this.timeWindow) {
this.amplitudeBuffer.shift();
this.phaseBuffer.shift();
}
// SNR estimate
let signalPower = 0, noisePower = 0;
for (let i = 0; i < this.subcarriers; i++) {
signalPower += amp[i] * amp[i];
noisePower += this._noiseState[i] * this._noiseState[i];
}
const snr = noisePower > 0 ? 10 * Math.log10(signalPower / noisePower) : 30;
this.frameCount++;
return { amplitude: amp, phase, snr: Math.max(0, Math.min(40, snr)) };
}
/**
* Build 3-channel pseudo-image for CNN input
* @param {number} targetSize - Output image dimension (square)
* @returns {Uint8Array} RGB data (targetSize * targetSize * 3)
*/
buildPseudoImage(targetSize = 56) {
const buf = this.amplitudeBuffer;
const pBuf = this.phaseBuffer;
const frames = buf.length;
if (frames < 2) {
return new Uint8Array(targetSize * targetSize * 3);
}
const rgb = new Uint8Array(targetSize * targetSize * 3);
for (let y = 0; y < targetSize; y++) {
const fi = Math.min(Math.floor(y / targetSize * frames), frames - 1);
for (let x = 0; x < targetSize; x++) {
const si = Math.min(Math.floor(x / targetSize * this.subcarriers), this.subcarriers - 1);
const idx = (y * targetSize + x) * 3;
// R: Amplitude (normalized to 0-255)
const ampVal = buf[fi][si];
rgb[idx] = Math.min(255, Math.max(0, Math.floor(ampVal * 255)));
// G: Phase (wrapped to 0-255)
const phaseVal = (pBuf[fi][si] % (2 * Math.PI) + 2 * Math.PI) % (2 * Math.PI);
rgb[idx + 1] = Math.floor(phaseVal / (2 * Math.PI) * 255);
// B: Temporal difference
if (fi > 0) {
const diff = Math.abs(buf[fi][si] - buf[fi - 1][si]);
rgb[idx + 2] = Math.min(255, Math.floor(diff * 500));
}
}
}
return rgb;
}
/**
* Get heatmap data for visualization
* @returns {{ data: Float32Array, width: number, height: number }}
*/
getHeatmapData() {
const frames = this.amplitudeBuffer.length;
const w = this.subcarriers;
const h = Math.min(frames, this.timeWindow);
const data = new Float32Array(w * h);
for (let y = 0; y < h; y++) {
const fi = frames - h + y;
if (fi >= 0 && fi < frames) {
for (let x = 0; x < w; x++) {
data[y * w + x] = this.amplitudeBuffer[fi][x];
}
}
}
return { data, width: w, height: h };
}
// === Private ===
_generateDemoFrame(amp, phase, elapsed) {
const rng = this._rng;
const presence = this.personPresence;
const motion = this.personMotion;
const px = this.personX;
for (let i = 0; i < this.subcarriers; i++) {
// Base CSI profile (frequency-selective channel)
let a = this._baseAmplitude[i];
let p = this._basePhase[i] + elapsed * 0.05;
// Environmental noise (correlated across subcarriers)
this._noiseState[i] = 0.95 * this._noiseState[i] + 0.05 * (rng() * 2 - 1) * 0.03;
a += this._noiseState[i];
// Person-induced CSI perturbation
if (presence > 0.1) {
// Subcarrier-dependent body reflection (Fresnel zone model)
const freqOffset = (i - this.subcarriers * px) / (this.subcarriers * 0.3);
const bodyReflection = presence * 0.25 * Math.exp(-freqOffset * freqOffset);
// Motion causes amplitude fluctuation
const motionEffect = motion * 0.15 * Math.sin(elapsed * 3.5 + i * 0.3);
// Breathing modulation (0.2-0.3 Hz)
const breathing = presence * 0.02 * Math.sin(elapsed * 1.5 + i * 0.05);
a += bodyReflection + motionEffect + breathing;
p += presence * 0.4 * Math.sin(elapsed * 2.1 + i * 0.15);
}
amp[i] = Math.max(0, Math.min(1, a));
phase[i] = p;
}
}
_handleLiveFrame(data) {
const view = new DataView(data);
// Check ADR-018 magic: 0xC5110001
if (data.byteLength < 20) return;
const magic = view.getUint32(0, true);
if (magic !== 0xC5110001) return;
const numSub = Math.min(view.getUint16(8, true), this.subcarriers);
this._liveAmplitude = new Float32Array(this.subcarriers);
this._livePhase = new Float32Array(this.subcarriers);
const headerSize = 20;
for (let i = 0; i < numSub && (headerSize + i * 4 + 3) < data.byteLength; i++) {
const real = view.getInt16(headerSize + i * 4, true);
const imag = view.getInt16(headerSize + i * 4 + 2, true);
this._liveAmplitude[i] = Math.sqrt(real * real + imag * imag) / 2048;
this._livePhase[i] = Math.atan2(imag, real);
}
}
_mulberry32(seed) {
return function() {
let t = (seed += 0x6D2B79F5);
t = Math.imul(t ^ (t >>> 15), t | 1);
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
}
+166
View File
@@ -0,0 +1,166 @@
/**
* FusionEngine — Attention-weighted dual-modal embedding fusion.
*
* Combines visual (camera) and CSI (WiFi) embeddings with dynamic
* confidence gating based on signal quality.
*/
export class FusionEngine {
/**
* @param {number} embeddingDim
*/
constructor(embeddingDim = 128) {
this.embeddingDim = embeddingDim;
// Learnable attention weights (initialized to balanced 0.5)
// In production, these would be loaded from trained JSON
this.attentionWeights = new Float32Array(embeddingDim).fill(0.5);
// Dynamic modality confidence [0, 1]
this.videoConfidence = 1.0;
this.csiConfidence = 0.0;
this.fusedConfidence = 0.5;
// Smoothing for confidence transitions
this._smoothAlpha = 0.85;
// Embedding history for visualization
this.recentVideoEmbeddings = [];
this.recentCsiEmbeddings = [];
this.recentFusedEmbeddings = [];
this.maxHistory = 50;
}
/**
* Update quality-based confidence scores
* @param {number} videoBrightness - [0,1] video brightness quality
* @param {number} videoMotion - [0,1] motion detected
* @param {number} csiSnr - CSI signal-to-noise ratio in dB
* @param {boolean} csiActive - Whether CSI source is connected
*/
updateConfidence(videoBrightness, videoMotion, csiSnr, csiActive) {
// Video confidence: drops with low brightness, boosted by motion
let vc = 0;
if (videoBrightness > 0.05) {
vc = Math.min(1, videoBrightness * 1.5) * 0.7 + Math.min(1, videoMotion * 3) * 0.3;
}
// CSI confidence: based on SNR and connection status
let cc = 0;
if (csiActive) {
cc = Math.min(1, csiSnr / 25); // 25dB = full confidence
}
// Smooth transitions
this.videoConfidence = this._smoothAlpha * this.videoConfidence + (1 - this._smoothAlpha) * vc;
this.csiConfidence = this._smoothAlpha * this.csiConfidence + (1 - this._smoothAlpha) * cc;
// Fused confidence is the max of either (fusion can only help)
this.fusedConfidence = Math.min(1, Math.sqrt(
this.videoConfidence * this.videoConfidence + this.csiConfidence * this.csiConfidence
));
}
/**
* Fuse video and CSI embeddings
* @param {Float32Array|null} videoEmb - Visual embedding (or null if video-off)
* @param {Float32Array|null} csiEmb - CSI embedding (or null if CSI-off)
* @param {string} mode - 'dual' | 'video' | 'csi'
* @returns {Float32Array} Fused embedding
*/
fuse(videoEmb, csiEmb, mode = 'dual') {
const dim = this.embeddingDim;
const fused = new Float32Array(dim);
if (mode === 'video' || !csiEmb) {
if (videoEmb) fused.set(videoEmb);
this._recordEmbedding(videoEmb, null, fused);
return fused;
}
if (mode === 'csi' || !videoEmb) {
if (csiEmb) fused.set(csiEmb);
this._recordEmbedding(null, csiEmb, fused);
return fused;
}
// Dual mode: attention-weighted fusion with confidence gating
const totalConf = this.videoConfidence + this.csiConfidence;
const videoWeight = totalConf > 0 ? this.videoConfidence / totalConf : 0.5;
for (let i = 0; i < dim; i++) {
const alpha = this.attentionWeights[i] * videoWeight +
(1 - this.attentionWeights[i]) * (1 - videoWeight);
fused[i] = alpha * videoEmb[i] + (1 - alpha) * csiEmb[i];
}
// Re-normalize
let norm = 0;
for (let i = 0; i < dim; i++) norm += fused[i] * fused[i];
norm = Math.sqrt(norm);
if (norm > 1e-8) {
for (let i = 0; i < dim; i++) fused[i] /= norm;
}
this._recordEmbedding(videoEmb, csiEmb, fused);
return fused;
}
/**
* Get embedding pairs for 2D visualization (PCA projection)
* @returns {{ video: Array, csi: Array, fused: Array }}
*/
getEmbeddingPoints() {
// Simple 2D projection using first two principal components (approximated)
const project = (emb) => {
if (!emb || emb.length < 4) return null;
// Use pairs of dimensions as crude 2D projection
let x = 0, y = 0;
for (let i = 0; i < emb.length; i += 2) {
x += emb[i] * (i % 4 < 2 ? 1 : -1);
if (i + 1 < emb.length) {
y += emb[i + 1] * (i % 4 < 2 ? 1 : -1);
}
}
return [x * 2, y * 2]; // Scale for visibility
};
return {
video: this.recentVideoEmbeddings.map(project).filter(Boolean),
csi: this.recentCsiEmbeddings.map(project).filter(Boolean),
fused: this.recentFusedEmbeddings.map(project).filter(Boolean)
};
}
/**
* Cross-modal similarity score
* @returns {number} Cosine similarity between latest video and CSI embeddings
*/
getCrossModalSimilarity() {
const v = this.recentVideoEmbeddings[this.recentVideoEmbeddings.length - 1];
const c = this.recentCsiEmbeddings[this.recentCsiEmbeddings.length - 1];
if (!v || !c) return 0;
let dot = 0, na = 0, nb = 0;
for (let i = 0; i < v.length; i++) {
dot += v[i] * c[i];
na += v[i] * v[i];
nb += c[i] * c[i];
}
na = Math.sqrt(na); nb = Math.sqrt(nb);
return (na > 1e-8 && nb > 1e-8) ? dot / (na * nb) : 0;
}
_recordEmbedding(video, csi, fused) {
if (video) {
this.recentVideoEmbeddings.push(new Float32Array(video));
if (this.recentVideoEmbeddings.length > this.maxHistory) this.recentVideoEmbeddings.shift();
}
if (csi) {
this.recentCsiEmbeddings.push(new Float32Array(csi));
if (this.recentCsiEmbeddings.length > this.maxHistory) this.recentCsiEmbeddings.shift();
}
this.recentFusedEmbeddings.push(new Float32Array(fused));
if (this.recentFusedEmbeddings.length > this.maxHistory) this.recentFusedEmbeddings.shift();
}
}
+315
View File
@@ -0,0 +1,315 @@
/**
* WiFi-DensePose — Dual-Modal Pose Estimation Demo
*
* Main orchestration: video capture → CNN embedding → CSI processing → fusion → rendering
*/
import { VideoCapture } from './video-capture.js';
import { CsiSimulator } from './csi-simulator.js';
import { CnnEmbedder } from './cnn-embedder.js';
import { FusionEngine } from './fusion-engine.js';
import { PoseDecoder } from './pose-decoder.js';
import { CanvasRenderer } from './canvas-renderer.js';
// === State ===
let mode = 'dual'; // 'dual' | 'video' | 'csi'
let isRunning = false;
let isPaused = false;
let startTime = 0;
let frameCount = 0;
let fps = 0;
let lastFpsTime = 0;
let confidenceThreshold = 0.3;
// Latency tracking
const latency = { video: 0, csi: 0, fusion: 0, total: 0 };
// === Components ===
const videoCapture = new VideoCapture(document.getElementById('webcam'));
const csiSimulator = new CsiSimulator({ subcarriers: 52, timeWindow: 56 });
const visualCnn = new CnnEmbedder({ inputSize: 56, embeddingDim: 128, seed: 42 });
const csiCnn = new CnnEmbedder({ inputSize: 56, embeddingDim: 128, seed: 137 });
const fusionEngine = new FusionEngine(128);
const poseDecoder = new PoseDecoder(128);
const renderer = new CanvasRenderer();
// === Canvas Elements ===
const skeletonCanvas = document.getElementById('skeleton-canvas');
const skeletonCtx = skeletonCanvas.getContext('2d');
const csiCanvas = document.getElementById('csi-canvas');
const csiCtx = csiCanvas.getContext('2d');
const embeddingCanvas = document.getElementById('embedding-canvas');
const embeddingCtx = embeddingCanvas.getContext('2d');
// === UI Elements ===
const modeSelect = document.getElementById('mode-select');
const statusDot = document.getElementById('status-dot');
const statusLabel = document.getElementById('status-label');
const fpsDisplay = document.getElementById('fps-display');
const cameraPrompt = document.getElementById('camera-prompt');
const startCameraBtn = document.getElementById('start-camera-btn');
const pauseBtn = document.getElementById('pause-btn');
const confSlider = document.getElementById('confidence-slider');
const confValue = document.getElementById('confidence-value');
const wsUrlInput = document.getElementById('ws-url');
const connectWsBtn = document.getElementById('connect-ws-btn');
// Fusion bar elements
const videoBar = document.getElementById('video-bar');
const csiBar = document.getElementById('csi-bar');
const fusedBar = document.getElementById('fused-bar');
const videoBarVal = document.getElementById('video-bar-val');
const csiBarVal = document.getElementById('csi-bar-val');
const fusedBarVal = document.getElementById('fused-bar-val');
// Latency elements
const latVideoEl = document.getElementById('lat-video');
const latCsiEl = document.getElementById('lat-csi');
const latFusionEl = document.getElementById('lat-fusion');
const latTotalEl = document.getElementById('lat-total');
// Cross-modal similarity
const crossModalEl = document.getElementById('cross-modal-sim');
// === Initialize ===
function init() {
resizeCanvases();
window.addEventListener('resize', resizeCanvases);
// Mode change
modeSelect.addEventListener('change', (e) => {
mode = e.target.value;
updateModeUI();
});
// Camera start
startCameraBtn.addEventListener('click', startCamera);
// Pause
pauseBtn.addEventListener('click', () => {
isPaused = !isPaused;
pauseBtn.textContent = isPaused ? '▶ Resume' : '⏸ Pause';
pauseBtn.classList.toggle('active', isPaused);
});
// Confidence slider
confSlider.addEventListener('input', (e) => {
confidenceThreshold = parseFloat(e.target.value);
confValue.textContent = confidenceThreshold.toFixed(2);
});
// WebSocket connect
connectWsBtn.addEventListener('click', async () => {
const url = wsUrlInput.value.trim();
if (!url) return;
connectWsBtn.textContent = 'Connecting...';
const ok = await csiSimulator.connectLive(url);
connectWsBtn.textContent = ok ? '✓ Connected' : 'Connect';
if (ok) {
connectWsBtn.classList.add('active');
}
});
// Try to load WASM embedders (non-blocking)
// Resolve relative to this JS module file (in pose-fusion/js/) → ../pkg/
const wasmBase = new URL('../pkg/ruvector_cnn_wasm', import.meta.url).href;
visualCnn.tryLoadWasm(wasmBase);
csiCnn.tryLoadWasm(wasmBase);
// Auto-connect to local sensing server WebSocket if available
const defaultWsUrl = 'ws://localhost:8765/ws/sensing';
if (wsUrlInput) wsUrlInput.value = defaultWsUrl;
csiSimulator.connectLive(defaultWsUrl).then(ok => {
if (ok && connectWsBtn) {
connectWsBtn.textContent = '✓ Live ESP32';
connectWsBtn.classList.add('active');
statusLabel.textContent = 'LIVE CSI';
statusDot.classList.remove('offline');
}
});
// Auto-start camera for video/dual modes
updateModeUI();
startTime = performance.now() / 1000;
isRunning = true;
requestAnimationFrame(mainLoop);
}
async function startCamera() {
cameraPrompt.style.display = 'none';
const ok = await videoCapture.start();
if (ok) {
statusDot.classList.remove('offline');
statusLabel.textContent = 'LIVE';
resizeCanvases();
} else {
cameraPrompt.style.display = 'flex';
cameraPrompt.querySelector('p').textContent = 'Camera access denied. Try CSI-only mode.';
}
}
function updateModeUI() {
const needsVideo = mode !== 'csi';
const needsCsi = mode !== 'video';
// Show/hide camera prompt
if (needsVideo && !videoCapture.isActive) {
cameraPrompt.style.display = 'flex';
} else {
cameraPrompt.style.display = 'none';
}
}
function resizeCanvases() {
const videoPanel = document.querySelector('.video-panel');
if (videoPanel) {
const rect = videoPanel.getBoundingClientRect();
skeletonCanvas.width = rect.width;
skeletonCanvas.height = rect.height;
}
// CSI canvas
csiCanvas.width = csiCanvas.parentElement.clientWidth;
csiCanvas.height = 120;
// Embedding canvas
embeddingCanvas.width = embeddingCanvas.parentElement.clientWidth;
embeddingCanvas.height = 140;
}
// === Main Loop ===
function mainLoop(timestamp) {
if (!isRunning) return;
requestAnimationFrame(mainLoop);
if (isPaused) return;
const elapsed = performance.now() / 1000 - startTime;
const totalStart = performance.now();
// --- Video Pipeline ---
let videoEmb = null;
let motionRegion = null;
if (mode !== 'csi' && videoCapture.isActive) {
const t0 = performance.now();
const frame = videoCapture.captureFrame(56, 56);
if (frame) {
videoEmb = visualCnn.extract(frame.rgb, frame.width, frame.height);
motionRegion = videoCapture.detectMotionRegion(56, 56);
// Feed motion to CSI simulator for correlated demo data
// When detected=false, CSI simulator handles through-wall persistence
csiSimulator.updatePersonState(
motionRegion.detected ? 1.0 : 0,
motionRegion.detected ? motionRegion.x + motionRegion.w / 2 : 0.5,
motionRegion.detected ? motionRegion.y + motionRegion.h / 2 : 0.5,
frame.motion
);
fusionEngine.updateConfidence(
frame.brightness, frame.motion,
0, csiSimulator.isLive || mode === 'dual'
);
}
latency.video = performance.now() - t0;
}
// --- CSI Pipeline ---
let csiEmb = null;
if (mode !== 'video') {
const t0 = performance.now();
const csiFrame = csiSimulator.nextFrame(elapsed);
const pseudoImage = csiSimulator.buildPseudoImage(56);
csiEmb = csiCnn.extract(pseudoImage, 56, 56);
fusionEngine.updateConfidence(
videoCapture.brightnessScore,
videoCapture.motionScore,
csiFrame.snr,
true
);
// Draw CSI heatmap
const heatmap = csiSimulator.getHeatmapData();
renderer.drawCsiHeatmap(csiCtx, heatmap, csiCanvas.width, csiCanvas.height);
latency.csi = performance.now() - t0;
}
// --- Fusion ---
const t0f = performance.now();
const fusedEmb = fusionEngine.fuse(videoEmb, csiEmb, mode);
latency.fusion = performance.now() - t0f;
// --- Pose Decode ---
// For CSI-only mode, generate a synthetic motion region from CSI energy
if (mode === 'csi' && (!motionRegion || !motionRegion.detected)) {
const csiPresence = csiSimulator.personPresence;
if (csiPresence > 0.1) {
motionRegion = {
detected: true,
x: 0.25, y: 0.15, w: 0.5, h: 0.7,
coverage: csiPresence,
motionGrid: null,
gridCols: 10,
gridRows: 8
};
}
}
// CSI state for through-wall tracking
const csiState = {
csiPresence: csiSimulator.personPresence,
isLive: csiSimulator.isLive
};
const keypoints = poseDecoder.decode(fusedEmb, motionRegion, elapsed, csiState);
// --- Render Skeleton ---
const labelMap = { dual: 'DUAL FUSION', video: 'VIDEO ONLY', csi: 'CSI ONLY' };
renderer.drawSkeleton(skeletonCtx, keypoints, skeletonCanvas.width, skeletonCanvas.height, {
minConfidence: confidenceThreshold,
color: mode === 'csi' ? 'amber' : 'green',
label: labelMap[mode]
});
// --- Render Embedding Space ---
const embPoints = fusionEngine.getEmbeddingPoints();
renderer.drawEmbeddingSpace(embeddingCtx, embPoints, embeddingCanvas.width, embeddingCanvas.height);
// --- Update UI ---
latency.total = performance.now() - totalStart;
// FPS
frameCount++;
if (timestamp - lastFpsTime > 500) {
fps = Math.round(frameCount * 1000 / (timestamp - lastFpsTime));
lastFpsTime = timestamp;
frameCount = 0;
fpsDisplay.textContent = `${fps} FPS`;
}
// Fusion bars
const vc = fusionEngine.videoConfidence;
const cc = fusionEngine.csiConfidence;
const fc = fusionEngine.fusedConfidence;
videoBar.style.width = `${vc * 100}%`;
csiBar.style.width = `${cc * 100}%`;
fusedBar.style.width = `${fc * 100}%`;
videoBarVal.textContent = `${Math.round(vc * 100)}%`;
csiBarVal.textContent = `${Math.round(cc * 100)}%`;
fusedBarVal.textContent = `${Math.round(fc * 100)}%`;
// Latency
latVideoEl.textContent = `${latency.video.toFixed(1)}ms`;
latCsiEl.textContent = `${latency.csi.toFixed(1)}ms`;
latFusionEl.textContent = `${latency.fusion.toFixed(1)}ms`;
latTotalEl.textContent = `${latency.total.toFixed(1)}ms`;
// Cross-modal similarity
const sim = fusionEngine.getCrossModalSimilarity();
crossModalEl.textContent = sim.toFixed(3);
}
// Boot
document.addEventListener('DOMContentLoaded', init);
+373
View File
@@ -0,0 +1,373 @@
/**
* PoseDecoder — Maps motion detection grid → 17 COCO keypoints.
*
* Uses per-cell motion intensity to track actual body part positions:
* - Head: top-center motion cluster
* - Shoulders/Elbows/Wrists: lateral motion in upper body zone
* - Hips/Knees/Ankles: lower body motion distribution
*
* When person exits frame, CSI data continues tracking (through-wall mode).
*/
// COCO keypoint definitions
export const KEYPOINT_NAMES = [
'nose', 'left_eye', 'right_eye', 'left_ear', 'right_ear',
'left_shoulder', 'right_shoulder', 'left_elbow', 'right_elbow',
'left_wrist', 'right_wrist', 'left_hip', 'right_hip',
'left_knee', 'right_knee', 'left_ankle', 'right_ankle'
];
// Skeleton connections (pairs of keypoint indices)
export const SKELETON_CONNECTIONS = [
[0, 1], [0, 2], [1, 3], [2, 4], // Head
[5, 6], // Shoulders
[5, 7], [7, 9], // Left arm
[6, 8], [8, 10], // Right arm
[5, 11], [6, 12], // Torso
[11, 12], // Hips
[11, 13], [13, 15], // Left leg
[12, 14], [14, 16], // Right leg
];
// Standard body proportions (relative to body height)
const PROPORTIONS = {
headToShoulder: 0.15,
shoulderWidth: 0.25,
shoulderToElbow: 0.18,
elbowToWrist: 0.16,
shoulderToHip: 0.30,
hipWidth: 0.18,
hipToKnee: 0.24,
kneeToAnkle: 0.24,
eyeSpacing: 0.04,
earSpacing: 0.07,
};
export class PoseDecoder {
constructor(embeddingDim = 128) {
this.embeddingDim = embeddingDim;
this.smoothedKeypoints = null;
this.smoothingFactor = 0.45; // Lower = more responsive to movement
this._time = 0;
// Through-wall tracking state
this._lastBodyState = null;
this._ghostState = null;
this._ghostConfidence = 0;
this._ghostVelocity = { x: 0, y: 0 };
// Arm tracking history (smoothed positions)
this._leftArmY = 0.5;
this._rightArmY = 0.5;
this._leftArmX = 0;
this._rightArmX = 0;
this._headOffsetX = 0;
}
/**
* Decode motion data into 17 keypoints
* @param {Float32Array} embedding - Fused embedding vector
* @param {{ detected, x, y, w, h, motionGrid, gridCols, gridRows, motionCx, motionCy, exitDirection }} motionRegion
* @param {number} elapsed - Time in seconds
* @param {{ csiPresence: number }} csiState - CSI sensing state for through-wall
* @returns {Array<{x: number, y: number, confidence: number, name: string}>}
*/
decode(embedding, motionRegion, elapsed, csiState = {}) {
this._time = elapsed;
const hasMotion = motionRegion && motionRegion.detected;
const hasCsi = csiState && csiState.csiPresence > 0.1;
if (hasMotion) {
// Active tracking from video motion grid
this._ghostConfidence = 0;
const rawKeypoints = this._trackFromMotionGrid(motionRegion, embedding, elapsed);
this._lastBodyState = { keypoints: rawKeypoints.map(kp => ({...kp})), time: elapsed };
// Track exit velocity
if (motionRegion.exitDirection) {
const speed = 0.008;
this._ghostVelocity = {
x: motionRegion.exitDirection === 'left' ? -speed : motionRegion.exitDirection === 'right' ? speed : 0,
y: motionRegion.exitDirection === 'up' ? -speed : motionRegion.exitDirection === 'down' ? speed : 0
};
}
// Apply temporal smoothing
if (this.smoothedKeypoints && this.smoothedKeypoints.length === rawKeypoints.length) {
const alpha = this.smoothingFactor;
for (let i = 0; i < rawKeypoints.length; i++) {
rawKeypoints[i].x = alpha * this.smoothedKeypoints[i].x + (1 - alpha) * rawKeypoints[i].x;
rawKeypoints[i].y = alpha * this.smoothedKeypoints[i].y + (1 - alpha) * rawKeypoints[i].y;
}
}
this.smoothedKeypoints = rawKeypoints;
return rawKeypoints;
} else if (this._lastBodyState && (hasCsi || this._ghostConfidence > 0.05)) {
// Through-wall mode: person left frame but CSI still senses them
return this._trackThroughWall(elapsed, csiState);
} else if (this.smoothedKeypoints) {
// Fade out
const faded = this.smoothedKeypoints.map(kp => ({
...kp,
confidence: kp.confidence * 0.88
})).filter(kp => kp.confidence > 0.05);
if (faded.length === 0) this.smoothedKeypoints = null;
else this.smoothedKeypoints = faded;
return faded;
}
return [];
}
/**
* Track body parts from the motion grid.
* The grid tells us WHERE motion is happening → we map that to joint positions.
*/
_trackFromMotionGrid(region, embedding, elapsed) {
const grid = region.motionGrid;
const cols = region.gridCols || 10;
const rows = region.gridRows || 8;
// Body bounding box
const cx = region.x + region.w / 2;
const cy = region.y + region.h / 2;
const bodyH = Math.max(region.h, 0.3);
const bodyW = Math.max(region.w, 0.15);
// Analyze the motion grid to find arm positions
// Divide body into zones: head (top 20%), arms (top 60% sides), torso (center), legs (bottom 40%)
if (grid) {
const armAnalysis = this._analyzeArmMotion(grid, cols, rows, region);
// Smooth arm tracking
this._leftArmY = 0.6 * this._leftArmY + 0.4 * armAnalysis.leftArmHeight;
this._rightArmY = 0.6 * this._rightArmY + 0.4 * armAnalysis.rightArmHeight;
this._leftArmX = 0.6 * this._leftArmX + 0.4 * armAnalysis.leftArmSpread;
this._rightArmX = 0.6 * this._rightArmX + 0.4 * armAnalysis.rightArmSpread;
this._headOffsetX = 0.7 * this._headOffsetX + 0.3 * armAnalysis.headOffsetX;
}
const P = PROPORTIONS;
const halfW = P.shoulderWidth * bodyH / 2;
const hipHalfW = P.hipWidth * bodyH / 2;
// Breathing (subtle)
const breathe = Math.sin(elapsed * 1.5) * 0.002;
// Core body positions from detection center
const hipY = cy + bodyH * 0.15;
const shoulderY = hipY - P.shoulderToHip * bodyH + breathe;
const headY = shoulderY - P.headToShoulder * bodyH;
const kneeY = hipY + P.hipToKnee * bodyH;
const ankleY = kneeY + P.kneeToAnkle * bodyH;
// HEAD follows motion centroid
const headX = cx + this._headOffsetX * bodyW * 0.3;
// ARM POSITIONS driven by motion grid analysis
// leftArmY: 0 = arm down at side, 1 = arm fully raised
// leftArmSpread: how far out the arm extends
const leftArmRaise = this._leftArmY; // 0-1
const rightArmRaise = this._rightArmY;
const leftSpread = 0.02 + this._leftArmX * 0.12;
const rightSpread = 0.02 + this._rightArmX * 0.12;
// Elbow: interpolate between "at side" and "raised"
const lElbowY = shoulderY + P.shoulderToElbow * bodyH * (1 - leftArmRaise * 0.9);
const rElbowY = shoulderY + P.shoulderToElbow * bodyH * (1 - rightArmRaise * 0.9);
const lElbowX = cx - halfW - leftSpread;
const rElbowX = cx + halfW + rightSpread;
// Wrist: extends further when raised
const lWristY = lElbowY + P.elbowToWrist * bodyH * (1 - leftArmRaise * 1.1);
const rWristY = rElbowY + P.elbowToWrist * bodyH * (1 - rightArmRaise * 1.1);
const lWristX = lElbowX - leftSpread * 0.6;
const rWristX = rElbowX + rightSpread * 0.6;
// Leg motion from lower grid cells
const legMotion = grid ? this._analyzeLegMotion(grid, cols, rows) : { left: 0, right: 0 };
const legSwing = 0.015;
const keypoints = [
// 0: nose
{ x: headX, y: headY + 0.01, confidence: 0.92 },
// 1: left_eye
{ x: headX - P.eyeSpacing * bodyH, y: headY - 0.005, confidence: 0.88 },
// 2: right_eye
{ x: headX + P.eyeSpacing * bodyH, y: headY - 0.005, confidence: 0.88 },
// 3: left_ear
{ x: headX - P.earSpacing * bodyH, y: headY + 0.005, confidence: 0.72 },
// 4: right_ear
{ x: headX + P.earSpacing * bodyH, y: headY + 0.005, confidence: 0.72 },
// 5: left_shoulder
{ x: cx - halfW, y: shoulderY, confidence: 0.94 },
// 6: right_shoulder
{ x: cx + halfW, y: shoulderY, confidence: 0.94 },
// 7: left_elbow
{ x: lElbowX, y: lElbowY, confidence: 0.87 },
// 8: right_elbow
{ x: rElbowX, y: rElbowY, confidence: 0.87 },
// 9: left_wrist
{ x: lWristX, y: lWristY, confidence: 0.82 },
// 10: right_wrist
{ x: rWristX, y: rWristY, confidence: 0.82 },
// 11: left_hip
{ x: cx - hipHalfW, y: hipY, confidence: 0.91 },
// 12: right_hip
{ x: cx + hipHalfW, y: hipY, confidence: 0.91 },
// 13: left_knee
{ x: cx - hipHalfW + legMotion.left * legSwing, y: kneeY, confidence: 0.88 },
// 14: right_knee
{ x: cx + hipHalfW + legMotion.right * legSwing, y: kneeY, confidence: 0.88 },
// 15: left_ankle
{ x: cx - hipHalfW + legMotion.left * legSwing * 1.3, y: ankleY, confidence: 0.83 },
// 16: right_ankle
{ x: cx + hipHalfW + legMotion.right * legSwing * 1.3, y: ankleY, confidence: 0.83 },
];
for (let i = 0; i < keypoints.length; i++) {
keypoints[i].name = KEYPOINT_NAMES[i];
}
return keypoints;
}
/**
* Analyze the motion grid to determine arm positions.
* Left side of grid = left side of body, etc.
*/
_analyzeArmMotion(grid, cols, rows, region) {
// Body center column
const centerCol = Math.floor(cols / 2);
// Upper body rows (top 60% of detected region)
const upperEnd = Math.floor(rows * 0.6);
// Compute motion intensity for left vs right, at different heights
let leftUpperMotion = 0, leftMidMotion = 0;
let rightUpperMotion = 0, rightMidMotion = 0;
let leftCount = 0, rightCount = 0;
let headMotionX = 0, headMotionWeight = 0;
for (let r = 0; r < upperEnd; r++) {
const heightWeight = 1.0 - (r / upperEnd) * 0.3; // Upper rows weighted more
// Head zone: top 25%, center 40% of width
if (r < Math.floor(rows * 0.25)) {
const headLeft = Math.floor(cols * 0.3);
const headRight = Math.floor(cols * 0.7);
for (let c = headLeft; c <= headRight; c++) {
const val = grid[r][c];
headMotionX += (c / cols - 0.5) * val;
headMotionWeight += val;
}
}
// Left arm zone: left 40% of grid
for (let c = 0; c < Math.floor(cols * 0.4); c++) {
const val = grid[r][c];
if (r < rows * 0.3) leftUpperMotion += val * heightWeight;
else leftMidMotion += val * heightWeight;
leftCount++;
}
// Right arm zone: right 40% of grid
for (let c = Math.floor(cols * 0.6); c < cols; c++) {
const val = grid[r][c];
if (r < rows * 0.3) rightUpperMotion += val * heightWeight;
else rightMidMotion += val * heightWeight;
rightCount++;
}
}
// Normalize
const leftTotal = leftUpperMotion + leftMidMotion;
const rightTotal = rightUpperMotion + rightMidMotion;
const maxMotion = 0.15; // Calibration threshold
// Arm height: 0 = at side, 1 = raised
// High motion in upper-left → left arm is raised
const leftArmHeight = Math.min(1, (leftUpperMotion / maxMotion) * 2);
const rightArmHeight = Math.min(1, (rightUpperMotion / maxMotion) * 2);
// Arm spread: how far out from body
const leftArmSpread = Math.min(1, leftTotal / maxMotion);
const rightArmSpread = Math.min(1, rightTotal / maxMotion);
// Head offset
const headOffsetX = headMotionWeight > 0.01 ? headMotionX / headMotionWeight : 0;
return { leftArmHeight, rightArmHeight, leftArmSpread, rightArmSpread, headOffsetX };
}
/**
* Analyze lower grid for leg motion.
*/
_analyzeLegMotion(grid, cols, rows) {
const lowerStart = Math.floor(rows * 0.6);
let leftMotion = 0, rightMotion = 0;
for (let r = lowerStart; r < rows; r++) {
for (let c = 0; c < Math.floor(cols / 2); c++) {
leftMotion += grid[r][c];
}
for (let c = Math.floor(cols / 2); c < cols; c++) {
rightMotion += grid[r][c];
}
}
// Return as -1 to 1 range (asymmetry indicates which leg is moving)
const total = leftMotion + rightMotion + 0.001;
return {
left: (leftMotion - rightMotion) / total,
right: (rightMotion - leftMotion) / total
};
}
/**
* Through-wall tracking: continue showing pose via CSI when person left video frame.
* The skeleton drifts in the exit direction with decreasing confidence.
*/
_trackThroughWall(elapsed, csiState) {
if (!this._lastBodyState) return [];
const dt = elapsed - this._lastBodyState.time;
const csiPresence = csiState.csiPresence || 0;
// Initialize ghost on first call
if (this._ghostConfidence <= 0.05) {
this._ghostConfidence = 0.8;
this._ghostState = this._lastBodyState.keypoints.map(kp => ({...kp}));
}
// Ghost confidence decays, but CSI presence sustains it
const csiBoost = Math.min(0.7, csiPresence * 0.8);
this._ghostConfidence = Math.max(0.05, this._ghostConfidence * 0.995 - 0.001 + csiBoost * 0.002);
// Drift the ghost in exit direction
const vx = this._ghostVelocity.x;
const vy = this._ghostVelocity.y;
// Breathing continues via CSI
const breathe = Math.sin(elapsed * 1.5) * 0.003 * csiPresence;
const keypoints = this._ghostState.map((kp, i) => {
return {
x: kp.x + vx * dt * 0.3,
y: kp.y + vy * dt * 0.3 + (i >= 5 && i <= 6 ? breathe : 0),
confidence: kp.confidence * this._ghostConfidence * (0.5 + csiPresence * 0.5),
name: kp.name
};
});
// Slow down drift over time
this._ghostVelocity.x *= 0.998;
this._ghostVelocity.y *= 0.998;
this.smoothedKeypoints = keypoints;
return keypoints;
}
}
+235
View File
@@ -0,0 +1,235 @@
/**
* VideoCapture — getUserMedia webcam capture with frame extraction.
* Provides quality metrics (brightness, motion) for fusion confidence gating.
*/
export class VideoCapture {
constructor(videoElement) {
this.video = videoElement;
this.stream = null;
this.offscreen = document.createElement('canvas');
this.offCtx = this.offscreen.getContext('2d', { willReadFrequently: true });
this.prevFrame = null;
this.motionScore = 0;
this.brightnessScore = 0;
}
async start(constraints = {}) {
const defaultConstraints = {
video: {
width: { ideal: 640 },
height: { ideal: 480 },
facingMode: 'user',
frameRate: { ideal: 30 }
},
audio: false
};
try {
this.stream = await navigator.mediaDevices.getUserMedia(
Object.keys(constraints).length ? constraints : defaultConstraints
);
this.video.srcObject = this.stream;
await this.video.play();
this.offscreen.width = this.video.videoWidth;
this.offscreen.height = this.video.videoHeight;
return true;
} catch (err) {
console.error('[Video] Camera access failed:', err.message);
return false;
}
}
stop() {
if (this.stream) {
this.stream.getTracks().forEach(t => t.stop());
this.stream = null;
}
this.video.srcObject = null;
}
get isActive() {
return this.stream !== null && this.video.readyState >= 2;
}
get width() { return this.video.videoWidth || 640; }
get height() { return this.video.videoHeight || 480; }
/**
* Capture current frame as RGB Uint8Array + compute quality metrics.
* @param {number} targetW - Target width for CNN input
* @param {number} targetH - Target height for CNN input
* @returns {{ rgb: Uint8Array, width: number, height: number, motion: number, brightness: number }}
*/
captureFrame(targetW = 56, targetH = 56) {
if (!this.isActive) return null;
// Draw to offscreen at target resolution
this.offscreen.width = targetW;
this.offscreen.height = targetH;
this.offCtx.drawImage(this.video, 0, 0, targetW, targetH);
const imageData = this.offCtx.getImageData(0, 0, targetW, targetH);
const rgba = imageData.data;
// Convert RGBA → RGB
const pixels = targetW * targetH;
const rgb = new Uint8Array(pixels * 3);
let brightnessSum = 0;
let motionSum = 0;
for (let i = 0; i < pixels; i++) {
const r = rgba[i * 4];
const g = rgba[i * 4 + 1];
const b = rgba[i * 4 + 2];
rgb[i * 3] = r;
rgb[i * 3 + 1] = g;
rgb[i * 3 + 2] = b;
// Luminance for brightness
const lum = 0.299 * r + 0.587 * g + 0.114 * b;
brightnessSum += lum;
// Motion: diff from previous frame
if (this.prevFrame) {
const pr = this.prevFrame[i * 3];
const pg = this.prevFrame[i * 3 + 1];
const pb = this.prevFrame[i * 3 + 2];
motionSum += Math.abs(r - pr) + Math.abs(g - pg) + Math.abs(b - pb);
}
}
this.brightnessScore = brightnessSum / (pixels * 255);
this.motionScore = this.prevFrame ? Math.min(1, motionSum / (pixels * 100)) : 0;
this.prevFrame = new Uint8Array(rgb);
return {
rgb,
width: targetW,
height: targetH,
motion: this.motionScore,
brightness: this.brightnessScore
};
}
/**
* Capture full-resolution RGBA for overlay rendering
* @returns {ImageData|null}
*/
captureFullFrame() {
if (!this.isActive) return null;
this.offscreen.width = this.width;
this.offscreen.height = this.height;
this.offCtx.drawImage(this.video, 0, 0);
return this.offCtx.getImageData(0, 0, this.width, this.height);
}
/**
* Detect motion region + detailed motion grid for body-part tracking.
* Returns bounding box + a grid showing WHERE motion is concentrated.
* @returns {{ x, y, w, h, detected: boolean, motionGrid: number[][], gridCols: number, gridRows: number, exitDirection: string|null }}
*/
detectMotionRegion(targetW = 56, targetH = 56) {
if (!this.isActive || !this.prevFrame) return { detected: false, motionGrid: null };
this.offscreen.width = targetW;
this.offscreen.height = targetH;
this.offCtx.drawImage(this.video, 0, 0, targetW, targetH);
const rgba = this.offCtx.getImageData(0, 0, targetW, targetH).data;
let minX = targetW, minY = targetH, maxX = 0, maxY = 0;
let motionPixels = 0;
const threshold = 25;
// Motion grid: divide frame into cells and track motion intensity per cell
const gridCols = 10;
const gridRows = 8;
const cellW = targetW / gridCols;
const cellH = targetH / gridRows;
const motionGrid = Array.from({ length: gridRows }, () => new Float32Array(gridCols));
const cellPixels = cellW * cellH;
// Also track motion centroid weighted by intensity
let motionCxSum = 0, motionCySum = 0, motionWeightSum = 0;
for (let y = 0; y < targetH; y++) {
for (let x = 0; x < targetW; x++) {
const i = y * targetW + x;
const r = rgba[i * 4], g = rgba[i * 4 + 1], b = rgba[i * 4 + 2];
const pr = this.prevFrame[i * 3], pg = this.prevFrame[i * 3 + 1], pb = this.prevFrame[i * 3 + 2];
const diff = Math.abs(r - pr) + Math.abs(g - pg) + Math.abs(b - pb);
if (diff > threshold * 3) {
motionPixels++;
if (x < minX) minX = x;
if (y < minY) minY = y;
if (x > maxX) maxX = x;
if (y > maxY) maxY = y;
}
// Accumulate per-cell motion intensity
const gc = Math.min(Math.floor(x / cellW), gridCols - 1);
const gr = Math.min(Math.floor(y / cellH), gridRows - 1);
const intensity = diff / (3 * 255); // Normalize 0-1
motionGrid[gr][gc] += intensity / cellPixels;
// Weighted centroid
if (diff > threshold) {
motionCxSum += x * diff;
motionCySum += y * diff;
motionWeightSum += diff;
}
}
}
const detected = motionPixels > (targetW * targetH * 0.02);
// Motion centroid (normalized 0-1)
const motionCx = motionWeightSum > 0 ? motionCxSum / (motionWeightSum * targetW) : 0.5;
const motionCy = motionWeightSum > 0 ? motionCySum / (motionWeightSum * targetH) : 0.5;
// Detect exit direction: if centroid is near edges
let exitDirection = null;
if (detected && motionCx < 0.1) exitDirection = 'left';
else if (detected && motionCx > 0.9) exitDirection = 'right';
else if (detected && motionCy < 0.1) exitDirection = 'up';
else if (detected && motionCy > 0.9) exitDirection = 'down';
// Track last known position for through-wall persistence
if (detected) {
this._lastDetected = {
x: minX / targetW,
y: minY / targetH,
w: (maxX - minX) / targetW,
h: (maxY - minY) / targetH,
cx: motionCx,
cy: motionCy,
exitDirection,
time: performance.now()
};
}
return {
detected,
x: minX / targetW,
y: minY / targetH,
w: (maxX - minX) / targetW,
h: (maxY - minY) / targetH,
coverage: motionPixels / (targetW * targetH),
motionGrid,
gridCols,
gridRows,
motionCx,
motionCy,
exitDirection
};
}
/**
* Get the last known detection info (for through-wall persistence)
*/
get lastDetection() {
return this._lastDetected || null;
}
}
@@ -0,0 +1,26 @@
{
"name": "ruvector-cnn-wasm",
"type": "module",
"description": "WASM bindings for ruvector-cnn - CNN feature extraction for image embeddings",
"version": "0.1.0",
"license": "MIT OR Apache-2.0",
"repository": {
"type": "git",
"url": "https://github.com/ruvnet/ruvector"
},
"files": [
"ruvector_cnn_wasm_bg.wasm",
"ruvector_cnn_wasm.js"
],
"main": "ruvector_cnn_wasm.js",
"sideEffects": [
"./snippets/*"
],
"keywords": [
"cnn",
"embeddings",
"wasm",
"simd",
"machine-learning"
]
}
@@ -0,0 +1,802 @@
/**
* Configuration for CNN embedder
*/
export class EmbedderConfig {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
EmbedderConfigFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_embedderconfig_free(ptr, 0);
}
constructor() {
const ret = wasm.embedderconfig_new();
this.__wbg_ptr = ret >>> 0;
EmbedderConfigFinalization.register(this, this.__wbg_ptr, this);
return this;
}
/**
* Output embedding dimension
* @returns {number}
*/
get embedding_dim() {
const ret = wasm.__wbg_get_embedderconfig_embedding_dim(this.__wbg_ptr);
return ret >>> 0;
}
/**
* Input image size (square)
* @returns {number}
*/
get input_size() {
const ret = wasm.__wbg_get_embedderconfig_input_size(this.__wbg_ptr);
return ret >>> 0;
}
/**
* Whether to L2 normalize embeddings
* @returns {boolean}
*/
get normalize() {
const ret = wasm.__wbg_get_embedderconfig_normalize(this.__wbg_ptr);
return ret !== 0;
}
/**
* Output embedding dimension
* @param {number} arg0
*/
set embedding_dim(arg0) {
wasm.__wbg_set_embedderconfig_embedding_dim(this.__wbg_ptr, arg0);
}
/**
* Input image size (square)
* @param {number} arg0
*/
set input_size(arg0) {
wasm.__wbg_set_embedderconfig_input_size(this.__wbg_ptr, arg0);
}
/**
* Whether to L2 normalize embeddings
* @param {boolean} arg0
*/
set normalize(arg0) {
wasm.__wbg_set_embedderconfig_normalize(this.__wbg_ptr, arg0);
}
}
if (Symbol.dispose) EmbedderConfig.prototype[Symbol.dispose] = EmbedderConfig.prototype.free;
/**
* Layer operations for building custom networks
*/
export class LayerOps {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
LayerOpsFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_layerops_free(ptr, 0);
}
/**
* Apply batch normalization (returns new array)
* @param {Float32Array} input
* @param {Float32Array} gamma
* @param {Float32Array} beta
* @param {Float32Array} mean
* @param {Float32Array} _var
* @param {number} epsilon
* @returns {Float32Array}
*/
static batch_norm(input, gamma, beta, mean, _var, epsilon) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(input, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
const ptr1 = passArrayF32ToWasm0(gamma, wasm.__wbindgen_export2);
const len1 = WASM_VECTOR_LEN;
const ptr2 = passArrayF32ToWasm0(beta, wasm.__wbindgen_export2);
const len2 = WASM_VECTOR_LEN;
const ptr3 = passArrayF32ToWasm0(mean, wasm.__wbindgen_export2);
const len3 = WASM_VECTOR_LEN;
const ptr4 = passArrayF32ToWasm0(_var, wasm.__wbindgen_export2);
const len4 = WASM_VECTOR_LEN;
wasm.layerops_batch_norm(retptr, ptr0, len0, ptr1, len1, ptr2, len2, ptr3, len3, ptr4, len4, epsilon);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var v6 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v6;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Apply global average pooling
* Returns one value per channel
* @param {Float32Array} input
* @param {number} height
* @param {number} width
* @param {number} channels
* @returns {Float32Array}
*/
static global_avg_pool(input, height, width, channels) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(input, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.layerops_global_avg_pool(retptr, ptr0, len0, height, width, channels);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var v2 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v2;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
}
if (Symbol.dispose) LayerOps.prototype[Symbol.dispose] = LayerOps.prototype.free;
/**
* SIMD-optimized operations
*/
export class SimdOps {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
SimdOpsFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_simdops_free(ptr, 0);
}
/**
* Dot product of two vectors
* @param {Float32Array} a
* @param {Float32Array} b
* @returns {number}
*/
static dot_product(a, b) {
const ptr0 = passArrayF32ToWasm0(a, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
const ptr1 = passArrayF32ToWasm0(b, wasm.__wbindgen_export2);
const len1 = WASM_VECTOR_LEN;
const ret = wasm.simdops_dot_product(ptr0, len0, ptr1, len1);
return ret;
}
/**
* L2 normalize a vector (returns new array)
* @param {Float32Array} data
* @returns {Float32Array}
*/
static l2_normalize(data) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(data, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.simdops_l2_normalize(retptr, ptr0, len0);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var v2 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v2;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* ReLU activation (returns new array)
* @param {Float32Array} data
* @returns {Float32Array}
*/
static relu(data) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(data, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.simdops_relu(retptr, ptr0, len0);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var v2 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v2;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* ReLU6 activation (returns new array)
* @param {Float32Array} data
* @returns {Float32Array}
*/
static relu6(data) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(data, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.simdops_relu6(retptr, ptr0, len0);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var v2 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v2;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
}
if (Symbol.dispose) SimdOps.prototype[Symbol.dispose] = SimdOps.prototype.free;
/**
* WASM CNN Embedder for image feature extraction
*/
export class WasmCnnEmbedder {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
WasmCnnEmbedderFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_wasmcnnembedder_free(ptr, 0);
}
/**
* Compute cosine similarity between two embeddings
* @param {Float32Array} a
* @param {Float32Array} b
* @returns {number}
*/
cosine_similarity(a, b) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(a, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
const ptr1 = passArrayF32ToWasm0(b, wasm.__wbindgen_export2);
const len1 = WASM_VECTOR_LEN;
wasm.wasmcnnembedder_cosine_similarity(retptr, this.__wbg_ptr, ptr0, len0, ptr1, len1);
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
if (r2) {
throw takeObject(r1);
}
return r0;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Get the embedding dimension
* @returns {number}
*/
get embedding_dim() {
const ret = wasm.wasmcnnembedder_embedding_dim(this.__wbg_ptr);
return ret >>> 0;
}
/**
* Extract embedding from image data (RGB format, row-major)
* @param {Uint8Array} image_data
* @param {number} width
* @param {number} height
* @returns {Float32Array}
*/
extract(image_data, width, height) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArray8ToWasm0(image_data, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.wasmcnnembedder_extract(retptr, this.__wbg_ptr, ptr0, len0, width, height);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
var r3 = getDataViewMemory0().getInt32(retptr + 4 * 3, true);
if (r3) {
throw takeObject(r2);
}
var v2 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v2;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Create a new CNN embedder
* @param {EmbedderConfig | null} [config]
*/
constructor(config) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
let ptr0 = 0;
if (!isLikeNone(config)) {
_assertClass(config, EmbedderConfig);
ptr0 = config.__destroy_into_raw();
}
wasm.wasmcnnembedder_new(retptr, ptr0);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
if (r2) {
throw takeObject(r1);
}
this.__wbg_ptr = r0 >>> 0;
WasmCnnEmbedderFinalization.register(this, this.__wbg_ptr, this);
return this;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
}
if (Symbol.dispose) WasmCnnEmbedder.prototype[Symbol.dispose] = WasmCnnEmbedder.prototype.free;
/**
* InfoNCE loss for contrastive learning (SimCLR style)
*/
export class WasmInfoNCELoss {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
WasmInfoNCELossFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_wasminfonceloss_free(ptr, 0);
}
/**
* Compute loss for a batch of embedding pairs
* embeddings: [2N, D] flattened where (i, i+N) are positive pairs
* @param {Float32Array} embeddings
* @param {number} batch_size
* @param {number} dim
* @returns {number}
*/
forward(embeddings, batch_size, dim) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(embeddings, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.wasminfonceloss_forward(retptr, this.__wbg_ptr, ptr0, len0, batch_size, dim);
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
if (r2) {
throw takeObject(r1);
}
return r0;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Create new InfoNCE loss with temperature parameter
* @param {number} temperature
*/
constructor(temperature) {
const ret = wasm.wasminfonceloss_new(temperature);
this.__wbg_ptr = ret >>> 0;
WasmInfoNCELossFinalization.register(this, this.__wbg_ptr, this);
return this;
}
/**
* Get the temperature parameter
* @returns {number}
*/
get temperature() {
const ret = wasm.wasminfonceloss_temperature(this.__wbg_ptr);
return ret;
}
}
if (Symbol.dispose) WasmInfoNCELoss.prototype[Symbol.dispose] = WasmInfoNCELoss.prototype.free;
/**
* Triplet loss for metric learning
*/
export class WasmTripletLoss {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
WasmTripletLossFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_wasmtripletloss_free(ptr, 0);
}
/**
* Compute loss for a batch of triplets
* @param {Float32Array} anchors
* @param {Float32Array} positives
* @param {Float32Array} negatives
* @param {number} dim
* @returns {number}
*/
forward(anchors, positives, negatives, dim) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(anchors, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
const ptr1 = passArrayF32ToWasm0(positives, wasm.__wbindgen_export2);
const len1 = WASM_VECTOR_LEN;
const ptr2 = passArrayF32ToWasm0(negatives, wasm.__wbindgen_export2);
const len2 = WASM_VECTOR_LEN;
wasm.wasmtripletloss_forward(retptr, this.__wbg_ptr, ptr0, len0, ptr1, len1, ptr2, len2, dim);
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
if (r2) {
throw takeObject(r1);
}
return r0;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Compute loss for a single triplet
* @param {Float32Array} anchor
* @param {Float32Array} positive
* @param {Float32Array} negative
* @returns {number}
*/
forward_single(anchor, positive, negative) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(anchor, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
const ptr1 = passArrayF32ToWasm0(positive, wasm.__wbindgen_export2);
const len1 = WASM_VECTOR_LEN;
const ptr2 = passArrayF32ToWasm0(negative, wasm.__wbindgen_export2);
const len2 = WASM_VECTOR_LEN;
wasm.wasmtripletloss_forward_single(retptr, this.__wbg_ptr, ptr0, len0, ptr1, len1, ptr2, len2);
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
if (r2) {
throw takeObject(r1);
}
return r0;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Get the margin parameter
* @returns {number}
*/
get margin() {
const ret = wasm.wasmtripletloss_margin(this.__wbg_ptr);
return ret;
}
/**
* Create new triplet loss with margin
* @param {number} margin
*/
constructor(margin) {
const ret = wasm.wasmtripletloss_new(margin);
this.__wbg_ptr = ret >>> 0;
WasmTripletLossFinalization.register(this, this.__wbg_ptr, this);
return this;
}
}
if (Symbol.dispose) WasmTripletLoss.prototype[Symbol.dispose] = WasmTripletLoss.prototype.free;
/**
* Initialize panic hook for better error messages
*/
export function init() {
wasm.init();
}
function __wbg_get_imports() {
const import0 = {
__proto__: null,
__wbg___wbindgen_throw_39bc967c0e5a9b58: function(arg0, arg1) {
throw new Error(getStringFromWasm0(arg0, arg1));
},
__wbg_error_a6fa202b58aa1cd3: function(arg0, arg1) {
let deferred0_0;
let deferred0_1;
try {
deferred0_0 = arg0;
deferred0_1 = arg1;
console.error(getStringFromWasm0(arg0, arg1));
} finally {
wasm.__wbindgen_export(deferred0_0, deferred0_1, 1);
}
},
__wbg_new_227d7c05414eb861: function() {
const ret = new Error();
return addHeapObject(ret);
},
__wbg_stack_3b0d974bbf31e44f: function(arg0, arg1) {
const ret = getObject(arg1).stack;
const ptr1 = passStringToWasm0(ret, wasm.__wbindgen_export2, wasm.__wbindgen_export3);
const len1 = WASM_VECTOR_LEN;
getDataViewMemory0().setInt32(arg0 + 4 * 1, len1, true);
getDataViewMemory0().setInt32(arg0 + 4 * 0, ptr1, true);
},
__wbindgen_cast_0000000000000001: function(arg0, arg1) {
// Cast intrinsic for `Ref(String) -> Externref`.
const ret = getStringFromWasm0(arg0, arg1);
return addHeapObject(ret);
},
__wbindgen_object_drop_ref: function(arg0) {
takeObject(arg0);
},
};
return {
__proto__: null,
"./ruvector_cnn_wasm_bg.js": import0,
};
}
const EmbedderConfigFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_embedderconfig_free(ptr >>> 0, 1));
const LayerOpsFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_layerops_free(ptr >>> 0, 1));
const SimdOpsFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_simdops_free(ptr >>> 0, 1));
const WasmCnnEmbedderFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_wasmcnnembedder_free(ptr >>> 0, 1));
const WasmInfoNCELossFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_wasminfonceloss_free(ptr >>> 0, 1));
const WasmTripletLossFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_wasmtripletloss_free(ptr >>> 0, 1));
function addHeapObject(obj) {
if (heap_next === heap.length) heap.push(heap.length + 1);
const idx = heap_next;
heap_next = heap[idx];
heap[idx] = obj;
return idx;
}
function _assertClass(instance, klass) {
if (!(instance instanceof klass)) {
throw new Error(`expected instance of ${klass.name}`);
}
}
function dropObject(idx) {
if (idx < 1028) return;
heap[idx] = heap_next;
heap_next = idx;
}
function getArrayF32FromWasm0(ptr, len) {
ptr = ptr >>> 0;
return getFloat32ArrayMemory0().subarray(ptr / 4, ptr / 4 + len);
}
let cachedDataViewMemory0 = null;
function getDataViewMemory0() {
if (cachedDataViewMemory0 === null || cachedDataViewMemory0.buffer.detached === true || (cachedDataViewMemory0.buffer.detached === undefined && cachedDataViewMemory0.buffer !== wasm.memory.buffer)) {
cachedDataViewMemory0 = new DataView(wasm.memory.buffer);
}
return cachedDataViewMemory0;
}
let cachedFloat32ArrayMemory0 = null;
function getFloat32ArrayMemory0() {
if (cachedFloat32ArrayMemory0 === null || cachedFloat32ArrayMemory0.byteLength === 0) {
cachedFloat32ArrayMemory0 = new Float32Array(wasm.memory.buffer);
}
return cachedFloat32ArrayMemory0;
}
function getStringFromWasm0(ptr, len) {
ptr = ptr >>> 0;
return decodeText(ptr, len);
}
let cachedUint8ArrayMemory0 = null;
function getUint8ArrayMemory0() {
if (cachedUint8ArrayMemory0 === null || cachedUint8ArrayMemory0.byteLength === 0) {
cachedUint8ArrayMemory0 = new Uint8Array(wasm.memory.buffer);
}
return cachedUint8ArrayMemory0;
}
function getObject(idx) { return heap[idx]; }
let heap = new Array(1024).fill(undefined);
heap.push(undefined, null, true, false);
let heap_next = heap.length;
function isLikeNone(x) {
return x === undefined || x === null;
}
function passArray8ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 1, 1) >>> 0;
getUint8ArrayMemory0().set(arg, ptr / 1);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
function passArrayF32ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 4, 4) >>> 0;
getFloat32ArrayMemory0().set(arg, ptr / 4);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
function passStringToWasm0(arg, malloc, realloc) {
if (realloc === undefined) {
const buf = cachedTextEncoder.encode(arg);
const ptr = malloc(buf.length, 1) >>> 0;
getUint8ArrayMemory0().subarray(ptr, ptr + buf.length).set(buf);
WASM_VECTOR_LEN = buf.length;
return ptr;
}
let len = arg.length;
let ptr = malloc(len, 1) >>> 0;
const mem = getUint8ArrayMemory0();
let offset = 0;
for (; offset < len; offset++) {
const code = arg.charCodeAt(offset);
if (code > 0x7F) break;
mem[ptr + offset] = code;
}
if (offset !== len) {
if (offset !== 0) {
arg = arg.slice(offset);
}
ptr = realloc(ptr, len, len = offset + arg.length * 3, 1) >>> 0;
const view = getUint8ArrayMemory0().subarray(ptr + offset, ptr + len);
const ret = cachedTextEncoder.encodeInto(arg, view);
offset += ret.written;
ptr = realloc(ptr, len, offset, 1) >>> 0;
}
WASM_VECTOR_LEN = offset;
return ptr;
}
function takeObject(idx) {
const ret = getObject(idx);
dropObject(idx);
return ret;
}
let cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });
cachedTextDecoder.decode();
const MAX_SAFARI_DECODE_BYTES = 2146435072;
let numBytesDecoded = 0;
function decodeText(ptr, len) {
numBytesDecoded += len;
if (numBytesDecoded >= MAX_SAFARI_DECODE_BYTES) {
cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });
cachedTextDecoder.decode();
numBytesDecoded = len;
}
return cachedTextDecoder.decode(getUint8ArrayMemory0().subarray(ptr, ptr + len));
}
const cachedTextEncoder = new TextEncoder();
if (!('encodeInto' in cachedTextEncoder)) {
cachedTextEncoder.encodeInto = function (arg, view) {
const buf = cachedTextEncoder.encode(arg);
view.set(buf);
return {
read: arg.length,
written: buf.length
};
};
}
let WASM_VECTOR_LEN = 0;
let wasmModule, wasm;
function __wbg_finalize_init(instance, module) {
wasm = instance.exports;
wasmModule = module;
cachedDataViewMemory0 = null;
cachedFloat32ArrayMemory0 = null;
cachedUint8ArrayMemory0 = null;
wasm.__wbindgen_start();
return wasm;
}
async function __wbg_load(module, imports) {
if (typeof Response === 'function' && module instanceof Response) {
if (typeof WebAssembly.instantiateStreaming === 'function') {
try {
return await WebAssembly.instantiateStreaming(module, imports);
} catch (e) {
const validResponse = module.ok && expectedResponseType(module.type);
if (validResponse && module.headers.get('Content-Type') !== 'application/wasm') {
console.warn("`WebAssembly.instantiateStreaming` failed because your server does not serve Wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n", e);
} else { throw e; }
}
}
const bytes = await module.arrayBuffer();
return await WebAssembly.instantiate(bytes, imports);
} else {
const instance = await WebAssembly.instantiate(module, imports);
if (instance instanceof WebAssembly.Instance) {
return { instance, module };
} else {
return instance;
}
}
function expectedResponseType(type) {
switch (type) {
case 'basic': case 'cors': case 'default': return true;
}
return false;
}
}
function initSync(module) {
if (wasm !== undefined) return wasm;
if (module !== undefined) {
if (Object.getPrototypeOf(module) === Object.prototype) {
({module} = module)
} else {
console.warn('using deprecated parameters for `initSync()`; pass a single object instead')
}
}
const imports = __wbg_get_imports();
if (!(module instanceof WebAssembly.Module)) {
module = new WebAssembly.Module(module);
}
const instance = new WebAssembly.Instance(module, imports);
return __wbg_finalize_init(instance, module);
}
async function __wbg_init(module_or_path) {
if (wasm !== undefined) return wasm;
if (module_or_path !== undefined) {
if (Object.getPrototypeOf(module_or_path) === Object.prototype) {
({module_or_path} = module_or_path)
} else {
console.warn('using deprecated parameters for the initialization function; pass a single object instead')
}
}
if (module_or_path === undefined) {
module_or_path = new URL('ruvector_cnn_wasm_bg.wasm', import.meta.url);
}
const imports = __wbg_get_imports();
if (typeof module_or_path === 'string' || (typeof Request === 'function' && module_or_path instanceof Request) || (typeof URL === 'function' && module_or_path instanceof URL)) {
module_or_path = fetch(module_or_path);
}
const { instance, module } = await __wbg_load(await module_or_path, imports);
return __wbg_finalize_init(instance, module);
}
export { initSync, __wbg_init as default };