Compare commits

..

9 Commits

Author SHA1 Message Date
ruv 0223ef6d2e docs: add ADR-059 live ESP32 CSI pipeline + update README with demo links
- ADR-059: Documents end-to-end ESP32 → sensing server → browser pipeline
- README: Add dual-modal pose fusion demo link, update ADR count to 49
- References issue #245

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 17:40:16 -04:00
ruv 2f5e7ffb41 feat: live ESP32 CSI pipeline + auto-connect WebSocket
- Add auto-connect to local sensing server WebSocket (ws://localhost:8765)
- Demo shows "Live ESP32" when connected to real CSI data
- Add build_firmware.ps1 for native Windows ESP-IDF builds (no Docker)
- Add read_serial.ps1 for ESP32 serial monitor

Pipeline: ESP32 → UDP:5005 → sensing-server → WS:8765 → browser demo

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 17:37:27 -04:00
ruv 4ce8ffc465 fix: video fills available space + correct WASM path resolution
- Remove fixed aspect-ratio and max-height from video panel so it
  fills the available viewport space without scrolling
- Grid uses 1fr row for content area, overflow:hidden on main grid
- Fix WASM path: resolve relative to JS module file using import.meta.url
  instead of hardcoded ./pkg/ which resolved incorrectly on gh-pages
- Responsive: mobile still gets aspect-ratio constraint

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 16:16:07 -04:00
ruv 3be63a7589 fix: motion-responsive skeleton + through-wall CSI tracking
- Pose decoder now uses per-cell motion grid to track actual arm/head
  positions — raising arms moves the skeleton's arms, head follows
  lateral movement
- Motion grid (10x8 cells) tracks intensity per body zone: head,
  left/right arm upper/mid, legs
- Through-wall mode: when person exits frame, CSI maintains presence
  with slow decay (~10s) and skeleton drifts in exit direction
- CSI simulator persists sensing after video loss, ghost pose renders
  with decreasing confidence
- Reduced temporal smoothing (0.45) for faster response to movement

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 16:10:29 -04:00
ruv c4e640c812 feat: dual-modal WASM browser pose estimation demo (ADR-058)
Live webcam video + WiFi CSI fusion for real-time pose estimation.
Two parallel CNN pipelines (ruvector-cnn-wasm) with attention-weighted
fusion and dynamic confidence gating. Three modes: Dual, Video-only,
CSI-only. Includes pre-built WASM package (~52KB) for browser deployment.

- ADR-058: Dual-modal architecture design
- ui/pose-fusion.html: Main demo page with dark theme UI
- 7 JS modules: video-capture, csi-simulator, cnn-embedder, fusion-engine,
  pose-decoder, canvas-renderer, main orchestrator
- Pre-built ruvector-cnn-wasm WASM package for browser
- CSI heatmap, embedding space visualization, latency metrics
- WebSocket support for live ESP32 CSI data
- Navigation link added to main dashboard

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 14:26:39 -04:00
ruv 6e03a47867 docs: update user guide with v0.4.1 firmware release and CSI troubleshooting
- Add v0.4.1 to firmware release table as recommended stable release
- Update flash command with correct partition offsets (8MB, OTA)
- Add "CSI not enabled" troubleshooting entry
- Add warning about pre-v0.4.1 firmware CSI bug

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 13:49:20 -04:00
ruv 9d1140de2d docs: update README firmware release table with v0.4.1
Add v0.4.1-esp32 as the recommended stable release and update the
flash command to match the current partition layout.

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 13:49:20 -04:00
ruv 952f27a1ce fix(firmware): enable CSI in sdkconfig and add build guard (ADR-057)
The committed sdkconfig had CONFIG_ESP_WIFI_CSI_ENABLED disabled, causing
all builds to crash at runtime with "CSI not enabled in menuconfig".
Root cause: sdkconfig.defaults.template existed but ESP-IDF only reads
sdkconfig.defaults (no .template suffix).

Fixes:
- Add sdkconfig.defaults with CONFIG_ESP_WIFI_CSI_ENABLED=y
- Add #error compile guard in csi_collector.c to prevent recurrence
- Fix NVS encryption default (requires eFuse, breaks clean builds)

Verified: Docker build + flash to ESP32-S3 + CSI callbacks confirmed.

Closes #241
Relates to #223, #238, #234, #210, #190

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-12 13:49:20 -04:00
Reuven f7d043d727 docs: fix Docker commands to use CSI_SOURCE environment variable
The Docker image uses CSI_SOURCE env var to select the data source,
not command-line arguments appended after the image name.

Fixed:
- ESP32 mode examples now use -e CSI_SOURCE=esp32
- Training mode example now uses --entrypoint override
- Added CSI_SOURCE value table in Docker section

Fixes #226

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-03-10 12:16:06 -04:00
41 changed files with 3948 additions and 4030 deletions
+10 -4
View File
@@ -75,7 +75,7 @@ docker run -p 3000:3000 ruvnet/wifi-densepose:latest
|----------|-------------|
| [User Guide](docs/user-guide.md) | Step-by-step guide: installation, first run, API usage, hardware setup, training |
| [Build Guide](docs/build-guide.md) | Building from source (Rust and Python) |
| [Architecture Decisions](docs/adr/README.md) | 48 ADRs — why each technical choice was made, organized by domain (hardware, signal processing, ML, platform, infrastructure) |
| [Architecture Decisions](docs/adr/README.md) | 49 ADRs — why each technical choice was made, organized by domain (hardware, signal processing, ML, platform, infrastructure) |
| [Domain Models](docs/ddd/README.md) | 7 DDD models (RuvSense, Signal Processing, Training Pipeline, Hardware Platform, Sensing Server, WiFi-Mat, CHCI) — bounded contexts, aggregates, domain events, and ubiquitous language |
| [Desktop App](rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop/README.md) | **WIP** — Tauri v2 desktop app for node management, OTA updates, WASM deployment, and mesh visualization |
@@ -89,8 +89,12 @@ docker run -p 3000:3000 ruvnet/wifi-densepose:latest
<em>Real-time pose skeleton from WiFi CSI signals — no cameras, no wearables</em>
<br>
<a href="https://ruvnet.github.io/RuView/"><strong>▶ Live Observatory Demo</strong></a>
&nbsp;|&nbsp;
<a href="https://ruvnet.github.io/RuView/pose-fusion.html"><strong>▶ Dual-Modal Pose Fusion Demo</strong></a>
> The [server](#-quick-start) is optional for visualization and aggregation — the ESP32 [runs independently](#esp32-s3-hardware-pipeline) for presence detection, vital signs, and fall alerts.
>
> **Live ESP32 pipeline**: Connect an ESP32-S3 node → run the [sensing server](#sensing-server) → open the [pose fusion demo](https://ruvnet.github.io/RuView/pose-fusion.html) for real-time dual-modal pose estimation (webcam + WiFi CSI). See [ADR-059](docs/adr/ADR-059-live-esp32-csi-pipeline.md).
## 🚀 Key Features
@@ -1043,14 +1047,16 @@ Download a pre-built binary — no build toolchain needed:
| Release | What's included | Tag |
|---------|-----------------|-----|
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Stable — raw CSI streaming, multi-node TDM, channel hopping | `v0.2.0-esp32` |
| [v0.4.1](https://github.com/ruvnet/RuView/releases/tag/v0.4.1-esp32) | **Stable** — CSI build fix, compile guard, AMOLED display, edge intelligence ([ADR-057](docs/adr/ADR-057-firmware-csi-build-guard.md)) | `v0.4.1-esp32` |
| [v0.3.0-alpha](https://github.com/ruvnet/RuView/releases/tag/v0.3.0-alpha-esp32) | Alpha — adds on-device edge intelligence and WASM modules ([ADR-039](docs/adr/ADR-039-esp32-edge-intelligence.md), [ADR-040](docs/adr/ADR-040-wasm-programmable-sensing.md)) | `v0.3.0-alpha-esp32` |
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Raw CSI streaming, multi-node TDM, channel hopping | `v0.2.0-esp32` |
```bash
# 1. Flash the firmware to your ESP32-S3
python -m esptool --chip esp32s3 --port COM7 --baud 460800 \
write_flash --flash_mode dio --flash_size 8MB \
0x0 bootloader.bin 0x8000 partition-table.bin 0x10000 esp32-csi-node.bin
write_flash --flash-mode dio --flash-size 8MB --flash-freq 80m \
0x0 bootloader.bin 0x8000 partition-table.bin \
0xf000 ota_data_initial.bin 0x20000 esp32-csi-node.bin
# 2. Set WiFi credentials and server address (stored in flash, survives reboots)
python firmware/esp32-csi-node/provision.py --port COM7 \
@@ -1,240 +0,0 @@
# ADR-057: Desktop App Training & RuVector Integration
| Field | Value |
|-------|-------|
| Status | Proposed |
| Date | 2026-03-10 |
| Authors | RuView Team |
| Reviewers | - |
| Related | ADR-016, ADR-017, ADR-024, ADR-027 |
## Context
The RuView desktop application currently provides device discovery, firmware flashing, OTA updates, and real-time sensing visualization. However, users cannot train models or configure RuVector signal processing modules directly from the desktop app.
The following crates exist in the workspace but are not exposed in the desktop UI:
### Training Crate (`wifi-densepose-train`)
- Dataset management (MM-Fi, Wi-Pose formats)
- Model architectures (CSI encoder, pose decoder)
- Training loops with metrics tracking
- Checkpoint save/load
- ruview_metrics integration
### RuVector Crates (5 modules)
1. **ruvector-mincut** - Graph-based person segmentation, DynamicPersonMatcher
2. **ruvector-attn-mincut** - Attention-weighted antenna selection
3. **ruvector-temporal-tensor** - Temporal CSI compression, breathing detection
4. **ruvector-solver** - Sparse interpolation, triangulation
5. **ruvector-attention** - Spatial attention, BVP extraction
## Decision
Add a new **"Training"** page to the desktop application with tabbed navigation:
### Tab Structure
```
┌─────────────────────────────────────────────────────────────┐
│ Training & Models │
├──────────┬──────────┬──────────┬──────────┬────────────────┤
│ Datasets │ Models │ Training │ RuVector │ Metrics │
└──────────┴──────────┴──────────┴──────────┴────────────────┘
```
### Tab 1: Datasets
- **Download** standard datasets (MM-Fi, Wi-Pose)
- **Import** custom CSI recordings
- **Preview** dataset samples (CSI heatmaps, labels)
- **Split** into train/val/test sets
- **Statistics** - sample counts, class distribution
### Tab 2: Models
- **Browse** available architectures:
- CSI Encoder (CNN, Transformer)
- Pose Decoder (LSTM, GRU)
- AETHER embedding network (ADR-024)
- MERIDIAN domain adaptor (ADR-027)
- **Load** checkpoints from disk
- **View** model summary (params, layers, memory)
- **Export** to ONNX/TorchScript
### Tab 3: Training
- **Configure** training:
- Learning rate, batch size, epochs
- Optimizer (Adam, SGD, AdamW)
- Loss function selection
- Data augmentation toggles
- **GPU Detection** - CUDA/Metal availability
- **Start/Stop** training jobs
- **Progress** - live loss curves, ETA
- **Checkpointing** - auto-save best model
### Tab 4: RuVector
- **Module Configuration**:
- MinCut graph parameters
- Attention weights
- Temporal compression ratio
- Solver interpolation settings
- **Live Testing** - apply to real-time CSI stream
- **Comparison** - A/B test configurations
- **Export** - save optimal config
### Tab 5: Metrics
- **Loss Curves** - training/validation over epochs
- **Evaluation** - PCK, mAP, IoU scores
- **Confusion Matrix** - per-joint accuracy
- **Export** - CSV, JSON, TensorBoard format
## Architecture
### Backend (Rust/Tauri)
```
wifi-densepose-desktop/
├── src/
│ ├── commands/
│ │ ├── training.rs # NEW: Training job management
│ │ ├── datasets.rs # NEW: Dataset download/import
│ │ ├── models.rs # NEW: Model loading/export
│ │ ├── ruvector.rs # NEW: RuVector config
│ │ └── metrics.rs # NEW: Metrics retrieval
│ └── domain/
│ ├── training.rs # Training state machine
│ └── ruvector.rs # RuVector config types
```
### Frontend (React/TypeScript)
```
ui/src/pages/
├── Training/
│ ├── index.tsx # Tab container
│ ├── DatasetsTab.tsx # Dataset management
│ ├── ModelsTab.tsx # Model browser
│ ├── TrainingTab.tsx # Training control
│ ├── RuVectorTab.tsx # Signal processing config
│ └── MetricsTab.tsx # Visualization
```
### Tauri Commands
| Command | Description |
|---------|-------------|
| `list_datasets` | Get available datasets |
| `download_dataset` | Download standard dataset |
| `import_dataset` | Import custom recordings |
| `list_models` | Get model architectures |
| `load_checkpoint` | Load model weights |
| `export_model` | Export to ONNX |
| `detect_gpu` | Check CUDA/Metal |
| `start_training` | Begin training job |
| `stop_training` | Cancel training |
| `training_progress` | Get current status |
| `get_ruvector_config` | Load RuVector settings |
| `set_ruvector_config` | Update settings |
| `test_ruvector_live` | Apply to live CSI |
| `get_metrics` | Retrieve training metrics |
### Event System
Training progress updates via Tauri events:
```rust
#[derive(Serialize, Clone)]
pub struct TrainingProgress {
pub epoch: u32,
pub total_epochs: u32,
pub batch: u32,
pub total_batches: u32,
pub train_loss: f32,
pub val_loss: Option<f32>,
pub learning_rate: f32,
pub eta_secs: u64,
pub gpu_memory_mb: Option<u64>,
}
// Emit every batch
app.emit("training:progress", progress)?;
// Emit on completion
app.emit("training:complete", result)?;
```
## Implementation Plan
### Phase 1: Foundation (Week 1-2)
1. Create `Training` page skeleton with tabs
2. Implement `detect_gpu` command
3. Add dataset listing/download commands
4. Design TypeScript types for all entities
### Phase 2: Dataset Management (Week 3)
1. MM-Fi dataset downloader
2. Wi-Pose dataset downloader
3. Custom dataset import (CSV/NPZ)
4. Dataset preview component
### Phase 3: Model Management (Week 4)
1. Model architecture browser
2. Checkpoint loading
3. Model summary display
4. ONNX export
### Phase 4: Training Loop (Week 5-6)
1. Training configuration UI
2. Background training thread
3. Progress event emission
4. Checkpoint auto-save
5. Training history persistence
### Phase 5: RuVector Integration (Week 7)
1. RuVector config UI
2. Live CSI testing
3. A/B comparison mode
4. Config export/import
### Phase 6: Metrics & Polish (Week 8)
1. Loss curve visualization (Chart.js/Recharts)
2. Evaluation metrics display
3. Export functionality
4. Error handling & edge cases
## Risks & Mitigations
| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| No GPU available | Medium | High | CPU fallback with warning |
| Large dataset downloads | High | Medium | Resume support, progress UI |
| Training crashes | Medium | High | Checkpoint recovery, error reporting |
| Memory exhaustion | Low | High | Batch size auto-tuning |
| UI blocking | Medium | High | All training in background thread |
## Success Criteria
1. User can download MM-Fi dataset from UI
2. User can start training with GPU detection
3. Live progress updates without UI freeze
4. Training can be paused/resumed
5. RuVector config changes apply to live CSI
6. Metrics display updates in real-time
7. Models can be exported to ONNX
## Alternatives Considered
### 1. Separate Training App
- **Rejected**: Fragments user experience, duplicates code
### 2. Web-based Training Dashboard
- **Rejected**: Requires server, no offline support
### 3. CLI-only Training
- **Rejected**: Poor UX for non-technical users
## References
- ADR-016: RuVector Training Pipeline Integration
- ADR-017: RuVector Signal + MAT Integration
- ADR-024: AETHER Contrastive CSI Embedding
- ADR-027: MERIDIAN Domain Generalization
- Tauri v2 Events: https://v2.tauri.app/develop/calling-rust/#events
@@ -0,0 +1,82 @@
# ADR-057: Firmware CSI Build Guard and sdkconfig.defaults
| Field | Value |
|-------------|---------------------------------------------|
| **Status** | Accepted |
| **Date** | 2026-03-12 |
| **Authors** | ruv |
| **Issues** | #223, #238, #234, #210, #190 |
## Context
Multiple GitHub issues (#223, #238, #234, #210, #190) report firmware problems
that fall into two categories:
1. **CSI not enabled at runtime** — The committed `sdkconfig` had
`# CONFIG_ESP_WIFI_CSI_ENABLED is not set` (line 1135), meaning users who
built from source or used pre-built binaries got the runtime error:
`E (6700) wifi:CSI not enabled in menuconfig!`
Root cause: `sdkconfig.defaults.template` existed with the correct setting
(`CONFIG_ESP_WIFI_CSI_ENABLED=y`) but ESP-IDF only reads
`sdkconfig.defaults` — not `.template` suffixed files. No `sdkconfig.defaults`
file was committed.
2. **Unsupported ESP32 variants** — Users attempting to use original ESP32
(D0WD) and ESP32-C3 boards. The firmware targets ESP32-S3 only
(`CONFIG_IDF_TARGET="esp32s3"`, Xtensa architecture) and this was not
surfaced clearly enough in documentation or build errors.
## Decision
### Fix 1: Commit `sdkconfig.defaults` (not just template)
Copy `sdkconfig.defaults.template``sdkconfig.defaults` so that ESP-IDF
applies the correct defaults (including `CONFIG_ESP_WIFI_CSI_ENABLED=y`)
automatically when `sdkconfig` is regenerated.
### Fix 2: `#error` compile-time guard in `csi_collector.c`
Add a preprocessor guard:
```c
#ifndef CONFIG_ESP_WIFI_CSI_ENABLED
#error "CONFIG_ESP_WIFI_CSI_ENABLED must be set in sdkconfig."
#endif
```
This turns a confusing runtime crash into a clear compile-time error with
instructions on how to fix it.
### Fix 3: Fix committed `sdkconfig`
Change line 1135 from `# CONFIG_ESP_WIFI_CSI_ENABLED is not set` to
`CONFIG_ESP_WIFI_CSI_ENABLED=y`.
## Consequences
- **Positive**: New builds will always have CSI enabled. Users building from
source will get a clear compile error if CSI is somehow disabled.
- **Positive**: Pre-built release binaries will include CSI support.
- **Neutral**: Original ESP32 and ESP32-C3 remain unsupported. This is by
design — only ESP32-S3 has the CSI API surface we depend on. Future ADRs
may address multi-target support if demand warrants it.
- **Negative**: None identified.
## Hardware Support Matrix
| Variant | CSI Support | Firmware Target | Status |
|--------------|-------------|-----------------|---------------|
| ESP32-S3 | Yes | Yes | Supported |
| ESP32 (orig) | Partial | No | Unsupported |
| ESP32-C3 | Yes (IDF 5.1+) | No | Unsupported |
| ESP32-C6 | Yes | No | Unsupported |
## Notes
- ESP32-C3 and C6 use RISC-V architecture; a separate build target
(`idf.py set-target esp32c3`) would be needed.
- Original ESP32 has limited CSI (no STBC HT-LTF2, fewer subcarriers).
- Users on unsupported hardware can still write custom firmware using the
ADR-018 binary frame format (magic `0xC5110001`) for interop with the
Rust aggregator.
@@ -0,0 +1,392 @@
# ADR-058: Dual-Modal WASM Browser Pose Estimation — Live Video + WiFi CSI Fusion
- **Status**: Proposed
- **Date**: 2026-03-12
- **Deciders**: ruv
- **Tags**: wasm, browser, cnn, pose-estimation, ruvector, video, multimodal, fusion
## Context
WiFi-DensePose estimates human poses from WiFi CSI (Channel State Information).
The `ruvector-cnn` crate provides a pure Rust CNN (MobileNet-V3) with WASM bindings.
Both modalities exist independently — what's missing is **fusing live webcam video
with WiFi CSI** in a single browser demo to achieve robust pose estimation that
works even when one modality degrades (occlusion, signal noise, poor lighting).
Existing assets:
1. **`wifi-densepose-wasm`** — CSI signal processing compiled to WASM
2. **`wifi-densepose-sensing-server`** — Axum server streaming live CSI via WebSocket
3. **`ruvector-cnn`** — Pure Rust CNN with MobileNet-V3 backbones, SIMD, contrastive learning
4. **`ruvector-cnn-wasm`** — wasm-bindgen bindings: `WasmCnnEmbedder`, `SimdOps`, `LayerOps`, contrastive losses
5. **`vendor/ruvector/examples/wasm-vanilla/`** — Reference vanilla JS WASM example
Research shows multi-modal fusion (camera + WiFi) significantly outperforms either alone:
- Camera fails under occlusion, poor lighting, privacy constraints
- WiFi CSI fails with signal noise, multipath, low spatial resolution
- Fusion compensates: WiFi provides through-wall coverage, camera provides fine-grained detail
## Decision
Build a **dual-modal browser demo** at `examples/wasm-browser-pose/` that:
1. Captures **live webcam video** via `getUserMedia` API
2. Receives **live WiFi CSI** via WebSocket from the sensing server
3. Processes **both streams** through separate CNN pipelines in `ruvector-cnn-wasm`
4. **Fuses embeddings** with learned attention weights for combined pose estimation
5. Renders **video overlay** with skeleton + WiFi confidence heatmap on Canvas
6. Runs entirely in the browser — all inference client-side via WASM
### Architecture
```
┌──────────────────────────────────────────────────────────────────┐
│ Browser │
│ │
│ ┌────────────┐ ┌────────────────┐ ┌───────────────────┐ │
│ │ getUserMedia│───▶│ Video Frame │───▶│ CNN WASM │ │
│ │ (Webcam) │ │ Capture │ │ (Visual Embedder) │ │
│ └────────────┘ │ 224×224 RGB │ │ → 512-dim │ │
│ └────────────────┘ └────────┬──────────┘ │
│ │ │
│ visual_embedding │
│ │ │
│ ┌──────▼──────┐ │
│ ┌────────────┐ ┌────────────────┐ │ │ │
│ │ WebSocket │───▶│ CSI WASM │ │ Attention │ │
│ │ Client │ │ (densepose- │ │ Fusion │ │
│ │ │ │ wasm) │ │ Module │ │
│ └────────────┘ └───────┬────────┘ │ │ │
│ │ └──────┬──────┘ │
│ ┌───────▼────────┐ │ │
│ │ CNN WASM │ fused_embedding │
│ │ (CSI Embedder) │ │ │
│ │ → 512-dim │ ┌──────▼──────┐ │
│ └───────┬────────┘ │ Pose │ │
│ │ │ Decoder │ │
│ csi_embedding │ → 17 kpts │ │
│ │ └──────┬──────┘ │
│ └──────────────────────┘ │
│ │ │
│ ┌──────────────┐ ┌─────▼──────┐ │
│ │ Video Canvas │◀────────│ Overlay │ │
│ │ + Skeleton │ │ Renderer │ │
│ │ + Heatmap │ └────────────┘ │
│ └──────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────┘
▲ ▲
│ getUserMedia │ WebSocket
│ (camera) │ (ws://host:3030/ws/csi)
│ │
┌────┴────┐ ┌───────┴─────────┐
│ Webcam │ │ Sensing Server │
└─────────┘ └─────────────────┘
```
### Dual Pipeline Design
Two parallel CNN pipelines run on each frame tick (~30 FPS):
| Pipeline | Input | Preprocessing | CNN Config | Output |
|----------|-------|---------------|------------|--------|
| **Visual** | Webcam frame (640×480) | Resize to 224×224 RGB, ImageNet normalize | MobileNet-V3 Small, 512-dim | Visual embedding |
| **CSI** | CSI frame (ADR-018 binary) | Amplitude/phase/delta → 224×224 pseudo-RGB | MobileNet-V3 Small, 512-dim | CSI embedding |
Both use the same `WasmCnnEmbedder` but with separate instances and weight sets.
### Fusion Strategy
**Learned attention-weighted fusion** combines the two 512-dim embeddings:
```javascript
// Attention fusion: learn which modality to trust per-dimension
// α ∈ [0,1]^512 — attention weights (shipped as JSON, trained offline)
// visual_emb, csi_emb ∈ R^512
function fuseEmbeddings(visual_emb, csi_emb, attention_weights) {
const fused = new Float32Array(512);
for (let i = 0; i < 512; i++) {
const α = attention_weights[i];
fused[i] = α * visual_emb[i] + (1 - α) * csi_emb[i];
}
return fused;
}
```
**Dynamic confidence gating** adjusts fusion based on signal quality:
| Condition | Behavior |
|-----------|----------|
| Good video + good CSI | Balanced fusion (α ≈ 0.5) |
| Poor lighting / occlusion | CSI-dominant (α → 0, WiFi takes over) |
| CSI noise / no ESP32 | Video-dominant (α → 1, camera only) |
| Video-only mode (no WiFi) | α = 1.0, pure visual CNN pose estimation |
| CSI-only mode (no camera) | α = 0.0, pure WiFi pose estimation |
Quality detection:
- **Video quality**: Frame brightness variance (dark = low quality), motion blur score
- **CSI quality**: Signal-to-noise ratio from `wifi-densepose-wasm`, coherence gate output
### CSI-to-Image Encoding
CSI data encoded as 3-channel pseudo-image for the CSI CNN pipeline:
| Channel | Data | Normalization |
|---------|------|---------------|
| R | CSI amplitude (subcarrier × time window) | Min-max to [0, 255] |
| G | CSI phase (unwrapped, subcarrier × time window) | Min-max to [0, 255] |
| B | Temporal difference (frame-to-frame Δ amplitude) | Abs, min-max to [0, 255] |
### Video Processing
Webcam frames processed through standard ImageNet pipeline:
```javascript
// Capture frame from video element
const frame = captureVideoFrame(videoElement, 224, 224); // Returns Uint8Array RGB
// ImageNet normalization happens inside WasmCnnEmbedder.extract()
const visual_embedding = visual_embedder.extract(frame, 224, 224);
```
### Pose Keypoint Mapping
17 COCO-format keypoints decoded from the fused 512-dim embedding:
```
0: nose 1: left_eye 2: right_eye
3: left_ear 4: right_ear 5: left_shoulder
6: right_shoulder 7: left_elbow 8: right_elbow
9: left_wrist 10: right_wrist 11: left_hip
12: right_hip 13: left_knee 14: right_knee
15: left_ankle 16: right_ankle
```
Each keypoint decoded as (x, y, confidence) = 51 values from the 512-dim embedding
via a learned linear projection.
### Operating Modes
The demo supports three modes, selectable in the UI:
| Mode | Video | CSI | Fusion | Use Case |
|------|-------|-----|--------|----------|
| **Dual (default)** | ✅ | ✅ | Attention-weighted | Best accuracy, full demo |
| **Video Only** | ✅ | ❌ | α = 1.0 | No ESP32 available, quick demo |
| **CSI Only** | ❌ | ✅ | α = 0.0 | Privacy mode, through-wall sensing |
**Video Only mode works without any hardware** — just a webcam — making the demo
instantly accessible for anyone wanting to try it.
### File Layout
```
examples/wasm-browser-pose/
├── index.html # Single-page app (vanilla JS, no bundler)
├── js/
│ ├── app.js # Main entry, mode selection, orchestration
│ ├── video-capture.js # getUserMedia, frame extraction, quality detection
│ ├── csi-processor.js # WebSocket CSI client, frame parsing, pseudo-image encoding
│ ├── fusion.js # Attention-weighted embedding fusion, confidence gating
│ ├── pose-decoder.js # Fused embedding → 17 keypoints
│ └── canvas-renderer.js # Video overlay, skeleton, CSI heatmap, confidence bars
├── data/
│ ├── visual-weights.json # Visual CNN → embedding projection (placeholder until trained)
│ ├── csi-weights.json # CSI CNN → embedding projection (placeholder until trained)
│ ├── fusion-weights.json # Attention fusion α weights (512 values)
│ └── pose-weights.json # Fused embedding → keypoint projection
├── css/
│ └── style.css # Dark theme UI styling
├── pkg/ # Built WASM packages (gitignored, built by script)
│ ├── wifi_densepose_wasm/
│ └── ruvector_cnn_wasm/
├── build.sh # wasm-pack build script for both packages
└── README.md # Setup and usage instructions
```
### Build Pipeline
```bash
#!/bin/bash
# build.sh — builds both WASM packages into pkg/
set -e
# Build wifi-densepose-wasm (CSI processing)
wasm-pack build ../../rust-port/wifi-densepose-rs/crates/wifi-densepose-wasm \
--target web --out-dir "$(pwd)/pkg/wifi_densepose_wasm" --no-typescript
# Build ruvector-cnn-wasm (CNN inference for both video and CSI)
wasm-pack build ../../vendor/ruvector/crates/ruvector-cnn-wasm \
--target web --out-dir "$(pwd)/pkg/ruvector_cnn_wasm" --no-typescript
echo "Build complete. Serve with: python3 -m http.server 8080"
```
### UI Layout
```
┌─────────────────────────────────────────────────────────┐
│ WiFi-DensePose — Live Dual-Modal Pose Estimation │
│ [Dual Mode ▼] [⚙ Settings] FPS: 28 ◉ Live │
├───────────────────────────┬─────────────────────────────┤
│ │ │
│ ┌───────────────────┐ │ ┌───────────────────┐ │
│ │ │ │ │ │ │
│ │ Video + Skeleton │ │ │ CSI Heatmap │ │
│ │ Overlay │ │ │ (amplitude × │ │
│ │ (main canvas) │ │ │ subcarrier) │ │
│ │ │ │ │ │ │
│ └───────────────────┘ │ └───────────────────┘ │
│ │ │
├───────────────────────────┴─────────────────────────────┤
│ Fusion Confidence: ████████░░ 78% │
│ Video: ██████████ 95% │ CSI: ██████░░░░ 61% │
├─────────────────────────────────────────────────────────┤
│ ┌─────────────────────────────────────────────────┐ │
│ │ Embedding Space (2D projection) │ │
│ │ · · · │ │
│ │ · · · · · · (color = pose cluster) │ │
│ │ · · · · │ │
│ └─────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────┤
│ Latency: Video 12ms │ CSI 8ms │ Fusion 1ms │ Total 21ms│
│ [▶ Record] [📷 Snapshot] [Confidence: ████ 0.6] │
└─────────────────────────────────────────────────────────┘
```
### WASM Module Structure
| Package | Source Crate | Provides | Size (est.) |
|---------|-------------|----------|-------------|
| `wifi_densepose_wasm` | `wifi-densepose-wasm` | CSI frame parsing, signal processing, feature extraction | ~200KB |
| `ruvector_cnn_wasm` | `ruvector-cnn-wasm` | `WasmCnnEmbedder` (×2 instances), `SimdOps`, `LayerOps`, contrastive losses | ~150KB |
Two `WasmCnnEmbedder` instances are created — one for video frames, one for CSI pseudo-images.
They share the same WASM module but have independent state.
### Browser API Requirements
| API | Purpose | Required | Fallback |
|-----|---------|----------|----------|
| `getUserMedia` | Webcam capture | For video mode | CSI-only mode |
| WebAssembly | CNN inference | Yes | None (hard requirement) |
| WASM SIMD128 | Accelerated inference | No | Scalar fallback (~2× slower) |
| WebSocket | CSI data stream | For CSI mode | Video-only mode |
| Canvas 2D | Rendering | Yes | None |
| `requestAnimationFrame` | Render loop | Yes | `setTimeout` fallback |
| ES Modules | Code organization | Yes | None |
Target: Chrome 89+, Firefox 89+, Safari 15+, Edge 89+
### Performance Budget
| Stage | Target Latency | Notes |
|-------|---------------|-------|
| Video frame capture + resize | <3ms | `drawImage` to offscreen canvas |
| Video CNN embedding | <15ms | 224×224 RGB → 512-dim |
| CSI receive + parse | <2ms | Binary WebSocket message |
| CSI pseudo-image encoding | <3ms | Amplitude/phase/delta channels |
| CSI CNN embedding | <15ms | 224×224 pseudo-RGB → 512-dim |
| Attention fusion | <1ms | Element-wise weighted sum |
| Pose decoding | <1ms | Linear projection |
| Canvas overlay render | <3ms | Video + skeleton + heatmap |
| **Total (dual mode)** | **<33ms** | **30 FPS capable** |
| **Total (video only)** | **<22ms** | **45 FPS capable** |
Note: Video and CSI CNN pipelines can run in parallel using Web Workers,
reducing dual-mode latency to ~max(15, 15) + 5 = ~20ms (50 FPS).
### Contrastive Learning Integration
The demo optionally shows real-time contrastive learning in the browser:
- **InfoNCE loss** (`WasmInfoNCELoss`): Compare video vs CSI embeddings for the same pose — trains cross-modal alignment
- **Triplet loss** (`WasmTripletLoss`): Push apart different poses, pull together same pose across modalities
- **SimdOps**: Accelerated dot products for real-time similarity computation
- **Embedding space panel**: Live 2D projection shows video and CSI embeddings converging when viewing the same person
### Relationship to Existing Crates
| Existing Crate | Role in This Demo |
|---------------|-------------------|
| `ruvector-cnn-wasm` | CNN inference for **both** video frames and CSI pseudo-images |
| `wifi-densepose-wasm` | CSI frame parsing and signal processing |
| `wifi-densepose-sensing-server` | WebSocket CSI data source |
| `wifi-densepose-core` | ADR-018 frame format definitions |
| `ruvector-cnn` | Underlying MobileNet-V3, layers, contrastive learning |
No new Rust crates are needed. The example is pure HTML/JS consuming existing WASM packages.
## Consequences
### Positive
- **Instant demo**: Video-only mode works with just a webcam — no ESP32 needed
- **Multi-modal showcase**: Demonstrates camera + WiFi fusion, the core innovation of the project
- **Graceful degradation**: Works with video-only, CSI-only, or both
- **Through-wall capability**: CSI mode shows pose estimation where cameras cannot reach
- **Zero-install**: Anyone with a browser can try it
- **Training data collection**: Can record paired (video, CSI) data for offline model training
- **Reusable**: JS modules embed directly in the Tauri desktop app's webview
### Negative
- **Model weights**: Requires offline-trained weights for visual CNN, CSI CNN, fusion, and pose decoder (~200KB total JSON)
- **WASM size**: Two WASM modules total ~350KB (acceptable)
- **No GPU**: CPU-only WASM inference; adequate at 224×224 but limits resolution scaling
- **Camera privacy**: Video mode requires camera permission (mitigated: CSI-only mode available)
- **Two CNN instances**: Memory footprint doubles vs single-modal (~10MB total, acceptable for desktop browsers)
### Risks
- **Cross-modal alignment**: Video and CSI embeddings must be trained jointly for fusion to work;
without proper training, fusion may be worse than either modality alone
- **Latency on mobile**: Dual CNN on mobile browsers may exceed 33ms; implement automatic quality reduction
- **WebSocket drops**: Network jitter → CSI frame gaps; buffer last 3 frames, interpolate missing data
## Implementation Plan
1. **Phase 1 — Scaffold**: File layout, build.sh, index.html shell, mode selector UI
2. **Phase 2 — Video pipeline**: getUserMedia → frame capture → CNN embedding → basic pose display
3. **Phase 3 — CSI pipeline**: WebSocket client → CSI parsing → pseudo-image → CNN embedding
4. **Phase 4 — Fusion**: Attention-weighted combination, confidence gating, mode switching
5. **Phase 5 — Pose decoder**: Linear projection with placeholder weights → 17 keypoints
6. **Phase 6 — Overlay renderer**: Video canvas with skeleton overlay, CSI heatmap panel
7. **Phase 7 — Training**: Use `wifi-densepose-train` to generate real weights for both CNNs + fusion + decoder
8. **Phase 8 — Contrastive demo**: Embedding space visualization, cross-modal similarity display
9. **Phase 9 — Web Workers**: Move CNN inference to workers for parallel video + CSI processing
10. **Phase 10 — Polish**: Recording, snapshots, adaptive quality, mobile optimization
## Alternatives Considered
### 1. CSI-Only (No Video)
Rejected: Misses the opportunity to show multi-modal fusion and makes the demo less
accessible (requires ESP32 hardware). Video-only mode as a fallback is strictly better.
### 2. Server-Side Video Inference
Rejected: Adds latency, requires webcam stream upload (privacy concern), and defeats
the WASM-first architecture. All inference must be client-side.
### 3. TensorFlow.js for Video, ruvector-cnn-wasm for CSI
Rejected: Would require two different ML frameworks. Using `ruvector-cnn-wasm` for both
keeps a single WASM module, unified embedding space, and simpler fusion.
### 4. Pre-recorded Video Demo
Rejected: Live webcam input is far more compelling for demonstrations.
Pre-recorded mode can be added as a secondary option.
### 5. React/Vue Framework
Rejected: Adds build tooling. Vanilla JS + ES modules keeps the demo self-contained.
## References
- [ADR-018: Binary CSI Frame Format](ADR-018-binary-csi-frame-format.md)
- [ADR-024: Contrastive CSI Embedding / AETHER](ADR-024-contrastive-csi-embedding.md)
- [ADR-055: Integrated Sensing Server](ADR-055-integrated-sensing-server.md)
- `vendor/ruvector/crates/ruvector-cnn/src/lib.rs` — CNN embedder implementation
- `vendor/ruvector/crates/ruvector-cnn-wasm/src/lib.rs` — WASM bindings
- `vendor/ruvector/examples/wasm-vanilla/index.html` — Reference vanilla JS WASM pattern
- Person-in-WiFi: Fine-grained Person Perception using WiFi (ICCV 2019) — camera+WiFi fusion precedent
- WiPose: Multi-Person WiFi Pose Estimation (TMC 2022) — cross-modal embedding approach
@@ -0,0 +1,83 @@
# ADR-059: Live ESP32 CSI Pipeline Integration
## Status
Accepted
## Date
2026-03-12
## Context
ADR-058 established a dual-modal browser demo combining webcam video and WiFi CSI for pose estimation. However, it used simulated CSI data. To demonstrate real-world capability, we need an end-to-end pipeline from physical ESP32 hardware through to the browser visualization.
The ESP32-S3 firmware (`firmware/esp32-csi-node/`) already supports CSI collection and UDP streaming (ADR-018). The sensing server (`wifi-densepose-sensing-server`) already supports UDP ingestion and WebSocket bridging. The missing piece was connecting these components and enabling the browser demo to consume live data.
## Decision
Implement a complete live CSI pipeline:
```
ESP32-S3 (CSI capture) → UDP:5005 → sensing-server (Rust/Axum) → WS:8765 → browser demo
```
### Components
1. **ESP32 Firmware** — Rebuilt with native Windows ESP-IDF v5.4.0 toolchain (no Docker). Configured for target network and PC IP via `sdkconfig`. Helper scripts added:
- `build_firmware.ps1` — Sets up IDF environment, cleans, builds, and flashes
- `read_serial.ps1` — Serial monitor with DTR/RTS reset capability
2. **Sensing Server**`wifi-densepose-sensing-server` started with:
- `--source esp32` — Expect real ESP32 UDP frames
- `--bind-addr 0.0.0.0` — Accept connections from any interface
- `--ui-path <path>` — Serve the demo UI via HTTP
3. **Browser Demo**`main.js` updated to auto-connect to `ws://localhost:8765/ws/sensing` on page load. Falls back to simulated CSI if the WebSocket is unavailable (GitHub Pages).
### Network Configuration
The ESP32 sends UDP packets to a configured target IP. If the PC's IP doesn't match the firmware's compiled target, a secondary IP alias can be added:
```powershell
# PowerShell (Admin)
New-NetIPAddress -IPAddress 192.168.1.100 -PrefixLength 24 -InterfaceAlias "Wi-Fi"
```
### Data Flow
| Stage | Protocol | Format | Rate |
|-------|----------|--------|------|
| ESP32 → Server | UDP | ADR-018 binary frame (magic `0xC5110001`, I/Q pairs) | ~100 Hz |
| Server → Browser | WebSocket | ADR-018 binary frame (forwarded) | ~10 Hz (tick-ms=100) |
| Browser decode | JavaScript | Float32 amplitude/phase arrays | Per frame |
### Build Environment (Windows)
ESP-IDF v5.4.0 on Windows requires:
- IDF_PATH pointing to the ESP-IDF framework
- IDF_TOOLS_PATH pointing to toolchain binaries
- MSYS/MinGW environment variables removed (ESP-IDF rejects them)
- Python venv from ESP-IDF tools for `idf.py` execution
The `build_firmware.ps1` script handles all of this automatically.
## Consequences
### Positive
- First end-to-end demonstration of real WiFi CSI → pose estimation in a browser
- No Docker required for firmware builds on Windows
- Demo gracefully degrades to simulated CSI when no server is available
- Same demo works on GitHub Pages (simulated) and locally (live ESP32)
### Negative
- ESP32 target IP is compiled into firmware; changing it requires a rebuild or NVS override
- Windows firewall may block UDP:5005; user must allow it
- Mixed content restrictions prevent HTTPS pages from connecting to ws:// (local only)
## Related
- [ADR-018](ADR-018-esp32-dev-implementation.md) — ESP32 CSI frame format and UDP streaming
- [ADR-058](ADR-058-ruvector-wasm-browser-pose-example.md) — Dual-modal WASM browser pose demo
- [ADR-039](ADR-039-edge-intelligence-framework.md) — Edge intelligence on ESP32
- Issue [#245](https://github.com/ruvnet/RuView/issues/245) — Tracking issue
+33 -11
View File
@@ -78,6 +78,17 @@ docker pull ruvnet/wifi-densepose:latest
Multi-architecture image (amd64 + arm64). Works on Intel/AMD and Apple Silicon Macs. Contains the Rust sensing server, Three.js UI, and all signal processing.
**Data source selection:** Use the `CSI_SOURCE` environment variable to select the sensing mode:
| Value | Description |
|-------|-------------|
| `auto` | (default) Probe for ESP32 on UDP 5005, fall back to simulation |
| `esp32` | Receive real CSI frames from ESP32 devices over UDP |
| `simulated` | Generate synthetic CSI frames (no hardware required) |
| `wifi` | Host Wi-Fi RSSI (not available inside containers) |
Example: `docker run -e CSI_SOURCE=esp32 -p 3000:3000 -p 5005:5005/udp ruvnet/wifi-densepose:latest`
### From Source (Rust)
```bash
@@ -267,8 +278,8 @@ Real Channel State Information at 20 Hz with 56-192 subcarriers. Required for po
# From source
./target/release/sensing-server --source esp32 --udp-port 5005 --http-port 3000 --ws-port 3001
# Docker
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp ruvnet/wifi-densepose:latest --source esp32
# Docker (use CSI_SOURCE environment variable)
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp -e CSI_SOURCE=esp32 ruvnet/wifi-densepose:latest
```
The ESP32 nodes stream binary CSI frames over UDP to port 5005. See [Hardware Setup](#esp32-s3-mesh) for flashing instructions.
@@ -679,9 +690,11 @@ Download the dataset files and place them in a `data/` directory.
./target/release/sensing-server --train --dataset data/ --dataset-type mmfi --epochs 100 --save-rvf model.rvf
# Via Docker (mount your data directory)
# Note: Training mode requires overriding the default entrypoint
docker run --rm \
-v $(pwd)/data:/data \
-v $(pwd)/output:/output \
--entrypoint /app/sensing-server \
ruvnet/wifi-densepose:latest \
--train --dataset /data --epochs 100 --export-rvf /output/model.rvf
```
@@ -797,14 +810,18 @@ Pre-built binaries are available at [Releases](https://github.com/ruvnet/RuView/
| Release | What It Includes | Tag |
|---------|-----------------|-----|
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Stable — raw CSI streaming, TDM, channel hopping, QUIC mesh | `v0.2.0-esp32` |
| [v0.4.1](https://github.com/ruvnet/RuView/releases/tag/v0.4.1-esp32) | **Stable** — CSI build fix, compile guard, AMOLED display, edge intelligence ([ADR-057](../docs/adr/ADR-057-firmware-csi-build-guard.md)) | `v0.4.1-esp32` |
| [v0.3.0-alpha](https://github.com/ruvnet/RuView/releases/tag/v0.3.0-alpha-esp32) | Alpha — adds on-device edge intelligence (ADR-039) | `v0.3.0-alpha-esp32` |
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Raw CSI streaming, TDM, channel hopping, QUIC mesh | `v0.2.0-esp32` |
> **Important:** Firmware versions prior to v0.4.1 had CSI **disabled** in the build config, causing a runtime error (`E wifi:CSI not enabled in menuconfig!`). Always use v0.4.1 or later.
```bash
# Flash an ESP32-S3 (requires esptool: pip install esptool)
python -m esptool --chip esp32s3 --port COM7 --baud 460800 \
write-flash --flash-mode dio --flash-size 4MB \
0x0 bootloader.bin 0x8000 partition-table.bin 0x10000 esp32-csi-node.bin
write-flash --flash-mode dio --flash-size 8MB --flash-freq 80m \
0x0 bootloader.bin 0x8000 partition-table.bin \
0xf000 ota_data_initial.bin 0x20000 esp32-csi-node.bin
```
**Provisioning:**
@@ -885,8 +902,8 @@ Binary size: 777 KB (24% free in the 1 MB app partition).
# From source
./target/release/sensing-server --source esp32 --udp-port 5005 --http-port 3000 --ws-port 3001
# Docker
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp ruvnet/wifi-densepose:latest --source esp32
# Docker (use CSI_SOURCE environment variable)
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp -e CSI_SOURCE=esp32 ruvnet/wifi-densepose:latest
```
See [ADR-018](../docs/adr/ADR-018-esp32-dev-implementation.md), [ADR-029](../docs/adr/ADR-029-ruvsense-multistatic-sensing-mode.md), and [Tutorial #34](https://github.com/ruvnet/RuView/issues/34).
@@ -953,12 +970,17 @@ Add the WebSocket port mapping:
docker run -p 3000:3000 -p 3001:3001 ruvnet/wifi-densepose:latest
```
### ESP32: "CSI not enabled in menuconfig"
Firmware versions prior to v0.4.1 had `CONFIG_ESP_WIFI_CSI_ENABLED` disabled in the build config. Upgrade to [v0.4.1](https://github.com/ruvnet/RuView/releases/tag/v0.4.1-esp32) or later. If building from source, ensure `sdkconfig.defaults` exists (not just `sdkconfig.defaults.template`). See [ADR-057](../docs/adr/ADR-057-firmware-csi-build-guard.md).
### ESP32: No data arriving
1. Verify the ESP32 is connected to the same WiFi network
2. Check the target IP matches the sensing server machine: `python firmware/esp32-csi-node/provision.py --port COM7 --target-ip <YOUR_IP>`
3. Verify UDP port 5005 is not blocked by firewall
4. Test with: `nc -lu 5005` (Linux) or similar UDP listener
1. Verify firmware is v0.4.1+ (older versions had CSI disabled — see above)
2. Verify the ESP32 is connected to the same WiFi network
3. Check the target IP matches the sensing server machine: `python firmware/esp32-csi-node/provision.py --port COM7 --target-ip <YOUR_IP>`
4. Verify UDP port 5005 is not blocked by firewall
5. Test with: `nc -lu 5005` (Linux) or similar UDP listener
### Build: Rust compilation errors
@@ -0,0 +1,31 @@
# Remove MSYS environment variables that trigger ESP-IDF's MinGW rejection
Remove-Item env:MSYSTEM -ErrorAction SilentlyContinue
Remove-Item env:MSYSTEM_CARCH -ErrorAction SilentlyContinue
Remove-Item env:MSYSTEM_CHOST -ErrorAction SilentlyContinue
Remove-Item env:MSYSTEM_PREFIX -ErrorAction SilentlyContinue
Remove-Item env:MINGW_CHOST -ErrorAction SilentlyContinue
Remove-Item env:MINGW_PACKAGE_PREFIX -ErrorAction SilentlyContinue
Remove-Item env:MINGW_PREFIX -ErrorAction SilentlyContinue
$env:IDF_PATH = "C:\Users\ruv\esp\v5.4\esp-idf"
$env:IDF_TOOLS_PATH = "C:\Espressif\tools"
$env:IDF_PYTHON_ENV_PATH = "C:\Espressif\tools\python\v5.4\venv"
$env:PATH = "C:\Espressif\tools\xtensa-esp-elf\esp-14.2.0_20241119\xtensa-esp-elf\bin;C:\Espressif\tools\cmake\3.30.2\cmake-3.30.2-windows-x86_64\bin;C:\Espressif\tools\ninja\1.12.1;C:\Espressif\tools\ccache\4.10.2\ccache-4.10.2-windows-x86_64;C:\Espressif\tools\idf-exe\1.0.3;C:\Espressif\tools\python\v5.4\venv\Scripts;$env:PATH"
Set-Location "C:\Users\ruv\Projects\wifi-densepose\firmware\esp32-csi-node"
$python = "$env:IDF_PYTHON_ENV_PATH\Scripts\python.exe"
$idf = "$env:IDF_PATH\tools\idf.py"
Write-Host "=== Cleaning stale build cache ==="
& $python $idf fullclean
Write-Host "=== Building firmware (SSID=ruv.net, target=192.168.1.20:5005) ==="
& $python $idf build
if ($LASTEXITCODE -eq 0) {
Write-Host "=== Build succeeded! Flashing to COM7 ==="
& $python $idf -p COM7 flash
} else {
Write-Host "=== Build failed with exit code $LASTEXITCODE ==="
}
@@ -21,6 +21,16 @@
#include "esp_timer.h"
#include "sdkconfig.h"
/* ADR-057: Build-time guard — fail early if CSI is not enabled in sdkconfig.
* Without this, the firmware compiles but crashes at runtime with:
* "E (xxxx) wifi:CSI not enabled in menuconfig!"
* which is confusing for users flashing pre-built binaries. */
#ifndef CONFIG_ESP_WIFI_CSI_ENABLED
#error "CONFIG_ESP_WIFI_CSI_ENABLED must be set in sdkconfig. " \
"Run: idf.py menuconfig -> Component config -> Wi-Fi -> Enable WiFi CSI, " \
"or copy sdkconfig.defaults.template to sdkconfig.defaults before building."
#endif
static const char *TAG = "csi_collector";
static uint32_t s_sequence = 0;
+14
View File
@@ -0,0 +1,14 @@
$p = New-Object System.IO.Ports.SerialPort('COM7', 115200)
$p.ReadTimeout = 5000
$p.Open()
Start-Sleep -Milliseconds 200
for ($i = 0; $i -lt 60; $i++) {
try {
$line = $p.ReadLine()
Write-Host $line
} catch {
break
}
}
$p.Close()
@@ -0,0 +1,33 @@
# ESP32-S3 CSI Node — Default SDK Configuration
# This file is applied automatically by idf.py when no sdkconfig exists.
# Target: ESP32-S3
CONFIG_IDF_TARGET="esp32s3"
# Use custom partition table (8MB flash with OTA — ADR-045)
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions_display.csv"
# Flash configuration: 8MB (Quad SPI)
CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y
CONFIG_ESPTOOLPY_FLASHSIZE="8MB"
# Compiler optimization: optimize for size to reduce binary
CONFIG_COMPILER_OPTIMIZATION_SIZE=y
# Enable CSI (Channel State Information) in WiFi driver
CONFIG_ESP_WIFI_CSI_ENABLED=y
# NVS encryption disabled by default (requires eFuse provisioning).
# Enable only after burning HMAC key to eFuse block.
# CONFIG_NVS_ENCRYPTION is not set
# Disable unused features to reduce binary size
CONFIG_BOOTLOADER_LOG_LEVEL_WARN=y
CONFIG_LOG_DEFAULT_LEVEL_INFO=y
# LWIP: enable extended socket options for UDP multicast
CONFIG_LWIP_SO_RCVBUF=y
# FreeRTOS: increase task stack for CSI processing
CONFIG_ESP_MAIN_TASK_STACK_SIZE=8192
@@ -18,8 +18,9 @@ CONFIG_COMPILER_OPTIMIZATION_SIZE=y
# Enable CSI (Channel State Information) in WiFi driver
CONFIG_ESP_WIFI_CSI_ENABLED=y
# Enable NVS encryption for secure credential storage
CONFIG_NVS_ENCRYPTION=y
# NVS encryption disabled by default (requires eFuse provisioning).
# Enable only after burning HMAC key to eFuse block.
# CONFIG_NVS_ENCRYPTION is not set
# Disable unused features to reduce binary size
CONFIG_BOOTLOADER_LOG_LEVEL_WARN=y
@@ -4,5 +4,4 @@ pub mod ota;
pub mod provision;
pub mod server;
pub mod settings;
pub mod training;
pub mod wasm;
@@ -1,482 +0,0 @@
//! Training commands for the desktop application.
//!
//! Provides Tauri commands for:
//! - GPU detection
//! - Dataset management
//! - Model/checkpoint operations
//! - Training job control
//! - RuVector configuration
//! - Metrics retrieval
use crate::domain::training::{
CheckpointInfo, DatasetFormat, DatasetInfo, EpochMetrics, EvaluationMetrics,
GpuBackend, GpuInfo, JointAccuracy, LiveTestMetrics,
ModelInfo, ModelType, RuVectorConfig, TrainingConfig, TrainingJob,
TrainingProgress, TrainingStatus,
};
use crate::state::AppState;
use tauri::State;
// ============================================================================
// Standard Datasets (built-in)
// ============================================================================
fn get_standard_datasets() -> Vec<DatasetInfo> {
vec![
DatasetInfo {
id: "mmfi".into(),
name: "MM-Fi Dataset".into(),
description: "Multi-modal WiFi sensing dataset with 40 subjects, 27 activities".into(),
format: DatasetFormat::MmFi,
size_mb: 2400.0,
samples: 320000,
downloaded: false,
path: None,
url: Some("https://ntu-aiot-lab.github.io/mm-fi".into()),
},
DatasetInfo {
id: "wipose".into(),
name: "Wi-Pose Dataset".into(),
description: "WiFi-based pose estimation with 3D skeleton annotations".into(),
format: DatasetFormat::WiPose,
size_mb: 1800.0,
samples: 150000,
downloaded: false,
path: None,
url: Some("https://github.com/Wi-Pose".into()),
},
DatasetInfo {
id: "wiar".into(),
name: "WiAR Dataset".into(),
description: "WiFi activity recognition with CSI data".into(),
format: DatasetFormat::Wiar,
size_mb: 500.0,
samples: 45000,
downloaded: false,
path: None,
url: Some("https://github.com/WiAR".into()),
},
]
}
// ============================================================================
// Standard Model Architectures
// ============================================================================
fn get_standard_models() -> Vec<ModelInfo> {
vec![
ModelInfo {
id: "csi-encoder-cnn".into(),
name: "CSI Encoder (CNN)".into(),
model_type: ModelType::Encoder,
description: "Convolutional encoder for CSI amplitude/phase features".into(),
params_m: 2.3,
memory_mb: 128,
paper: None,
},
ModelInfo {
id: "csi-encoder-transformer".into(),
name: "CSI Encoder (Transformer)".into(),
model_type: ModelType::Encoder,
description: "Self-attention based CSI feature extraction".into(),
params_m: 8.5,
memory_mb: 384,
paper: Some("WiFi-ViT 2024".into()),
},
ModelInfo {
id: "pose-decoder-lstm".into(),
name: "Pose Decoder (LSTM)".into(),
model_type: ModelType::Decoder,
description: "Recurrent decoder for temporal pose estimation".into(),
params_m: 1.8,
memory_mb: 96,
paper: None,
},
ModelInfo {
id: "pose-decoder-gru".into(),
name: "Pose Decoder (GRU)".into(),
model_type: ModelType::Decoder,
description: "Gated recurrent unit pose decoder (faster)".into(),
params_m: 1.2,
memory_mb: 64,
paper: None,
},
ModelInfo {
id: "aether-embedding".into(),
name: "AETHER Embedding".into(),
model_type: ModelType::Embedding,
description: "Contrastive CSI embedding for person re-identification (ADR-024)".into(),
params_m: 4.2,
memory_mb: 192,
paper: Some("AETHER 2025".into()),
},
ModelInfo {
id: "meridian-adaptor".into(),
name: "MERIDIAN Adaptor".into(),
model_type: ModelType::Adaptor,
description: "Cross-environment domain generalization module (ADR-027)".into(),
params_m: 3.1,
memory_mb: 144,
paper: Some("MERIDIAN 2025".into()),
},
]
}
// ============================================================================
// GPU Detection Commands
// ============================================================================
/// Detect available GPU(s) and return information.
#[tauri::command]
pub async fn detect_gpu(state: State<'_, AppState>) -> Result<GpuInfo, String> {
// Check for cached GPU info
if let Ok(training) = state.training.lock() {
if let Some(ref info) = training.gpu_info {
return Ok(info.clone());
}
}
// Detect GPU
let info = detect_gpu_internal();
// Cache the result
if let Ok(mut training) = state.training.lock() {
training.gpu_info = Some(info.clone());
}
Ok(info)
}
fn detect_gpu_internal() -> GpuInfo {
// Check for Metal on macOS
#[cfg(target_os = "macos")]
{
// Check if system has Apple Silicon or discrete GPU
let has_metal = std::process::Command::new("system_profiler")
.args(["SPDisplaysDataType", "-json"])
.output()
.map(|o| {
let output = String::from_utf8_lossy(&o.stdout);
output.contains("Metal") || output.contains("Apple M")
})
.unwrap_or(false);
if has_metal {
// Try to get GPU name
let name = std::process::Command::new("system_profiler")
.args(["SPDisplaysDataType"])
.output()
.ok()
.and_then(|o| {
let output = String::from_utf8_lossy(&o.stdout);
// Parse chipset name
for line in output.lines() {
if line.contains("Chipset Model:") {
return line.split(':').nth(1).map(|s| s.trim().to_string());
}
}
None
});
return GpuInfo {
available: true,
backend: GpuBackend::Metal,
name,
memory_mb: None, // Metal doesn't easily expose this
cuda_version: None,
metal_supported: true,
};
}
}
// Check for CUDA on Linux/Windows
#[cfg(any(target_os = "linux", target_os = "windows"))]
{
// Try nvidia-smi for CUDA detection
if let Ok(output) = std::process::Command::new("nvidia-smi")
.args(["--query-gpu=name,memory.total", "--format=csv,noheader,nounits"])
.output()
{
if output.status.success() {
let stdout = String::from_utf8_lossy(&output.stdout);
let parts: Vec<&str> = stdout.trim().split(',').collect();
let name = parts.first().map(|s| s.trim().to_string());
let memory_mb = parts.get(1)
.and_then(|s| s.trim().parse::<u64>().ok());
// Get CUDA version
let cuda_version = std::process::Command::new("nvidia-smi")
.output()
.ok()
.and_then(|o| {
let output = String::from_utf8_lossy(&o.stdout);
for line in output.lines() {
if line.contains("CUDA Version:") {
return line.split("CUDA Version:")
.nth(1)
.map(|s| s.split_whitespace().next().unwrap_or("").to_string());
}
}
None
});
return GpuInfo {
available: true,
backend: GpuBackend::Cuda,
name,
memory_mb,
cuda_version,
metal_supported: false,
};
}
}
}
// Fall back to CPU
GpuInfo {
available: false,
backend: GpuBackend::Cpu,
name: None,
memory_mb: None,
cuda_version: None,
metal_supported: false,
}
}
// ============================================================================
// Dataset Commands
// ============================================================================
/// List available datasets (both standard and downloaded).
#[tauri::command]
pub async fn list_datasets(state: State<'_, AppState>) -> Result<Vec<String>, String> {
let training = state.training.lock().map_err(|e| e.to_string())?;
// Return IDs of downloaded datasets
Ok(training.datasets.iter()
.filter(|d| d.downloaded)
.map(|d| d.id.clone())
.collect())
}
/// Get full dataset information.
#[tauri::command]
pub async fn get_datasets(state: State<'_, AppState>) -> Result<Vec<DatasetInfo>, String> {
let mut training = state.training.lock().map_err(|e| e.to_string())?;
// Initialize with standard datasets if empty
if training.datasets.is_empty() {
training.datasets = get_standard_datasets();
}
Ok(training.datasets.clone())
}
/// Download a dataset (placeholder - actual download would need async HTTP).
#[tauri::command]
pub async fn download_dataset(
dataset_id: String,
state: State<'_, AppState>,
) -> Result<DatasetInfo, String> {
let mut training = state.training.lock().map_err(|e| e.to_string())?;
// Find the dataset
let dataset = training.datasets.iter_mut()
.find(|d| d.id == dataset_id)
.ok_or_else(|| format!("Dataset not found: {}", dataset_id))?;
// Simulate download completion
dataset.downloaded = true;
dataset.path = Some(format!("~/.ruview/datasets/{}", dataset_id));
Ok(dataset.clone())
}
// ============================================================================
// Model/Checkpoint Commands
// ============================================================================
/// List available model architectures.
#[tauri::command]
pub async fn list_models() -> Result<Vec<ModelInfo>, String> {
Ok(get_standard_models())
}
/// List saved checkpoints.
#[tauri::command]
pub async fn list_checkpoints(state: State<'_, AppState>) -> Result<Vec<CheckpointInfo>, String> {
let training = state.training.lock().map_err(|e| e.to_string())?;
Ok(training.checkpoints.clone())
}
/// Export a model checkpoint to ONNX or TorchScript.
#[tauri::command]
pub async fn export_model(
checkpoint_id: String,
format: String,
state: State<'_, AppState>,
) -> Result<String, String> {
let training = state.training.lock().map_err(|e| e.to_string())?;
let checkpoint = training.checkpoints.iter()
.find(|c| c.id == checkpoint_id)
.ok_or_else(|| format!("Checkpoint not found: {}", checkpoint_id))?;
let output_path = match format.as_str() {
"onnx" => format!("{}.onnx", checkpoint.path.trim_end_matches(".pt")),
"torchscript" => format!("{}.ts", checkpoint.path.trim_end_matches(".pt")),
_ => return Err(format!("Unsupported format: {}", format)),
};
// In a real implementation, this would call the actual export logic
Ok(output_path)
}
// ============================================================================
// Training Job Commands
// ============================================================================
/// Start a training job.
#[tauri::command]
pub async fn start_training(
config: TrainingConfig,
state: State<'_, AppState>,
) -> Result<String, String> {
let mut training = state.training.lock().map_err(|e| e.to_string())?;
// Create a new job
let job_id = uuid::Uuid::new_v4().to_string();
let job = TrainingJob {
id: job_id.clone(),
config,
status: TrainingStatus::Running,
started_at: Some(chrono::Utc::now().to_rfc3339()),
progress: TrainingProgress::default(),
loss_history: Vec::new(),
};
training.current_job = Some(job);
// In a real implementation, this would spawn a background training thread
// and emit progress events via Tauri's event system
Ok(job_id)
}
/// Stop the current training job.
#[tauri::command]
pub async fn stop_training(state: State<'_, AppState>) -> Result<(), String> {
let mut training = state.training.lock().map_err(|e| e.to_string())?;
if let Some(ref mut job) = training.current_job {
job.status = TrainingStatus::Paused;
}
Ok(())
}
/// Get current training progress.
#[tauri::command]
pub async fn training_progress(state: State<'_, AppState>) -> Result<Option<TrainingProgress>, String> {
let training = state.training.lock().map_err(|e| e.to_string())?;
Ok(training.current_job.as_ref().map(|j| j.progress.clone()))
}
// ============================================================================
// RuVector Configuration Commands
// ============================================================================
/// Get current RuVector configuration.
#[tauri::command]
pub async fn get_ruvector_config(state: State<'_, AppState>) -> Result<RuVectorConfig, String> {
let training = state.training.lock().map_err(|e| e.to_string())?;
Ok(training.ruvector_config.clone())
}
/// Set RuVector configuration.
#[tauri::command]
pub async fn set_ruvector_config(
config: RuVectorConfig,
state: State<'_, AppState>,
) -> Result<(), String> {
let mut training = state.training.lock().map_err(|e| e.to_string())?;
training.ruvector_config = config;
Ok(())
}
/// Test RuVector modules on live CSI data.
#[tauri::command]
pub async fn test_ruvector_live(
_state: State<'_, AppState>,
) -> Result<LiveTestMetrics, String> {
// In a real implementation, this would process live CSI data
// through the RuVector pipeline and return metrics
Ok(LiveTestMetrics {
fps: 30.0,
latency_ms: 15.0,
persons_detected: 1,
})
}
// ============================================================================
// Metrics Commands
// ============================================================================
/// Get training history (loss/accuracy per epoch).
#[tauri::command]
pub async fn get_training_history(state: State<'_, AppState>) -> Result<Vec<EpochMetrics>, String> {
let training = state.training.lock().map_err(|e| e.to_string())?;
Ok(training.training_history.clone())
}
/// Get evaluation metrics.
#[tauri::command]
pub async fn get_evaluation_metrics(state: State<'_, AppState>) -> Result<Option<EvaluationMetrics>, String> {
let training = state.training.lock().map_err(|e| e.to_string())?;
Ok(training.evaluation_metrics.clone())
}
/// Get per-joint accuracy metrics.
#[tauri::command]
pub async fn get_joint_accuracies(state: State<'_, AppState>) -> Result<Vec<JointAccuracy>, String> {
let training = state.training.lock().map_err(|e| e.to_string())?;
Ok(training.joint_accuracies.clone())
}
// ============================================================================
// Tests
// ============================================================================
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_standard_datasets() {
let datasets = get_standard_datasets();
assert_eq!(datasets.len(), 3);
assert!(datasets.iter().any(|d| d.id == "mmfi"));
}
#[test]
fn test_standard_models() {
let models = get_standard_models();
assert_eq!(models.len(), 6);
assert!(models.iter().any(|m| m.id == "csi-encoder-cnn"));
}
#[test]
fn test_detect_gpu_internal() {
let info = detect_gpu_internal();
// Just verify it returns valid data
assert!(matches!(info.backend, GpuBackend::Cpu | GpuBackend::Cuda | GpuBackend::Metal));
}
#[test]
fn test_ruvector_config_default() {
let config = RuVectorConfig::default();
assert!(config.mincut_enabled);
assert_eq!(config.attention_heads, 4);
}
}
@@ -1,4 +1,3 @@
pub mod config;
pub mod firmware;
pub mod node;
pub mod training;
@@ -1,312 +0,0 @@
//! Training domain types for the desktop application.
use serde::{Deserialize, Serialize};
/// GPU backend type.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
#[serde(rename_all = "lowercase")]
pub enum GpuBackend {
Cuda,
Metal,
#[default]
Cpu,
}
/// GPU information.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct GpuInfo {
pub available: bool,
pub backend: GpuBackend,
pub name: Option<String>,
pub memory_mb: Option<u64>,
pub cuda_version: Option<String>,
pub metal_supported: bool,
}
/// Dataset format type.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
#[serde(rename_all = "lowercase")]
pub enum DatasetFormat {
#[default]
MmFi,
WiPose,
Wiar,
Custom,
}
/// Dataset information.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct DatasetInfo {
pub id: String,
pub name: String,
pub description: String,
pub format: DatasetFormat,
pub size_mb: f64,
pub samples: u64,
pub downloaded: bool,
pub path: Option<String>,
pub url: Option<String>,
}
/// Model architecture type.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
#[serde(rename_all = "lowercase")]
pub enum ModelType {
#[default]
Encoder,
Decoder,
Embedding,
Adaptor,
}
/// Model architecture information.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ModelInfo {
pub id: String,
pub name: String,
pub model_type: ModelType,
pub description: String,
pub params_m: f64,
pub memory_mb: u64,
pub paper: Option<String>,
}
/// Checkpoint information.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CheckpointInfo {
pub id: String,
pub model_id: String,
pub name: String,
pub epoch: u32,
pub val_loss: f64,
pub created_at: String,
pub path: String,
pub size_mb: f64,
}
/// Training configuration.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TrainingConfig {
pub dataset_id: String,
pub model_id: String,
pub epochs: u32,
pub batch_size: u32,
pub learning_rate: f64,
pub optimizer: OptimizerType,
pub weight_decay: f64,
pub use_augmentation: bool,
pub checkpoint_every: u32,
}
impl Default for TrainingConfig {
fn default() -> Self {
Self {
dataset_id: "mmfi".into(),
model_id: "csi-encoder-cnn".into(),
epochs: 100,
batch_size: 32,
learning_rate: 0.001,
optimizer: OptimizerType::Adam,
weight_decay: 0.0001,
use_augmentation: true,
checkpoint_every: 10,
}
}
}
/// Optimizer type.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
#[serde(rename_all = "lowercase")]
pub enum OptimizerType {
#[default]
Adam,
AdamW,
Sgd,
}
/// Training job status.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
#[serde(rename_all = "lowercase")]
pub enum TrainingStatus {
#[default]
Pending,
Running,
Paused,
Completed,
Failed,
}
/// Training progress.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct TrainingProgress {
pub epoch: u32,
pub total_epochs: u32,
pub batch: u32,
pub total_batches: u32,
pub train_loss: f64,
pub val_loss: Option<f64>,
pub learning_rate: f64,
pub eta_secs: u64,
pub gpu_memory_mb: Option<u64>,
}
/// Training job.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct TrainingJob {
pub id: String,
pub config: TrainingConfig,
pub status: TrainingStatus,
pub started_at: Option<String>,
pub progress: TrainingProgress,
pub loss_history: Vec<EpochMetrics>,
}
/// Metrics for a single epoch.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct EpochMetrics {
pub epoch: u32,
pub train_loss: f64,
pub val_loss: f64,
pub train_acc: f64,
pub val_acc: f64,
pub learning_rate: f64,
pub timestamp: String,
}
/// Evaluation metrics.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct EvaluationMetrics {
pub pck_05: f64,
pub pck_10: f64,
pub pck_20: f64,
pub map_50: f64,
pub map_75: f64,
pub iou: f64,
}
/// Per-joint accuracy.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct JointAccuracy {
pub joint: String,
pub accuracy: f64,
}
/// RuVector interpolation mode.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
#[serde(rename_all = "lowercase")]
pub enum InterpolationMode {
Linear,
Cubic,
#[default]
Sparse,
}
/// RuVector module configuration.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RuVectorConfig {
// MinCut parameters
pub mincut_enabled: bool,
pub mincut_threshold: f64,
pub mincut_max_persons: u32,
// Attention parameters
pub attention_enabled: bool,
pub attention_heads: u32,
pub attention_dropout: f64,
// Temporal parameters
pub temporal_enabled: bool,
pub temporal_window_ms: u32,
pub temporal_compression_ratio: u32,
// Solver parameters
pub solver_enabled: bool,
pub solver_interpolation: InterpolationMode,
pub solver_subcarrier_count: u32,
// BVP parameters
pub bvp_enabled: bool,
pub bvp_filter_hz: (f64, f64),
}
impl Default for RuVectorConfig {
fn default() -> Self {
Self {
mincut_enabled: true,
mincut_threshold: 0.5,
mincut_max_persons: 5,
attention_enabled: true,
attention_heads: 4,
attention_dropout: 0.1,
temporal_enabled: true,
temporal_window_ms: 500,
temporal_compression_ratio: 4,
solver_enabled: true,
solver_interpolation: InterpolationMode::Sparse,
solver_subcarrier_count: 56,
bvp_enabled: false,
bvp_filter_hz: (0.7, 4.0),
}
}
}
/// Live test metrics from RuVector processing.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct LiveTestMetrics {
pub fps: f64,
pub latency_ms: f64,
pub persons_detected: u32,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_gpu_info_default() {
let info = GpuInfo::default();
assert!(!info.available);
assert_eq!(info.backend, GpuBackend::Cpu);
}
#[test]
fn test_training_config_default() {
let config = TrainingConfig::default();
assert_eq!(config.epochs, 100);
assert_eq!(config.batch_size, 32);
assert_eq!(config.optimizer, OptimizerType::Adam);
}
#[test]
fn test_ruvector_config_default() {
let config = RuVectorConfig::default();
assert!(config.mincut_enabled);
assert_eq!(config.mincut_threshold, 0.5);
assert_eq!(config.attention_heads, 4);
}
#[test]
fn test_serialization() {
let config = TrainingConfig::default();
let json = serde_json::to_string(&config).unwrap();
let parsed: TrainingConfig = serde_json::from_str(&json).unwrap();
assert_eq!(parsed.epochs, config.epochs);
}
#[test]
fn test_dataset_info() {
let dataset = DatasetInfo {
id: "mmfi".into(),
name: "MM-Fi Dataset".into(),
description: "Multi-modal WiFi sensing".into(),
format: DatasetFormat::MmFi,
size_mb: 2400.0,
samples: 320000,
downloaded: false,
path: None,
url: Some("https://example.com/mmfi.zip".into()),
};
assert_eq!(dataset.id, "mmfi");
assert!(!dataset.downloaded);
}
}
@@ -2,7 +2,7 @@ pub mod commands;
pub mod domain;
pub mod state;
use commands::{discovery, flash, ota, provision, server, settings, training, wasm};
use commands::{discovery, flash, ota, provision, server, settings, wasm};
pub fn run() {
tauri::Builder::default()
@@ -46,23 +46,6 @@ pub fn run() {
// Settings
settings::get_settings,
settings::save_settings,
// Training
training::detect_gpu,
training::list_datasets,
training::get_datasets,
training::download_dataset,
training::list_models,
training::list_checkpoints,
training::export_model,
training::start_training,
training::stop_training,
training::training_progress,
training::get_ruvector_config,
training::set_ruvector_config,
training::test_ruvector_live,
training::get_training_history,
training::get_evaluation_metrics,
training::get_joint_accuracies,
])
.run(tauri::generate_context!())
.expect("error while running tauri application");
@@ -3,10 +3,6 @@ use std::sync::Mutex;
use std::time::Instant;
use crate::domain::node::DiscoveredNode;
use crate::domain::training::{
CheckpointInfo, DatasetInfo, EpochMetrics, EvaluationMetrics,
GpuInfo, JointAccuracy, RuVectorConfig, TrainingJob,
};
/// Sub-state for discovered nodes.
#[derive(Default)]
@@ -91,33 +87,6 @@ impl Default for SettingsState {
}
}
/// Sub-state for training operations.
pub struct TrainingState {
pub gpu_info: Option<GpuInfo>,
pub datasets: Vec<DatasetInfo>,
pub checkpoints: Vec<CheckpointInfo>,
pub current_job: Option<TrainingJob>,
pub ruvector_config: RuVectorConfig,
pub training_history: Vec<EpochMetrics>,
pub evaluation_metrics: Option<EvaluationMetrics>,
pub joint_accuracies: Vec<JointAccuracy>,
}
impl Default for TrainingState {
fn default() -> Self {
Self {
gpu_info: None,
datasets: Vec::new(),
checkpoints: Vec::new(),
current_job: None,
ruvector_config: RuVectorConfig::default(),
training_history: Vec::new(),
evaluation_metrics: None,
joint_accuracies: Vec::new(),
}
}
}
/// Top-level application state managed by Tauri.
pub struct AppState {
pub discovery: Mutex<DiscoveryState>,
@@ -125,7 +94,6 @@ pub struct AppState {
pub flash: Mutex<FlashState>,
pub ota: Mutex<OtaState>,
pub settings: Mutex<SettingsState>,
pub training: Mutex<TrainingState>,
}
impl Default for AppState {
@@ -136,7 +104,6 @@ impl Default for AppState {
flash: Mutex::new(FlashState::default()),
ota: Mutex::new(OtaState::default()),
settings: Mutex::new(SettingsState::default()),
training: Mutex::new(TrainingState::default()),
}
}
}
@@ -168,9 +135,6 @@ impl AppState {
if let Ok(mut settings) = self.settings.lock() {
*settings = SettingsState::default();
}
if let Ok(mut training) = self.training.lock() {
*training = TrainingState::default();
}
}
}
@@ -1,7 +1,7 @@
{
"$schema": "https://raw.githubusercontent.com/tauri-apps/tauri/dev/crates/tauri-config-schema/schema.json",
"productName": "RuView Desktop",
"version": "0.5.0",
"version": "0.4.4",
"identifier": "net.ruv.ruview",
"build": {
"frontendDist": "ui/dist",
@@ -1,7 +1,7 @@
{
"name": "ruview-desktop-ui",
"private": true,
"version": "0.5.0",
"version": "0.4.4",
"type": "module",
"scripts": {
"dev": "vite",
@@ -8,7 +8,6 @@ import { OtaUpdate } from "./pages/OtaUpdate";
import { EdgeModules } from "./pages/EdgeModules";
import { Sensing } from "./pages/Sensing";
import { MeshView } from "./pages/MeshView";
import Training from "./pages/Training";
import { Settings } from "./pages/Settings";
type Page =
@@ -20,7 +19,6 @@ type Page =
| "wasm"
| "sensing"
| "mesh"
| "training"
| "settings";
interface NavItem {
@@ -38,7 +36,6 @@ const NAV_ITEMS: NavItem[] = [
{ id: "wasm", label: "Edge Modules", icon: "\u2B21" },
{ id: "sensing", label: "Sensing", icon: "\u2248" },
{ id: "mesh", label: "Mesh View", icon: "\u2B2F" },
{ id: "training", label: "Training", icon: "\u2B50" },
{ id: "settings", label: "Settings", icon: "\u2699" },
];
@@ -102,7 +99,6 @@ const App: React.FC = () => {
case "wasm": return <EdgeModules />;
case "sensing": return <Sensing />;
case "mesh": return <MeshView />;
case "training": return <Training />;
case "settings": return <Settings />;
}
};
@@ -1,369 +0,0 @@
import React, { useState, useEffect } from "react";
import { invoke } from "@tauri-apps/api/core";
interface Dataset {
id: string;
name: string;
description: string;
size_mb: number;
samples: number;
downloaded: boolean;
path: string | null;
}
const STANDARD_DATASETS: Omit<Dataset, "downloaded" | "path">[] = [
{
id: "mmfi",
name: "MM-Fi Dataset",
description: "Multi-modal WiFi sensing dataset with 40 subjects, 27 activities",
size_mb: 2400,
samples: 320000,
},
{
id: "wipose",
name: "Wi-Pose Dataset",
description: "WiFi-based pose estimation with 3D skeleton annotations",
size_mb: 1800,
samples: 150000,
},
{
id: "wiar",
name: "WiAR Dataset",
description: "WiFi activity recognition with CSI data",
size_mb: 500,
samples: 45000,
},
];
const DatasetsTab: React.FC = () => {
const [datasets, setDatasets] = useState<Dataset[]>([]);
const [downloading, setDownloading] = useState<string | null>(null);
const [downloadProgress, setDownloadProgress] = useState<number>(0);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
loadDatasets();
}, []);
const loadDatasets = async () => {
try {
const downloaded = await invoke<string[]>("list_datasets");
const ds = STANDARD_DATASETS.map((d) => ({
...d,
downloaded: downloaded.includes(d.id),
path: downloaded.includes(d.id) ? `~/.ruview/datasets/${d.id}` : null,
}));
setDatasets(ds);
} catch (err) {
// If command not implemented yet, show placeholders
setDatasets(
STANDARD_DATASETS.map((d) => ({
...d,
downloaded: false,
path: null,
}))
);
}
};
const handleDownload = async (datasetId: string) => {
setDownloading(datasetId);
setDownloadProgress(0);
setError(null);
try {
// Simulate download progress for now
for (let i = 0; i <= 100; i += 10) {
setDownloadProgress(i);
await new Promise((r) => setTimeout(r, 500));
}
// TODO: Call actual download command
// await invoke("download_dataset", { datasetId });
setDatasets((prev) =>
prev.map((d) =>
d.id === datasetId
? { ...d, downloaded: true, path: `~/.ruview/datasets/${d.id}` }
: d
)
);
} catch (err) {
setError(`Download failed: ${err}`);
} finally {
setDownloading(null);
}
};
return (
<div>
{/* Stats Row */}
<div
style={{
display: "grid",
gridTemplateColumns: "repeat(3, 1fr)",
gap: "var(--space-4)",
marginBottom: "var(--space-5)",
}}
>
<StatCard
label="Available Datasets"
value={datasets.length}
/>
<StatCard
label="Downloaded"
value={datasets.filter((d) => d.downloaded).length}
color="var(--status-online)"
/>
<StatCard
label="Total Samples"
value={`${(datasets.reduce((acc, d) => acc + (d.downloaded ? d.samples : 0), 0) / 1000).toFixed(0)}K`}
/>
</div>
{error && (
<div
style={{
background: "rgba(248, 81, 73, 0.1)",
border: "1px solid rgba(248, 81, 73, 0.3)",
borderRadius: 6,
padding: "var(--space-3)",
marginBottom: "var(--space-4)",
fontSize: 13,
color: "var(--status-error)",
}}
>
{error}
</div>
)}
{/* Dataset Cards */}
<div
style={{
display: "grid",
gridTemplateColumns: "repeat(auto-fill, minmax(340px, 1fr))",
gap: "var(--space-4)",
}}
>
{datasets.map((dataset) => (
<div
key={dataset.id}
className="card"
style={{
padding: "var(--space-4)",
opacity: dataset.downloaded ? 1 : 0.85,
}}
>
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "start",
marginBottom: "var(--space-3)",
}}
>
<div>
<h3 style={{ margin: 0, fontSize: 15, fontWeight: 600 }}>
{dataset.name}
</h3>
<p
style={{
fontSize: 12,
color: "var(--text-muted)",
marginTop: 4,
lineHeight: 1.4,
}}
>
{dataset.description}
</p>
</div>
{dataset.downloaded && (
<span
style={{
background: "rgba(63, 185, 80, 0.15)",
color: "var(--status-online)",
padding: "2px 8px",
borderRadius: 4,
fontSize: 10,
fontWeight: 600,
}}
>
DOWNLOADED
</span>
)}
</div>
<div
style={{
display: "flex",
gap: "var(--space-4)",
fontSize: 12,
color: "var(--text-secondary)",
marginBottom: "var(--space-3)",
}}
>
<span>📦 {(dataset.size_mb / 1024).toFixed(1)} GB</span>
<span>📊 {(dataset.samples / 1000).toFixed(0)}K samples</span>
</div>
{downloading === dataset.id ? (
<div>
<div
style={{
height: 4,
background: "var(--border)",
borderRadius: 2,
overflow: "hidden",
}}
>
<div
style={{
width: `${downloadProgress}%`,
height: "100%",
background: "var(--accent)",
transition: "width 0.3s",
}}
/>
</div>
<div
style={{
fontSize: 11,
color: "var(--text-muted)",
marginTop: 4,
textAlign: "center",
}}
>
Downloading... {downloadProgress}%
</div>
</div>
) : (
<div style={{ display: "flex", gap: "var(--space-2)" }}>
{dataset.downloaded ? (
<>
<button
style={{
flex: 1,
padding: "8px 12px",
background: "rgba(56, 139, 253, 0.1)",
border: "1px solid rgba(56, 139, 253, 0.3)",
borderRadius: 6,
color: "var(--accent)",
fontSize: 12,
fontWeight: 600,
cursor: "pointer",
}}
>
Preview
</button>
<button
style={{
padding: "8px 12px",
background: "transparent",
border: "1px solid var(--border)",
borderRadius: 6,
color: "var(--text-secondary)",
fontSize: 12,
fontWeight: 600,
cursor: "pointer",
}}
>
Delete
</button>
</>
) : (
<button
onClick={() => handleDownload(dataset.id)}
className="btn-gradient"
style={{ flex: 1, fontSize: 12 }}
>
Download Dataset
</button>
)}
</div>
)}
</div>
))}
</div>
{/* Import Custom Dataset */}
<div
className="card"
style={{
marginTop: "var(--space-5)",
padding: "var(--space-4)",
border: "2px dashed var(--border)",
textAlign: "center",
}}
>
<div style={{ fontSize: 32, marginBottom: "var(--space-2)" }}>📁</div>
<h4 style={{ margin: 0, fontSize: 14, fontWeight: 600 }}>
Import Custom Dataset
</h4>
<p
style={{
fontSize: 12,
color: "var(--text-muted)",
marginTop: 4,
marginBottom: "var(--space-3)",
}}
>
Import CSI recordings in CSV, NPZ, or HDF5 format
</p>
<button
style={{
padding: "8px 20px",
background: "transparent",
border: "1px solid var(--border)",
borderRadius: 6,
color: "var(--text-secondary)",
fontSize: 12,
fontWeight: 600,
cursor: "pointer",
}}
>
Browse Files
</button>
</div>
</div>
);
};
function StatCard({
label,
value,
color,
}: {
label: string;
value: number | string;
color?: string;
}) {
return (
<div className="card-glow" style={{ padding: "var(--space-4)" }}>
<div
style={{
fontSize: 10,
textTransform: "uppercase",
letterSpacing: "0.06em",
color: "var(--text-muted)",
marginBottom: "var(--space-2)",
fontWeight: 600,
}}
>
{label}
</div>
<div
style={{
fontFamily: "var(--font-mono)",
fontSize: 28,
fontWeight: 600,
color: color || "var(--text-primary)",
letterSpacing: "-0.02em",
}}
>
{value}
</div>
</div>
);
}
export default DatasetsTab;
@@ -1,609 +0,0 @@
import React, { useState, useEffect } from "react";
import { invoke } from "@tauri-apps/api/core";
interface TrainingMetrics {
epoch: number;
train_loss: number;
val_loss: number;
train_acc: number;
val_acc: number;
learning_rate: number;
timestamp: string;
}
interface EvaluationMetrics {
pck_05: number;
pck_10: number;
pck_20: number;
map_50: number;
map_75: number;
iou: number;
}
interface JointAccuracy {
joint: string;
accuracy: number;
}
const JOINT_NAMES = [
"nose",
"left_eye",
"right_eye",
"left_ear",
"right_ear",
"left_shoulder",
"right_shoulder",
"left_elbow",
"right_elbow",
"left_wrist",
"right_wrist",
"left_hip",
"right_hip",
"left_knee",
"right_knee",
"left_ankle",
"right_ankle",
];
const MetricsTab: React.FC = () => {
const [trainingHistory, setTrainingHistory] = useState<TrainingMetrics[]>([]);
const [evaluation, setEvaluation] = useState<EvaluationMetrics | null>(null);
const [jointAccuracies, setJointAccuracies] = useState<JointAccuracy[]>([]);
const [selectedMetric, setSelectedMetric] = useState<"loss" | "accuracy">("loss");
const [exporting, setExporting] = useState(false);
useEffect(() => {
loadMetrics();
}, []);
const loadMetrics = async () => {
try {
const metrics = await invoke<TrainingMetrics[]>("get_training_history");
setTrainingHistory(metrics);
const evalMetrics = await invoke<EvaluationMetrics>("get_evaluation_metrics");
setEvaluation(evalMetrics);
const joints = await invoke<JointAccuracy[]>("get_joint_accuracies");
setJointAccuracies(joints);
} catch (err) {
// Generate mock data for demonstration
const mockHistory: TrainingMetrics[] = [];
for (let i = 1; i <= 50; i++) {
mockHistory.push({
epoch: i,
train_loss: 0.5 * Math.exp(-i / 20) + 0.02 + Math.random() * 0.01,
val_loss: 0.55 * Math.exp(-i / 18) + 0.025 + Math.random() * 0.015,
train_acc: 1 - 0.5 * Math.exp(-i / 15) - Math.random() * 0.02,
val_acc: 1 - 0.55 * Math.exp(-i / 15) - Math.random() * 0.025,
learning_rate: 0.001 * Math.pow(0.95, Math.floor(i / 10)),
timestamp: new Date(Date.now() - (50 - i) * 60000).toISOString(),
});
}
setTrainingHistory(mockHistory);
setEvaluation({
pck_05: 0.72,
pck_10: 0.89,
pck_20: 0.96,
map_50: 0.84,
map_75: 0.71,
iou: 0.78,
});
setJointAccuracies(
JOINT_NAMES.map((joint) => ({
joint,
accuracy: 0.7 + Math.random() * 0.25,
}))
);
}
};
const exportMetrics = async (format: "csv" | "json" | "tensorboard") => {
setExporting(true);
try {
if (format === "json") {
const data = {
training: trainingHistory,
evaluation,
joints: jointAccuracies,
};
const blob = new Blob([JSON.stringify(data, null, 2)], { type: "application/json" });
downloadBlob(blob, "metrics.json");
} else if (format === "csv") {
const headers = "epoch,train_loss,val_loss,train_acc,val_acc,learning_rate\n";
const rows = trainingHistory
.map(
(m) =>
`${m.epoch},${m.train_loss.toFixed(6)},${m.val_loss.toFixed(6)},${m.train_acc.toFixed(4)},${m.val_acc.toFixed(4)},${m.learning_rate.toExponential(2)}`
)
.join("\n");
const blob = new Blob([headers + rows], { type: "text/csv" });
downloadBlob(blob, "training_history.csv");
} else {
// TensorBoard format would require server-side handling
alert("TensorBoard export requires running the backend server");
}
} finally {
setExporting(false);
}
};
const downloadBlob = (blob: Blob, filename: string) => {
const url = URL.createObjectURL(blob);
const a = document.createElement("a");
a.href = url;
a.download = filename;
a.click();
URL.revokeObjectURL(url);
};
const maxLoss = Math.max(
...trainingHistory.map((m) => Math.max(m.train_loss, m.val_loss)),
0.1
);
return (
<div>
{/* Summary Stats */}
<div
style={{
display: "grid",
gridTemplateColumns: "repeat(4, 1fr)",
gap: "var(--space-4)",
marginBottom: "var(--space-5)",
}}
>
<StatCard
label="Epochs Trained"
value={trainingHistory.length}
/>
<StatCard
label="Best Val Loss"
value={
trainingHistory.length > 0
? Math.min(...trainingHistory.map((m) => m.val_loss)).toFixed(4)
: "—"
}
color="var(--status-online)"
/>
<StatCard
label="Best Val Acc"
value={
trainingHistory.length > 0
? `${(Math.max(...trainingHistory.map((m) => m.val_acc)) * 100).toFixed(1)}%`
: "—"
}
color="var(--accent)"
/>
<StatCard
label="PCK@0.1"
value={evaluation ? `${(evaluation.pck_10 * 100).toFixed(1)}%` : "—"}
/>
</div>
<div style={{ display: "grid", gridTemplateColumns: "2fr 1fr", gap: "var(--space-5)" }}>
{/* Loss/Accuracy Charts */}
<div className="card" style={{ padding: "var(--space-4)" }}>
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "center",
marginBottom: "var(--space-4)",
}}
>
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600 }}>Training Curves</h3>
<div style={{ display: "flex", gap: "var(--space-2)" }}>
<button
onClick={() => setSelectedMetric("loss")}
style={{
padding: "6px 12px",
background: selectedMetric === "loss" ? "var(--accent)" : "transparent",
border: `1px solid ${selectedMetric === "loss" ? "var(--accent)" : "var(--border)"}`,
borderRadius: 4,
color: selectedMetric === "loss" ? "white" : "var(--text-secondary)",
fontSize: 11,
fontWeight: 600,
cursor: "pointer",
}}
>
Loss
</button>
<button
onClick={() => setSelectedMetric("accuracy")}
style={{
padding: "6px 12px",
background: selectedMetric === "accuracy" ? "var(--accent)" : "transparent",
border: `1px solid ${selectedMetric === "accuracy" ? "var(--accent)" : "var(--border)"}`,
borderRadius: 4,
color: selectedMetric === "accuracy" ? "white" : "var(--text-secondary)",
fontSize: 11,
fontWeight: 600,
cursor: "pointer",
}}
>
Accuracy
</button>
</div>
</div>
{/* Chart Area */}
<div
style={{
height: 250,
position: "relative",
background: "var(--bg-secondary)",
borderRadius: 8,
padding: "var(--space-3)",
}}
>
{trainingHistory.length === 0 ? (
<div
style={{
display: "flex",
flexDirection: "column",
alignItems: "center",
justifyContent: "center",
height: "100%",
color: "var(--text-muted)",
}}
>
<span style={{ fontSize: 32 }}>📊</span>
<p style={{ fontSize: 13, marginTop: "var(--space-2)" }}>
No training data yet
</p>
</div>
) : (
<svg width="100%" height="100%" viewBox="0 0 500 200" preserveAspectRatio="none">
{/* Grid lines */}
{[0, 0.25, 0.5, 0.75, 1].map((y) => (
<line
key={y}
x1="0"
y1={y * 180}
x2="500"
y2={y * 180}
stroke="var(--border)"
strokeWidth="0.5"
strokeDasharray="4"
/>
))}
{/* Train line */}
<polyline
fill="none"
stroke="var(--accent)"
strokeWidth="2"
points={trainingHistory
.map((m, i) => {
const x = (i / (trainingHistory.length - 1)) * 500;
const value = selectedMetric === "loss" ? m.train_loss : m.train_acc;
const y =
selectedMetric === "loss"
? (value / maxLoss) * 180
: (1 - value) * 180;
return `${x},${y}`;
})
.join(" ")}
/>
{/* Val line */}
<polyline
fill="none"
stroke="var(--status-online)"
strokeWidth="2"
points={trainingHistory
.map((m, i) => {
const x = (i / (trainingHistory.length - 1)) * 500;
const value = selectedMetric === "loss" ? m.val_loss : m.val_acc;
const y =
selectedMetric === "loss"
? (value / maxLoss) * 180
: (1 - value) * 180;
return `${x},${y}`;
})
.join(" ")}
/>
</svg>
)}
{/* Legend */}
<div
style={{
position: "absolute",
top: "var(--space-2)",
right: "var(--space-2)",
display: "flex",
gap: "var(--space-3)",
fontSize: 11,
}}
>
<span style={{ display: "flex", alignItems: "center", gap: 4 }}>
<span
style={{
width: 12,
height: 3,
background: "var(--accent)",
borderRadius: 2,
}}
/>
Train
</span>
<span style={{ display: "flex", alignItems: "center", gap: 4 }}>
<span
style={{
width: 12,
height: 3,
background: "var(--status-online)",
borderRadius: 2,
}}
/>
Validation
</span>
</div>
</div>
</div>
{/* Evaluation Metrics */}
<div className="card" style={{ padding: "var(--space-4)" }}>
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
Evaluation Metrics
</h3>
{!evaluation ? (
<div
style={{
display: "flex",
flexDirection: "column",
alignItems: "center",
justifyContent: "center",
height: 200,
color: "var(--text-muted)",
}}
>
<span style={{ fontSize: 32 }}>📏</span>
<p style={{ fontSize: 13, marginTop: "var(--space-2)" }}>
Run evaluation to see metrics
</p>
</div>
) : (
<div style={{ display: "flex", flexDirection: "column", gap: "var(--space-3)" }}>
<MetricBar label="PCK@0.05" value={evaluation.pck_05} color="#f59e0b" />
<MetricBar label="PCK@0.10" value={evaluation.pck_10} color="var(--accent)" />
<MetricBar label="PCK@0.20" value={evaluation.pck_20} color="var(--status-online)" />
<div style={{ height: 1, background: "var(--border)", margin: "var(--space-2) 0" }} />
<MetricBar label="mAP@0.50" value={evaluation.map_50} color="#a855f7" />
<MetricBar label="mAP@0.75" value={evaluation.map_75} color="#ec4899" />
<MetricBar label="IoU" value={evaluation.iou} color="#06b6d4" />
</div>
)}
</div>
</div>
{/* Joint-wise Accuracy */}
<div className="card" style={{ marginTop: "var(--space-5)", padding: "var(--space-4)" }}>
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
Per-Joint Accuracy
</h3>
{jointAccuracies.length === 0 ? (
<div
style={{
textAlign: "center",
padding: "var(--space-5)",
color: "var(--text-muted)",
}}
>
No joint accuracy data available
</div>
) : (
<div
style={{
display: "grid",
gridTemplateColumns: "repeat(auto-fill, minmax(140px, 1fr))",
gap: "var(--space-3)",
}}
>
{jointAccuracies.map((ja) => (
<div
key={ja.joint}
style={{
padding: "var(--space-3)",
background: "var(--bg-secondary)",
borderRadius: 6,
textAlign: "center",
}}
>
<div
style={{
fontSize: 11,
color: "var(--text-muted)",
marginBottom: 4,
textTransform: "capitalize",
}}
>
{ja.joint.replace("_", " ")}
</div>
<div
style={{
fontFamily: "var(--font-mono)",
fontSize: 18,
fontWeight: 600,
color:
ja.accuracy > 0.9
? "var(--status-online)"
: ja.accuracy > 0.8
? "var(--accent)"
: ja.accuracy > 0.7
? "#f59e0b"
: "var(--status-error)",
}}
>
{(ja.accuracy * 100).toFixed(1)}%
</div>
</div>
))}
</div>
)}
</div>
{/* Export Section */}
<div
className="card"
style={{
marginTop: "var(--space-5)",
padding: "var(--space-4)",
display: "flex",
justifyContent: "space-between",
alignItems: "center",
}}
>
<div>
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600 }}>Export Metrics</h3>
<p style={{ fontSize: 12, color: "var(--text-muted)", marginTop: 4 }}>
Download training history and evaluation results
</p>
</div>
<div style={{ display: "flex", gap: "var(--space-2)" }}>
<button
onClick={() => exportMetrics("csv")}
disabled={exporting || trainingHistory.length === 0}
style={{
padding: "8px 16px",
background: "rgba(56, 139, 253, 0.1)",
border: "1px solid rgba(56, 139, 253, 0.3)",
borderRadius: 6,
color: "var(--accent)",
fontSize: 12,
fontWeight: 600,
cursor: trainingHistory.length === 0 ? "not-allowed" : "pointer",
opacity: trainingHistory.length === 0 ? 0.5 : 1,
}}
>
CSV
</button>
<button
onClick={() => exportMetrics("json")}
disabled={exporting || trainingHistory.length === 0}
style={{
padding: "8px 16px",
background: "rgba(56, 139, 253, 0.1)",
border: "1px solid rgba(56, 139, 253, 0.3)",
borderRadius: 6,
color: "var(--accent)",
fontSize: 12,
fontWeight: 600,
cursor: trainingHistory.length === 0 ? "not-allowed" : "pointer",
opacity: trainingHistory.length === 0 ? 0.5 : 1,
}}
>
JSON
</button>
<button
onClick={() => exportMetrics("tensorboard")}
disabled={exporting || trainingHistory.length === 0}
style={{
padding: "8px 16px",
background: "transparent",
border: "1px solid var(--border)",
borderRadius: 6,
color: "var(--text-secondary)",
fontSize: 12,
fontWeight: 600,
cursor: trainingHistory.length === 0 ? "not-allowed" : "pointer",
opacity: trainingHistory.length === 0 ? 0.5 : 1,
}}
>
TensorBoard
</button>
</div>
</div>
</div>
);
};
function StatCard({
label,
value,
color,
}: {
label: string;
value: number | string;
color?: string;
}) {
return (
<div className="card-glow" style={{ padding: "var(--space-4)" }}>
<div
style={{
fontSize: 10,
textTransform: "uppercase",
letterSpacing: "0.06em",
color: "var(--text-muted)",
marginBottom: "var(--space-2)",
fontWeight: 600,
}}
>
{label}
</div>
<div
style={{
fontFamily: "var(--font-mono)",
fontSize: 28,
fontWeight: 600,
color: color || "var(--text-primary)",
letterSpacing: "-0.02em",
}}
>
{value}
</div>
</div>
);
}
function MetricBar({
label,
value,
color,
}: {
label: string;
value: number;
color: string;
}) {
return (
<div>
<div
style={{
display: "flex",
justifyContent: "space-between",
fontSize: 12,
marginBottom: 4,
}}
>
<span>{label}</span>
<span style={{ fontFamily: "var(--font-mono)", fontWeight: 600 }}>
{(value * 100).toFixed(1)}%
</span>
</div>
<div
style={{
height: 6,
background: "var(--bg-secondary)",
borderRadius: 3,
overflow: "hidden",
}}
>
<div
style={{
width: `${value * 100}%`,
height: "100%",
background: color,
borderRadius: 3,
transition: "width 0.5s",
}}
/>
</div>
</div>
);
}
export default MetricsTab;
@@ -1,405 +0,0 @@
import React, { useState, useEffect } from "react";
import { invoke } from "@tauri-apps/api/core";
interface ModelArchitecture {
id: string;
name: string;
type: "encoder" | "decoder" | "embedding" | "adaptor";
description: string;
params_m: number;
memory_mb: number;
paper?: string;
}
interface Checkpoint {
id: string;
model_id: string;
name: string;
epoch: number;
val_loss: number;
created_at: string;
path: string;
size_mb: number;
}
const MODEL_ARCHITECTURES: ModelArchitecture[] = [
{
id: "csi-encoder-cnn",
name: "CSI Encoder (CNN)",
type: "encoder",
description: "Convolutional encoder for CSI amplitude/phase features",
params_m: 2.3,
memory_mb: 128,
},
{
id: "csi-encoder-transformer",
name: "CSI Encoder (Transformer)",
type: "encoder",
description: "Self-attention based CSI feature extraction",
params_m: 8.5,
memory_mb: 384,
paper: "WiFi-ViT 2024",
},
{
id: "pose-decoder-lstm",
name: "Pose Decoder (LSTM)",
type: "decoder",
description: "Recurrent decoder for temporal pose estimation",
params_m: 1.8,
memory_mb: 96,
},
{
id: "pose-decoder-gru",
name: "Pose Decoder (GRU)",
type: "decoder",
description: "Gated recurrent unit pose decoder (faster)",
params_m: 1.2,
memory_mb: 64,
},
{
id: "aether-embedding",
name: "AETHER Embedding",
type: "embedding",
description: "Contrastive CSI embedding for person re-identification (ADR-024)",
params_m: 4.2,
memory_mb: 192,
paper: "AETHER 2025",
},
{
id: "meridian-adaptor",
name: "MERIDIAN Adaptor",
type: "adaptor",
description: "Cross-environment domain generalization module (ADR-027)",
params_m: 3.1,
memory_mb: 144,
paper: "MERIDIAN 2025",
},
];
const ModelsTab: React.FC = () => {
const [checkpoints, setCheckpoints] = useState<Checkpoint[]>([]);
const [selectedModel, setSelectedModel] = useState<string | null>(null);
const [exporting, setExporting] = useState<string | null>(null);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
loadCheckpoints();
}, []);
const loadCheckpoints = async () => {
try {
const loaded = await invoke<Checkpoint[]>("list_checkpoints");
setCheckpoints(loaded);
} catch (err) {
// Mock data if command not implemented
setCheckpoints([
{
id: "ckpt-001",
model_id: "csi-encoder-cnn",
name: "CSI-CNN v1.2",
epoch: 50,
val_loss: 0.0234,
created_at: "2026-03-08T14:30:00Z",
path: "~/.ruview/models/csi-cnn-v1.2.pt",
size_mb: 12.4,
},
{
id: "ckpt-002",
model_id: "pose-decoder-gru",
name: "Pose-GRU v2.0",
epoch: 100,
val_loss: 0.0189,
created_at: "2026-03-09T09:15:00Z",
path: "~/.ruview/models/pose-gru-v2.pt",
size_mb: 8.2,
},
]);
}
};
const handleExport = async (checkpointId: string, format: "onnx" | "torchscript") => {
setExporting(checkpointId);
setError(null);
try {
await invoke("export_model", { checkpointId, format });
// Success notification would go here
} catch (err) {
setError(`Export failed: ${err}`);
} finally {
setExporting(null);
}
};
const getTypeColor = (type: ModelArchitecture["type"]) => {
switch (type) {
case "encoder":
return "var(--accent)";
case "decoder":
return "var(--status-online)";
case "embedding":
return "#a855f7";
case "adaptor":
return "#f59e0b";
}
};
return (
<div>
{/* Stats Row */}
<div
style={{
display: "grid",
gridTemplateColumns: "repeat(4, 1fr)",
gap: "var(--space-4)",
marginBottom: "var(--space-5)",
}}
>
<StatCard label="Architectures" value={MODEL_ARCHITECTURES.length} />
<StatCard
label="Checkpoints"
value={checkpoints.length}
color="var(--status-online)"
/>
<StatCard
label="Total Params"
value={`${MODEL_ARCHITECTURES.reduce((acc, m) => acc + m.params_m, 0).toFixed(1)}M`}
/>
<StatCard
label="Storage Used"
value={`${checkpoints.reduce((acc, c) => acc + c.size_mb, 0).toFixed(1)} MB`}
/>
</div>
{error && (
<div
style={{
background: "rgba(248, 81, 73, 0.1)",
border: "1px solid rgba(248, 81, 73, 0.3)",
borderRadius: 6,
padding: "var(--space-3)",
marginBottom: "var(--space-4)",
fontSize: 13,
color: "var(--status-error)",
}}
>
{error}
</div>
)}
{/* Model Architectures */}
<h3 style={{ fontSize: 14, fontWeight: 600, marginBottom: "var(--space-3)" }}>
Available Architectures
</h3>
<div
style={{
display: "grid",
gridTemplateColumns: "repeat(auto-fill, minmax(300px, 1fr))",
gap: "var(--space-3)",
marginBottom: "var(--space-5)",
}}
>
{MODEL_ARCHITECTURES.map((model) => (
<div
key={model.id}
className="card"
style={{
padding: "var(--space-3)",
cursor: "pointer",
border:
selectedModel === model.id
? "1px solid var(--accent)"
: "1px solid transparent",
}}
onClick={() => setSelectedModel(model.id)}
>
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "start",
marginBottom: "var(--space-2)",
}}
>
<div>
<h4 style={{ margin: 0, fontSize: 14, fontWeight: 600 }}>
{model.name}
</h4>
<span
style={{
display: "inline-block",
marginTop: 4,
padding: "1px 6px",
borderRadius: 3,
fontSize: 10,
fontWeight: 600,
textTransform: "uppercase",
background: `${getTypeColor(model.type)}20`,
color: getTypeColor(model.type),
}}
>
{model.type}
</span>
</div>
{model.paper && (
<span
style={{
fontSize: 10,
color: "var(--text-muted)",
fontStyle: "italic",
}}
>
{model.paper}
</span>
)}
</div>
<p
style={{
fontSize: 11,
color: "var(--text-muted)",
margin: "var(--space-2) 0",
lineHeight: 1.4,
}}
>
{model.description}
</p>
<div
style={{
display: "flex",
gap: "var(--space-3)",
fontSize: 11,
color: "var(--text-secondary)",
}}
>
<span>🧮 {model.params_m}M params</span>
<span>💾 {model.memory_mb} MB</span>
</div>
</div>
))}
</div>
{/* Checkpoints */}
<h3 style={{ fontSize: 14, fontWeight: 600, marginBottom: "var(--space-3)" }}>
Saved Checkpoints
</h3>
{checkpoints.length === 0 ? (
<div
className="card"
style={{
padding: "var(--space-5)",
textAlign: "center",
color: "var(--text-muted)",
}}
>
<div style={{ fontSize: 32, marginBottom: "var(--space-2)" }}>📦</div>
<p style={{ fontSize: 13 }}>No checkpoints saved yet</p>
<p style={{ fontSize: 12 }}>Train a model to create checkpoints</p>
</div>
) : (
<div style={{ display: "flex", flexDirection: "column", gap: "var(--space-2)" }}>
{checkpoints.map((ckpt) => (
<div
key={ckpt.id}
className="card"
style={{
padding: "var(--space-3)",
display: "flex",
justifyContent: "space-between",
alignItems: "center",
}}
>
<div>
<div style={{ fontWeight: 600, fontSize: 13 }}>{ckpt.name}</div>
<div
style={{
fontSize: 11,
color: "var(--text-muted)",
marginTop: 2,
}}
>
Epoch {ckpt.epoch} Val Loss: {ckpt.val_loss.toFixed(4)} {" "}
{ckpt.size_mb.toFixed(1)} MB
</div>
</div>
<div style={{ display: "flex", gap: "var(--space-2)" }}>
<button
onClick={() => handleExport(ckpt.id, "onnx")}
disabled={exporting === ckpt.id}
style={{
padding: "6px 12px",
background: "rgba(56, 139, 253, 0.1)",
border: "1px solid rgba(56, 139, 253, 0.3)",
borderRadius: 4,
color: "var(--accent)",
fontSize: 11,
fontWeight: 600,
cursor: exporting === ckpt.id ? "wait" : "pointer",
opacity: exporting === ckpt.id ? 0.6 : 1,
}}
>
{exporting === ckpt.id ? "Exporting..." : "ONNX"}
</button>
<button
onClick={() => handleExport(ckpt.id, "torchscript")}
disabled={exporting === ckpt.id}
style={{
padding: "6px 12px",
background: "transparent",
border: "1px solid var(--border)",
borderRadius: 4,
color: "var(--text-secondary)",
fontSize: 11,
fontWeight: 600,
cursor: exporting === ckpt.id ? "wait" : "pointer",
opacity: exporting === ckpt.id ? 0.6 : 1,
}}
>
TorchScript
</button>
</div>
</div>
))}
</div>
)}
</div>
);
};
function StatCard({
label,
value,
color,
}: {
label: string;
value: number | string;
color?: string;
}) {
return (
<div className="card-glow" style={{ padding: "var(--space-4)" }}>
<div
style={{
fontSize: 10,
textTransform: "uppercase",
letterSpacing: "0.06em",
color: "var(--text-muted)",
marginBottom: "var(--space-2)",
fontWeight: 600,
}}
>
{label}
</div>
<div
style={{
fontFamily: "var(--font-mono)",
fontSize: 28,
fontWeight: 600,
color: color || "var(--text-primary)",
letterSpacing: "-0.02em",
}}
>
{value}
</div>
</div>
);
}
export default ModelsTab;
@@ -1,767 +0,0 @@
import React, { useState, useEffect } from "react";
import { invoke } from "@tauri-apps/api/core";
interface RuVectorConfig {
// MinCut Parameters
mincut_enabled: boolean;
mincut_threshold: number;
mincut_max_persons: number;
// Attention Parameters
attention_enabled: boolean;
attention_heads: number;
attention_dropout: number;
// Temporal Parameters
temporal_enabled: boolean;
temporal_window_ms: number;
temporal_compression_ratio: number;
// Solver Parameters
solver_enabled: boolean;
solver_interpolation: "linear" | "cubic" | "sparse";
solver_subcarrier_count: number;
// BVP Parameters
bvp_enabled: boolean;
bvp_filter_hz: [number, number];
}
const DEFAULT_CONFIG: RuVectorConfig = {
mincut_enabled: true,
mincut_threshold: 0.5,
mincut_max_persons: 5,
attention_enabled: true,
attention_heads: 4,
attention_dropout: 0.1,
temporal_enabled: true,
temporal_window_ms: 500,
temporal_compression_ratio: 4,
solver_enabled: true,
solver_interpolation: "sparse",
solver_subcarrier_count: 56,
bvp_enabled: false,
bvp_filter_hz: [0.7, 4.0],
};
const MODULES = [
{
id: "mincut",
name: "MinCut Segmentation",
crate: "ruvector-mincut",
description: "Graph-based person segmentation using DynamicPersonMatcher",
icon: "✂️",
},
{
id: "attention",
name: "Spatial Attention",
crate: "ruvector-attention",
description: "Attention-weighted antenna selection and BVP extraction",
icon: "🎯",
},
{
id: "temporal",
name: "Temporal Tensor",
crate: "ruvector-temporal-tensor",
description: "Temporal CSI compression and breathing detection",
icon: "⏱️",
},
{
id: "solver",
name: "Sparse Solver",
crate: "ruvector-solver",
description: "Sparse interpolation (114→56 subcarriers) and triangulation",
icon: "🧮",
},
{
id: "attn-mincut",
name: "Attention MinCut",
crate: "ruvector-attn-mincut",
description: "Combined attention-weighted graph segmentation",
icon: "🔀",
},
];
const RuVectorTab: React.FC = () => {
const [config, setConfig] = useState<RuVectorConfig>(DEFAULT_CONFIG);
const [testingLive, setTestingLive] = useState(false);
const [liveMetrics, setLiveMetrics] = useState<{
fps: number;
latency_ms: number;
persons_detected: number;
} | null>(null);
const [saved, setSaved] = useState(true);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
loadConfig();
}, []);
const loadConfig = async () => {
try {
const loaded = await invoke<RuVectorConfig>("get_ruvector_config");
setConfig(loaded);
} catch (err) {
// Use defaults if not implemented
}
};
const saveConfig = async () => {
setError(null);
try {
await invoke("set_ruvector_config", { config });
setSaved(true);
} catch (err) {
setError(`Failed to save: ${err}`);
}
};
const handleChange = <K extends keyof RuVectorConfig>(
key: K,
value: RuVectorConfig[K]
) => {
setConfig((prev) => ({ ...prev, [key]: value }));
setSaved(false);
};
const startLiveTest = async () => {
setTestingLive(true);
setError(null);
try {
// Simulate live testing metrics
const interval = setInterval(() => {
setLiveMetrics({
fps: 25 + Math.random() * 10,
latency_ms: 15 + Math.random() * 10,
persons_detected: Math.floor(Math.random() * 3) + 1,
});
}, 500);
// Stop after 10 seconds for demo
setTimeout(() => {
clearInterval(interval);
setTestingLive(false);
setLiveMetrics(null);
}, 10000);
} catch (err) {
setError(`Live test failed: ${err}`);
setTestingLive(false);
}
};
const exportConfig = () => {
const blob = new Blob([JSON.stringify(config, null, 2)], {
type: "application/json",
});
const url = URL.createObjectURL(blob);
const a = document.createElement("a");
a.href = url;
a.download = "ruvector-config.json";
a.click();
URL.revokeObjectURL(url);
};
return (
<div>
{/* Module Cards */}
<div
style={{
display: "grid",
gridTemplateColumns: "repeat(auto-fill, minmax(200px, 1fr))",
gap: "var(--space-3)",
marginBottom: "var(--space-5)",
}}
>
{MODULES.map((mod) => {
const isEnabled =
config[`${mod.id.replace("-", "_")}_enabled` as keyof RuVectorConfig] ?? true;
return (
<div
key={mod.id}
className="card"
style={{
padding: "var(--space-3)",
opacity: isEnabled ? 1 : 0.5,
transition: "opacity 0.2s",
}}
>
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "start",
}}
>
<span style={{ fontSize: 24 }}>{mod.icon}</span>
<span
style={{
fontSize: 9,
padding: "2px 6px",
borderRadius: 3,
background: isEnabled
? "rgba(63, 185, 80, 0.15)"
: "rgba(139, 148, 158, 0.15)",
color: isEnabled ? "var(--status-online)" : "var(--text-muted)",
fontWeight: 600,
}}
>
{isEnabled ? "ON" : "OFF"}
</span>
</div>
<h4 style={{ margin: "var(--space-2) 0 4px", fontSize: 13, fontWeight: 600 }}>
{mod.name}
</h4>
<p
style={{
fontSize: 11,
color: "var(--text-muted)",
margin: 0,
lineHeight: 1.4,
}}
>
{mod.description}
</p>
<div
style={{
marginTop: "var(--space-2)",
fontFamily: "var(--font-mono)",
fontSize: 10,
color: "var(--text-secondary)",
}}
>
{mod.crate}
</div>
</div>
);
})}
</div>
{error && (
<div
style={{
background: "rgba(248, 81, 73, 0.1)",
border: "1px solid rgba(248, 81, 73, 0.3)",
borderRadius: 6,
padding: "var(--space-3)",
marginBottom: "var(--space-4)",
fontSize: 13,
color: "var(--status-error)",
}}
>
{error}
</div>
)}
<div style={{ display: "grid", gridTemplateColumns: "1fr 1fr", gap: "var(--space-5)" }}>
{/* Configuration Panel */}
<div className="card" style={{ padding: "var(--space-4)" }}>
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
Module Configuration
</h3>
{/* MinCut Section */}
<ConfigSection title="MinCut Segmentation">
<ToggleRow
label="Enable MinCut"
checked={config.mincut_enabled}
onChange={(v) => handleChange("mincut_enabled", v)}
/>
<SliderRow
label="Threshold"
value={config.mincut_threshold}
min={0.1}
max={1.0}
step={0.05}
onChange={(v) => handleChange("mincut_threshold", v)}
disabled={!config.mincut_enabled}
/>
<NumberRow
label="Max Persons"
value={config.mincut_max_persons}
min={1}
max={10}
onChange={(v) => handleChange("mincut_max_persons", v)}
disabled={!config.mincut_enabled}
/>
</ConfigSection>
{/* Attention Section */}
<ConfigSection title="Spatial Attention">
<ToggleRow
label="Enable Attention"
checked={config.attention_enabled}
onChange={(v) => handleChange("attention_enabled", v)}
/>
<NumberRow
label="Attention Heads"
value={config.attention_heads}
min={1}
max={16}
onChange={(v) => handleChange("attention_heads", v)}
disabled={!config.attention_enabled}
/>
<SliderRow
label="Dropout"
value={config.attention_dropout}
min={0}
max={0.5}
step={0.05}
onChange={(v) => handleChange("attention_dropout", v)}
disabled={!config.attention_enabled}
/>
</ConfigSection>
{/* Temporal Section */}
<ConfigSection title="Temporal Processing">
<ToggleRow
label="Enable Temporal"
checked={config.temporal_enabled}
onChange={(v) => handleChange("temporal_enabled", v)}
/>
<NumberRow
label="Window (ms)"
value={config.temporal_window_ms}
min={100}
max={2000}
step={100}
onChange={(v) => handleChange("temporal_window_ms", v)}
disabled={!config.temporal_enabled}
/>
<NumberRow
label="Compression Ratio"
value={config.temporal_compression_ratio}
min={1}
max={16}
onChange={(v) => handleChange("temporal_compression_ratio", v)}
disabled={!config.temporal_enabled}
/>
</ConfigSection>
{/* Solver Section */}
<ConfigSection title="Sparse Solver">
<ToggleRow
label="Enable Solver"
checked={config.solver_enabled}
onChange={(v) => handleChange("solver_enabled", v)}
/>
<div style={{ marginBottom: "var(--space-2)" }}>
<label style={labelStyle}>Interpolation</label>
<select
value={config.solver_interpolation}
onChange={(e) =>
handleChange(
"solver_interpolation",
e.target.value as RuVectorConfig["solver_interpolation"]
)
}
disabled={!config.solver_enabled}
style={{
...inputStyle,
opacity: config.solver_enabled ? 1 : 0.5,
}}
>
<option value="linear">Linear</option>
<option value="cubic">Cubic</option>
<option value="sparse">Sparse (L1)</option>
</select>
</div>
<NumberRow
label="Subcarrier Count"
value={config.solver_subcarrier_count}
min={28}
max={114}
onChange={(v) => handleChange("solver_subcarrier_count", v)}
disabled={!config.solver_enabled}
/>
</ConfigSection>
{/* Action Buttons */}
<div
style={{
display: "flex",
gap: "var(--space-2)",
marginTop: "var(--space-4)",
}}
>
<button
onClick={saveConfig}
className="btn-gradient"
style={{
flex: 1,
padding: "10px",
fontSize: 12,
opacity: saved ? 0.6 : 1,
}}
disabled={saved}
>
{saved ? "Saved" : "Save Configuration"}
</button>
<button
onClick={exportConfig}
style={{
padding: "10px 16px",
background: "transparent",
border: "1px solid var(--border)",
borderRadius: 6,
color: "var(--text-secondary)",
fontSize: 12,
fontWeight: 600,
cursor: "pointer",
}}
>
Export
</button>
</div>
</div>
{/* Live Testing Panel */}
<div className="card" style={{ padding: "var(--space-4)" }}>
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
Live Testing
</h3>
<div
style={{
display: "flex",
flexDirection: "column",
alignItems: "center",
justifyContent: "center",
minHeight: 200,
background: "var(--bg-secondary)",
borderRadius: 8,
marginBottom: "var(--space-4)",
}}
>
{testingLive ? (
<>
<div
style={{
fontSize: 48,
animation: "pulse 1s infinite",
}}
>
📡
</div>
<p style={{ fontSize: 13, color: "var(--text-secondary)", marginTop: "var(--space-2)" }}>
Processing live CSI stream...
</p>
</>
) : (
<>
<div style={{ fontSize: 48, opacity: 0.5 }}>📡</div>
<p style={{ fontSize: 13, color: "var(--text-muted)", marginTop: "var(--space-2)" }}>
Start live test to apply config to real CSI data
</p>
</>
)}
</div>
{liveMetrics && (
<div
style={{
display: "grid",
gridTemplateColumns: "repeat(3, 1fr)",
gap: "var(--space-3)",
marginBottom: "var(--space-4)",
}}
>
<MetricCard label="FPS" value={liveMetrics.fps.toFixed(1)} />
<MetricCard label="Latency" value={`${liveMetrics.latency_ms.toFixed(0)}ms`} />
<MetricCard label="Persons" value={liveMetrics.persons_detected.toString()} />
</div>
)}
<button
onClick={testingLive ? () => setTestingLive(false) : startLiveTest}
style={{
width: "100%",
padding: "12px",
background: testingLive
? "rgba(248, 81, 73, 0.1)"
: "rgba(56, 139, 253, 0.1)",
border: `1px solid ${testingLive ? "rgba(248, 81, 73, 0.3)" : "rgba(56, 139, 253, 0.3)"}`,
borderRadius: 6,
color: testingLive ? "var(--status-error)" : "var(--accent)",
fontSize: 13,
fontWeight: 600,
cursor: "pointer",
}}
>
{testingLive ? "Stop Test" : "Start Live Test"}
</button>
{/* Presets */}
<div style={{ marginTop: "var(--space-5)" }}>
<h4 style={{ fontSize: 12, fontWeight: 600, marginBottom: "var(--space-3)" }}>
Quick Presets
</h4>
<div style={{ display: "flex", flexDirection: "column", gap: "var(--space-2)" }}>
<PresetButton
label="Low Latency"
description="Minimal processing for real-time"
onClick={() => {
setConfig({
...DEFAULT_CONFIG,
attention_heads: 2,
temporal_compression_ratio: 8,
solver_subcarrier_count: 28,
});
setSaved(false);
}}
/>
<PresetButton
label="High Accuracy"
description="Maximum quality, higher latency"
onClick={() => {
setConfig({
...DEFAULT_CONFIG,
attention_heads: 8,
temporal_compression_ratio: 2,
solver_subcarrier_count: 114,
solver_interpolation: "cubic",
});
setSaved(false);
}}
/>
<PresetButton
label="Balanced"
description="Default recommended settings"
onClick={() => {
setConfig(DEFAULT_CONFIG);
setSaved(false);
}}
/>
</div>
</div>
</div>
</div>
<style>{`
@keyframes pulse {
0%, 100% { transform: scale(1); }
50% { transform: scale(1.1); }
}
`}</style>
</div>
);
};
// Helper Components
function ConfigSection({ title, children }: { title: string; children: React.ReactNode }) {
return (
<div style={{ marginBottom: "var(--space-4)" }}>
<h4
style={{
fontSize: 11,
fontWeight: 600,
color: "var(--text-muted)",
textTransform: "uppercase",
letterSpacing: "0.04em",
marginBottom: "var(--space-2)",
}}
>
{title}
</h4>
{children}
</div>
);
}
function ToggleRow({
label,
checked,
onChange,
}: {
label: string;
checked: boolean;
onChange: (v: boolean) => void;
}) {
return (
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "center",
marginBottom: "var(--space-2)",
}}
>
<span style={{ fontSize: 12 }}>{label}</span>
<button
onClick={() => onChange(!checked)}
style={{
width: 40,
height: 22,
borderRadius: 11,
border: "none",
background: checked ? "var(--accent)" : "var(--border)",
position: "relative",
cursor: "pointer",
transition: "background 0.2s",
}}
>
<span
style={{
position: "absolute",
top: 2,
left: checked ? 20 : 2,
width: 18,
height: 18,
borderRadius: "50%",
background: "white",
transition: "left 0.2s",
}}
/>
</button>
</div>
);
}
function SliderRow({
label,
value,
min,
max,
step,
onChange,
disabled,
}: {
label: string;
value: number;
min: number;
max: number;
step: number;
onChange: (v: number) => void;
disabled?: boolean;
}) {
return (
<div style={{ marginBottom: "var(--space-2)", opacity: disabled ? 0.5 : 1 }}>
<div style={{ display: "flex", justifyContent: "space-between", marginBottom: 4 }}>
<span style={{ fontSize: 12 }}>{label}</span>
<span style={{ fontSize: 11, fontFamily: "var(--font-mono)", color: "var(--text-muted)" }}>
{value.toFixed(2)}
</span>
</div>
<input
type="range"
value={value}
min={min}
max={max}
step={step}
onChange={(e) => onChange(parseFloat(e.target.value))}
disabled={disabled}
style={{ width: "100%", cursor: disabled ? "not-allowed" : "pointer" }}
/>
</div>
);
}
function NumberRow({
label,
value,
min,
max,
step = 1,
onChange,
disabled,
}: {
label: string;
value: number;
min: number;
max: number;
step?: number;
onChange: (v: number) => void;
disabled?: boolean;
}) {
return (
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "center",
marginBottom: "var(--space-2)",
opacity: disabled ? 0.5 : 1,
}}
>
<span style={{ fontSize: 12 }}>{label}</span>
<input
type="number"
value={value}
min={min}
max={max}
step={step}
onChange={(e) => onChange(parseInt(e.target.value) || min)}
disabled={disabled}
style={{
width: 70,
padding: "4px 8px",
background: "var(--bg-secondary)",
border: "1px solid var(--border)",
borderRadius: 4,
color: "var(--text-primary)",
fontSize: 12,
textAlign: "right",
cursor: disabled ? "not-allowed" : "text",
}}
/>
</div>
);
}
function MetricCard({ label, value }: { label: string; value: string }) {
return (
<div className="card" style={{ padding: "var(--space-3)", textAlign: "center" }}>
<div style={{ fontSize: 10, color: "var(--text-muted)", marginBottom: 2 }}>{label}</div>
<div style={{ fontFamily: "var(--font-mono)", fontSize: 18, fontWeight: 600 }}>{value}</div>
</div>
);
}
function PresetButton({
label,
description,
onClick,
}: {
label: string;
description: string;
onClick: () => void;
}) {
return (
<button
onClick={onClick}
style={{
display: "flex",
flexDirection: "column",
alignItems: "start",
padding: "var(--space-3)",
background: "var(--bg-secondary)",
border: "1px solid var(--border)",
borderRadius: 6,
cursor: "pointer",
textAlign: "left",
}}
>
<span style={{ fontSize: 12, fontWeight: 600, color: "var(--text-primary)" }}>{label}</span>
<span style={{ fontSize: 11, color: "var(--text-muted)" }}>{description}</span>
</button>
);
}
const labelStyle: React.CSSProperties = {
display: "block",
fontSize: 11,
fontWeight: 600,
color: "var(--text-muted)",
marginBottom: 4,
};
const inputStyle: React.CSSProperties = {
width: "100%",
padding: "8px 12px",
background: "var(--bg-secondary)",
border: "1px solid var(--border)",
borderRadius: 6,
color: "var(--text-primary)",
fontSize: 13,
};
export default RuVectorTab;
@@ -1,601 +0,0 @@
import React, { useState, useEffect } from "react";
import { invoke } from "@tauri-apps/api/core";
import { listen, UnlistenFn } from "@tauri-apps/api/event";
interface TrainingConfig {
dataset_id: string;
model_id: string;
epochs: number;
batch_size: number;
learning_rate: number;
optimizer: "adam" | "sgd" | "adamw";
weight_decay: number;
use_augmentation: boolean;
checkpoint_every: number;
}
interface TrainingProgress {
epoch: number;
total_epochs: number;
batch: number;
total_batches: number;
train_loss: number;
val_loss: number | null;
learning_rate: number;
eta_secs: number;
gpu_memory_mb: number | null;
}
interface TrainingJob {
id: string;
status: "running" | "paused" | "completed" | "failed";
started_at: string;
progress: TrainingProgress;
}
const DEFAULT_CONFIG: TrainingConfig = {
dataset_id: "mmfi",
model_id: "csi-encoder-cnn",
epochs: 100,
batch_size: 32,
learning_rate: 0.001,
optimizer: "adam",
weight_decay: 0.0001,
use_augmentation: true,
checkpoint_every: 10,
};
interface TrainingTabProps {
gpuAvailable: boolean;
}
const TrainingTab: React.FC<TrainingTabProps> = ({ gpuAvailable }) => {
const [config, setConfig] = useState<TrainingConfig>(DEFAULT_CONFIG);
const [currentJob, setCurrentJob] = useState<TrainingJob | null>(null);
const [lossHistory, setLossHistory] = useState<{ epoch: number; train: number; val: number }[]>(
[]
);
const [error, setError] = useState<string | null>(null);
useEffect(() => {
let unlisten: UnlistenFn | undefined;
const setupListener = async () => {
try {
unlisten = await listen<TrainingProgress>("training:progress", (event) => {
const progress = event.payload;
setCurrentJob((prev) =>
prev
? { ...prev, progress }
: {
id: "job-1",
status: "running",
started_at: new Date().toISOString(),
progress,
}
);
if (progress.val_loss !== null && progress.batch === progress.total_batches) {
setLossHistory((prev) => [
...prev,
{
epoch: progress.epoch,
train: progress.train_loss,
val: progress.val_loss!,
},
]);
}
});
} catch (err) {
console.error("Failed to setup training listener:", err);
}
};
setupListener();
return () => {
if (unlisten) unlisten();
};
}, []);
const handleStartTraining = async () => {
setError(null);
try {
await invoke("start_training", { config });
setCurrentJob({
id: `job-${Date.now()}`,
status: "running",
started_at: new Date().toISOString(),
progress: {
epoch: 0,
total_epochs: config.epochs,
batch: 0,
total_batches: 0,
train_loss: 0,
val_loss: null,
learning_rate: config.learning_rate,
eta_secs: 0,
gpu_memory_mb: null,
},
});
} catch (err) {
setError(`Failed to start training: ${err}`);
}
};
const handleStopTraining = async () => {
try {
await invoke("stop_training");
setCurrentJob((prev) => (prev ? { ...prev, status: "paused" } : null));
} catch (err) {
setError(`Failed to stop training: ${err}`);
}
};
const formatEta = (seconds: number) => {
if (seconds < 60) return `${seconds}s`;
if (seconds < 3600) return `${Math.floor(seconds / 60)}m ${seconds % 60}s`;
const hours = Math.floor(seconds / 3600);
const mins = Math.floor((seconds % 3600) / 60);
return `${hours}h ${mins}m`;
};
const progress = currentJob?.progress;
const epochProgress = progress ? (progress.epoch / progress.total_epochs) * 100 : 0;
const batchProgress = progress && progress.total_batches > 0
? (progress.batch / progress.total_batches) * 100
: 0;
return (
<div>
{/* GPU Warning */}
{!gpuAvailable && (
<div
style={{
background: "rgba(245, 158, 11, 0.1)",
border: "1px solid rgba(245, 158, 11, 0.3)",
borderRadius: 6,
padding: "var(--space-3)",
marginBottom: "var(--space-4)",
display: "flex",
alignItems: "center",
gap: "var(--space-3)",
}}
>
<span style={{ fontSize: 18 }}></span>
<div>
<div style={{ fontWeight: 600, fontSize: 13, color: "#f59e0b" }}>
GPU Not Available
</div>
<div style={{ fontSize: 12, color: "var(--text-muted)" }}>
Training will use CPU, which is significantly slower. Consider using a
machine with CUDA or Metal support.
</div>
</div>
</div>
)}
{error && (
<div
style={{
background: "rgba(248, 81, 73, 0.1)",
border: "1px solid rgba(248, 81, 73, 0.3)",
borderRadius: 6,
padding: "var(--space-3)",
marginBottom: "var(--space-4)",
fontSize: 13,
color: "var(--status-error)",
}}
>
{error}
</div>
)}
<div style={{ display: "grid", gridTemplateColumns: "1fr 1fr", gap: "var(--space-5)" }}>
{/* Configuration Panel */}
<div className="card" style={{ padding: "var(--space-4)" }}>
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
Training Configuration
</h3>
<div style={{ display: "flex", flexDirection: "column", gap: "var(--space-3)" }}>
<div>
<label style={labelStyle}>Dataset</label>
<select
value={config.dataset_id}
onChange={(e) => setConfig({ ...config, dataset_id: e.target.value })}
style={inputStyle}
>
<option value="mmfi">MM-Fi Dataset</option>
<option value="wipose">Wi-Pose Dataset</option>
<option value="wiar">WiAR Dataset</option>
</select>
</div>
<div>
<label style={labelStyle}>Model Architecture</label>
<select
value={config.model_id}
onChange={(e) => setConfig({ ...config, model_id: e.target.value })}
style={inputStyle}
>
<option value="csi-encoder-cnn">CSI Encoder (CNN)</option>
<option value="csi-encoder-transformer">CSI Encoder (Transformer)</option>
<option value="pose-decoder-lstm">Pose Decoder (LSTM)</option>
<option value="pose-decoder-gru">Pose Decoder (GRU)</option>
</select>
</div>
<div style={{ display: "grid", gridTemplateColumns: "1fr 1fr", gap: "var(--space-3)" }}>
<div>
<label style={labelStyle}>Epochs</label>
<input
type="number"
value={config.epochs}
onChange={(e) => setConfig({ ...config, epochs: parseInt(e.target.value) || 1 })}
min={1}
max={1000}
style={inputStyle}
/>
</div>
<div>
<label style={labelStyle}>Batch Size</label>
<input
type="number"
value={config.batch_size}
onChange={(e) =>
setConfig({ ...config, batch_size: parseInt(e.target.value) || 1 })
}
min={1}
max={512}
style={inputStyle}
/>
</div>
</div>
<div style={{ display: "grid", gridTemplateColumns: "1fr 1fr", gap: "var(--space-3)" }}>
<div>
<label style={labelStyle}>Learning Rate</label>
<input
type="number"
value={config.learning_rate}
onChange={(e) =>
setConfig({ ...config, learning_rate: parseFloat(e.target.value) || 0.001 })
}
step={0.0001}
min={0.00001}
max={1}
style={inputStyle}
/>
</div>
<div>
<label style={labelStyle}>Optimizer</label>
<select
value={config.optimizer}
onChange={(e) =>
setConfig({ ...config, optimizer: e.target.value as TrainingConfig["optimizer"] })
}
style={inputStyle}
>
<option value="adam">Adam</option>
<option value="adamw">AdamW</option>
<option value="sgd">SGD</option>
</select>
</div>
</div>
<div style={{ display: "grid", gridTemplateColumns: "1fr 1fr", gap: "var(--space-3)" }}>
<div>
<label style={labelStyle}>Weight Decay</label>
<input
type="number"
value={config.weight_decay}
onChange={(e) =>
setConfig({ ...config, weight_decay: parseFloat(e.target.value) || 0 })
}
step={0.0001}
min={0}
max={1}
style={inputStyle}
/>
</div>
<div>
<label style={labelStyle}>Checkpoint Every</label>
<input
type="number"
value={config.checkpoint_every}
onChange={(e) =>
setConfig({ ...config, checkpoint_every: parseInt(e.target.value) || 1 })
}
min={1}
max={100}
style={inputStyle}
/>
</div>
</div>
<div style={{ display: "flex", alignItems: "center", gap: "var(--space-2)" }}>
<input
type="checkbox"
id="augmentation"
checked={config.use_augmentation}
onChange={(e) => setConfig({ ...config, use_augmentation: e.target.checked })}
style={{ width: 16, height: 16 }}
/>
<label htmlFor="augmentation" style={{ fontSize: 13, cursor: "pointer" }}>
Enable Data Augmentation
</label>
</div>
<div style={{ marginTop: "var(--space-3)" }}>
{currentJob?.status === "running" ? (
<button
onClick={handleStopTraining}
style={{
width: "100%",
padding: "12px",
background: "rgba(248, 81, 73, 0.1)",
border: "1px solid rgba(248, 81, 73, 0.3)",
borderRadius: 6,
color: "var(--status-error)",
fontSize: 13,
fontWeight: 600,
cursor: "pointer",
}}
>
Stop Training
</button>
) : (
<button
onClick={handleStartTraining}
className="btn-gradient"
style={{ width: "100%", padding: "12px", fontSize: 13 }}
>
Start Training
</button>
)}
</div>
</div>
</div>
{/* Progress Panel */}
<div className="card" style={{ padding: "var(--space-4)" }}>
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
Training Progress
</h3>
{!currentJob ? (
<div
style={{
display: "flex",
flexDirection: "column",
alignItems: "center",
justifyContent: "center",
height: 300,
color: "var(--text-muted)",
}}
>
<div style={{ fontSize: 48, marginBottom: "var(--space-3)" }}>🎯</div>
<p style={{ fontSize: 13 }}>No training job running</p>
<p style={{ fontSize: 12 }}>Configure and start training to begin</p>
</div>
) : (
<div style={{ display: "flex", flexDirection: "column", gap: "var(--space-4)" }}>
{/* Status */}
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "center",
}}
>
<div style={{ display: "flex", alignItems: "center", gap: "var(--space-2)" }}>
<span
style={{
width: 8,
height: 8,
borderRadius: "50%",
background:
currentJob.status === "running"
? "var(--status-online)"
: currentJob.status === "paused"
? "#f59e0b"
: "var(--status-error)",
animation: currentJob.status === "running" ? "pulse 1.5s infinite" : "none",
}}
/>
<span style={{ fontSize: 13, fontWeight: 600, textTransform: "capitalize" }}>
{currentJob.status}
</span>
</div>
<span style={{ fontSize: 12, color: "var(--text-muted)" }}>
ETA: {formatEta(progress?.eta_secs ?? 0)}
</span>
</div>
{/* Epoch Progress */}
<div>
<div
style={{
display: "flex",
justifyContent: "space-between",
fontSize: 12,
marginBottom: 4,
}}
>
<span>Epoch</span>
<span>
{progress?.epoch ?? 0} / {progress?.total_epochs ?? config.epochs}
</span>
</div>
<div
style={{
height: 6,
background: "var(--border)",
borderRadius: 3,
overflow: "hidden",
}}
>
<div
style={{
width: `${epochProgress}%`,
height: "100%",
background: "var(--accent)",
transition: "width 0.3s",
}}
/>
</div>
</div>
{/* Batch Progress */}
<div>
<div
style={{
display: "flex",
justifyContent: "space-between",
fontSize: 12,
marginBottom: 4,
}}
>
<span>Batch</span>
<span>
{progress?.batch ?? 0} / {progress?.total_batches ?? 0}
</span>
</div>
<div
style={{
height: 4,
background: "var(--border)",
borderRadius: 2,
overflow: "hidden",
}}
>
<div
style={{
width: `${batchProgress}%`,
height: "100%",
background: "rgba(56, 139, 253, 0.5)",
transition: "width 0.1s",
}}
/>
</div>
</div>
{/* Stats Grid */}
<div
style={{
display: "grid",
gridTemplateColumns: "repeat(2, 1fr)",
gap: "var(--space-3)",
}}
>
<div className="card" style={{ padding: "var(--space-3)" }}>
<div style={{ fontSize: 10, color: "var(--text-muted)", marginBottom: 4 }}>
Train Loss
</div>
<div style={{ fontFamily: "var(--font-mono)", fontSize: 20, fontWeight: 600 }}>
{progress?.train_loss.toFixed(4) ?? "—"}
</div>
</div>
<div className="card" style={{ padding: "var(--space-3)" }}>
<div style={{ fontSize: 10, color: "var(--text-muted)", marginBottom: 4 }}>
Val Loss
</div>
<div
style={{
fontFamily: "var(--font-mono)",
fontSize: 20,
fontWeight: 600,
color: "var(--status-online)",
}}
>
{progress?.val_loss?.toFixed(4) ?? "—"}
</div>
</div>
<div className="card" style={{ padding: "var(--space-3)" }}>
<div style={{ fontSize: 10, color: "var(--text-muted)", marginBottom: 4 }}>
Learning Rate
</div>
<div style={{ fontFamily: "var(--font-mono)", fontSize: 14, fontWeight: 600 }}>
{progress?.learning_rate.toExponential(2) ?? "—"}
</div>
</div>
<div className="card" style={{ padding: "var(--space-3)" }}>
<div style={{ fontSize: 10, color: "var(--text-muted)", marginBottom: 4 }}>
GPU Memory
</div>
<div style={{ fontFamily: "var(--font-mono)", fontSize: 14, fontWeight: 600 }}>
{progress?.gpu_memory_mb ? `${progress.gpu_memory_mb} MB` : "N/A"}
</div>
</div>
</div>
{/* Mini Loss Chart */}
{lossHistory.length > 0 && (
<div>
<div style={{ fontSize: 12, fontWeight: 600, marginBottom: "var(--space-2)" }}>
Loss History
</div>
<div
style={{
height: 80,
display: "flex",
alignItems: "flex-end",
gap: 2,
padding: "var(--space-2)",
background: "var(--bg-secondary)",
borderRadius: 4,
}}
>
{lossHistory.slice(-20).map((h, i) => (
<div
key={i}
style={{
flex: 1,
height: `${Math.max(5, Math.min(100, (1 - h.train) * 100))}%`,
background: "var(--accent)",
borderRadius: 2,
opacity: 0.6 + (i / 20) * 0.4,
}}
title={`Epoch ${h.epoch}: Train=${h.train.toFixed(4)}, Val=${h.val.toFixed(4)}`}
/>
))}
</div>
</div>
)}
</div>
)}
</div>
</div>
<style>{`
@keyframes pulse {
0%, 100% { opacity: 1; }
50% { opacity: 0.5; }
}
`}</style>
</div>
);
};
const labelStyle: React.CSSProperties = {
display: "block",
fontSize: 11,
fontWeight: 600,
color: "var(--text-muted)",
marginBottom: 4,
textTransform: "uppercase",
letterSpacing: "0.04em",
};
const inputStyle: React.CSSProperties = {
width: "100%",
padding: "8px 12px",
background: "var(--bg-secondary)",
border: "1px solid var(--border)",
borderRadius: 6,
color: "var(--text-primary)",
fontSize: 13,
};
export default TrainingTab;
@@ -1,165 +0,0 @@
import React, { useState, useEffect } from "react";
import { invoke } from "@tauri-apps/api/core";
import DatasetsTab from "./DatasetsTab";
import ModelsTab from "./ModelsTab";
import TrainingTab from "./TrainingTab";
import RuVectorTab from "./RuVectorTab";
import MetricsTab from "./MetricsTab";
type TrainingTabType = "datasets" | "models" | "training" | "ruvector" | "metrics";
interface GpuInfo {
available: boolean;
name: string | null;
memory_mb: number | null;
cuda_version: string | null;
metal_supported: boolean;
}
const Training: React.FC = () => {
const [activeTab, setActiveTab] = useState<TrainingTabType>("datasets");
const [gpuInfo, setGpuInfo] = useState<GpuInfo | null>(null);
const [loading, setLoading] = useState(true);
useEffect(() => {
detectGpu();
}, []);
const detectGpu = async () => {
try {
const info = await invoke<GpuInfo>("detect_gpu");
setGpuInfo(info);
} catch (err) {
console.error("GPU detection failed:", err);
setGpuInfo({
available: false,
name: null,
memory_mb: null,
cuda_version: null,
metal_supported: false,
});
} finally {
setLoading(false);
}
};
const tabs: { id: TrainingTabType; label: string; icon: string }[] = [
{ id: "datasets", label: "Datasets", icon: "📊" },
{ id: "models", label: "Models", icon: "🧠" },
{ id: "training", label: "Training", icon: "⚡" },
{ id: "ruvector", label: "RuVector", icon: "📡" },
{ id: "metrics", label: "Metrics", icon: "📈" },
];
return (
<div style={{ padding: "var(--space-5)", maxWidth: 1400 }}>
{/* Header */}
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "center",
marginBottom: "var(--space-5)",
}}
>
<div>
<h1 className="heading-lg" style={{ margin: 0 }}>
Training & Models
</h1>
<p
style={{
fontSize: 13,
color: "var(--text-secondary)",
marginTop: 4,
}}
>
Train pose estimation models and configure RuVector signal processing
</p>
</div>
{/* GPU Status */}
<div
style={{
display: "flex",
alignItems: "center",
gap: "var(--space-3)",
padding: "var(--space-3) var(--space-4)",
background: gpuInfo?.available
? "rgba(63, 185, 80, 0.1)"
: "rgba(139, 148, 158, 0.1)",
border: `1px solid ${gpuInfo?.available ? "rgba(63, 185, 80, 0.3)" : "rgba(139, 148, 158, 0.3)"}`,
borderRadius: 8,
}}
>
<span style={{ fontSize: 18 }}>{gpuInfo?.available ? "🎮" : "💻"}</span>
<div>
<div style={{ fontSize: 12, fontWeight: 600, color: "var(--text-primary)" }}>
{loading
? "Detecting GPU..."
: gpuInfo?.available
? gpuInfo.name || "GPU Available"
: "CPU Mode"}
</div>
<div style={{ fontSize: 11, color: "var(--text-muted)" }}>
{gpuInfo?.cuda_version
? `CUDA ${gpuInfo.cuda_version}`
: gpuInfo?.metal_supported
? "Metal Supported"
: "No GPU acceleration"}
{gpuInfo?.memory_mb && `${Math.round(gpuInfo.memory_mb / 1024)}GB`}
</div>
</div>
</div>
</div>
{/* Tabs */}
<div
style={{
display: "flex",
gap: "var(--space-1)",
borderBottom: "1px solid var(--border)",
marginBottom: "var(--space-5)",
}}
>
{tabs.map((tab) => (
<button
key={tab.id}
onClick={() => setActiveTab(tab.id)}
style={{
display: "flex",
alignItems: "center",
gap: 6,
padding: "12px 20px",
border: "none",
background: "transparent",
color: activeTab === tab.id ? "var(--accent)" : "var(--text-secondary)",
fontSize: 13,
fontWeight: 600,
cursor: "pointer",
borderBottom:
activeTab === tab.id
? "2px solid var(--accent)"
: "2px solid transparent",
marginBottom: -1,
transition: "color 0.15s, border-color 0.15s",
}}
>
<span>{tab.icon}</span>
{tab.label}
</button>
))}
</div>
{/* Tab Content */}
<div>
{activeTab === "datasets" && <DatasetsTab />}
{activeTab === "models" && <ModelsTab />}
{activeTab === "training" && <TrainingTab gpuAvailable={gpuInfo?.available ?? false} />}
{activeTab === "ruvector" && <RuVectorTab />}
{activeTab === "metrics" && <MetricsTab />}
</div>
</div>
);
};
export default Training;
@@ -1,2 +1,2 @@
// Application version - single source of truth
export const APP_VERSION = "0.5.0";
export const APP_VERSION = "0.4.4";
+1
View File
@@ -29,6 +29,7 @@
<button class="nav-tab" data-tab="applications">Applications</button>
<button class="nav-tab" data-tab="sensing">Sensing</button>
<button class="nav-tab" data-tab="training">Training</button>
<a href="pose-fusion.html" class="nav-tab" style="text-decoration:none">Pose Fusion</a>
<a href="observatory.html" class="nav-tab" style="text-decoration:none">Observatory</a>
</nav>
+160
View File
@@ -0,0 +1,160 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>WiFi-DensePose — Dual-Modal Pose Estimation</title>
<link rel="stylesheet" href="pose-fusion/css/style.css">
</head>
<body>
<!-- Header -->
<header class="header">
<div class="header-left">
<div class="logo"><span class="pi">&pi;</span> DensePose</div>
<div class="header-title">Dual-Modal Pose Estimation — Live Video + WiFi CSI Fusion</div>
</div>
<div class="header-right">
<select id="mode-select" class="mode-select">
<option value="dual">Dual Mode (Video + CSI)</option>
<option value="video">Video Only</option>
<option value="csi">CSI Only (WiFi)</option>
</select>
<div class="status-badge">
<span id="status-dot" class="status-dot offline"></span>
<span id="status-label">READY</span>
</div>
<span id="fps-display" class="fps-badge">-- FPS</span>
<a href="index.html" class="back-link">&larr; Dashboard</a>
<a href="observatory.html" class="back-link">Observatory &rarr;</a>
</div>
</header>
<!-- Main Grid -->
<div class="main-grid">
<!-- Video + Skeleton Panel -->
<div class="video-panel">
<video id="webcam" autoplay playsinline muted></video>
<canvas id="skeleton-canvas"></canvas>
<div class="video-overlay-label" id="mode-label">DUAL FUSION</div>
<div id="camera-prompt" class="camera-prompt">
<p>Enable your webcam for live video pose estimation.<br>
Or switch to <strong>CSI Only</strong> mode for WiFi-based sensing.</p>
<button id="start-camera-btn">Enable Camera</button>
</div>
</div>
<!-- Side Panels -->
<div class="side-panels">
<!-- Fusion Confidence -->
<div class="panel">
<div class="panel-title">&#9670; Fusion Confidence</div>
<div class="fusion-bars">
<div class="bar-row">
<span class="bar-label">Video</span>
<div class="bar-track"><div class="bar-fill video" id="video-bar" style="width:0%"></div></div>
<span class="bar-value" id="video-bar-val">0%</span>
</div>
<div class="bar-row">
<span class="bar-label">CSI</span>
<div class="bar-track"><div class="bar-fill csi" id="csi-bar" style="width:0%"></div></div>
<span class="bar-value" id="csi-bar-val">0%</span>
</div>
<div class="bar-row">
<span class="bar-label">Fused</span>
<div class="bar-track"><div class="bar-fill fused" id="fused-bar" style="width:0%"></div></div>
<span class="bar-value" id="fused-bar-val">0%</span>
</div>
</div>
<div style="margin-top:8px; font-size:10px; color:var(--text-label)">
Cross-modal: <span id="cross-modal-sim" style="color:var(--green-glow)">0.000</span>
</div>
</div>
<!-- CSI Heatmap -->
<div class="panel">
<div class="panel-title">&#9670; CSI Amplitude Heatmap</div>
<div class="csi-canvas-wrapper">
<canvas id="csi-canvas" width="320" height="120"></canvas>
</div>
</div>
<!-- Embedding Space -->
<div class="panel">
<div class="panel-title">&#9670; Embedding Space (2D Projection)</div>
<div class="embedding-canvas-wrapper">
<canvas id="embedding-canvas" width="320" height="140"></canvas>
</div>
</div>
<!-- Latency -->
<div class="panel">
<div class="panel-title">&#9670; Pipeline Latency</div>
<div class="latency-grid">
<div class="latency-item">
<div class="latency-value" id="lat-video">--</div>
<div class="latency-label">Video CNN</div>
</div>
<div class="latency-item">
<div class="latency-value" id="lat-csi">--</div>
<div class="latency-label">CSI CNN</div>
</div>
<div class="latency-item">
<div class="latency-value" id="lat-fusion">--</div>
<div class="latency-label">Fusion</div>
</div>
<div class="latency-item">
<div class="latency-value" id="lat-total">--</div>
<div class="latency-label">Total</div>
</div>
</div>
</div>
<!-- Controls -->
<div class="panel">
<div class="panel-title">&#9670; Controls</div>
<div class="controls-row">
<button class="btn" id="pause-btn">⏸ Pause</button>
</div>
<div class="slider-row">
<label>Confidence</label>
<input type="range" id="confidence-slider" min="0" max="1" step="0.05" value="0.3">
<span class="slider-val" id="confidence-value">0.30</span>
</div>
<div style="margin-top:10px">
<div class="panel-title" style="margin-bottom:6px">&#9670; Live CSI Source</div>
<div style="display:flex;gap:6px">
<input type="text" id="ws-url" placeholder="ws://localhost:3030/ws/csi"
style="flex:1;background:rgba(255,255,255,0.05);border:1px solid var(--bg-panel-border);
color:var(--text-primary);padding:5px 8px;border-radius:4px;font-size:11px;
font-family:'JetBrains Mono',monospace">
<button class="btn" id="connect-ws-btn">Connect</button>
</div>
</div>
</div>
</div><!-- /side-panels -->
<!-- Bottom Bar -->
<div class="bottom-bar">
<div>
WiFi-DensePose &middot; Dual-Modal Pose Estimation &middot;
Architecture: MobileNet-V3 &times; 2 &rarr; Attention Fusion &rarr; 17-Keypoint COCO
</div>
<div>
<a href="https://github.com/ruvnet/wifi-densepose">GitHub</a> &middot;
CNN: ruvector-cnn (JS fallback) &middot;
<a href="observatory.html">Observatory</a>
</div>
</div>
</div><!-- /main-grid -->
<script type="module" src="pose-fusion/js/main.js"></script>
</body>
</html>
+30
View File
@@ -0,0 +1,30 @@
#!/bin/bash
# Build WASM packages for the dual-modal pose estimation demo.
# Requires: wasm-pack (cargo install wasm-pack)
#
# Usage: ./build.sh
#
# Output: pkg/ruvector_cnn_wasm/ — WASM CNN embedder for browser
set -e
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
VENDOR_DIR="$SCRIPT_DIR/../../vendor/ruvector"
OUT_DIR="$SCRIPT_DIR/pkg/ruvector_cnn_wasm"
echo "Building ruvector-cnn-wasm..."
wasm-pack build "$VENDOR_DIR/crates/ruvector-cnn-wasm" \
--target web \
--out-dir "$OUT_DIR" \
--no-typescript
# Remove .gitignore so we can commit the build output for GitHub Pages
rm -f "$OUT_DIR/.gitignore"
echo ""
echo "Build complete!"
echo " WASM: $(du -sh "$OUT_DIR/ruvector_cnn_wasm_bg.wasm" | cut -f1)"
echo " JS: $(du -sh "$OUT_DIR/ruvector_cnn_wasm.js" | cut -f1)"
echo ""
echo "Serve the demo: cd $SCRIPT_DIR/.. && python3 -m http.server 8080"
echo "Open: http://localhost:8080/pose-fusion.html"
+405
View File
@@ -0,0 +1,405 @@
/* WiFi-DensePose — Dual-Modal Pose Fusion Demo
Dark theme matching Observatory */
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@300;400;600;700&family=JetBrains+Mono:wght@400;600&display=swap');
:root {
--bg-deep: #080c14;
--bg-panel: rgba(8, 16, 28, 0.92);
--bg-panel-border: rgba(0, 210, 120, 0.25);
--green-glow: #00d878;
--green-bright:#3eff8a;
--green-dim: #0a6b3a;
--amber: #ffb020;
--amber-dim: #a06800;
--blue-signal: #2090ff;
--blue-dim: #0a3060;
--red-alert: #ff3040;
--cyan: #00e5ff;
--text-primary: #e8ece0;
--text-secondary: rgba(232,236,224, 0.55);
--text-label: rgba(232,236,224, 0.35);
--radius: 8px;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
background: var(--bg-deep);
font-family: 'Inter', -apple-system, sans-serif;
color: var(--text-primary);
-webkit-font-smoothing: antialiased;
overflow-x: hidden;
min-height: 100vh;
}
/* === Header === */
.header {
display: flex;
align-items: center;
justify-content: space-between;
padding: 16px 24px;
border-bottom: 1px solid var(--bg-panel-border);
background: var(--bg-panel);
backdrop-filter: blur(12px);
}
.header-left {
display: flex;
align-items: center;
gap: 16px;
}
.logo {
font-weight: 700;
font-size: 24px;
color: var(--green-glow);
}
.logo .pi { font-style: normal; }
.header-title {
font-size: 14px;
color: var(--text-secondary);
font-weight: 300;
}
.header-right {
display: flex;
align-items: center;
gap: 16px;
}
.mode-select {
background: rgba(0,210,120,0.1);
border: 1px solid var(--bg-panel-border);
color: var(--text-primary);
padding: 6px 12px;
border-radius: var(--radius);
font-family: inherit;
font-size: 13px;
cursor: pointer;
}
.mode-select option { background: #0c1420; }
.status-badge {
display: flex;
align-items: center;
gap: 6px;
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
padding: 4px 10px;
border-radius: 12px;
background: rgba(0,210,120,0.1);
border: 1px solid var(--bg-panel-border);
}
.status-dot {
width: 8px; height: 8px;
border-radius: 50%;
background: var(--green-glow);
box-shadow: 0 0 8px var(--green-glow);
animation: pulse-dot 2s ease infinite;
}
.status-dot.offline { background: #555; box-shadow: none; animation: none; }
.status-dot.warning { background: var(--amber); box-shadow: 0 0 8px var(--amber); }
@keyframes pulse-dot {
0%, 100% { opacity: 1; }
50% { opacity: 0.5; }
}
.fps-badge {
font-family: 'JetBrains Mono', monospace;
font-size: 12px;
color: var(--green-glow);
}
.back-link {
color: var(--text-secondary);
text-decoration: none;
font-size: 13px;
transition: color 0.2s;
}
.back-link:hover { color: var(--green-glow); }
/* === Main Layout === */
.main-grid {
display: grid;
grid-template-columns: 1fr 360px;
grid-template-rows: 1fr auto;
gap: 16px;
padding: 16px 24px;
height: calc(100vh - 72px);
overflow: hidden;
}
/* === Video Panel === */
.video-panel {
position: relative;
background: #000;
border-radius: var(--radius);
border: 1px solid var(--bg-panel-border);
overflow: hidden;
min-height: 0;
}
.video-panel video {
width: 100%;
height: 100%;
object-fit: cover;
transform: scaleX(-1);
}
.video-panel canvas {
position: absolute;
top: 0; left: 0;
width: 100%;
height: 100%;
transform: scaleX(-1);
}
.video-overlay-label {
position: absolute;
top: 12px; left: 12px;
font-family: 'JetBrains Mono', monospace;
font-size: 11px;
padding: 4px 8px;
background: rgba(0,0,0,0.7);
border-radius: 4px;
color: var(--green-glow);
z-index: 5;
transform: scaleX(-1);
}
.camera-prompt {
position: absolute;
top: 50%; left: 50%;
transform: translate(-50%, -50%);
text-align: center;
color: var(--text-secondary);
}
.camera-prompt button {
margin-top: 12px;
padding: 10px 24px;
background: var(--green-glow);
color: #000;
border: none;
border-radius: var(--radius);
font-family: inherit;
font-weight: 600;
font-size: 14px;
cursor: pointer;
transition: background 0.2s;
}
.camera-prompt button:hover { background: var(--green-bright); }
/* === Side Panels === */
.side-panels {
display: flex;
flex-direction: column;
gap: 12px;
overflow-y: auto;
min-height: 0;
}
.panel {
background: var(--bg-panel);
border: 1px solid var(--bg-panel-border);
border-radius: var(--radius);
padding: 14px;
}
.panel-title {
font-size: 11px;
text-transform: uppercase;
letter-spacing: 1.2px;
color: var(--text-label);
margin-bottom: 10px;
display: flex;
align-items: center;
gap: 6px;
}
/* === CSI Heatmap === */
.csi-canvas-wrapper {
position: relative;
border-radius: 4px;
overflow: hidden;
background: #000;
}
.csi-canvas-wrapper canvas {
width: 100%;
display: block;
}
/* === Fusion Bars === */
.fusion-bars {
display: flex;
flex-direction: column;
gap: 8px;
}
.bar-row {
display: flex;
align-items: center;
gap: 8px;
}
.bar-label {
font-family: 'JetBrains Mono', monospace;
font-size: 11px;
color: var(--text-secondary);
width: 55px;
text-align: right;
}
.bar-track {
flex: 1;
height: 6px;
background: rgba(255,255,255,0.06);
border-radius: 3px;
overflow: hidden;
}
.bar-fill {
height: 100%;
border-radius: 3px;
transition: width 0.3s ease;
}
.bar-fill.video { background: var(--cyan); }
.bar-fill.csi { background: var(--amber); }
.bar-fill.fused { background: var(--green-glow); box-shadow: 0 0 8px var(--green-glow); }
.bar-value {
font-family: 'JetBrains Mono', monospace;
font-size: 11px;
color: var(--text-primary);
width: 36px;
}
/* === Embedding Space === */
.embedding-canvas-wrapper {
position: relative;
background: #000;
border-radius: 4px;
overflow: hidden;
}
.embedding-canvas-wrapper canvas {
width: 100%;
display: block;
}
/* === Latency Panel === */
.latency-grid {
display: grid;
grid-template-columns: repeat(4, 1fr);
gap: 6px;
}
.latency-item {
text-align: center;
padding: 6px 0;
}
.latency-value {
font-family: 'JetBrains Mono', monospace;
font-size: 16px;
font-weight: 600;
color: var(--green-glow);
}
.latency-label {
font-size: 10px;
color: var(--text-label);
margin-top: 2px;
}
/* === Controls === */
.controls-row {
display: flex;
gap: 8px;
flex-wrap: wrap;
}
.btn {
padding: 6px 14px;
border: 1px solid var(--bg-panel-border);
background: rgba(0,210,120,0.08);
color: var(--text-primary);
border-radius: var(--radius);
font-family: inherit;
font-size: 12px;
cursor: pointer;
transition: all 0.2s;
}
.btn:hover { background: rgba(0,210,120,0.2); }
.btn.active { background: var(--green-glow); color: #000; font-weight: 600; }
.slider-row {
display: flex;
align-items: center;
gap: 8px;
margin-top: 8px;
}
.slider-row label {
font-size: 11px;
color: var(--text-secondary);
white-space: nowrap;
}
.slider-row input[type=range] {
flex: 1;
accent-color: var(--green-glow);
}
.slider-row .slider-val {
font-family: 'JetBrains Mono', monospace;
font-size: 11px;
width: 32px;
color: var(--green-glow);
}
/* === Bottom Bar === */
.bottom-bar {
grid-column: 1 / -1;
display: flex;
align-items: center;
justify-content: space-between;
padding: 10px 16px;
background: var(--bg-panel);
border: 1px solid var(--bg-panel-border);
border-radius: var(--radius);
font-family: 'JetBrains Mono', monospace;
font-size: 11px;
color: var(--text-secondary);
}
.bottom-bar a {
color: var(--green-glow);
text-decoration: none;
}
/* === Skeleton colors === */
.skeleton-joint { fill: var(--green-glow); }
.skeleton-limb { stroke: var(--green-bright); }
.skeleton-joint-csi { fill: var(--amber); }
.skeleton-limb-csi { stroke: var(--amber); }
/* === Responsive === */
@media (max-width: 900px) {
.main-grid {
grid-template-columns: 1fr;
height: auto;
overflow: auto;
}
.video-panel { aspect-ratio: 16/9; max-height: 50vh; }
.side-panels { max-height: none; overflow: visible; }
}
+247
View File
@@ -0,0 +1,247 @@
/**
* CanvasRenderer — Renders skeleton overlay on video, CSI heatmap,
* embedding space visualization, and fusion confidence bars.
*/
import { SKELETON_CONNECTIONS } from './pose-decoder.js';
export class CanvasRenderer {
constructor() {
this.colors = {
joint: '#00d878',
jointGlow: 'rgba(0, 216, 120, 0.4)',
limb: '#3eff8a',
limbGlow: 'rgba(62, 255, 138, 0.15)',
csiJoint: '#ffb020',
csiLimb: '#ffc850',
fused: '#00e5ff',
confidence: 'rgba(255,255,255,0.3)',
videoEmb: '#00e5ff',
csiEmb: '#ffb020',
fusedEmb: '#00d878',
};
}
/**
* Draw skeleton overlay on the video canvas
* @param {CanvasRenderingContext2D} ctx
* @param {Array<{x,y,confidence}>} keypoints - Normalized [0,1] coordinates
* @param {number} width - Canvas width
* @param {number} height - Canvas height
* @param {object} opts
*/
drawSkeleton(ctx, keypoints, width, height, opts = {}) {
const minConf = opts.minConfidence || 0.3;
const color = opts.color || 'green';
const jointColor = color === 'amber' ? this.colors.csiJoint : this.colors.joint;
const limbColor = color === 'amber' ? this.colors.csiLimb : this.colors.limb;
const glowColor = color === 'amber' ? 'rgba(255,176,32,0.4)' : this.colors.jointGlow;
ctx.clearRect(0, 0, width, height);
if (!keypoints || keypoints.length === 0) return;
// Draw limbs first (behind joints)
ctx.lineWidth = 3;
ctx.lineCap = 'round';
for (const [i, j] of SKELETON_CONNECTIONS) {
const kpA = keypoints[i];
const kpB = keypoints[j];
if (!kpA || !kpB || kpA.confidence < minConf || kpB.confidence < minConf) continue;
const ax = kpA.x * width, ay = kpA.y * height;
const bx = kpB.x * width, by = kpB.y * height;
const avgConf = (kpA.confidence + kpB.confidence) / 2;
// Glow
ctx.strokeStyle = this.colors.limbGlow;
ctx.lineWidth = 8;
ctx.globalAlpha = avgConf * 0.4;
ctx.beginPath();
ctx.moveTo(ax, ay);
ctx.lineTo(bx, by);
ctx.stroke();
// Main line
ctx.strokeStyle = limbColor;
ctx.lineWidth = 2.5;
ctx.globalAlpha = avgConf;
ctx.beginPath();
ctx.moveTo(ax, ay);
ctx.lineTo(bx, by);
ctx.stroke();
}
// Draw joints
ctx.globalAlpha = 1;
for (const kp of keypoints) {
if (!kp || kp.confidence < minConf) continue;
const x = kp.x * width;
const y = kp.y * height;
const r = 3 + kp.confidence * 3;
// Glow
ctx.beginPath();
ctx.arc(x, y, r + 4, 0, Math.PI * 2);
ctx.fillStyle = glowColor;
ctx.globalAlpha = kp.confidence * 0.6;
ctx.fill();
// Joint dot
ctx.beginPath();
ctx.arc(x, y, r, 0, Math.PI * 2);
ctx.fillStyle = jointColor;
ctx.globalAlpha = kp.confidence;
ctx.fill();
// White center
ctx.beginPath();
ctx.arc(x, y, r * 0.4, 0, Math.PI * 2);
ctx.fillStyle = '#fff';
ctx.globalAlpha = kp.confidence * 0.8;
ctx.fill();
}
ctx.globalAlpha = 1;
// Confidence label
if (opts.label) {
ctx.font = '11px "JetBrains Mono", monospace';
ctx.fillStyle = jointColor;
ctx.globalAlpha = 0.8;
ctx.fillText(opts.label, 8, height - 8);
ctx.globalAlpha = 1;
}
}
/**
* Draw CSI amplitude heatmap
* @param {CanvasRenderingContext2D} ctx
* @param {{ data: Float32Array, width: number, height: number }} heatmap
* @param {number} canvasW
* @param {number} canvasH
*/
drawCsiHeatmap(ctx, heatmap, canvasW, canvasH) {
ctx.clearRect(0, 0, canvasW, canvasH);
if (!heatmap || !heatmap.data || heatmap.height < 2) {
ctx.fillStyle = '#0a0e18';
ctx.fillRect(0, 0, canvasW, canvasH);
ctx.font = '11px "JetBrains Mono", monospace';
ctx.fillStyle = 'rgba(255,255,255,0.3)';
ctx.fillText('Waiting for CSI data...', 8, canvasH / 2);
return;
}
const { data, width: dw, height: dh } = heatmap;
const cellW = canvasW / dw;
const cellH = canvasH / dh;
for (let y = 0; y < dh; y++) {
for (let x = 0; x < dw; x++) {
const val = Math.min(1, Math.max(0, data[y * dw + x]));
ctx.fillStyle = this._heatmapColor(val);
ctx.fillRect(x * cellW, y * cellH, cellW + 0.5, cellH + 0.5);
}
}
// Axis labels
ctx.font = '9px "JetBrains Mono", monospace';
ctx.fillStyle = 'rgba(255,255,255,0.4)';
ctx.fillText('Subcarrier →', 4, canvasH - 4);
ctx.save();
ctx.translate(canvasW - 4, canvasH - 4);
ctx.rotate(-Math.PI / 2);
ctx.fillText('Time ↑', 0, 0);
ctx.restore();
}
/**
* Draw embedding space 2D projection
* @param {CanvasRenderingContext2D} ctx
* @param {{ video: Array, csi: Array, fused: Array }} points
* @param {number} w
* @param {number} h
*/
drawEmbeddingSpace(ctx, points, w, h) {
ctx.fillStyle = '#050810';
ctx.fillRect(0, 0, w, h);
// Grid
ctx.strokeStyle = 'rgba(255,255,255,0.05)';
ctx.lineWidth = 0.5;
for (let i = 0; i <= 4; i++) {
const x = (i / 4) * w;
ctx.beginPath(); ctx.moveTo(x, 0); ctx.lineTo(x, h); ctx.stroke();
const y = (i / 4) * h;
ctx.beginPath(); ctx.moveTo(0, y); ctx.lineTo(w, y); ctx.stroke();
}
// Axes
ctx.strokeStyle = 'rgba(255,255,255,0.1)';
ctx.lineWidth = 1;
ctx.beginPath(); ctx.moveTo(w / 2, 0); ctx.lineTo(w / 2, h); ctx.stroke();
ctx.beginPath(); ctx.moveTo(0, h / 2); ctx.lineTo(w, h / 2); ctx.stroke();
const drawPoints = (pts, color, size) => {
if (!pts || pts.length === 0) return;
const len = pts.length;
for (let i = 0; i < len; i++) {
const p = pts[i];
if (!p) continue;
const age = 1 - (i / len) * 0.7; // Fade older points
const px = w / 2 + p[0] * w * 0.35;
const py = h / 2 + p[1] * h * 0.35;
if (px < 0 || px > w || py < 0 || py > h) continue;
ctx.beginPath();
ctx.arc(px, py, size, 0, Math.PI * 2);
ctx.fillStyle = color;
ctx.globalAlpha = age * 0.7;
ctx.fill();
}
};
drawPoints(points.video, this.colors.videoEmb, 3);
drawPoints(points.csi, this.colors.csiEmb, 3);
drawPoints(points.fused, this.colors.fusedEmb, 4);
ctx.globalAlpha = 1;
// Legend
ctx.font = '9px "JetBrains Mono", monospace';
const legends = [
{ color: this.colors.videoEmb, label: 'Video' },
{ color: this.colors.csiEmb, label: 'CSI' },
{ color: this.colors.fusedEmb, label: 'Fused' },
];
legends.forEach((l, i) => {
const ly = 12 + i * 14;
ctx.fillStyle = l.color;
ctx.beginPath();
ctx.arc(10, ly - 3, 3, 0, Math.PI * 2);
ctx.fill();
ctx.fillStyle = 'rgba(255,255,255,0.5)';
ctx.fillText(l.label, 18, ly);
});
}
_heatmapColor(val) {
// Dark blue → cyan → green → yellow → red
if (val < 0.25) {
const t = val / 0.25;
return `rgb(${Math.floor(t * 20)}, ${Math.floor(20 + t * 60)}, ${Math.floor(60 + t * 100)})`;
} else if (val < 0.5) {
const t = (val - 0.25) / 0.25;
return `rgb(${Math.floor(20 + t * 20)}, ${Math.floor(80 + t * 100)}, ${Math.floor(160 - t * 60)})`;
} else if (val < 0.75) {
const t = (val - 0.5) / 0.25;
return `rgb(${Math.floor(40 + t * 180)}, ${Math.floor(180 + t * 75)}, ${Math.floor(100 - t * 80)})`;
} else {
const t = (val - 0.75) / 0.25;
return `rgb(${Math.floor(220 + t * 35)}, ${Math.floor(255 - t * 120)}, ${Math.floor(20 - t * 20)})`;
}
}
}
+226
View File
@@ -0,0 +1,226 @@
/**
* CNN Embedder — Lightweight MobileNet-V3-style feature extractor.
*
* Architecture mirrors ruvector-cnn: Conv2D → BatchNorm → ReLU → Pool → Project → L2 Normalize
* Uses pre-seeded random weights (deterministic). When ruvector-cnn-wasm is available,
* transparently delegates to the WASM implementation.
*
* Two instances are created: one for video frames, one for CSI pseudo-images.
*/
// Seeded PRNG for deterministic weight initialization
function mulberry32(seed) {
return function() {
let t = (seed += 0x6D2B79F5);
t = Math.imul(t ^ (t >>> 15), t | 1);
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
export class CnnEmbedder {
/**
* @param {object} opts
* @param {number} opts.inputSize - Square input dimension (default 56 for speed)
* @param {number} opts.embeddingDim - Output embedding dimension (default 128)
* @param {boolean} opts.normalize - L2 normalize output
* @param {number} opts.seed - PRNG seed for weight init
*/
constructor(opts = {}) {
this.inputSize = opts.inputSize || 56;
this.embeddingDim = opts.embeddingDim || 128;
this.normalize = opts.normalize !== false;
this.wasmEmbedder = null;
// Initialize weights with deterministic PRNG
const rng = mulberry32(opts.seed || 42);
const randRange = (lo, hi) => lo + rng() * (hi - lo);
// Conv 3x3: 3 input channels → 16 output channels
this.convWeights = new Float32Array(3 * 3 * 3 * 16);
for (let i = 0; i < this.convWeights.length; i++) {
this.convWeights[i] = randRange(-0.15, 0.15);
}
// BatchNorm params (16 channels)
this.bnGamma = new Float32Array(16).fill(1.0);
this.bnBeta = new Float32Array(16).fill(0.0);
this.bnMean = new Float32Array(16).fill(0.0);
this.bnVar = new Float32Array(16).fill(1.0);
// Projection: 16 → embeddingDim
this.projWeights = new Float32Array(16 * this.embeddingDim);
for (let i = 0; i < this.projWeights.length; i++) {
this.projWeights[i] = randRange(-0.1, 0.1);
}
}
/**
* Try to load WASM embedder from ruvector-cnn-wasm package
* @param {string} wasmPath - Path to the WASM package directory
*/
async tryLoadWasm(wasmPath) {
try {
const mod = await import(`${wasmPath}/ruvector_cnn_wasm.js`);
await mod.default();
const config = new mod.EmbedderConfig();
config.input_size = this.inputSize;
config.embedding_dim = this.embeddingDim;
config.normalize = this.normalize;
this.wasmEmbedder = new mod.WasmCnnEmbedder(config);
console.log('[CNN] WASM embedder loaded successfully');
return true;
} catch (e) {
console.log('[CNN] WASM not available, using JS fallback:', e.message);
return false;
}
}
/**
* Extract embedding from RGB image data
* @param {Uint8Array} rgbData - RGB pixel data (H*W*3)
* @param {number} width
* @param {number} height
* @returns {Float32Array} embedding vector
*/
extract(rgbData, width, height) {
if (this.wasmEmbedder) {
try {
const result = this.wasmEmbedder.extract(rgbData, width, height);
return new Float32Array(result);
} catch (_) { /* fallback to JS */ }
}
return this._extractJS(rgbData, width, height);
}
_extractJS(rgbData, width, height) {
// 1. Resize to inputSize × inputSize if needed
const sz = this.inputSize;
let input;
if (width === sz && height === sz) {
input = new Float32Array(rgbData.length);
for (let i = 0; i < rgbData.length; i++) input[i] = rgbData[i] / 255.0;
} else {
input = this._resize(rgbData, width, height, sz, sz);
}
// 2. ImageNet normalization
const mean = [0.485, 0.456, 0.406];
const std = [0.229, 0.224, 0.225];
const pixels = sz * sz;
for (let i = 0; i < pixels; i++) {
input[i * 3] = (input[i * 3] - mean[0]) / std[0];
input[i * 3 + 1] = (input[i * 3 + 1] - mean[1]) / std[1];
input[i * 3 + 2] = (input[i * 3 + 2] - mean[2]) / std[2];
}
// 3. Conv2D 3x3 (3 → 16 channels)
const convOut = this._conv2d3x3(input, sz, sz, 3, 16);
// 4. BatchNorm
this._batchNorm(convOut, 16);
// 5. ReLU
for (let i = 0; i < convOut.length; i++) {
if (convOut[i] < 0) convOut[i] = 0;
}
// 6. Global average pooling → 16-dim
const outH = sz - 2, outW = sz - 2;
const pooled = new Float32Array(16);
const spatial = outH * outW;
for (let i = 0; i < spatial; i++) {
for (let c = 0; c < 16; c++) {
pooled[c] += convOut[i * 16 + c];
}
}
for (let c = 0; c < 16; c++) pooled[c] /= spatial;
// 7. Linear projection → embeddingDim
const emb = new Float32Array(this.embeddingDim);
for (let o = 0; o < this.embeddingDim; o++) {
let sum = 0;
for (let i = 0; i < 16; i++) {
sum += pooled[i] * this.projWeights[i * this.embeddingDim + o];
}
emb[o] = sum;
}
// 8. L2 normalize
if (this.normalize) {
let norm = 0;
for (let i = 0; i < emb.length; i++) norm += emb[i] * emb[i];
norm = Math.sqrt(norm);
if (norm > 1e-8) {
for (let i = 0; i < emb.length; i++) emb[i] /= norm;
}
}
return emb;
}
_conv2d3x3(input, H, W, Cin, Cout) {
const outH = H - 2, outW = W - 2;
const output = new Float32Array(outH * outW * Cout);
for (let y = 0; y < outH; y++) {
for (let x = 0; x < outW; x++) {
for (let co = 0; co < Cout; co++) {
let sum = 0;
for (let ky = 0; ky < 3; ky++) {
for (let kx = 0; kx < 3; kx++) {
for (let ci = 0; ci < Cin; ci++) {
const px = ((y + ky) * W + (x + kx)) * Cin + ci;
const wt = (((ky * 3 + kx) * Cin) + ci) * Cout + co;
sum += input[px] * this.convWeights[wt];
}
}
}
output[(y * outW + x) * Cout + co] = sum;
}
}
}
return output;
}
_batchNorm(data, channels) {
const spatial = data.length / channels;
for (let i = 0; i < spatial; i++) {
for (let c = 0; c < channels; c++) {
const idx = i * channels + c;
data[idx] = this.bnGamma[c] * (data[idx] - this.bnMean[c]) / Math.sqrt(this.bnVar[c] + 1e-5) + this.bnBeta[c];
}
}
}
_resize(rgbData, srcW, srcH, dstW, dstH) {
const output = new Float32Array(dstW * dstH * 3);
const xRatio = srcW / dstW;
const yRatio = srcH / dstH;
for (let y = 0; y < dstH; y++) {
for (let x = 0; x < dstW; x++) {
const sx = Math.min(Math.floor(x * xRatio), srcW - 1);
const sy = Math.min(Math.floor(y * yRatio), srcH - 1);
const srcIdx = (sy * srcW + sx) * 3;
const dstIdx = (y * dstW + x) * 3;
output[dstIdx] = rgbData[srcIdx] / 255.0;
output[dstIdx + 1] = rgbData[srcIdx + 1] / 255.0;
output[dstIdx + 2] = rgbData[srcIdx + 2] / 255.0;
}
}
return output;
}
/** Cosine similarity between two embeddings */
static cosineSimilarity(a, b) {
let dot = 0, normA = 0, normB = 0;
for (let i = 0; i < a.length; i++) {
dot += a[i] * b[i];
normA += a[i] * a[i];
normB += b[i] * b[i];
}
normA = Math.sqrt(normA);
normB = Math.sqrt(normB);
if (normA < 1e-8 || normB < 1e-8) return 0;
return dot / (normA * normB);
}
}
+267
View File
@@ -0,0 +1,267 @@
/**
* CSI Simulator — Generates realistic WiFi Channel State Information data.
*
* In live mode, connects to the sensing server via WebSocket.
* In demo mode, generates synthetic CSI that correlates with detected motion.
*
* Outputs: 3-channel pseudo-image (amplitude, phase, temporal diff)
* matching the ADR-018 frame format expectations.
*/
export class CsiSimulator {
constructor(opts = {}) {
this.subcarriers = opts.subcarriers || 52; // 802.11n HT20
this.timeWindow = opts.timeWindow || 56; // frames in sliding window
this.mode = 'demo'; // 'demo' | 'live'
this.ws = null;
// Circular buffer for CSI frames
this.amplitudeBuffer = [];
this.phaseBuffer = [];
this.frameCount = 0;
// Noise parameters
this._rng = this._mulberry32(opts.seed || 7);
this._noiseState = new Float32Array(this.subcarriers);
this._baseAmplitude = new Float32Array(this.subcarriers);
this._basePhase = new Float32Array(this.subcarriers);
// Initialize base CSI profile (empty room)
for (let i = 0; i < this.subcarriers; i++) {
this._baseAmplitude[i] = 0.5 + 0.3 * Math.sin(i * 0.12);
this._basePhase[i] = (i / this.subcarriers) * Math.PI * 2;
}
// Person influence (updated from video motion)
this.personPresence = 0;
this.personX = 0.5;
this.personY = 0.5;
this.personMotion = 0;
}
/**
* Connect to live sensing server WebSocket
* @param {string} url - WebSocket URL (e.g. ws://localhost:3030/ws/csi)
*/
async connectLive(url) {
return new Promise((resolve) => {
try {
this.ws = new WebSocket(url);
this.ws.binaryType = 'arraybuffer';
this.ws.onmessage = (evt) => this._handleLiveFrame(evt.data);
this.ws.onopen = () => { this.mode = 'live'; resolve(true); };
this.ws.onerror = () => resolve(false);
this.ws.onclose = () => { this.mode = 'demo'; };
// Timeout after 3s
setTimeout(() => { if (this.mode !== 'live') resolve(false); }, 3000);
} catch {
resolve(false);
}
});
}
disconnect() {
if (this.ws) { this.ws.close(); this.ws = null; }
this.mode = 'demo';
}
get isLive() { return this.mode === 'live'; }
/**
* Update person state from video detection (for correlated demo data).
* When person exits frame, CSI maintains presence with slow decay
* (simulating through-wall sensing capability).
*/
updatePersonState(presence, x, y, motion) {
if (presence > 0.1) {
// Person detected in video — update CSI state directly
this.personPresence = presence;
this.personX = x;
this.personY = y;
this.personMotion = motion;
this._lastSeenTime = performance.now();
this._lastSeenX = x;
this._lastSeenY = y;
} else if (this._lastSeenTime) {
// Person NOT in video — CSI "through-wall" persistence
const elapsed = (performance.now() - this._lastSeenTime) / 1000;
// CSI can sense through walls for ~10 seconds with decaying confidence
const decayRate = 0.15; // Lose ~15% per second
this.personPresence = Math.max(0, 1.0 - elapsed * decayRate);
// Position slowly drifts (person walking behind wall)
this.personX = this._lastSeenX;
this.personY = this._lastSeenY;
this.personMotion = Math.max(0, motion * 0.5 + this.personPresence * 0.2);
if (this.personPresence < 0.05) {
this._lastSeenTime = null;
}
} else {
this.personPresence = 0;
this.personMotion = 0;
}
}
/**
* Generate next CSI frame (demo mode) or return latest live frame
* @param {number} elapsed - Time in seconds
* @returns {{ amplitude: Float32Array, phase: Float32Array, snr: number }}
*/
nextFrame(elapsed) {
const amp = new Float32Array(this.subcarriers);
const phase = new Float32Array(this.subcarriers);
if (this.mode === 'live' && this._liveAmplitude) {
amp.set(this._liveAmplitude);
phase.set(this._livePhase);
} else {
this._generateDemoFrame(amp, phase, elapsed);
}
// Push to circular buffer
this.amplitudeBuffer.push(new Float32Array(amp));
this.phaseBuffer.push(new Float32Array(phase));
if (this.amplitudeBuffer.length > this.timeWindow) {
this.amplitudeBuffer.shift();
this.phaseBuffer.shift();
}
// SNR estimate
let signalPower = 0, noisePower = 0;
for (let i = 0; i < this.subcarriers; i++) {
signalPower += amp[i] * amp[i];
noisePower += this._noiseState[i] * this._noiseState[i];
}
const snr = noisePower > 0 ? 10 * Math.log10(signalPower / noisePower) : 30;
this.frameCount++;
return { amplitude: amp, phase, snr: Math.max(0, Math.min(40, snr)) };
}
/**
* Build 3-channel pseudo-image for CNN input
* @param {number} targetSize - Output image dimension (square)
* @returns {Uint8Array} RGB data (targetSize * targetSize * 3)
*/
buildPseudoImage(targetSize = 56) {
const buf = this.amplitudeBuffer;
const pBuf = this.phaseBuffer;
const frames = buf.length;
if (frames < 2) {
return new Uint8Array(targetSize * targetSize * 3);
}
const rgb = new Uint8Array(targetSize * targetSize * 3);
for (let y = 0; y < targetSize; y++) {
const fi = Math.min(Math.floor(y / targetSize * frames), frames - 1);
for (let x = 0; x < targetSize; x++) {
const si = Math.min(Math.floor(x / targetSize * this.subcarriers), this.subcarriers - 1);
const idx = (y * targetSize + x) * 3;
// R: Amplitude (normalized to 0-255)
const ampVal = buf[fi][si];
rgb[idx] = Math.min(255, Math.max(0, Math.floor(ampVal * 255)));
// G: Phase (wrapped to 0-255)
const phaseVal = (pBuf[fi][si] % (2 * Math.PI) + 2 * Math.PI) % (2 * Math.PI);
rgb[idx + 1] = Math.floor(phaseVal / (2 * Math.PI) * 255);
// B: Temporal difference
if (fi > 0) {
const diff = Math.abs(buf[fi][si] - buf[fi - 1][si]);
rgb[idx + 2] = Math.min(255, Math.floor(diff * 500));
}
}
}
return rgb;
}
/**
* Get heatmap data for visualization
* @returns {{ data: Float32Array, width: number, height: number }}
*/
getHeatmapData() {
const frames = this.amplitudeBuffer.length;
const w = this.subcarriers;
const h = Math.min(frames, this.timeWindow);
const data = new Float32Array(w * h);
for (let y = 0; y < h; y++) {
const fi = frames - h + y;
if (fi >= 0 && fi < frames) {
for (let x = 0; x < w; x++) {
data[y * w + x] = this.amplitudeBuffer[fi][x];
}
}
}
return { data, width: w, height: h };
}
// === Private ===
_generateDemoFrame(amp, phase, elapsed) {
const rng = this._rng;
const presence = this.personPresence;
const motion = this.personMotion;
const px = this.personX;
for (let i = 0; i < this.subcarriers; i++) {
// Base CSI profile (frequency-selective channel)
let a = this._baseAmplitude[i];
let p = this._basePhase[i] + elapsed * 0.05;
// Environmental noise (correlated across subcarriers)
this._noiseState[i] = 0.95 * this._noiseState[i] + 0.05 * (rng() * 2 - 1) * 0.03;
a += this._noiseState[i];
// Person-induced CSI perturbation
if (presence > 0.1) {
// Subcarrier-dependent body reflection (Fresnel zone model)
const freqOffset = (i - this.subcarriers * px) / (this.subcarriers * 0.3);
const bodyReflection = presence * 0.25 * Math.exp(-freqOffset * freqOffset);
// Motion causes amplitude fluctuation
const motionEffect = motion * 0.15 * Math.sin(elapsed * 3.5 + i * 0.3);
// Breathing modulation (0.2-0.3 Hz)
const breathing = presence * 0.02 * Math.sin(elapsed * 1.5 + i * 0.05);
a += bodyReflection + motionEffect + breathing;
p += presence * 0.4 * Math.sin(elapsed * 2.1 + i * 0.15);
}
amp[i] = Math.max(0, Math.min(1, a));
phase[i] = p;
}
}
_handleLiveFrame(data) {
const view = new DataView(data);
// Check ADR-018 magic: 0xC5110001
if (data.byteLength < 20) return;
const magic = view.getUint32(0, true);
if (magic !== 0xC5110001) return;
const numSub = Math.min(view.getUint16(8, true), this.subcarriers);
this._liveAmplitude = new Float32Array(this.subcarriers);
this._livePhase = new Float32Array(this.subcarriers);
const headerSize = 20;
for (let i = 0; i < numSub && (headerSize + i * 4 + 3) < data.byteLength; i++) {
const real = view.getInt16(headerSize + i * 4, true);
const imag = view.getInt16(headerSize + i * 4 + 2, true);
this._liveAmplitude[i] = Math.sqrt(real * real + imag * imag) / 2048;
this._livePhase[i] = Math.atan2(imag, real);
}
}
_mulberry32(seed) {
return function() {
let t = (seed += 0x6D2B79F5);
t = Math.imul(t ^ (t >>> 15), t | 1);
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
}
+166
View File
@@ -0,0 +1,166 @@
/**
* FusionEngine — Attention-weighted dual-modal embedding fusion.
*
* Combines visual (camera) and CSI (WiFi) embeddings with dynamic
* confidence gating based on signal quality.
*/
export class FusionEngine {
/**
* @param {number} embeddingDim
*/
constructor(embeddingDim = 128) {
this.embeddingDim = embeddingDim;
// Learnable attention weights (initialized to balanced 0.5)
// In production, these would be loaded from trained JSON
this.attentionWeights = new Float32Array(embeddingDim).fill(0.5);
// Dynamic modality confidence [0, 1]
this.videoConfidence = 1.0;
this.csiConfidence = 0.0;
this.fusedConfidence = 0.5;
// Smoothing for confidence transitions
this._smoothAlpha = 0.85;
// Embedding history for visualization
this.recentVideoEmbeddings = [];
this.recentCsiEmbeddings = [];
this.recentFusedEmbeddings = [];
this.maxHistory = 50;
}
/**
* Update quality-based confidence scores
* @param {number} videoBrightness - [0,1] video brightness quality
* @param {number} videoMotion - [0,1] motion detected
* @param {number} csiSnr - CSI signal-to-noise ratio in dB
* @param {boolean} csiActive - Whether CSI source is connected
*/
updateConfidence(videoBrightness, videoMotion, csiSnr, csiActive) {
// Video confidence: drops with low brightness, boosted by motion
let vc = 0;
if (videoBrightness > 0.05) {
vc = Math.min(1, videoBrightness * 1.5) * 0.7 + Math.min(1, videoMotion * 3) * 0.3;
}
// CSI confidence: based on SNR and connection status
let cc = 0;
if (csiActive) {
cc = Math.min(1, csiSnr / 25); // 25dB = full confidence
}
// Smooth transitions
this.videoConfidence = this._smoothAlpha * this.videoConfidence + (1 - this._smoothAlpha) * vc;
this.csiConfidence = this._smoothAlpha * this.csiConfidence + (1 - this._smoothAlpha) * cc;
// Fused confidence is the max of either (fusion can only help)
this.fusedConfidence = Math.min(1, Math.sqrt(
this.videoConfidence * this.videoConfidence + this.csiConfidence * this.csiConfidence
));
}
/**
* Fuse video and CSI embeddings
* @param {Float32Array|null} videoEmb - Visual embedding (or null if video-off)
* @param {Float32Array|null} csiEmb - CSI embedding (or null if CSI-off)
* @param {string} mode - 'dual' | 'video' | 'csi'
* @returns {Float32Array} Fused embedding
*/
fuse(videoEmb, csiEmb, mode = 'dual') {
const dim = this.embeddingDim;
const fused = new Float32Array(dim);
if (mode === 'video' || !csiEmb) {
if (videoEmb) fused.set(videoEmb);
this._recordEmbedding(videoEmb, null, fused);
return fused;
}
if (mode === 'csi' || !videoEmb) {
if (csiEmb) fused.set(csiEmb);
this._recordEmbedding(null, csiEmb, fused);
return fused;
}
// Dual mode: attention-weighted fusion with confidence gating
const totalConf = this.videoConfidence + this.csiConfidence;
const videoWeight = totalConf > 0 ? this.videoConfidence / totalConf : 0.5;
for (let i = 0; i < dim; i++) {
const alpha = this.attentionWeights[i] * videoWeight +
(1 - this.attentionWeights[i]) * (1 - videoWeight);
fused[i] = alpha * videoEmb[i] + (1 - alpha) * csiEmb[i];
}
// Re-normalize
let norm = 0;
for (let i = 0; i < dim; i++) norm += fused[i] * fused[i];
norm = Math.sqrt(norm);
if (norm > 1e-8) {
for (let i = 0; i < dim; i++) fused[i] /= norm;
}
this._recordEmbedding(videoEmb, csiEmb, fused);
return fused;
}
/**
* Get embedding pairs for 2D visualization (PCA projection)
* @returns {{ video: Array, csi: Array, fused: Array }}
*/
getEmbeddingPoints() {
// Simple 2D projection using first two principal components (approximated)
const project = (emb) => {
if (!emb || emb.length < 4) return null;
// Use pairs of dimensions as crude 2D projection
let x = 0, y = 0;
for (let i = 0; i < emb.length; i += 2) {
x += emb[i] * (i % 4 < 2 ? 1 : -1);
if (i + 1 < emb.length) {
y += emb[i + 1] * (i % 4 < 2 ? 1 : -1);
}
}
return [x * 2, y * 2]; // Scale for visibility
};
return {
video: this.recentVideoEmbeddings.map(project).filter(Boolean),
csi: this.recentCsiEmbeddings.map(project).filter(Boolean),
fused: this.recentFusedEmbeddings.map(project).filter(Boolean)
};
}
/**
* Cross-modal similarity score
* @returns {number} Cosine similarity between latest video and CSI embeddings
*/
getCrossModalSimilarity() {
const v = this.recentVideoEmbeddings[this.recentVideoEmbeddings.length - 1];
const c = this.recentCsiEmbeddings[this.recentCsiEmbeddings.length - 1];
if (!v || !c) return 0;
let dot = 0, na = 0, nb = 0;
for (let i = 0; i < v.length; i++) {
dot += v[i] * c[i];
na += v[i] * v[i];
nb += c[i] * c[i];
}
na = Math.sqrt(na); nb = Math.sqrt(nb);
return (na > 1e-8 && nb > 1e-8) ? dot / (na * nb) : 0;
}
_recordEmbedding(video, csi, fused) {
if (video) {
this.recentVideoEmbeddings.push(new Float32Array(video));
if (this.recentVideoEmbeddings.length > this.maxHistory) this.recentVideoEmbeddings.shift();
}
if (csi) {
this.recentCsiEmbeddings.push(new Float32Array(csi));
if (this.recentCsiEmbeddings.length > this.maxHistory) this.recentCsiEmbeddings.shift();
}
this.recentFusedEmbeddings.push(new Float32Array(fused));
if (this.recentFusedEmbeddings.length > this.maxHistory) this.recentFusedEmbeddings.shift();
}
}
+315
View File
@@ -0,0 +1,315 @@
/**
* WiFi-DensePose — Dual-Modal Pose Estimation Demo
*
* Main orchestration: video capture → CNN embedding → CSI processing → fusion → rendering
*/
import { VideoCapture } from './video-capture.js';
import { CsiSimulator } from './csi-simulator.js';
import { CnnEmbedder } from './cnn-embedder.js';
import { FusionEngine } from './fusion-engine.js';
import { PoseDecoder } from './pose-decoder.js';
import { CanvasRenderer } from './canvas-renderer.js';
// === State ===
let mode = 'dual'; // 'dual' | 'video' | 'csi'
let isRunning = false;
let isPaused = false;
let startTime = 0;
let frameCount = 0;
let fps = 0;
let lastFpsTime = 0;
let confidenceThreshold = 0.3;
// Latency tracking
const latency = { video: 0, csi: 0, fusion: 0, total: 0 };
// === Components ===
const videoCapture = new VideoCapture(document.getElementById('webcam'));
const csiSimulator = new CsiSimulator({ subcarriers: 52, timeWindow: 56 });
const visualCnn = new CnnEmbedder({ inputSize: 56, embeddingDim: 128, seed: 42 });
const csiCnn = new CnnEmbedder({ inputSize: 56, embeddingDim: 128, seed: 137 });
const fusionEngine = new FusionEngine(128);
const poseDecoder = new PoseDecoder(128);
const renderer = new CanvasRenderer();
// === Canvas Elements ===
const skeletonCanvas = document.getElementById('skeleton-canvas');
const skeletonCtx = skeletonCanvas.getContext('2d');
const csiCanvas = document.getElementById('csi-canvas');
const csiCtx = csiCanvas.getContext('2d');
const embeddingCanvas = document.getElementById('embedding-canvas');
const embeddingCtx = embeddingCanvas.getContext('2d');
// === UI Elements ===
const modeSelect = document.getElementById('mode-select');
const statusDot = document.getElementById('status-dot');
const statusLabel = document.getElementById('status-label');
const fpsDisplay = document.getElementById('fps-display');
const cameraPrompt = document.getElementById('camera-prompt');
const startCameraBtn = document.getElementById('start-camera-btn');
const pauseBtn = document.getElementById('pause-btn');
const confSlider = document.getElementById('confidence-slider');
const confValue = document.getElementById('confidence-value');
const wsUrlInput = document.getElementById('ws-url');
const connectWsBtn = document.getElementById('connect-ws-btn');
// Fusion bar elements
const videoBar = document.getElementById('video-bar');
const csiBar = document.getElementById('csi-bar');
const fusedBar = document.getElementById('fused-bar');
const videoBarVal = document.getElementById('video-bar-val');
const csiBarVal = document.getElementById('csi-bar-val');
const fusedBarVal = document.getElementById('fused-bar-val');
// Latency elements
const latVideoEl = document.getElementById('lat-video');
const latCsiEl = document.getElementById('lat-csi');
const latFusionEl = document.getElementById('lat-fusion');
const latTotalEl = document.getElementById('lat-total');
// Cross-modal similarity
const crossModalEl = document.getElementById('cross-modal-sim');
// === Initialize ===
function init() {
resizeCanvases();
window.addEventListener('resize', resizeCanvases);
// Mode change
modeSelect.addEventListener('change', (e) => {
mode = e.target.value;
updateModeUI();
});
// Camera start
startCameraBtn.addEventListener('click', startCamera);
// Pause
pauseBtn.addEventListener('click', () => {
isPaused = !isPaused;
pauseBtn.textContent = isPaused ? '▶ Resume' : '⏸ Pause';
pauseBtn.classList.toggle('active', isPaused);
});
// Confidence slider
confSlider.addEventListener('input', (e) => {
confidenceThreshold = parseFloat(e.target.value);
confValue.textContent = confidenceThreshold.toFixed(2);
});
// WebSocket connect
connectWsBtn.addEventListener('click', async () => {
const url = wsUrlInput.value.trim();
if (!url) return;
connectWsBtn.textContent = 'Connecting...';
const ok = await csiSimulator.connectLive(url);
connectWsBtn.textContent = ok ? '✓ Connected' : 'Connect';
if (ok) {
connectWsBtn.classList.add('active');
}
});
// Try to load WASM embedders (non-blocking)
// Resolve relative to this JS module file (in pose-fusion/js/) → ../pkg/
const wasmBase = new URL('../pkg/ruvector_cnn_wasm', import.meta.url).href;
visualCnn.tryLoadWasm(wasmBase);
csiCnn.tryLoadWasm(wasmBase);
// Auto-connect to local sensing server WebSocket if available
const defaultWsUrl = 'ws://localhost:8765/ws/sensing';
if (wsUrlInput) wsUrlInput.value = defaultWsUrl;
csiSimulator.connectLive(defaultWsUrl).then(ok => {
if (ok && connectWsBtn) {
connectWsBtn.textContent = '✓ Live ESP32';
connectWsBtn.classList.add('active');
statusLabel.textContent = 'LIVE CSI';
statusDot.classList.remove('offline');
}
});
// Auto-start camera for video/dual modes
updateModeUI();
startTime = performance.now() / 1000;
isRunning = true;
requestAnimationFrame(mainLoop);
}
async function startCamera() {
cameraPrompt.style.display = 'none';
const ok = await videoCapture.start();
if (ok) {
statusDot.classList.remove('offline');
statusLabel.textContent = 'LIVE';
resizeCanvases();
} else {
cameraPrompt.style.display = 'flex';
cameraPrompt.querySelector('p').textContent = 'Camera access denied. Try CSI-only mode.';
}
}
function updateModeUI() {
const needsVideo = mode !== 'csi';
const needsCsi = mode !== 'video';
// Show/hide camera prompt
if (needsVideo && !videoCapture.isActive) {
cameraPrompt.style.display = 'flex';
} else {
cameraPrompt.style.display = 'none';
}
}
function resizeCanvases() {
const videoPanel = document.querySelector('.video-panel');
if (videoPanel) {
const rect = videoPanel.getBoundingClientRect();
skeletonCanvas.width = rect.width;
skeletonCanvas.height = rect.height;
}
// CSI canvas
csiCanvas.width = csiCanvas.parentElement.clientWidth;
csiCanvas.height = 120;
// Embedding canvas
embeddingCanvas.width = embeddingCanvas.parentElement.clientWidth;
embeddingCanvas.height = 140;
}
// === Main Loop ===
function mainLoop(timestamp) {
if (!isRunning) return;
requestAnimationFrame(mainLoop);
if (isPaused) return;
const elapsed = performance.now() / 1000 - startTime;
const totalStart = performance.now();
// --- Video Pipeline ---
let videoEmb = null;
let motionRegion = null;
if (mode !== 'csi' && videoCapture.isActive) {
const t0 = performance.now();
const frame = videoCapture.captureFrame(56, 56);
if (frame) {
videoEmb = visualCnn.extract(frame.rgb, frame.width, frame.height);
motionRegion = videoCapture.detectMotionRegion(56, 56);
// Feed motion to CSI simulator for correlated demo data
// When detected=false, CSI simulator handles through-wall persistence
csiSimulator.updatePersonState(
motionRegion.detected ? 1.0 : 0,
motionRegion.detected ? motionRegion.x + motionRegion.w / 2 : 0.5,
motionRegion.detected ? motionRegion.y + motionRegion.h / 2 : 0.5,
frame.motion
);
fusionEngine.updateConfidence(
frame.brightness, frame.motion,
0, csiSimulator.isLive || mode === 'dual'
);
}
latency.video = performance.now() - t0;
}
// --- CSI Pipeline ---
let csiEmb = null;
if (mode !== 'video') {
const t0 = performance.now();
const csiFrame = csiSimulator.nextFrame(elapsed);
const pseudoImage = csiSimulator.buildPseudoImage(56);
csiEmb = csiCnn.extract(pseudoImage, 56, 56);
fusionEngine.updateConfidence(
videoCapture.brightnessScore,
videoCapture.motionScore,
csiFrame.snr,
true
);
// Draw CSI heatmap
const heatmap = csiSimulator.getHeatmapData();
renderer.drawCsiHeatmap(csiCtx, heatmap, csiCanvas.width, csiCanvas.height);
latency.csi = performance.now() - t0;
}
// --- Fusion ---
const t0f = performance.now();
const fusedEmb = fusionEngine.fuse(videoEmb, csiEmb, mode);
latency.fusion = performance.now() - t0f;
// --- Pose Decode ---
// For CSI-only mode, generate a synthetic motion region from CSI energy
if (mode === 'csi' && (!motionRegion || !motionRegion.detected)) {
const csiPresence = csiSimulator.personPresence;
if (csiPresence > 0.1) {
motionRegion = {
detected: true,
x: 0.25, y: 0.15, w: 0.5, h: 0.7,
coverage: csiPresence,
motionGrid: null,
gridCols: 10,
gridRows: 8
};
}
}
// CSI state for through-wall tracking
const csiState = {
csiPresence: csiSimulator.personPresence,
isLive: csiSimulator.isLive
};
const keypoints = poseDecoder.decode(fusedEmb, motionRegion, elapsed, csiState);
// --- Render Skeleton ---
const labelMap = { dual: 'DUAL FUSION', video: 'VIDEO ONLY', csi: 'CSI ONLY' };
renderer.drawSkeleton(skeletonCtx, keypoints, skeletonCanvas.width, skeletonCanvas.height, {
minConfidence: confidenceThreshold,
color: mode === 'csi' ? 'amber' : 'green',
label: labelMap[mode]
});
// --- Render Embedding Space ---
const embPoints = fusionEngine.getEmbeddingPoints();
renderer.drawEmbeddingSpace(embeddingCtx, embPoints, embeddingCanvas.width, embeddingCanvas.height);
// --- Update UI ---
latency.total = performance.now() - totalStart;
// FPS
frameCount++;
if (timestamp - lastFpsTime > 500) {
fps = Math.round(frameCount * 1000 / (timestamp - lastFpsTime));
lastFpsTime = timestamp;
frameCount = 0;
fpsDisplay.textContent = `${fps} FPS`;
}
// Fusion bars
const vc = fusionEngine.videoConfidence;
const cc = fusionEngine.csiConfidence;
const fc = fusionEngine.fusedConfidence;
videoBar.style.width = `${vc * 100}%`;
csiBar.style.width = `${cc * 100}%`;
fusedBar.style.width = `${fc * 100}%`;
videoBarVal.textContent = `${Math.round(vc * 100)}%`;
csiBarVal.textContent = `${Math.round(cc * 100)}%`;
fusedBarVal.textContent = `${Math.round(fc * 100)}%`;
// Latency
latVideoEl.textContent = `${latency.video.toFixed(1)}ms`;
latCsiEl.textContent = `${latency.csi.toFixed(1)}ms`;
latFusionEl.textContent = `${latency.fusion.toFixed(1)}ms`;
latTotalEl.textContent = `${latency.total.toFixed(1)}ms`;
// Cross-modal similarity
const sim = fusionEngine.getCrossModalSimilarity();
crossModalEl.textContent = sim.toFixed(3);
}
// Boot
document.addEventListener('DOMContentLoaded', init);
+373
View File
@@ -0,0 +1,373 @@
/**
* PoseDecoder — Maps motion detection grid → 17 COCO keypoints.
*
* Uses per-cell motion intensity to track actual body part positions:
* - Head: top-center motion cluster
* - Shoulders/Elbows/Wrists: lateral motion in upper body zone
* - Hips/Knees/Ankles: lower body motion distribution
*
* When person exits frame, CSI data continues tracking (through-wall mode).
*/
// COCO keypoint definitions
export const KEYPOINT_NAMES = [
'nose', 'left_eye', 'right_eye', 'left_ear', 'right_ear',
'left_shoulder', 'right_shoulder', 'left_elbow', 'right_elbow',
'left_wrist', 'right_wrist', 'left_hip', 'right_hip',
'left_knee', 'right_knee', 'left_ankle', 'right_ankle'
];
// Skeleton connections (pairs of keypoint indices)
export const SKELETON_CONNECTIONS = [
[0, 1], [0, 2], [1, 3], [2, 4], // Head
[5, 6], // Shoulders
[5, 7], [7, 9], // Left arm
[6, 8], [8, 10], // Right arm
[5, 11], [6, 12], // Torso
[11, 12], // Hips
[11, 13], [13, 15], // Left leg
[12, 14], [14, 16], // Right leg
];
// Standard body proportions (relative to body height)
const PROPORTIONS = {
headToShoulder: 0.15,
shoulderWidth: 0.25,
shoulderToElbow: 0.18,
elbowToWrist: 0.16,
shoulderToHip: 0.30,
hipWidth: 0.18,
hipToKnee: 0.24,
kneeToAnkle: 0.24,
eyeSpacing: 0.04,
earSpacing: 0.07,
};
export class PoseDecoder {
constructor(embeddingDim = 128) {
this.embeddingDim = embeddingDim;
this.smoothedKeypoints = null;
this.smoothingFactor = 0.45; // Lower = more responsive to movement
this._time = 0;
// Through-wall tracking state
this._lastBodyState = null;
this._ghostState = null;
this._ghostConfidence = 0;
this._ghostVelocity = { x: 0, y: 0 };
// Arm tracking history (smoothed positions)
this._leftArmY = 0.5;
this._rightArmY = 0.5;
this._leftArmX = 0;
this._rightArmX = 0;
this._headOffsetX = 0;
}
/**
* Decode motion data into 17 keypoints
* @param {Float32Array} embedding - Fused embedding vector
* @param {{ detected, x, y, w, h, motionGrid, gridCols, gridRows, motionCx, motionCy, exitDirection }} motionRegion
* @param {number} elapsed - Time in seconds
* @param {{ csiPresence: number }} csiState - CSI sensing state for through-wall
* @returns {Array<{x: number, y: number, confidence: number, name: string}>}
*/
decode(embedding, motionRegion, elapsed, csiState = {}) {
this._time = elapsed;
const hasMotion = motionRegion && motionRegion.detected;
const hasCsi = csiState && csiState.csiPresence > 0.1;
if (hasMotion) {
// Active tracking from video motion grid
this._ghostConfidence = 0;
const rawKeypoints = this._trackFromMotionGrid(motionRegion, embedding, elapsed);
this._lastBodyState = { keypoints: rawKeypoints.map(kp => ({...kp})), time: elapsed };
// Track exit velocity
if (motionRegion.exitDirection) {
const speed = 0.008;
this._ghostVelocity = {
x: motionRegion.exitDirection === 'left' ? -speed : motionRegion.exitDirection === 'right' ? speed : 0,
y: motionRegion.exitDirection === 'up' ? -speed : motionRegion.exitDirection === 'down' ? speed : 0
};
}
// Apply temporal smoothing
if (this.smoothedKeypoints && this.smoothedKeypoints.length === rawKeypoints.length) {
const alpha = this.smoothingFactor;
for (let i = 0; i < rawKeypoints.length; i++) {
rawKeypoints[i].x = alpha * this.smoothedKeypoints[i].x + (1 - alpha) * rawKeypoints[i].x;
rawKeypoints[i].y = alpha * this.smoothedKeypoints[i].y + (1 - alpha) * rawKeypoints[i].y;
}
}
this.smoothedKeypoints = rawKeypoints;
return rawKeypoints;
} else if (this._lastBodyState && (hasCsi || this._ghostConfidence > 0.05)) {
// Through-wall mode: person left frame but CSI still senses them
return this._trackThroughWall(elapsed, csiState);
} else if (this.smoothedKeypoints) {
// Fade out
const faded = this.smoothedKeypoints.map(kp => ({
...kp,
confidence: kp.confidence * 0.88
})).filter(kp => kp.confidence > 0.05);
if (faded.length === 0) this.smoothedKeypoints = null;
else this.smoothedKeypoints = faded;
return faded;
}
return [];
}
/**
* Track body parts from the motion grid.
* The grid tells us WHERE motion is happening → we map that to joint positions.
*/
_trackFromMotionGrid(region, embedding, elapsed) {
const grid = region.motionGrid;
const cols = region.gridCols || 10;
const rows = region.gridRows || 8;
// Body bounding box
const cx = region.x + region.w / 2;
const cy = region.y + region.h / 2;
const bodyH = Math.max(region.h, 0.3);
const bodyW = Math.max(region.w, 0.15);
// Analyze the motion grid to find arm positions
// Divide body into zones: head (top 20%), arms (top 60% sides), torso (center), legs (bottom 40%)
if (grid) {
const armAnalysis = this._analyzeArmMotion(grid, cols, rows, region);
// Smooth arm tracking
this._leftArmY = 0.6 * this._leftArmY + 0.4 * armAnalysis.leftArmHeight;
this._rightArmY = 0.6 * this._rightArmY + 0.4 * armAnalysis.rightArmHeight;
this._leftArmX = 0.6 * this._leftArmX + 0.4 * armAnalysis.leftArmSpread;
this._rightArmX = 0.6 * this._rightArmX + 0.4 * armAnalysis.rightArmSpread;
this._headOffsetX = 0.7 * this._headOffsetX + 0.3 * armAnalysis.headOffsetX;
}
const P = PROPORTIONS;
const halfW = P.shoulderWidth * bodyH / 2;
const hipHalfW = P.hipWidth * bodyH / 2;
// Breathing (subtle)
const breathe = Math.sin(elapsed * 1.5) * 0.002;
// Core body positions from detection center
const hipY = cy + bodyH * 0.15;
const shoulderY = hipY - P.shoulderToHip * bodyH + breathe;
const headY = shoulderY - P.headToShoulder * bodyH;
const kneeY = hipY + P.hipToKnee * bodyH;
const ankleY = kneeY + P.kneeToAnkle * bodyH;
// HEAD follows motion centroid
const headX = cx + this._headOffsetX * bodyW * 0.3;
// ARM POSITIONS driven by motion grid analysis
// leftArmY: 0 = arm down at side, 1 = arm fully raised
// leftArmSpread: how far out the arm extends
const leftArmRaise = this._leftArmY; // 0-1
const rightArmRaise = this._rightArmY;
const leftSpread = 0.02 + this._leftArmX * 0.12;
const rightSpread = 0.02 + this._rightArmX * 0.12;
// Elbow: interpolate between "at side" and "raised"
const lElbowY = shoulderY + P.shoulderToElbow * bodyH * (1 - leftArmRaise * 0.9);
const rElbowY = shoulderY + P.shoulderToElbow * bodyH * (1 - rightArmRaise * 0.9);
const lElbowX = cx - halfW - leftSpread;
const rElbowX = cx + halfW + rightSpread;
// Wrist: extends further when raised
const lWristY = lElbowY + P.elbowToWrist * bodyH * (1 - leftArmRaise * 1.1);
const rWristY = rElbowY + P.elbowToWrist * bodyH * (1 - rightArmRaise * 1.1);
const lWristX = lElbowX - leftSpread * 0.6;
const rWristX = rElbowX + rightSpread * 0.6;
// Leg motion from lower grid cells
const legMotion = grid ? this._analyzeLegMotion(grid, cols, rows) : { left: 0, right: 0 };
const legSwing = 0.015;
const keypoints = [
// 0: nose
{ x: headX, y: headY + 0.01, confidence: 0.92 },
// 1: left_eye
{ x: headX - P.eyeSpacing * bodyH, y: headY - 0.005, confidence: 0.88 },
// 2: right_eye
{ x: headX + P.eyeSpacing * bodyH, y: headY - 0.005, confidence: 0.88 },
// 3: left_ear
{ x: headX - P.earSpacing * bodyH, y: headY + 0.005, confidence: 0.72 },
// 4: right_ear
{ x: headX + P.earSpacing * bodyH, y: headY + 0.005, confidence: 0.72 },
// 5: left_shoulder
{ x: cx - halfW, y: shoulderY, confidence: 0.94 },
// 6: right_shoulder
{ x: cx + halfW, y: shoulderY, confidence: 0.94 },
// 7: left_elbow
{ x: lElbowX, y: lElbowY, confidence: 0.87 },
// 8: right_elbow
{ x: rElbowX, y: rElbowY, confidence: 0.87 },
// 9: left_wrist
{ x: lWristX, y: lWristY, confidence: 0.82 },
// 10: right_wrist
{ x: rWristX, y: rWristY, confidence: 0.82 },
// 11: left_hip
{ x: cx - hipHalfW, y: hipY, confidence: 0.91 },
// 12: right_hip
{ x: cx + hipHalfW, y: hipY, confidence: 0.91 },
// 13: left_knee
{ x: cx - hipHalfW + legMotion.left * legSwing, y: kneeY, confidence: 0.88 },
// 14: right_knee
{ x: cx + hipHalfW + legMotion.right * legSwing, y: kneeY, confidence: 0.88 },
// 15: left_ankle
{ x: cx - hipHalfW + legMotion.left * legSwing * 1.3, y: ankleY, confidence: 0.83 },
// 16: right_ankle
{ x: cx + hipHalfW + legMotion.right * legSwing * 1.3, y: ankleY, confidence: 0.83 },
];
for (let i = 0; i < keypoints.length; i++) {
keypoints[i].name = KEYPOINT_NAMES[i];
}
return keypoints;
}
/**
* Analyze the motion grid to determine arm positions.
* Left side of grid = left side of body, etc.
*/
_analyzeArmMotion(grid, cols, rows, region) {
// Body center column
const centerCol = Math.floor(cols / 2);
// Upper body rows (top 60% of detected region)
const upperEnd = Math.floor(rows * 0.6);
// Compute motion intensity for left vs right, at different heights
let leftUpperMotion = 0, leftMidMotion = 0;
let rightUpperMotion = 0, rightMidMotion = 0;
let leftCount = 0, rightCount = 0;
let headMotionX = 0, headMotionWeight = 0;
for (let r = 0; r < upperEnd; r++) {
const heightWeight = 1.0 - (r / upperEnd) * 0.3; // Upper rows weighted more
// Head zone: top 25%, center 40% of width
if (r < Math.floor(rows * 0.25)) {
const headLeft = Math.floor(cols * 0.3);
const headRight = Math.floor(cols * 0.7);
for (let c = headLeft; c <= headRight; c++) {
const val = grid[r][c];
headMotionX += (c / cols - 0.5) * val;
headMotionWeight += val;
}
}
// Left arm zone: left 40% of grid
for (let c = 0; c < Math.floor(cols * 0.4); c++) {
const val = grid[r][c];
if (r < rows * 0.3) leftUpperMotion += val * heightWeight;
else leftMidMotion += val * heightWeight;
leftCount++;
}
// Right arm zone: right 40% of grid
for (let c = Math.floor(cols * 0.6); c < cols; c++) {
const val = grid[r][c];
if (r < rows * 0.3) rightUpperMotion += val * heightWeight;
else rightMidMotion += val * heightWeight;
rightCount++;
}
}
// Normalize
const leftTotal = leftUpperMotion + leftMidMotion;
const rightTotal = rightUpperMotion + rightMidMotion;
const maxMotion = 0.15; // Calibration threshold
// Arm height: 0 = at side, 1 = raised
// High motion in upper-left → left arm is raised
const leftArmHeight = Math.min(1, (leftUpperMotion / maxMotion) * 2);
const rightArmHeight = Math.min(1, (rightUpperMotion / maxMotion) * 2);
// Arm spread: how far out from body
const leftArmSpread = Math.min(1, leftTotal / maxMotion);
const rightArmSpread = Math.min(1, rightTotal / maxMotion);
// Head offset
const headOffsetX = headMotionWeight > 0.01 ? headMotionX / headMotionWeight : 0;
return { leftArmHeight, rightArmHeight, leftArmSpread, rightArmSpread, headOffsetX };
}
/**
* Analyze lower grid for leg motion.
*/
_analyzeLegMotion(grid, cols, rows) {
const lowerStart = Math.floor(rows * 0.6);
let leftMotion = 0, rightMotion = 0;
for (let r = lowerStart; r < rows; r++) {
for (let c = 0; c < Math.floor(cols / 2); c++) {
leftMotion += grid[r][c];
}
for (let c = Math.floor(cols / 2); c < cols; c++) {
rightMotion += grid[r][c];
}
}
// Return as -1 to 1 range (asymmetry indicates which leg is moving)
const total = leftMotion + rightMotion + 0.001;
return {
left: (leftMotion - rightMotion) / total,
right: (rightMotion - leftMotion) / total
};
}
/**
* Through-wall tracking: continue showing pose via CSI when person left video frame.
* The skeleton drifts in the exit direction with decreasing confidence.
*/
_trackThroughWall(elapsed, csiState) {
if (!this._lastBodyState) return [];
const dt = elapsed - this._lastBodyState.time;
const csiPresence = csiState.csiPresence || 0;
// Initialize ghost on first call
if (this._ghostConfidence <= 0.05) {
this._ghostConfidence = 0.8;
this._ghostState = this._lastBodyState.keypoints.map(kp => ({...kp}));
}
// Ghost confidence decays, but CSI presence sustains it
const csiBoost = Math.min(0.7, csiPresence * 0.8);
this._ghostConfidence = Math.max(0.05, this._ghostConfidence * 0.995 - 0.001 + csiBoost * 0.002);
// Drift the ghost in exit direction
const vx = this._ghostVelocity.x;
const vy = this._ghostVelocity.y;
// Breathing continues via CSI
const breathe = Math.sin(elapsed * 1.5) * 0.003 * csiPresence;
const keypoints = this._ghostState.map((kp, i) => {
return {
x: kp.x + vx * dt * 0.3,
y: kp.y + vy * dt * 0.3 + (i >= 5 && i <= 6 ? breathe : 0),
confidence: kp.confidence * this._ghostConfidence * (0.5 + csiPresence * 0.5),
name: kp.name
};
});
// Slow down drift over time
this._ghostVelocity.x *= 0.998;
this._ghostVelocity.y *= 0.998;
this.smoothedKeypoints = keypoints;
return keypoints;
}
}
+235
View File
@@ -0,0 +1,235 @@
/**
* VideoCapture — getUserMedia webcam capture with frame extraction.
* Provides quality metrics (brightness, motion) for fusion confidence gating.
*/
export class VideoCapture {
constructor(videoElement) {
this.video = videoElement;
this.stream = null;
this.offscreen = document.createElement('canvas');
this.offCtx = this.offscreen.getContext('2d', { willReadFrequently: true });
this.prevFrame = null;
this.motionScore = 0;
this.brightnessScore = 0;
}
async start(constraints = {}) {
const defaultConstraints = {
video: {
width: { ideal: 640 },
height: { ideal: 480 },
facingMode: 'user',
frameRate: { ideal: 30 }
},
audio: false
};
try {
this.stream = await navigator.mediaDevices.getUserMedia(
Object.keys(constraints).length ? constraints : defaultConstraints
);
this.video.srcObject = this.stream;
await this.video.play();
this.offscreen.width = this.video.videoWidth;
this.offscreen.height = this.video.videoHeight;
return true;
} catch (err) {
console.error('[Video] Camera access failed:', err.message);
return false;
}
}
stop() {
if (this.stream) {
this.stream.getTracks().forEach(t => t.stop());
this.stream = null;
}
this.video.srcObject = null;
}
get isActive() {
return this.stream !== null && this.video.readyState >= 2;
}
get width() { return this.video.videoWidth || 640; }
get height() { return this.video.videoHeight || 480; }
/**
* Capture current frame as RGB Uint8Array + compute quality metrics.
* @param {number} targetW - Target width for CNN input
* @param {number} targetH - Target height for CNN input
* @returns {{ rgb: Uint8Array, width: number, height: number, motion: number, brightness: number }}
*/
captureFrame(targetW = 56, targetH = 56) {
if (!this.isActive) return null;
// Draw to offscreen at target resolution
this.offscreen.width = targetW;
this.offscreen.height = targetH;
this.offCtx.drawImage(this.video, 0, 0, targetW, targetH);
const imageData = this.offCtx.getImageData(0, 0, targetW, targetH);
const rgba = imageData.data;
// Convert RGBA → RGB
const pixels = targetW * targetH;
const rgb = new Uint8Array(pixels * 3);
let brightnessSum = 0;
let motionSum = 0;
for (let i = 0; i < pixels; i++) {
const r = rgba[i * 4];
const g = rgba[i * 4 + 1];
const b = rgba[i * 4 + 2];
rgb[i * 3] = r;
rgb[i * 3 + 1] = g;
rgb[i * 3 + 2] = b;
// Luminance for brightness
const lum = 0.299 * r + 0.587 * g + 0.114 * b;
brightnessSum += lum;
// Motion: diff from previous frame
if (this.prevFrame) {
const pr = this.prevFrame[i * 3];
const pg = this.prevFrame[i * 3 + 1];
const pb = this.prevFrame[i * 3 + 2];
motionSum += Math.abs(r - pr) + Math.abs(g - pg) + Math.abs(b - pb);
}
}
this.brightnessScore = brightnessSum / (pixels * 255);
this.motionScore = this.prevFrame ? Math.min(1, motionSum / (pixels * 100)) : 0;
this.prevFrame = new Uint8Array(rgb);
return {
rgb,
width: targetW,
height: targetH,
motion: this.motionScore,
brightness: this.brightnessScore
};
}
/**
* Capture full-resolution RGBA for overlay rendering
* @returns {ImageData|null}
*/
captureFullFrame() {
if (!this.isActive) return null;
this.offscreen.width = this.width;
this.offscreen.height = this.height;
this.offCtx.drawImage(this.video, 0, 0);
return this.offCtx.getImageData(0, 0, this.width, this.height);
}
/**
* Detect motion region + detailed motion grid for body-part tracking.
* Returns bounding box + a grid showing WHERE motion is concentrated.
* @returns {{ x, y, w, h, detected: boolean, motionGrid: number[][], gridCols: number, gridRows: number, exitDirection: string|null }}
*/
detectMotionRegion(targetW = 56, targetH = 56) {
if (!this.isActive || !this.prevFrame) return { detected: false, motionGrid: null };
this.offscreen.width = targetW;
this.offscreen.height = targetH;
this.offCtx.drawImage(this.video, 0, 0, targetW, targetH);
const rgba = this.offCtx.getImageData(0, 0, targetW, targetH).data;
let minX = targetW, minY = targetH, maxX = 0, maxY = 0;
let motionPixels = 0;
const threshold = 25;
// Motion grid: divide frame into cells and track motion intensity per cell
const gridCols = 10;
const gridRows = 8;
const cellW = targetW / gridCols;
const cellH = targetH / gridRows;
const motionGrid = Array.from({ length: gridRows }, () => new Float32Array(gridCols));
const cellPixels = cellW * cellH;
// Also track motion centroid weighted by intensity
let motionCxSum = 0, motionCySum = 0, motionWeightSum = 0;
for (let y = 0; y < targetH; y++) {
for (let x = 0; x < targetW; x++) {
const i = y * targetW + x;
const r = rgba[i * 4], g = rgba[i * 4 + 1], b = rgba[i * 4 + 2];
const pr = this.prevFrame[i * 3], pg = this.prevFrame[i * 3 + 1], pb = this.prevFrame[i * 3 + 2];
const diff = Math.abs(r - pr) + Math.abs(g - pg) + Math.abs(b - pb);
if (diff > threshold * 3) {
motionPixels++;
if (x < minX) minX = x;
if (y < minY) minY = y;
if (x > maxX) maxX = x;
if (y > maxY) maxY = y;
}
// Accumulate per-cell motion intensity
const gc = Math.min(Math.floor(x / cellW), gridCols - 1);
const gr = Math.min(Math.floor(y / cellH), gridRows - 1);
const intensity = diff / (3 * 255); // Normalize 0-1
motionGrid[gr][gc] += intensity / cellPixels;
// Weighted centroid
if (diff > threshold) {
motionCxSum += x * diff;
motionCySum += y * diff;
motionWeightSum += diff;
}
}
}
const detected = motionPixels > (targetW * targetH * 0.02);
// Motion centroid (normalized 0-1)
const motionCx = motionWeightSum > 0 ? motionCxSum / (motionWeightSum * targetW) : 0.5;
const motionCy = motionWeightSum > 0 ? motionCySum / (motionWeightSum * targetH) : 0.5;
// Detect exit direction: if centroid is near edges
let exitDirection = null;
if (detected && motionCx < 0.1) exitDirection = 'left';
else if (detected && motionCx > 0.9) exitDirection = 'right';
else if (detected && motionCy < 0.1) exitDirection = 'up';
else if (detected && motionCy > 0.9) exitDirection = 'down';
// Track last known position for through-wall persistence
if (detected) {
this._lastDetected = {
x: minX / targetW,
y: minY / targetH,
w: (maxX - minX) / targetW,
h: (maxY - minY) / targetH,
cx: motionCx,
cy: motionCy,
exitDirection,
time: performance.now()
};
}
return {
detected,
x: minX / targetW,
y: minY / targetH,
w: (maxX - minX) / targetW,
h: (maxY - minY) / targetH,
coverage: motionPixels / (targetW * targetH),
motionGrid,
gridCols,
gridRows,
motionCx,
motionCy,
exitDirection
};
}
/**
* Get the last known detection info (for through-wall persistence)
*/
get lastDetection() {
return this._lastDetected || null;
}
}
@@ -0,0 +1,26 @@
{
"name": "ruvector-cnn-wasm",
"type": "module",
"description": "WASM bindings for ruvector-cnn - CNN feature extraction for image embeddings",
"version": "0.1.0",
"license": "MIT OR Apache-2.0",
"repository": {
"type": "git",
"url": "https://github.com/ruvnet/ruvector"
},
"files": [
"ruvector_cnn_wasm_bg.wasm",
"ruvector_cnn_wasm.js"
],
"main": "ruvector_cnn_wasm.js",
"sideEffects": [
"./snippets/*"
],
"keywords": [
"cnn",
"embeddings",
"wasm",
"simd",
"machine-learning"
]
}
@@ -0,0 +1,802 @@
/**
* Configuration for CNN embedder
*/
export class EmbedderConfig {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
EmbedderConfigFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_embedderconfig_free(ptr, 0);
}
constructor() {
const ret = wasm.embedderconfig_new();
this.__wbg_ptr = ret >>> 0;
EmbedderConfigFinalization.register(this, this.__wbg_ptr, this);
return this;
}
/**
* Output embedding dimension
* @returns {number}
*/
get embedding_dim() {
const ret = wasm.__wbg_get_embedderconfig_embedding_dim(this.__wbg_ptr);
return ret >>> 0;
}
/**
* Input image size (square)
* @returns {number}
*/
get input_size() {
const ret = wasm.__wbg_get_embedderconfig_input_size(this.__wbg_ptr);
return ret >>> 0;
}
/**
* Whether to L2 normalize embeddings
* @returns {boolean}
*/
get normalize() {
const ret = wasm.__wbg_get_embedderconfig_normalize(this.__wbg_ptr);
return ret !== 0;
}
/**
* Output embedding dimension
* @param {number} arg0
*/
set embedding_dim(arg0) {
wasm.__wbg_set_embedderconfig_embedding_dim(this.__wbg_ptr, arg0);
}
/**
* Input image size (square)
* @param {number} arg0
*/
set input_size(arg0) {
wasm.__wbg_set_embedderconfig_input_size(this.__wbg_ptr, arg0);
}
/**
* Whether to L2 normalize embeddings
* @param {boolean} arg0
*/
set normalize(arg0) {
wasm.__wbg_set_embedderconfig_normalize(this.__wbg_ptr, arg0);
}
}
if (Symbol.dispose) EmbedderConfig.prototype[Symbol.dispose] = EmbedderConfig.prototype.free;
/**
* Layer operations for building custom networks
*/
export class LayerOps {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
LayerOpsFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_layerops_free(ptr, 0);
}
/**
* Apply batch normalization (returns new array)
* @param {Float32Array} input
* @param {Float32Array} gamma
* @param {Float32Array} beta
* @param {Float32Array} mean
* @param {Float32Array} _var
* @param {number} epsilon
* @returns {Float32Array}
*/
static batch_norm(input, gamma, beta, mean, _var, epsilon) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(input, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
const ptr1 = passArrayF32ToWasm0(gamma, wasm.__wbindgen_export2);
const len1 = WASM_VECTOR_LEN;
const ptr2 = passArrayF32ToWasm0(beta, wasm.__wbindgen_export2);
const len2 = WASM_VECTOR_LEN;
const ptr3 = passArrayF32ToWasm0(mean, wasm.__wbindgen_export2);
const len3 = WASM_VECTOR_LEN;
const ptr4 = passArrayF32ToWasm0(_var, wasm.__wbindgen_export2);
const len4 = WASM_VECTOR_LEN;
wasm.layerops_batch_norm(retptr, ptr0, len0, ptr1, len1, ptr2, len2, ptr3, len3, ptr4, len4, epsilon);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var v6 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v6;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Apply global average pooling
* Returns one value per channel
* @param {Float32Array} input
* @param {number} height
* @param {number} width
* @param {number} channels
* @returns {Float32Array}
*/
static global_avg_pool(input, height, width, channels) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(input, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.layerops_global_avg_pool(retptr, ptr0, len0, height, width, channels);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var v2 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v2;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
}
if (Symbol.dispose) LayerOps.prototype[Symbol.dispose] = LayerOps.prototype.free;
/**
* SIMD-optimized operations
*/
export class SimdOps {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
SimdOpsFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_simdops_free(ptr, 0);
}
/**
* Dot product of two vectors
* @param {Float32Array} a
* @param {Float32Array} b
* @returns {number}
*/
static dot_product(a, b) {
const ptr0 = passArrayF32ToWasm0(a, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
const ptr1 = passArrayF32ToWasm0(b, wasm.__wbindgen_export2);
const len1 = WASM_VECTOR_LEN;
const ret = wasm.simdops_dot_product(ptr0, len0, ptr1, len1);
return ret;
}
/**
* L2 normalize a vector (returns new array)
* @param {Float32Array} data
* @returns {Float32Array}
*/
static l2_normalize(data) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(data, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.simdops_l2_normalize(retptr, ptr0, len0);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var v2 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v2;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* ReLU activation (returns new array)
* @param {Float32Array} data
* @returns {Float32Array}
*/
static relu(data) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(data, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.simdops_relu(retptr, ptr0, len0);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var v2 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v2;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* ReLU6 activation (returns new array)
* @param {Float32Array} data
* @returns {Float32Array}
*/
static relu6(data) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(data, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.simdops_relu6(retptr, ptr0, len0);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var v2 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v2;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
}
if (Symbol.dispose) SimdOps.prototype[Symbol.dispose] = SimdOps.prototype.free;
/**
* WASM CNN Embedder for image feature extraction
*/
export class WasmCnnEmbedder {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
WasmCnnEmbedderFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_wasmcnnembedder_free(ptr, 0);
}
/**
* Compute cosine similarity between two embeddings
* @param {Float32Array} a
* @param {Float32Array} b
* @returns {number}
*/
cosine_similarity(a, b) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(a, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
const ptr1 = passArrayF32ToWasm0(b, wasm.__wbindgen_export2);
const len1 = WASM_VECTOR_LEN;
wasm.wasmcnnembedder_cosine_similarity(retptr, this.__wbg_ptr, ptr0, len0, ptr1, len1);
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
if (r2) {
throw takeObject(r1);
}
return r0;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Get the embedding dimension
* @returns {number}
*/
get embedding_dim() {
const ret = wasm.wasmcnnembedder_embedding_dim(this.__wbg_ptr);
return ret >>> 0;
}
/**
* Extract embedding from image data (RGB format, row-major)
* @param {Uint8Array} image_data
* @param {number} width
* @param {number} height
* @returns {Float32Array}
*/
extract(image_data, width, height) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArray8ToWasm0(image_data, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.wasmcnnembedder_extract(retptr, this.__wbg_ptr, ptr0, len0, width, height);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
var r3 = getDataViewMemory0().getInt32(retptr + 4 * 3, true);
if (r3) {
throw takeObject(r2);
}
var v2 = getArrayF32FromWasm0(r0, r1).slice();
wasm.__wbindgen_export(r0, r1 * 4, 4);
return v2;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Create a new CNN embedder
* @param {EmbedderConfig | null} [config]
*/
constructor(config) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
let ptr0 = 0;
if (!isLikeNone(config)) {
_assertClass(config, EmbedderConfig);
ptr0 = config.__destroy_into_raw();
}
wasm.wasmcnnembedder_new(retptr, ptr0);
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
if (r2) {
throw takeObject(r1);
}
this.__wbg_ptr = r0 >>> 0;
WasmCnnEmbedderFinalization.register(this, this.__wbg_ptr, this);
return this;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
}
if (Symbol.dispose) WasmCnnEmbedder.prototype[Symbol.dispose] = WasmCnnEmbedder.prototype.free;
/**
* InfoNCE loss for contrastive learning (SimCLR style)
*/
export class WasmInfoNCELoss {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
WasmInfoNCELossFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_wasminfonceloss_free(ptr, 0);
}
/**
* Compute loss for a batch of embedding pairs
* embeddings: [2N, D] flattened where (i, i+N) are positive pairs
* @param {Float32Array} embeddings
* @param {number} batch_size
* @param {number} dim
* @returns {number}
*/
forward(embeddings, batch_size, dim) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(embeddings, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
wasm.wasminfonceloss_forward(retptr, this.__wbg_ptr, ptr0, len0, batch_size, dim);
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
if (r2) {
throw takeObject(r1);
}
return r0;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Create new InfoNCE loss with temperature parameter
* @param {number} temperature
*/
constructor(temperature) {
const ret = wasm.wasminfonceloss_new(temperature);
this.__wbg_ptr = ret >>> 0;
WasmInfoNCELossFinalization.register(this, this.__wbg_ptr, this);
return this;
}
/**
* Get the temperature parameter
* @returns {number}
*/
get temperature() {
const ret = wasm.wasminfonceloss_temperature(this.__wbg_ptr);
return ret;
}
}
if (Symbol.dispose) WasmInfoNCELoss.prototype[Symbol.dispose] = WasmInfoNCELoss.prototype.free;
/**
* Triplet loss for metric learning
*/
export class WasmTripletLoss {
__destroy_into_raw() {
const ptr = this.__wbg_ptr;
this.__wbg_ptr = 0;
WasmTripletLossFinalization.unregister(this);
return ptr;
}
free() {
const ptr = this.__destroy_into_raw();
wasm.__wbg_wasmtripletloss_free(ptr, 0);
}
/**
* Compute loss for a batch of triplets
* @param {Float32Array} anchors
* @param {Float32Array} positives
* @param {Float32Array} negatives
* @param {number} dim
* @returns {number}
*/
forward(anchors, positives, negatives, dim) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(anchors, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
const ptr1 = passArrayF32ToWasm0(positives, wasm.__wbindgen_export2);
const len1 = WASM_VECTOR_LEN;
const ptr2 = passArrayF32ToWasm0(negatives, wasm.__wbindgen_export2);
const len2 = WASM_VECTOR_LEN;
wasm.wasmtripletloss_forward(retptr, this.__wbg_ptr, ptr0, len0, ptr1, len1, ptr2, len2, dim);
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
if (r2) {
throw takeObject(r1);
}
return r0;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Compute loss for a single triplet
* @param {Float32Array} anchor
* @param {Float32Array} positive
* @param {Float32Array} negative
* @returns {number}
*/
forward_single(anchor, positive, negative) {
try {
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
const ptr0 = passArrayF32ToWasm0(anchor, wasm.__wbindgen_export2);
const len0 = WASM_VECTOR_LEN;
const ptr1 = passArrayF32ToWasm0(positive, wasm.__wbindgen_export2);
const len1 = WASM_VECTOR_LEN;
const ptr2 = passArrayF32ToWasm0(negative, wasm.__wbindgen_export2);
const len2 = WASM_VECTOR_LEN;
wasm.wasmtripletloss_forward_single(retptr, this.__wbg_ptr, ptr0, len0, ptr1, len1, ptr2, len2);
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
if (r2) {
throw takeObject(r1);
}
return r0;
} finally {
wasm.__wbindgen_add_to_stack_pointer(16);
}
}
/**
* Get the margin parameter
* @returns {number}
*/
get margin() {
const ret = wasm.wasmtripletloss_margin(this.__wbg_ptr);
return ret;
}
/**
* Create new triplet loss with margin
* @param {number} margin
*/
constructor(margin) {
const ret = wasm.wasmtripletloss_new(margin);
this.__wbg_ptr = ret >>> 0;
WasmTripletLossFinalization.register(this, this.__wbg_ptr, this);
return this;
}
}
if (Symbol.dispose) WasmTripletLoss.prototype[Symbol.dispose] = WasmTripletLoss.prototype.free;
/**
* Initialize panic hook for better error messages
*/
export function init() {
wasm.init();
}
function __wbg_get_imports() {
const import0 = {
__proto__: null,
__wbg___wbindgen_throw_39bc967c0e5a9b58: function(arg0, arg1) {
throw new Error(getStringFromWasm0(arg0, arg1));
},
__wbg_error_a6fa202b58aa1cd3: function(arg0, arg1) {
let deferred0_0;
let deferred0_1;
try {
deferred0_0 = arg0;
deferred0_1 = arg1;
console.error(getStringFromWasm0(arg0, arg1));
} finally {
wasm.__wbindgen_export(deferred0_0, deferred0_1, 1);
}
},
__wbg_new_227d7c05414eb861: function() {
const ret = new Error();
return addHeapObject(ret);
},
__wbg_stack_3b0d974bbf31e44f: function(arg0, arg1) {
const ret = getObject(arg1).stack;
const ptr1 = passStringToWasm0(ret, wasm.__wbindgen_export2, wasm.__wbindgen_export3);
const len1 = WASM_VECTOR_LEN;
getDataViewMemory0().setInt32(arg0 + 4 * 1, len1, true);
getDataViewMemory0().setInt32(arg0 + 4 * 0, ptr1, true);
},
__wbindgen_cast_0000000000000001: function(arg0, arg1) {
// Cast intrinsic for `Ref(String) -> Externref`.
const ret = getStringFromWasm0(arg0, arg1);
return addHeapObject(ret);
},
__wbindgen_object_drop_ref: function(arg0) {
takeObject(arg0);
},
};
return {
__proto__: null,
"./ruvector_cnn_wasm_bg.js": import0,
};
}
const EmbedderConfigFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_embedderconfig_free(ptr >>> 0, 1));
const LayerOpsFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_layerops_free(ptr >>> 0, 1));
const SimdOpsFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_simdops_free(ptr >>> 0, 1));
const WasmCnnEmbedderFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_wasmcnnembedder_free(ptr >>> 0, 1));
const WasmInfoNCELossFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_wasminfonceloss_free(ptr >>> 0, 1));
const WasmTripletLossFinalization = (typeof FinalizationRegistry === 'undefined')
? { register: () => {}, unregister: () => {} }
: new FinalizationRegistry(ptr => wasm.__wbg_wasmtripletloss_free(ptr >>> 0, 1));
function addHeapObject(obj) {
if (heap_next === heap.length) heap.push(heap.length + 1);
const idx = heap_next;
heap_next = heap[idx];
heap[idx] = obj;
return idx;
}
function _assertClass(instance, klass) {
if (!(instance instanceof klass)) {
throw new Error(`expected instance of ${klass.name}`);
}
}
function dropObject(idx) {
if (idx < 1028) return;
heap[idx] = heap_next;
heap_next = idx;
}
function getArrayF32FromWasm0(ptr, len) {
ptr = ptr >>> 0;
return getFloat32ArrayMemory0().subarray(ptr / 4, ptr / 4 + len);
}
let cachedDataViewMemory0 = null;
function getDataViewMemory0() {
if (cachedDataViewMemory0 === null || cachedDataViewMemory0.buffer.detached === true || (cachedDataViewMemory0.buffer.detached === undefined && cachedDataViewMemory0.buffer !== wasm.memory.buffer)) {
cachedDataViewMemory0 = new DataView(wasm.memory.buffer);
}
return cachedDataViewMemory0;
}
let cachedFloat32ArrayMemory0 = null;
function getFloat32ArrayMemory0() {
if (cachedFloat32ArrayMemory0 === null || cachedFloat32ArrayMemory0.byteLength === 0) {
cachedFloat32ArrayMemory0 = new Float32Array(wasm.memory.buffer);
}
return cachedFloat32ArrayMemory0;
}
function getStringFromWasm0(ptr, len) {
ptr = ptr >>> 0;
return decodeText(ptr, len);
}
let cachedUint8ArrayMemory0 = null;
function getUint8ArrayMemory0() {
if (cachedUint8ArrayMemory0 === null || cachedUint8ArrayMemory0.byteLength === 0) {
cachedUint8ArrayMemory0 = new Uint8Array(wasm.memory.buffer);
}
return cachedUint8ArrayMemory0;
}
function getObject(idx) { return heap[idx]; }
let heap = new Array(1024).fill(undefined);
heap.push(undefined, null, true, false);
let heap_next = heap.length;
function isLikeNone(x) {
return x === undefined || x === null;
}
function passArray8ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 1, 1) >>> 0;
getUint8ArrayMemory0().set(arg, ptr / 1);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
function passArrayF32ToWasm0(arg, malloc) {
const ptr = malloc(arg.length * 4, 4) >>> 0;
getFloat32ArrayMemory0().set(arg, ptr / 4);
WASM_VECTOR_LEN = arg.length;
return ptr;
}
function passStringToWasm0(arg, malloc, realloc) {
if (realloc === undefined) {
const buf = cachedTextEncoder.encode(arg);
const ptr = malloc(buf.length, 1) >>> 0;
getUint8ArrayMemory0().subarray(ptr, ptr + buf.length).set(buf);
WASM_VECTOR_LEN = buf.length;
return ptr;
}
let len = arg.length;
let ptr = malloc(len, 1) >>> 0;
const mem = getUint8ArrayMemory0();
let offset = 0;
for (; offset < len; offset++) {
const code = arg.charCodeAt(offset);
if (code > 0x7F) break;
mem[ptr + offset] = code;
}
if (offset !== len) {
if (offset !== 0) {
arg = arg.slice(offset);
}
ptr = realloc(ptr, len, len = offset + arg.length * 3, 1) >>> 0;
const view = getUint8ArrayMemory0().subarray(ptr + offset, ptr + len);
const ret = cachedTextEncoder.encodeInto(arg, view);
offset += ret.written;
ptr = realloc(ptr, len, offset, 1) >>> 0;
}
WASM_VECTOR_LEN = offset;
return ptr;
}
function takeObject(idx) {
const ret = getObject(idx);
dropObject(idx);
return ret;
}
let cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });
cachedTextDecoder.decode();
const MAX_SAFARI_DECODE_BYTES = 2146435072;
let numBytesDecoded = 0;
function decodeText(ptr, len) {
numBytesDecoded += len;
if (numBytesDecoded >= MAX_SAFARI_DECODE_BYTES) {
cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });
cachedTextDecoder.decode();
numBytesDecoded = len;
}
return cachedTextDecoder.decode(getUint8ArrayMemory0().subarray(ptr, ptr + len));
}
const cachedTextEncoder = new TextEncoder();
if (!('encodeInto' in cachedTextEncoder)) {
cachedTextEncoder.encodeInto = function (arg, view) {
const buf = cachedTextEncoder.encode(arg);
view.set(buf);
return {
read: arg.length,
written: buf.length
};
};
}
let WASM_VECTOR_LEN = 0;
let wasmModule, wasm;
function __wbg_finalize_init(instance, module) {
wasm = instance.exports;
wasmModule = module;
cachedDataViewMemory0 = null;
cachedFloat32ArrayMemory0 = null;
cachedUint8ArrayMemory0 = null;
wasm.__wbindgen_start();
return wasm;
}
async function __wbg_load(module, imports) {
if (typeof Response === 'function' && module instanceof Response) {
if (typeof WebAssembly.instantiateStreaming === 'function') {
try {
return await WebAssembly.instantiateStreaming(module, imports);
} catch (e) {
const validResponse = module.ok && expectedResponseType(module.type);
if (validResponse && module.headers.get('Content-Type') !== 'application/wasm') {
console.warn("`WebAssembly.instantiateStreaming` failed because your server does not serve Wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n", e);
} else { throw e; }
}
}
const bytes = await module.arrayBuffer();
return await WebAssembly.instantiate(bytes, imports);
} else {
const instance = await WebAssembly.instantiate(module, imports);
if (instance instanceof WebAssembly.Instance) {
return { instance, module };
} else {
return instance;
}
}
function expectedResponseType(type) {
switch (type) {
case 'basic': case 'cors': case 'default': return true;
}
return false;
}
}
function initSync(module) {
if (wasm !== undefined) return wasm;
if (module !== undefined) {
if (Object.getPrototypeOf(module) === Object.prototype) {
({module} = module)
} else {
console.warn('using deprecated parameters for `initSync()`; pass a single object instead')
}
}
const imports = __wbg_get_imports();
if (!(module instanceof WebAssembly.Module)) {
module = new WebAssembly.Module(module);
}
const instance = new WebAssembly.Instance(module, imports);
return __wbg_finalize_init(instance, module);
}
async function __wbg_init(module_or_path) {
if (wasm !== undefined) return wasm;
if (module_or_path !== undefined) {
if (Object.getPrototypeOf(module_or_path) === Object.prototype) {
({module_or_path} = module_or_path)
} else {
console.warn('using deprecated parameters for the initialization function; pass a single object instead')
}
}
if (module_or_path === undefined) {
module_or_path = new URL('ruvector_cnn_wasm_bg.wasm', import.meta.url);
}
const imports = __wbg_get_imports();
if (typeof module_or_path === 'string' || (typeof Request === 'function' && module_or_path instanceof Request) || (typeof URL === 'function' && module_or_path instanceof URL)) {
module_or_path = fetch(module_or_path);
}
const { instance, module } = await __wbg_load(await module_or_path, imports);
return __wbg_finalize_init(instance, module);
}
export { initSync, __wbg_init as default };