mirror of
https://github.com/ruvnet/RuView
synced 2026-06-13 10:53:20 +00:00
Compare commits
3 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 377413e6a8 | |||
| b9e36a8be0 | |||
| 9e860c3a7a |
@@ -75,7 +75,7 @@ docker run -p 3000:3000 ruvnet/wifi-densepose:latest
|
||||
|----------|-------------|
|
||||
| [User Guide](docs/user-guide.md) | Step-by-step guide: installation, first run, API usage, hardware setup, training |
|
||||
| [Build Guide](docs/build-guide.md) | Building from source (Rust and Python) |
|
||||
| [Architecture Decisions](docs/adr/README.md) | 49 ADRs — why each technical choice was made, organized by domain (hardware, signal processing, ML, platform, infrastructure) |
|
||||
| [Architecture Decisions](docs/adr/README.md) | 48 ADRs — why each technical choice was made, organized by domain (hardware, signal processing, ML, platform, infrastructure) |
|
||||
| [Domain Models](docs/ddd/README.md) | 7 DDD models (RuvSense, Signal Processing, Training Pipeline, Hardware Platform, Sensing Server, WiFi-Mat, CHCI) — bounded contexts, aggregates, domain events, and ubiquitous language |
|
||||
| [Desktop App](rust-port/wifi-densepose-rs/crates/wifi-densepose-desktop/README.md) | **WIP** — Tauri v2 desktop app for node management, OTA updates, WASM deployment, and mesh visualization |
|
||||
|
||||
@@ -89,12 +89,8 @@ docker run -p 3000:3000 ruvnet/wifi-densepose:latest
|
||||
<em>Real-time pose skeleton from WiFi CSI signals — no cameras, no wearables</em>
|
||||
<br>
|
||||
<a href="https://ruvnet.github.io/RuView/"><strong>▶ Live Observatory Demo</strong></a>
|
||||
|
|
||||
<a href="https://ruvnet.github.io/RuView/pose-fusion.html"><strong>▶ Dual-Modal Pose Fusion Demo</strong></a>
|
||||
|
||||
> The [server](#-quick-start) is optional for visualization and aggregation — the ESP32 [runs independently](#esp32-s3-hardware-pipeline) for presence detection, vital signs, and fall alerts.
|
||||
>
|
||||
> **Live ESP32 pipeline**: Connect an ESP32-S3 node → run the [sensing server](#sensing-server) → open the [pose fusion demo](https://ruvnet.github.io/RuView/pose-fusion.html) for real-time dual-modal pose estimation (webcam + WiFi CSI). See [ADR-059](docs/adr/ADR-059-live-esp32-csi-pipeline.md).
|
||||
|
||||
|
||||
## 🚀 Key Features
|
||||
@@ -1047,16 +1043,14 @@ Download a pre-built binary — no build toolchain needed:
|
||||
|
||||
| Release | What's included | Tag |
|
||||
|---------|-----------------|-----|
|
||||
| [v0.4.1](https://github.com/ruvnet/RuView/releases/tag/v0.4.1-esp32) | **Stable** — CSI build fix, compile guard, AMOLED display, edge intelligence ([ADR-057](docs/adr/ADR-057-firmware-csi-build-guard.md)) | `v0.4.1-esp32` |
|
||||
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Stable — raw CSI streaming, multi-node TDM, channel hopping | `v0.2.0-esp32` |
|
||||
| [v0.3.0-alpha](https://github.com/ruvnet/RuView/releases/tag/v0.3.0-alpha-esp32) | Alpha — adds on-device edge intelligence and WASM modules ([ADR-039](docs/adr/ADR-039-esp32-edge-intelligence.md), [ADR-040](docs/adr/ADR-040-wasm-programmable-sensing.md)) | `v0.3.0-alpha-esp32` |
|
||||
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Raw CSI streaming, multi-node TDM, channel hopping | `v0.2.0-esp32` |
|
||||
|
||||
```bash
|
||||
# 1. Flash the firmware to your ESP32-S3
|
||||
python -m esptool --chip esp32s3 --port COM7 --baud 460800 \
|
||||
write_flash --flash-mode dio --flash-size 8MB --flash-freq 80m \
|
||||
0x0 bootloader.bin 0x8000 partition-table.bin \
|
||||
0xf000 ota_data_initial.bin 0x20000 esp32-csi-node.bin
|
||||
write_flash --flash_mode dio --flash_size 8MB \
|
||||
0x0 bootloader.bin 0x8000 partition-table.bin 0x10000 esp32-csi-node.bin
|
||||
|
||||
# 2. Set WiFi credentials and server address (stored in flash, survives reboots)
|
||||
python firmware/esp32-csi-node/provision.py --port COM7 \
|
||||
|
||||
@@ -0,0 +1,240 @@
|
||||
# ADR-057: Desktop App Training & RuVector Integration
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Status | Proposed |
|
||||
| Date | 2026-03-10 |
|
||||
| Authors | RuView Team |
|
||||
| Reviewers | - |
|
||||
| Related | ADR-016, ADR-017, ADR-024, ADR-027 |
|
||||
|
||||
## Context
|
||||
|
||||
The RuView desktop application currently provides device discovery, firmware flashing, OTA updates, and real-time sensing visualization. However, users cannot train models or configure RuVector signal processing modules directly from the desktop app.
|
||||
|
||||
The following crates exist in the workspace but are not exposed in the desktop UI:
|
||||
|
||||
### Training Crate (`wifi-densepose-train`)
|
||||
- Dataset management (MM-Fi, Wi-Pose formats)
|
||||
- Model architectures (CSI encoder, pose decoder)
|
||||
- Training loops with metrics tracking
|
||||
- Checkpoint save/load
|
||||
- ruview_metrics integration
|
||||
|
||||
### RuVector Crates (5 modules)
|
||||
1. **ruvector-mincut** - Graph-based person segmentation, DynamicPersonMatcher
|
||||
2. **ruvector-attn-mincut** - Attention-weighted antenna selection
|
||||
3. **ruvector-temporal-tensor** - Temporal CSI compression, breathing detection
|
||||
4. **ruvector-solver** - Sparse interpolation, triangulation
|
||||
5. **ruvector-attention** - Spatial attention, BVP extraction
|
||||
|
||||
## Decision
|
||||
|
||||
Add a new **"Training"** page to the desktop application with tabbed navigation:
|
||||
|
||||
### Tab Structure
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Training & Models │
|
||||
├──────────┬──────────┬──────────┬──────────┬────────────────┤
|
||||
│ Datasets │ Models │ Training │ RuVector │ Metrics │
|
||||
└──────────┴──────────┴──────────┴──────────┴────────────────┘
|
||||
```
|
||||
|
||||
### Tab 1: Datasets
|
||||
- **Download** standard datasets (MM-Fi, Wi-Pose)
|
||||
- **Import** custom CSI recordings
|
||||
- **Preview** dataset samples (CSI heatmaps, labels)
|
||||
- **Split** into train/val/test sets
|
||||
- **Statistics** - sample counts, class distribution
|
||||
|
||||
### Tab 2: Models
|
||||
- **Browse** available architectures:
|
||||
- CSI Encoder (CNN, Transformer)
|
||||
- Pose Decoder (LSTM, GRU)
|
||||
- AETHER embedding network (ADR-024)
|
||||
- MERIDIAN domain adaptor (ADR-027)
|
||||
- **Load** checkpoints from disk
|
||||
- **View** model summary (params, layers, memory)
|
||||
- **Export** to ONNX/TorchScript
|
||||
|
||||
### Tab 3: Training
|
||||
- **Configure** training:
|
||||
- Learning rate, batch size, epochs
|
||||
- Optimizer (Adam, SGD, AdamW)
|
||||
- Loss function selection
|
||||
- Data augmentation toggles
|
||||
- **GPU Detection** - CUDA/Metal availability
|
||||
- **Start/Stop** training jobs
|
||||
- **Progress** - live loss curves, ETA
|
||||
- **Checkpointing** - auto-save best model
|
||||
|
||||
### Tab 4: RuVector
|
||||
- **Module Configuration**:
|
||||
- MinCut graph parameters
|
||||
- Attention weights
|
||||
- Temporal compression ratio
|
||||
- Solver interpolation settings
|
||||
- **Live Testing** - apply to real-time CSI stream
|
||||
- **Comparison** - A/B test configurations
|
||||
- **Export** - save optimal config
|
||||
|
||||
### Tab 5: Metrics
|
||||
- **Loss Curves** - training/validation over epochs
|
||||
- **Evaluation** - PCK, mAP, IoU scores
|
||||
- **Confusion Matrix** - per-joint accuracy
|
||||
- **Export** - CSV, JSON, TensorBoard format
|
||||
|
||||
## Architecture
|
||||
|
||||
### Backend (Rust/Tauri)
|
||||
|
||||
```
|
||||
wifi-densepose-desktop/
|
||||
├── src/
|
||||
│ ├── commands/
|
||||
│ │ ├── training.rs # NEW: Training job management
|
||||
│ │ ├── datasets.rs # NEW: Dataset download/import
|
||||
│ │ ├── models.rs # NEW: Model loading/export
|
||||
│ │ ├── ruvector.rs # NEW: RuVector config
|
||||
│ │ └── metrics.rs # NEW: Metrics retrieval
|
||||
│ └── domain/
|
||||
│ ├── training.rs # Training state machine
|
||||
│ └── ruvector.rs # RuVector config types
|
||||
```
|
||||
|
||||
### Frontend (React/TypeScript)
|
||||
|
||||
```
|
||||
ui/src/pages/
|
||||
├── Training/
|
||||
│ ├── index.tsx # Tab container
|
||||
│ ├── DatasetsTab.tsx # Dataset management
|
||||
│ ├── ModelsTab.tsx # Model browser
|
||||
│ ├── TrainingTab.tsx # Training control
|
||||
│ ├── RuVectorTab.tsx # Signal processing config
|
||||
│ └── MetricsTab.tsx # Visualization
|
||||
```
|
||||
|
||||
### Tauri Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `list_datasets` | Get available datasets |
|
||||
| `download_dataset` | Download standard dataset |
|
||||
| `import_dataset` | Import custom recordings |
|
||||
| `list_models` | Get model architectures |
|
||||
| `load_checkpoint` | Load model weights |
|
||||
| `export_model` | Export to ONNX |
|
||||
| `detect_gpu` | Check CUDA/Metal |
|
||||
| `start_training` | Begin training job |
|
||||
| `stop_training` | Cancel training |
|
||||
| `training_progress` | Get current status |
|
||||
| `get_ruvector_config` | Load RuVector settings |
|
||||
| `set_ruvector_config` | Update settings |
|
||||
| `test_ruvector_live` | Apply to live CSI |
|
||||
| `get_metrics` | Retrieve training metrics |
|
||||
|
||||
### Event System
|
||||
|
||||
Training progress updates via Tauri events:
|
||||
|
||||
```rust
|
||||
#[derive(Serialize, Clone)]
|
||||
pub struct TrainingProgress {
|
||||
pub epoch: u32,
|
||||
pub total_epochs: u32,
|
||||
pub batch: u32,
|
||||
pub total_batches: u32,
|
||||
pub train_loss: f32,
|
||||
pub val_loss: Option<f32>,
|
||||
pub learning_rate: f32,
|
||||
pub eta_secs: u64,
|
||||
pub gpu_memory_mb: Option<u64>,
|
||||
}
|
||||
|
||||
// Emit every batch
|
||||
app.emit("training:progress", progress)?;
|
||||
|
||||
// Emit on completion
|
||||
app.emit("training:complete", result)?;
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Foundation (Week 1-2)
|
||||
1. Create `Training` page skeleton with tabs
|
||||
2. Implement `detect_gpu` command
|
||||
3. Add dataset listing/download commands
|
||||
4. Design TypeScript types for all entities
|
||||
|
||||
### Phase 2: Dataset Management (Week 3)
|
||||
1. MM-Fi dataset downloader
|
||||
2. Wi-Pose dataset downloader
|
||||
3. Custom dataset import (CSV/NPZ)
|
||||
4. Dataset preview component
|
||||
|
||||
### Phase 3: Model Management (Week 4)
|
||||
1. Model architecture browser
|
||||
2. Checkpoint loading
|
||||
3. Model summary display
|
||||
4. ONNX export
|
||||
|
||||
### Phase 4: Training Loop (Week 5-6)
|
||||
1. Training configuration UI
|
||||
2. Background training thread
|
||||
3. Progress event emission
|
||||
4. Checkpoint auto-save
|
||||
5. Training history persistence
|
||||
|
||||
### Phase 5: RuVector Integration (Week 7)
|
||||
1. RuVector config UI
|
||||
2. Live CSI testing
|
||||
3. A/B comparison mode
|
||||
4. Config export/import
|
||||
|
||||
### Phase 6: Metrics & Polish (Week 8)
|
||||
1. Loss curve visualization (Chart.js/Recharts)
|
||||
2. Evaluation metrics display
|
||||
3. Export functionality
|
||||
4. Error handling & edge cases
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Probability | Impact | Mitigation |
|
||||
|------|-------------|--------|------------|
|
||||
| No GPU available | Medium | High | CPU fallback with warning |
|
||||
| Large dataset downloads | High | Medium | Resume support, progress UI |
|
||||
| Training crashes | Medium | High | Checkpoint recovery, error reporting |
|
||||
| Memory exhaustion | Low | High | Batch size auto-tuning |
|
||||
| UI blocking | Medium | High | All training in background thread |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. User can download MM-Fi dataset from UI
|
||||
2. User can start training with GPU detection
|
||||
3. Live progress updates without UI freeze
|
||||
4. Training can be paused/resumed
|
||||
5. RuVector config changes apply to live CSI
|
||||
6. Metrics display updates in real-time
|
||||
7. Models can be exported to ONNX
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### 1. Separate Training App
|
||||
- **Rejected**: Fragments user experience, duplicates code
|
||||
|
||||
### 2. Web-based Training Dashboard
|
||||
- **Rejected**: Requires server, no offline support
|
||||
|
||||
### 3. CLI-only Training
|
||||
- **Rejected**: Poor UX for non-technical users
|
||||
|
||||
## References
|
||||
|
||||
- ADR-016: RuVector Training Pipeline Integration
|
||||
- ADR-017: RuVector Signal + MAT Integration
|
||||
- ADR-024: AETHER Contrastive CSI Embedding
|
||||
- ADR-027: MERIDIAN Domain Generalization
|
||||
- Tauri v2 Events: https://v2.tauri.app/develop/calling-rust/#events
|
||||
@@ -1,82 +0,0 @@
|
||||
# ADR-057: Firmware CSI Build Guard and sdkconfig.defaults
|
||||
|
||||
| Field | Value |
|
||||
|-------------|---------------------------------------------|
|
||||
| **Status** | Accepted |
|
||||
| **Date** | 2026-03-12 |
|
||||
| **Authors** | ruv |
|
||||
| **Issues** | #223, #238, #234, #210, #190 |
|
||||
|
||||
## Context
|
||||
|
||||
Multiple GitHub issues (#223, #238, #234, #210, #190) report firmware problems
|
||||
that fall into two categories:
|
||||
|
||||
1. **CSI not enabled at runtime** — The committed `sdkconfig` had
|
||||
`# CONFIG_ESP_WIFI_CSI_ENABLED is not set` (line 1135), meaning users who
|
||||
built from source or used pre-built binaries got the runtime error:
|
||||
`E (6700) wifi:CSI not enabled in menuconfig!`
|
||||
|
||||
Root cause: `sdkconfig.defaults.template` existed with the correct setting
|
||||
(`CONFIG_ESP_WIFI_CSI_ENABLED=y`) but ESP-IDF only reads
|
||||
`sdkconfig.defaults` — not `.template` suffixed files. No `sdkconfig.defaults`
|
||||
file was committed.
|
||||
|
||||
2. **Unsupported ESP32 variants** — Users attempting to use original ESP32
|
||||
(D0WD) and ESP32-C3 boards. The firmware targets ESP32-S3 only
|
||||
(`CONFIG_IDF_TARGET="esp32s3"`, Xtensa architecture) and this was not
|
||||
surfaced clearly enough in documentation or build errors.
|
||||
|
||||
## Decision
|
||||
|
||||
### Fix 1: Commit `sdkconfig.defaults` (not just template)
|
||||
|
||||
Copy `sdkconfig.defaults.template` → `sdkconfig.defaults` so that ESP-IDF
|
||||
applies the correct defaults (including `CONFIG_ESP_WIFI_CSI_ENABLED=y`)
|
||||
automatically when `sdkconfig` is regenerated.
|
||||
|
||||
### Fix 2: `#error` compile-time guard in `csi_collector.c`
|
||||
|
||||
Add a preprocessor guard:
|
||||
|
||||
```c
|
||||
#ifndef CONFIG_ESP_WIFI_CSI_ENABLED
|
||||
#error "CONFIG_ESP_WIFI_CSI_ENABLED must be set in sdkconfig."
|
||||
#endif
|
||||
```
|
||||
|
||||
This turns a confusing runtime crash into a clear compile-time error with
|
||||
instructions on how to fix it.
|
||||
|
||||
### Fix 3: Fix committed `sdkconfig`
|
||||
|
||||
Change line 1135 from `# CONFIG_ESP_WIFI_CSI_ENABLED is not set` to
|
||||
`CONFIG_ESP_WIFI_CSI_ENABLED=y`.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Positive**: New builds will always have CSI enabled. Users building from
|
||||
source will get a clear compile error if CSI is somehow disabled.
|
||||
- **Positive**: Pre-built release binaries will include CSI support.
|
||||
- **Neutral**: Original ESP32 and ESP32-C3 remain unsupported. This is by
|
||||
design — only ESP32-S3 has the CSI API surface we depend on. Future ADRs
|
||||
may address multi-target support if demand warrants it.
|
||||
- **Negative**: None identified.
|
||||
|
||||
## Hardware Support Matrix
|
||||
|
||||
| Variant | CSI Support | Firmware Target | Status |
|
||||
|--------------|-------------|-----------------|---------------|
|
||||
| ESP32-S3 | Yes | Yes | Supported |
|
||||
| ESP32 (orig) | Partial | No | Unsupported |
|
||||
| ESP32-C3 | Yes (IDF 5.1+) | No | Unsupported |
|
||||
| ESP32-C6 | Yes | No | Unsupported |
|
||||
|
||||
## Notes
|
||||
|
||||
- ESP32-C3 and C6 use RISC-V architecture; a separate build target
|
||||
(`idf.py set-target esp32c3`) would be needed.
|
||||
- Original ESP32 has limited CSI (no STBC HT-LTF2, fewer subcarriers).
|
||||
- Users on unsupported hardware can still write custom firmware using the
|
||||
ADR-018 binary frame format (magic `0xC5110001`) for interop with the
|
||||
Rust aggregator.
|
||||
@@ -1,392 +0,0 @@
|
||||
# ADR-058: Dual-Modal WASM Browser Pose Estimation — Live Video + WiFi CSI Fusion
|
||||
|
||||
- **Status**: Proposed
|
||||
- **Date**: 2026-03-12
|
||||
- **Deciders**: ruv
|
||||
- **Tags**: wasm, browser, cnn, pose-estimation, ruvector, video, multimodal, fusion
|
||||
|
||||
## Context
|
||||
|
||||
WiFi-DensePose estimates human poses from WiFi CSI (Channel State Information).
|
||||
The `ruvector-cnn` crate provides a pure Rust CNN (MobileNet-V3) with WASM bindings.
|
||||
Both modalities exist independently — what's missing is **fusing live webcam video
|
||||
with WiFi CSI** in a single browser demo to achieve robust pose estimation that
|
||||
works even when one modality degrades (occlusion, signal noise, poor lighting).
|
||||
|
||||
Existing assets:
|
||||
|
||||
1. **`wifi-densepose-wasm`** — CSI signal processing compiled to WASM
|
||||
2. **`wifi-densepose-sensing-server`** — Axum server streaming live CSI via WebSocket
|
||||
3. **`ruvector-cnn`** — Pure Rust CNN with MobileNet-V3 backbones, SIMD, contrastive learning
|
||||
4. **`ruvector-cnn-wasm`** — wasm-bindgen bindings: `WasmCnnEmbedder`, `SimdOps`, `LayerOps`, contrastive losses
|
||||
5. **`vendor/ruvector/examples/wasm-vanilla/`** — Reference vanilla JS WASM example
|
||||
|
||||
Research shows multi-modal fusion (camera + WiFi) significantly outperforms either alone:
|
||||
- Camera fails under occlusion, poor lighting, privacy constraints
|
||||
- WiFi CSI fails with signal noise, multipath, low spatial resolution
|
||||
- Fusion compensates: WiFi provides through-wall coverage, camera provides fine-grained detail
|
||||
|
||||
## Decision
|
||||
|
||||
Build a **dual-modal browser demo** at `examples/wasm-browser-pose/` that:
|
||||
|
||||
1. Captures **live webcam video** via `getUserMedia` API
|
||||
2. Receives **live WiFi CSI** via WebSocket from the sensing server
|
||||
3. Processes **both streams** through separate CNN pipelines in `ruvector-cnn-wasm`
|
||||
4. **Fuses embeddings** with learned attention weights for combined pose estimation
|
||||
5. Renders **video overlay** with skeleton + WiFi confidence heatmap on Canvas
|
||||
6. Runs entirely in the browser — all inference client-side via WASM
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ Browser │
|
||||
│ │
|
||||
│ ┌────────────┐ ┌────────────────┐ ┌───────────────────┐ │
|
||||
│ │ getUserMedia│───▶│ Video Frame │───▶│ CNN WASM │ │
|
||||
│ │ (Webcam) │ │ Capture │ │ (Visual Embedder) │ │
|
||||
│ └────────────┘ │ 224×224 RGB │ │ → 512-dim │ │
|
||||
│ └────────────────┘ └────────┬──────────┘ │
|
||||
│ │ │
|
||||
│ visual_embedding │
|
||||
│ │ │
|
||||
│ ┌──────▼──────┐ │
|
||||
│ ┌────────────┐ ┌────────────────┐ │ │ │
|
||||
│ │ WebSocket │───▶│ CSI WASM │ │ Attention │ │
|
||||
│ │ Client │ │ (densepose- │ │ Fusion │ │
|
||||
│ │ │ │ wasm) │ │ Module │ │
|
||||
│ └────────────┘ └───────┬────────┘ │ │ │
|
||||
│ │ └──────┬──────┘ │
|
||||
│ ┌───────▼────────┐ │ │
|
||||
│ │ CNN WASM │ fused_embedding │
|
||||
│ │ (CSI Embedder) │ │ │
|
||||
│ │ → 512-dim │ ┌──────▼──────┐ │
|
||||
│ └───────┬────────┘ │ Pose │ │
|
||||
│ │ │ Decoder │ │
|
||||
│ csi_embedding │ → 17 kpts │ │
|
||||
│ │ └──────┬──────┘ │
|
||||
│ └──────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────┐ ┌─────▼──────┐ │
|
||||
│ │ Video Canvas │◀────────│ Overlay │ │
|
||||
│ │ + Skeleton │ │ Renderer │ │
|
||||
│ │ + Heatmap │ └────────────┘ │
|
||||
│ └──────────────┘ │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
▲ ▲
|
||||
│ getUserMedia │ WebSocket
|
||||
│ (camera) │ (ws://host:3030/ws/csi)
|
||||
│ │
|
||||
┌────┴────┐ ┌───────┴─────────┐
|
||||
│ Webcam │ │ Sensing Server │
|
||||
└─────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
### Dual Pipeline Design
|
||||
|
||||
Two parallel CNN pipelines run on each frame tick (~30 FPS):
|
||||
|
||||
| Pipeline | Input | Preprocessing | CNN Config | Output |
|
||||
|----------|-------|---------------|------------|--------|
|
||||
| **Visual** | Webcam frame (640×480) | Resize to 224×224 RGB, ImageNet normalize | MobileNet-V3 Small, 512-dim | Visual embedding |
|
||||
| **CSI** | CSI frame (ADR-018 binary) | Amplitude/phase/delta → 224×224 pseudo-RGB | MobileNet-V3 Small, 512-dim | CSI embedding |
|
||||
|
||||
Both use the same `WasmCnnEmbedder` but with separate instances and weight sets.
|
||||
|
||||
### Fusion Strategy
|
||||
|
||||
**Learned attention-weighted fusion** combines the two 512-dim embeddings:
|
||||
|
||||
```javascript
|
||||
// Attention fusion: learn which modality to trust per-dimension
|
||||
// α ∈ [0,1]^512 — attention weights (shipped as JSON, trained offline)
|
||||
// visual_emb, csi_emb ∈ R^512
|
||||
|
||||
function fuseEmbeddings(visual_emb, csi_emb, attention_weights) {
|
||||
const fused = new Float32Array(512);
|
||||
for (let i = 0; i < 512; i++) {
|
||||
const α = attention_weights[i];
|
||||
fused[i] = α * visual_emb[i] + (1 - α) * csi_emb[i];
|
||||
}
|
||||
return fused;
|
||||
}
|
||||
```
|
||||
|
||||
**Dynamic confidence gating** adjusts fusion based on signal quality:
|
||||
|
||||
| Condition | Behavior |
|
||||
|-----------|----------|
|
||||
| Good video + good CSI | Balanced fusion (α ≈ 0.5) |
|
||||
| Poor lighting / occlusion | CSI-dominant (α → 0, WiFi takes over) |
|
||||
| CSI noise / no ESP32 | Video-dominant (α → 1, camera only) |
|
||||
| Video-only mode (no WiFi) | α = 1.0, pure visual CNN pose estimation |
|
||||
| CSI-only mode (no camera) | α = 0.0, pure WiFi pose estimation |
|
||||
|
||||
Quality detection:
|
||||
- **Video quality**: Frame brightness variance (dark = low quality), motion blur score
|
||||
- **CSI quality**: Signal-to-noise ratio from `wifi-densepose-wasm`, coherence gate output
|
||||
|
||||
### CSI-to-Image Encoding
|
||||
|
||||
CSI data encoded as 3-channel pseudo-image for the CSI CNN pipeline:
|
||||
|
||||
| Channel | Data | Normalization |
|
||||
|---------|------|---------------|
|
||||
| R | CSI amplitude (subcarrier × time window) | Min-max to [0, 255] |
|
||||
| G | CSI phase (unwrapped, subcarrier × time window) | Min-max to [0, 255] |
|
||||
| B | Temporal difference (frame-to-frame Δ amplitude) | Abs, min-max to [0, 255] |
|
||||
|
||||
### Video Processing
|
||||
|
||||
Webcam frames processed through standard ImageNet pipeline:
|
||||
|
||||
```javascript
|
||||
// Capture frame from video element
|
||||
const frame = captureVideoFrame(videoElement, 224, 224); // Returns Uint8Array RGB
|
||||
|
||||
// ImageNet normalization happens inside WasmCnnEmbedder.extract()
|
||||
const visual_embedding = visual_embedder.extract(frame, 224, 224);
|
||||
```
|
||||
|
||||
### Pose Keypoint Mapping
|
||||
|
||||
17 COCO-format keypoints decoded from the fused 512-dim embedding:
|
||||
|
||||
```
|
||||
0: nose 1: left_eye 2: right_eye
|
||||
3: left_ear 4: right_ear 5: left_shoulder
|
||||
6: right_shoulder 7: left_elbow 8: right_elbow
|
||||
9: left_wrist 10: right_wrist 11: left_hip
|
||||
12: right_hip 13: left_knee 14: right_knee
|
||||
15: left_ankle 16: right_ankle
|
||||
```
|
||||
|
||||
Each keypoint decoded as (x, y, confidence) = 51 values from the 512-dim embedding
|
||||
via a learned linear projection.
|
||||
|
||||
### Operating Modes
|
||||
|
||||
The demo supports three modes, selectable in the UI:
|
||||
|
||||
| Mode | Video | CSI | Fusion | Use Case |
|
||||
|------|-------|-----|--------|----------|
|
||||
| **Dual (default)** | ✅ | ✅ | Attention-weighted | Best accuracy, full demo |
|
||||
| **Video Only** | ✅ | ❌ | α = 1.0 | No ESP32 available, quick demo |
|
||||
| **CSI Only** | ❌ | ✅ | α = 0.0 | Privacy mode, through-wall sensing |
|
||||
|
||||
**Video Only mode works without any hardware** — just a webcam — making the demo
|
||||
instantly accessible for anyone wanting to try it.
|
||||
|
||||
### File Layout
|
||||
|
||||
```
|
||||
examples/wasm-browser-pose/
|
||||
├── index.html # Single-page app (vanilla JS, no bundler)
|
||||
├── js/
|
||||
│ ├── app.js # Main entry, mode selection, orchestration
|
||||
│ ├── video-capture.js # getUserMedia, frame extraction, quality detection
|
||||
│ ├── csi-processor.js # WebSocket CSI client, frame parsing, pseudo-image encoding
|
||||
│ ├── fusion.js # Attention-weighted embedding fusion, confidence gating
|
||||
│ ├── pose-decoder.js # Fused embedding → 17 keypoints
|
||||
│ └── canvas-renderer.js # Video overlay, skeleton, CSI heatmap, confidence bars
|
||||
├── data/
|
||||
│ ├── visual-weights.json # Visual CNN → embedding projection (placeholder until trained)
|
||||
│ ├── csi-weights.json # CSI CNN → embedding projection (placeholder until trained)
|
||||
│ ├── fusion-weights.json # Attention fusion α weights (512 values)
|
||||
│ └── pose-weights.json # Fused embedding → keypoint projection
|
||||
├── css/
|
||||
│ └── style.css # Dark theme UI styling
|
||||
├── pkg/ # Built WASM packages (gitignored, built by script)
|
||||
│ ├── wifi_densepose_wasm/
|
||||
│ └── ruvector_cnn_wasm/
|
||||
├── build.sh # wasm-pack build script for both packages
|
||||
└── README.md # Setup and usage instructions
|
||||
```
|
||||
|
||||
### Build Pipeline
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# build.sh — builds both WASM packages into pkg/
|
||||
|
||||
set -e
|
||||
|
||||
# Build wifi-densepose-wasm (CSI processing)
|
||||
wasm-pack build ../../rust-port/wifi-densepose-rs/crates/wifi-densepose-wasm \
|
||||
--target web --out-dir "$(pwd)/pkg/wifi_densepose_wasm" --no-typescript
|
||||
|
||||
# Build ruvector-cnn-wasm (CNN inference for both video and CSI)
|
||||
wasm-pack build ../../vendor/ruvector/crates/ruvector-cnn-wasm \
|
||||
--target web --out-dir "$(pwd)/pkg/ruvector_cnn_wasm" --no-typescript
|
||||
|
||||
echo "Build complete. Serve with: python3 -m http.server 8080"
|
||||
```
|
||||
|
||||
### UI Layout
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ WiFi-DensePose — Live Dual-Modal Pose Estimation │
|
||||
│ [Dual Mode ▼] [⚙ Settings] FPS: 28 ◉ Live │
|
||||
├───────────────────────────┬─────────────────────────────┤
|
||||
│ │ │
|
||||
│ ┌───────────────────┐ │ ┌───────────────────┐ │
|
||||
│ │ │ │ │ │ │
|
||||
│ │ Video + Skeleton │ │ │ CSI Heatmap │ │
|
||||
│ │ Overlay │ │ │ (amplitude × │ │
|
||||
│ │ (main canvas) │ │ │ subcarrier) │ │
|
||||
│ │ │ │ │ │ │
|
||||
│ └───────────────────┘ │ └───────────────────┘ │
|
||||
│ │ │
|
||||
├───────────────────────────┴─────────────────────────────┤
|
||||
│ Fusion Confidence: ████████░░ 78% │
|
||||
│ Video: ██████████ 95% │ CSI: ██████░░░░ 61% │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ ┌─────────────────────────────────────────────────┐ │
|
||||
│ │ Embedding Space (2D projection) │ │
|
||||
│ │ · · · │ │
|
||||
│ │ · · · · · · (color = pose cluster) │ │
|
||||
│ │ · · · · │ │
|
||||
│ └─────────────────────────────────────────────────┘ │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ Latency: Video 12ms │ CSI 8ms │ Fusion 1ms │ Total 21ms│
|
||||
│ [▶ Record] [📷 Snapshot] [Confidence: ████ 0.6] │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### WASM Module Structure
|
||||
|
||||
| Package | Source Crate | Provides | Size (est.) |
|
||||
|---------|-------------|----------|-------------|
|
||||
| `wifi_densepose_wasm` | `wifi-densepose-wasm` | CSI frame parsing, signal processing, feature extraction | ~200KB |
|
||||
| `ruvector_cnn_wasm` | `ruvector-cnn-wasm` | `WasmCnnEmbedder` (×2 instances), `SimdOps`, `LayerOps`, contrastive losses | ~150KB |
|
||||
|
||||
Two `WasmCnnEmbedder` instances are created — one for video frames, one for CSI pseudo-images.
|
||||
They share the same WASM module but have independent state.
|
||||
|
||||
### Browser API Requirements
|
||||
|
||||
| API | Purpose | Required | Fallback |
|
||||
|-----|---------|----------|----------|
|
||||
| `getUserMedia` | Webcam capture | For video mode | CSI-only mode |
|
||||
| WebAssembly | CNN inference | Yes | None (hard requirement) |
|
||||
| WASM SIMD128 | Accelerated inference | No | Scalar fallback (~2× slower) |
|
||||
| WebSocket | CSI data stream | For CSI mode | Video-only mode |
|
||||
| Canvas 2D | Rendering | Yes | None |
|
||||
| `requestAnimationFrame` | Render loop | Yes | `setTimeout` fallback |
|
||||
| ES Modules | Code organization | Yes | None |
|
||||
|
||||
Target: Chrome 89+, Firefox 89+, Safari 15+, Edge 89+
|
||||
|
||||
### Performance Budget
|
||||
|
||||
| Stage | Target Latency | Notes |
|
||||
|-------|---------------|-------|
|
||||
| Video frame capture + resize | <3ms | `drawImage` to offscreen canvas |
|
||||
| Video CNN embedding | <15ms | 224×224 RGB → 512-dim |
|
||||
| CSI receive + parse | <2ms | Binary WebSocket message |
|
||||
| CSI pseudo-image encoding | <3ms | Amplitude/phase/delta channels |
|
||||
| CSI CNN embedding | <15ms | 224×224 pseudo-RGB → 512-dim |
|
||||
| Attention fusion | <1ms | Element-wise weighted sum |
|
||||
| Pose decoding | <1ms | Linear projection |
|
||||
| Canvas overlay render | <3ms | Video + skeleton + heatmap |
|
||||
| **Total (dual mode)** | **<33ms** | **30 FPS capable** |
|
||||
| **Total (video only)** | **<22ms** | **45 FPS capable** |
|
||||
|
||||
Note: Video and CSI CNN pipelines can run in parallel using Web Workers,
|
||||
reducing dual-mode latency to ~max(15, 15) + 5 = ~20ms (50 FPS).
|
||||
|
||||
### Contrastive Learning Integration
|
||||
|
||||
The demo optionally shows real-time contrastive learning in the browser:
|
||||
|
||||
- **InfoNCE loss** (`WasmInfoNCELoss`): Compare video vs CSI embeddings for the same pose — trains cross-modal alignment
|
||||
- **Triplet loss** (`WasmTripletLoss`): Push apart different poses, pull together same pose across modalities
|
||||
- **SimdOps**: Accelerated dot products for real-time similarity computation
|
||||
- **Embedding space panel**: Live 2D projection shows video and CSI embeddings converging when viewing the same person
|
||||
|
||||
### Relationship to Existing Crates
|
||||
|
||||
| Existing Crate | Role in This Demo |
|
||||
|---------------|-------------------|
|
||||
| `ruvector-cnn-wasm` | CNN inference for **both** video frames and CSI pseudo-images |
|
||||
| `wifi-densepose-wasm` | CSI frame parsing and signal processing |
|
||||
| `wifi-densepose-sensing-server` | WebSocket CSI data source |
|
||||
| `wifi-densepose-core` | ADR-018 frame format definitions |
|
||||
| `ruvector-cnn` | Underlying MobileNet-V3, layers, contrastive learning |
|
||||
|
||||
No new Rust crates are needed. The example is pure HTML/JS consuming existing WASM packages.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- **Instant demo**: Video-only mode works with just a webcam — no ESP32 needed
|
||||
- **Multi-modal showcase**: Demonstrates camera + WiFi fusion, the core innovation of the project
|
||||
- **Graceful degradation**: Works with video-only, CSI-only, or both
|
||||
- **Through-wall capability**: CSI mode shows pose estimation where cameras cannot reach
|
||||
- **Zero-install**: Anyone with a browser can try it
|
||||
- **Training data collection**: Can record paired (video, CSI) data for offline model training
|
||||
- **Reusable**: JS modules embed directly in the Tauri desktop app's webview
|
||||
|
||||
### Negative
|
||||
|
||||
- **Model weights**: Requires offline-trained weights for visual CNN, CSI CNN, fusion, and pose decoder (~200KB total JSON)
|
||||
- **WASM size**: Two WASM modules total ~350KB (acceptable)
|
||||
- **No GPU**: CPU-only WASM inference; adequate at 224×224 but limits resolution scaling
|
||||
- **Camera privacy**: Video mode requires camera permission (mitigated: CSI-only mode available)
|
||||
- **Two CNN instances**: Memory footprint doubles vs single-modal (~10MB total, acceptable for desktop browsers)
|
||||
|
||||
### Risks
|
||||
|
||||
- **Cross-modal alignment**: Video and CSI embeddings must be trained jointly for fusion to work;
|
||||
without proper training, fusion may be worse than either modality alone
|
||||
- **Latency on mobile**: Dual CNN on mobile browsers may exceed 33ms; implement automatic quality reduction
|
||||
- **WebSocket drops**: Network jitter → CSI frame gaps; buffer last 3 frames, interpolate missing data
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
1. **Phase 1 — Scaffold**: File layout, build.sh, index.html shell, mode selector UI
|
||||
2. **Phase 2 — Video pipeline**: getUserMedia → frame capture → CNN embedding → basic pose display
|
||||
3. **Phase 3 — CSI pipeline**: WebSocket client → CSI parsing → pseudo-image → CNN embedding
|
||||
4. **Phase 4 — Fusion**: Attention-weighted combination, confidence gating, mode switching
|
||||
5. **Phase 5 — Pose decoder**: Linear projection with placeholder weights → 17 keypoints
|
||||
6. **Phase 6 — Overlay renderer**: Video canvas with skeleton overlay, CSI heatmap panel
|
||||
7. **Phase 7 — Training**: Use `wifi-densepose-train` to generate real weights for both CNNs + fusion + decoder
|
||||
8. **Phase 8 — Contrastive demo**: Embedding space visualization, cross-modal similarity display
|
||||
9. **Phase 9 — Web Workers**: Move CNN inference to workers for parallel video + CSI processing
|
||||
10. **Phase 10 — Polish**: Recording, snapshots, adaptive quality, mobile optimization
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### 1. CSI-Only (No Video)
|
||||
Rejected: Misses the opportunity to show multi-modal fusion and makes the demo less
|
||||
accessible (requires ESP32 hardware). Video-only mode as a fallback is strictly better.
|
||||
|
||||
### 2. Server-Side Video Inference
|
||||
Rejected: Adds latency, requires webcam stream upload (privacy concern), and defeats
|
||||
the WASM-first architecture. All inference must be client-side.
|
||||
|
||||
### 3. TensorFlow.js for Video, ruvector-cnn-wasm for CSI
|
||||
Rejected: Would require two different ML frameworks. Using `ruvector-cnn-wasm` for both
|
||||
keeps a single WASM module, unified embedding space, and simpler fusion.
|
||||
|
||||
### 4. Pre-recorded Video Demo
|
||||
Rejected: Live webcam input is far more compelling for demonstrations.
|
||||
Pre-recorded mode can be added as a secondary option.
|
||||
|
||||
### 5. React/Vue Framework
|
||||
Rejected: Adds build tooling. Vanilla JS + ES modules keeps the demo self-contained.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR-018: Binary CSI Frame Format](ADR-018-binary-csi-frame-format.md)
|
||||
- [ADR-024: Contrastive CSI Embedding / AETHER](ADR-024-contrastive-csi-embedding.md)
|
||||
- [ADR-055: Integrated Sensing Server](ADR-055-integrated-sensing-server.md)
|
||||
- `vendor/ruvector/crates/ruvector-cnn/src/lib.rs` — CNN embedder implementation
|
||||
- `vendor/ruvector/crates/ruvector-cnn-wasm/src/lib.rs` — WASM bindings
|
||||
- `vendor/ruvector/examples/wasm-vanilla/index.html` — Reference vanilla JS WASM pattern
|
||||
- Person-in-WiFi: Fine-grained Person Perception using WiFi (ICCV 2019) — camera+WiFi fusion precedent
|
||||
- WiPose: Multi-Person WiFi Pose Estimation (TMC 2022) — cross-modal embedding approach
|
||||
@@ -1,83 +0,0 @@
|
||||
# ADR-059: Live ESP32 CSI Pipeline Integration
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Date
|
||||
|
||||
2026-03-12
|
||||
|
||||
## Context
|
||||
|
||||
ADR-058 established a dual-modal browser demo combining webcam video and WiFi CSI for pose estimation. However, it used simulated CSI data. To demonstrate real-world capability, we need an end-to-end pipeline from physical ESP32 hardware through to the browser visualization.
|
||||
|
||||
The ESP32-S3 firmware (`firmware/esp32-csi-node/`) already supports CSI collection and UDP streaming (ADR-018). The sensing server (`wifi-densepose-sensing-server`) already supports UDP ingestion and WebSocket bridging. The missing piece was connecting these components and enabling the browser demo to consume live data.
|
||||
|
||||
## Decision
|
||||
|
||||
Implement a complete live CSI pipeline:
|
||||
|
||||
```
|
||||
ESP32-S3 (CSI capture) → UDP:5005 → sensing-server (Rust/Axum) → WS:8765 → browser demo
|
||||
```
|
||||
|
||||
### Components
|
||||
|
||||
1. **ESP32 Firmware** — Rebuilt with native Windows ESP-IDF v5.4.0 toolchain (no Docker). Configured for target network and PC IP via `sdkconfig`. Helper scripts added:
|
||||
- `build_firmware.ps1` — Sets up IDF environment, cleans, builds, and flashes
|
||||
- `read_serial.ps1` — Serial monitor with DTR/RTS reset capability
|
||||
|
||||
2. **Sensing Server** — `wifi-densepose-sensing-server` started with:
|
||||
- `--source esp32` — Expect real ESP32 UDP frames
|
||||
- `--bind-addr 0.0.0.0` — Accept connections from any interface
|
||||
- `--ui-path <path>` — Serve the demo UI via HTTP
|
||||
|
||||
3. **Browser Demo** — `main.js` updated to auto-connect to `ws://localhost:8765/ws/sensing` on page load. Falls back to simulated CSI if the WebSocket is unavailable (GitHub Pages).
|
||||
|
||||
### Network Configuration
|
||||
|
||||
The ESP32 sends UDP packets to a configured target IP. If the PC's IP doesn't match the firmware's compiled target, a secondary IP alias can be added:
|
||||
|
||||
```powershell
|
||||
# PowerShell (Admin)
|
||||
New-NetIPAddress -IPAddress 192.168.1.100 -PrefixLength 24 -InterfaceAlias "Wi-Fi"
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
| Stage | Protocol | Format | Rate |
|
||||
|-------|----------|--------|------|
|
||||
| ESP32 → Server | UDP | ADR-018 binary frame (magic `0xC5110001`, I/Q pairs) | ~100 Hz |
|
||||
| Server → Browser | WebSocket | ADR-018 binary frame (forwarded) | ~10 Hz (tick-ms=100) |
|
||||
| Browser decode | JavaScript | Float32 amplitude/phase arrays | Per frame |
|
||||
|
||||
### Build Environment (Windows)
|
||||
|
||||
ESP-IDF v5.4.0 on Windows requires:
|
||||
- IDF_PATH pointing to the ESP-IDF framework
|
||||
- IDF_TOOLS_PATH pointing to toolchain binaries
|
||||
- MSYS/MinGW environment variables removed (ESP-IDF rejects them)
|
||||
- Python venv from ESP-IDF tools for `idf.py` execution
|
||||
|
||||
The `build_firmware.ps1` script handles all of this automatically.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- First end-to-end demonstration of real WiFi CSI → pose estimation in a browser
|
||||
- No Docker required for firmware builds on Windows
|
||||
- Demo gracefully degrades to simulated CSI when no server is available
|
||||
- Same demo works on GitHub Pages (simulated) and locally (live ESP32)
|
||||
|
||||
### Negative
|
||||
- ESP32 target IP is compiled into firmware; changing it requires a rebuild or NVS override
|
||||
- Windows firewall may block UDP:5005; user must allow it
|
||||
- Mixed content restrictions prevent HTTPS pages from connecting to ws:// (local only)
|
||||
|
||||
## Related
|
||||
|
||||
- [ADR-018](ADR-018-esp32-dev-implementation.md) — ESP32 CSI frame format and UDP streaming
|
||||
- [ADR-058](ADR-058-ruvector-wasm-browser-pose-example.md) — Dual-modal WASM browser pose demo
|
||||
- [ADR-039](ADR-039-edge-intelligence-framework.md) — Edge intelligence on ESP32
|
||||
- Issue [#245](https://github.com/ruvnet/RuView/issues/245) — Tracking issue
|
||||
+11
-33
@@ -78,17 +78,6 @@ docker pull ruvnet/wifi-densepose:latest
|
||||
|
||||
Multi-architecture image (amd64 + arm64). Works on Intel/AMD and Apple Silicon Macs. Contains the Rust sensing server, Three.js UI, and all signal processing.
|
||||
|
||||
**Data source selection:** Use the `CSI_SOURCE` environment variable to select the sensing mode:
|
||||
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| `auto` | (default) Probe for ESP32 on UDP 5005, fall back to simulation |
|
||||
| `esp32` | Receive real CSI frames from ESP32 devices over UDP |
|
||||
| `simulated` | Generate synthetic CSI frames (no hardware required) |
|
||||
| `wifi` | Host Wi-Fi RSSI (not available inside containers) |
|
||||
|
||||
Example: `docker run -e CSI_SOURCE=esp32 -p 3000:3000 -p 5005:5005/udp ruvnet/wifi-densepose:latest`
|
||||
|
||||
### From Source (Rust)
|
||||
|
||||
```bash
|
||||
@@ -278,8 +267,8 @@ Real Channel State Information at 20 Hz with 56-192 subcarriers. Required for po
|
||||
# From source
|
||||
./target/release/sensing-server --source esp32 --udp-port 5005 --http-port 3000 --ws-port 3001
|
||||
|
||||
# Docker (use CSI_SOURCE environment variable)
|
||||
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp -e CSI_SOURCE=esp32 ruvnet/wifi-densepose:latest
|
||||
# Docker
|
||||
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp ruvnet/wifi-densepose:latest --source esp32
|
||||
```
|
||||
|
||||
The ESP32 nodes stream binary CSI frames over UDP to port 5005. See [Hardware Setup](#esp32-s3-mesh) for flashing instructions.
|
||||
@@ -690,11 +679,9 @@ Download the dataset files and place them in a `data/` directory.
|
||||
./target/release/sensing-server --train --dataset data/ --dataset-type mmfi --epochs 100 --save-rvf model.rvf
|
||||
|
||||
# Via Docker (mount your data directory)
|
||||
# Note: Training mode requires overriding the default entrypoint
|
||||
docker run --rm \
|
||||
-v $(pwd)/data:/data \
|
||||
-v $(pwd)/output:/output \
|
||||
--entrypoint /app/sensing-server \
|
||||
ruvnet/wifi-densepose:latest \
|
||||
--train --dataset /data --epochs 100 --export-rvf /output/model.rvf
|
||||
```
|
||||
@@ -810,18 +797,14 @@ Pre-built binaries are available at [Releases](https://github.com/ruvnet/RuView/
|
||||
|
||||
| Release | What It Includes | Tag |
|
||||
|---------|-----------------|-----|
|
||||
| [v0.4.1](https://github.com/ruvnet/RuView/releases/tag/v0.4.1-esp32) | **Stable** — CSI build fix, compile guard, AMOLED display, edge intelligence ([ADR-057](../docs/adr/ADR-057-firmware-csi-build-guard.md)) | `v0.4.1-esp32` |
|
||||
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Stable — raw CSI streaming, TDM, channel hopping, QUIC mesh | `v0.2.0-esp32` |
|
||||
| [v0.3.0-alpha](https://github.com/ruvnet/RuView/releases/tag/v0.3.0-alpha-esp32) | Alpha — adds on-device edge intelligence (ADR-039) | `v0.3.0-alpha-esp32` |
|
||||
| [v0.2.0](https://github.com/ruvnet/RuView/releases/tag/v0.2.0-esp32) | Raw CSI streaming, TDM, channel hopping, QUIC mesh | `v0.2.0-esp32` |
|
||||
|
||||
> **Important:** Firmware versions prior to v0.4.1 had CSI **disabled** in the build config, causing a runtime error (`E wifi:CSI not enabled in menuconfig!`). Always use v0.4.1 or later.
|
||||
|
||||
```bash
|
||||
# Flash an ESP32-S3 (requires esptool: pip install esptool)
|
||||
python -m esptool --chip esp32s3 --port COM7 --baud 460800 \
|
||||
write-flash --flash-mode dio --flash-size 8MB --flash-freq 80m \
|
||||
0x0 bootloader.bin 0x8000 partition-table.bin \
|
||||
0xf000 ota_data_initial.bin 0x20000 esp32-csi-node.bin
|
||||
write-flash --flash-mode dio --flash-size 4MB \
|
||||
0x0 bootloader.bin 0x8000 partition-table.bin 0x10000 esp32-csi-node.bin
|
||||
```
|
||||
|
||||
**Provisioning:**
|
||||
@@ -902,8 +885,8 @@ Binary size: 777 KB (24% free in the 1 MB app partition).
|
||||
# From source
|
||||
./target/release/sensing-server --source esp32 --udp-port 5005 --http-port 3000 --ws-port 3001
|
||||
|
||||
# Docker (use CSI_SOURCE environment variable)
|
||||
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp -e CSI_SOURCE=esp32 ruvnet/wifi-densepose:latest
|
||||
# Docker
|
||||
docker run -p 3000:3000 -p 3001:3001 -p 5005:5005/udp ruvnet/wifi-densepose:latest --source esp32
|
||||
```
|
||||
|
||||
See [ADR-018](../docs/adr/ADR-018-esp32-dev-implementation.md), [ADR-029](../docs/adr/ADR-029-ruvsense-multistatic-sensing-mode.md), and [Tutorial #34](https://github.com/ruvnet/RuView/issues/34).
|
||||
@@ -970,17 +953,12 @@ Add the WebSocket port mapping:
|
||||
docker run -p 3000:3000 -p 3001:3001 ruvnet/wifi-densepose:latest
|
||||
```
|
||||
|
||||
### ESP32: "CSI not enabled in menuconfig"
|
||||
|
||||
Firmware versions prior to v0.4.1 had `CONFIG_ESP_WIFI_CSI_ENABLED` disabled in the build config. Upgrade to [v0.4.1](https://github.com/ruvnet/RuView/releases/tag/v0.4.1-esp32) or later. If building from source, ensure `sdkconfig.defaults` exists (not just `sdkconfig.defaults.template`). See [ADR-057](../docs/adr/ADR-057-firmware-csi-build-guard.md).
|
||||
|
||||
### ESP32: No data arriving
|
||||
|
||||
1. Verify firmware is v0.4.1+ (older versions had CSI disabled — see above)
|
||||
2. Verify the ESP32 is connected to the same WiFi network
|
||||
3. Check the target IP matches the sensing server machine: `python firmware/esp32-csi-node/provision.py --port COM7 --target-ip <YOUR_IP>`
|
||||
4. Verify UDP port 5005 is not blocked by firewall
|
||||
5. Test with: `nc -lu 5005` (Linux) or similar UDP listener
|
||||
1. Verify the ESP32 is connected to the same WiFi network
|
||||
2. Check the target IP matches the sensing server machine: `python firmware/esp32-csi-node/provision.py --port COM7 --target-ip <YOUR_IP>`
|
||||
3. Verify UDP port 5005 is not blocked by firewall
|
||||
4. Test with: `nc -lu 5005` (Linux) or similar UDP listener
|
||||
|
||||
### Build: Rust compilation errors
|
||||
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
# Remove MSYS environment variables that trigger ESP-IDF's MinGW rejection
|
||||
Remove-Item env:MSYSTEM -ErrorAction SilentlyContinue
|
||||
Remove-Item env:MSYSTEM_CARCH -ErrorAction SilentlyContinue
|
||||
Remove-Item env:MSYSTEM_CHOST -ErrorAction SilentlyContinue
|
||||
Remove-Item env:MSYSTEM_PREFIX -ErrorAction SilentlyContinue
|
||||
Remove-Item env:MINGW_CHOST -ErrorAction SilentlyContinue
|
||||
Remove-Item env:MINGW_PACKAGE_PREFIX -ErrorAction SilentlyContinue
|
||||
Remove-Item env:MINGW_PREFIX -ErrorAction SilentlyContinue
|
||||
|
||||
$env:IDF_PATH = "C:\Users\ruv\esp\v5.4\esp-idf"
|
||||
$env:IDF_TOOLS_PATH = "C:\Espressif\tools"
|
||||
$env:IDF_PYTHON_ENV_PATH = "C:\Espressif\tools\python\v5.4\venv"
|
||||
$env:PATH = "C:\Espressif\tools\xtensa-esp-elf\esp-14.2.0_20241119\xtensa-esp-elf\bin;C:\Espressif\tools\cmake\3.30.2\cmake-3.30.2-windows-x86_64\bin;C:\Espressif\tools\ninja\1.12.1;C:\Espressif\tools\ccache\4.10.2\ccache-4.10.2-windows-x86_64;C:\Espressif\tools\idf-exe\1.0.3;C:\Espressif\tools\python\v5.4\venv\Scripts;$env:PATH"
|
||||
|
||||
Set-Location "C:\Users\ruv\Projects\wifi-densepose\firmware\esp32-csi-node"
|
||||
|
||||
$python = "$env:IDF_PYTHON_ENV_PATH\Scripts\python.exe"
|
||||
$idf = "$env:IDF_PATH\tools\idf.py"
|
||||
|
||||
Write-Host "=== Cleaning stale build cache ==="
|
||||
& $python $idf fullclean
|
||||
|
||||
Write-Host "=== Building firmware (SSID=ruv.net, target=192.168.1.20:5005) ==="
|
||||
& $python $idf build
|
||||
|
||||
if ($LASTEXITCODE -eq 0) {
|
||||
Write-Host "=== Build succeeded! Flashing to COM7 ==="
|
||||
& $python $idf -p COM7 flash
|
||||
} else {
|
||||
Write-Host "=== Build failed with exit code $LASTEXITCODE ==="
|
||||
}
|
||||
@@ -21,16 +21,6 @@
|
||||
#include "esp_timer.h"
|
||||
#include "sdkconfig.h"
|
||||
|
||||
/* ADR-057: Build-time guard — fail early if CSI is not enabled in sdkconfig.
|
||||
* Without this, the firmware compiles but crashes at runtime with:
|
||||
* "E (xxxx) wifi:CSI not enabled in menuconfig!"
|
||||
* which is confusing for users flashing pre-built binaries. */
|
||||
#ifndef CONFIG_ESP_WIFI_CSI_ENABLED
|
||||
#error "CONFIG_ESP_WIFI_CSI_ENABLED must be set in sdkconfig. " \
|
||||
"Run: idf.py menuconfig -> Component config -> Wi-Fi -> Enable WiFi CSI, " \
|
||||
"or copy sdkconfig.defaults.template to sdkconfig.defaults before building."
|
||||
#endif
|
||||
|
||||
static const char *TAG = "csi_collector";
|
||||
|
||||
static uint32_t s_sequence = 0;
|
||||
|
||||
@@ -1,14 +0,0 @@
|
||||
$p = New-Object System.IO.Ports.SerialPort('COM7', 115200)
|
||||
$p.ReadTimeout = 5000
|
||||
$p.Open()
|
||||
Start-Sleep -Milliseconds 200
|
||||
|
||||
for ($i = 0; $i -lt 60; $i++) {
|
||||
try {
|
||||
$line = $p.ReadLine()
|
||||
Write-Host $line
|
||||
} catch {
|
||||
break
|
||||
}
|
||||
}
|
||||
$p.Close()
|
||||
@@ -1,33 +0,0 @@
|
||||
# ESP32-S3 CSI Node — Default SDK Configuration
|
||||
# This file is applied automatically by idf.py when no sdkconfig exists.
|
||||
|
||||
# Target: ESP32-S3
|
||||
CONFIG_IDF_TARGET="esp32s3"
|
||||
|
||||
# Use custom partition table (8MB flash with OTA — ADR-045)
|
||||
CONFIG_PARTITION_TABLE_CUSTOM=y
|
||||
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions_display.csv"
|
||||
|
||||
# Flash configuration: 8MB (Quad SPI)
|
||||
CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y
|
||||
CONFIG_ESPTOOLPY_FLASHSIZE="8MB"
|
||||
|
||||
# Compiler optimization: optimize for size to reduce binary
|
||||
CONFIG_COMPILER_OPTIMIZATION_SIZE=y
|
||||
|
||||
# Enable CSI (Channel State Information) in WiFi driver
|
||||
CONFIG_ESP_WIFI_CSI_ENABLED=y
|
||||
|
||||
# NVS encryption disabled by default (requires eFuse provisioning).
|
||||
# Enable only after burning HMAC key to eFuse block.
|
||||
# CONFIG_NVS_ENCRYPTION is not set
|
||||
|
||||
# Disable unused features to reduce binary size
|
||||
CONFIG_BOOTLOADER_LOG_LEVEL_WARN=y
|
||||
CONFIG_LOG_DEFAULT_LEVEL_INFO=y
|
||||
|
||||
# LWIP: enable extended socket options for UDP multicast
|
||||
CONFIG_LWIP_SO_RCVBUF=y
|
||||
|
||||
# FreeRTOS: increase task stack for CSI processing
|
||||
CONFIG_ESP_MAIN_TASK_STACK_SIZE=8192
|
||||
@@ -18,9 +18,8 @@ CONFIG_COMPILER_OPTIMIZATION_SIZE=y
|
||||
# Enable CSI (Channel State Information) in WiFi driver
|
||||
CONFIG_ESP_WIFI_CSI_ENABLED=y
|
||||
|
||||
# NVS encryption disabled by default (requires eFuse provisioning).
|
||||
# Enable only after burning HMAC key to eFuse block.
|
||||
# CONFIG_NVS_ENCRYPTION is not set
|
||||
# Enable NVS encryption for secure credential storage
|
||||
CONFIG_NVS_ENCRYPTION=y
|
||||
|
||||
# Disable unused features to reduce binary size
|
||||
CONFIG_BOOTLOADER_LOG_LEVEL_WARN=y
|
||||
|
||||
@@ -4,4 +4,5 @@ pub mod ota;
|
||||
pub mod provision;
|
||||
pub mod server;
|
||||
pub mod settings;
|
||||
pub mod training;
|
||||
pub mod wasm;
|
||||
|
||||
@@ -0,0 +1,482 @@
|
||||
//! Training commands for the desktop application.
|
||||
//!
|
||||
//! Provides Tauri commands for:
|
||||
//! - GPU detection
|
||||
//! - Dataset management
|
||||
//! - Model/checkpoint operations
|
||||
//! - Training job control
|
||||
//! - RuVector configuration
|
||||
//! - Metrics retrieval
|
||||
|
||||
use crate::domain::training::{
|
||||
CheckpointInfo, DatasetFormat, DatasetInfo, EpochMetrics, EvaluationMetrics,
|
||||
GpuBackend, GpuInfo, JointAccuracy, LiveTestMetrics,
|
||||
ModelInfo, ModelType, RuVectorConfig, TrainingConfig, TrainingJob,
|
||||
TrainingProgress, TrainingStatus,
|
||||
};
|
||||
use crate::state::AppState;
|
||||
use tauri::State;
|
||||
|
||||
// ============================================================================
|
||||
// Standard Datasets (built-in)
|
||||
// ============================================================================
|
||||
|
||||
fn get_standard_datasets() -> Vec<DatasetInfo> {
|
||||
vec![
|
||||
DatasetInfo {
|
||||
id: "mmfi".into(),
|
||||
name: "MM-Fi Dataset".into(),
|
||||
description: "Multi-modal WiFi sensing dataset with 40 subjects, 27 activities".into(),
|
||||
format: DatasetFormat::MmFi,
|
||||
size_mb: 2400.0,
|
||||
samples: 320000,
|
||||
downloaded: false,
|
||||
path: None,
|
||||
url: Some("https://ntu-aiot-lab.github.io/mm-fi".into()),
|
||||
},
|
||||
DatasetInfo {
|
||||
id: "wipose".into(),
|
||||
name: "Wi-Pose Dataset".into(),
|
||||
description: "WiFi-based pose estimation with 3D skeleton annotations".into(),
|
||||
format: DatasetFormat::WiPose,
|
||||
size_mb: 1800.0,
|
||||
samples: 150000,
|
||||
downloaded: false,
|
||||
path: None,
|
||||
url: Some("https://github.com/Wi-Pose".into()),
|
||||
},
|
||||
DatasetInfo {
|
||||
id: "wiar".into(),
|
||||
name: "WiAR Dataset".into(),
|
||||
description: "WiFi activity recognition with CSI data".into(),
|
||||
format: DatasetFormat::Wiar,
|
||||
size_mb: 500.0,
|
||||
samples: 45000,
|
||||
downloaded: false,
|
||||
path: None,
|
||||
url: Some("https://github.com/WiAR".into()),
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Standard Model Architectures
|
||||
// ============================================================================
|
||||
|
||||
fn get_standard_models() -> Vec<ModelInfo> {
|
||||
vec![
|
||||
ModelInfo {
|
||||
id: "csi-encoder-cnn".into(),
|
||||
name: "CSI Encoder (CNN)".into(),
|
||||
model_type: ModelType::Encoder,
|
||||
description: "Convolutional encoder for CSI amplitude/phase features".into(),
|
||||
params_m: 2.3,
|
||||
memory_mb: 128,
|
||||
paper: None,
|
||||
},
|
||||
ModelInfo {
|
||||
id: "csi-encoder-transformer".into(),
|
||||
name: "CSI Encoder (Transformer)".into(),
|
||||
model_type: ModelType::Encoder,
|
||||
description: "Self-attention based CSI feature extraction".into(),
|
||||
params_m: 8.5,
|
||||
memory_mb: 384,
|
||||
paper: Some("WiFi-ViT 2024".into()),
|
||||
},
|
||||
ModelInfo {
|
||||
id: "pose-decoder-lstm".into(),
|
||||
name: "Pose Decoder (LSTM)".into(),
|
||||
model_type: ModelType::Decoder,
|
||||
description: "Recurrent decoder for temporal pose estimation".into(),
|
||||
params_m: 1.8,
|
||||
memory_mb: 96,
|
||||
paper: None,
|
||||
},
|
||||
ModelInfo {
|
||||
id: "pose-decoder-gru".into(),
|
||||
name: "Pose Decoder (GRU)".into(),
|
||||
model_type: ModelType::Decoder,
|
||||
description: "Gated recurrent unit pose decoder (faster)".into(),
|
||||
params_m: 1.2,
|
||||
memory_mb: 64,
|
||||
paper: None,
|
||||
},
|
||||
ModelInfo {
|
||||
id: "aether-embedding".into(),
|
||||
name: "AETHER Embedding".into(),
|
||||
model_type: ModelType::Embedding,
|
||||
description: "Contrastive CSI embedding for person re-identification (ADR-024)".into(),
|
||||
params_m: 4.2,
|
||||
memory_mb: 192,
|
||||
paper: Some("AETHER 2025".into()),
|
||||
},
|
||||
ModelInfo {
|
||||
id: "meridian-adaptor".into(),
|
||||
name: "MERIDIAN Adaptor".into(),
|
||||
model_type: ModelType::Adaptor,
|
||||
description: "Cross-environment domain generalization module (ADR-027)".into(),
|
||||
params_m: 3.1,
|
||||
memory_mb: 144,
|
||||
paper: Some("MERIDIAN 2025".into()),
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// GPU Detection Commands
|
||||
// ============================================================================
|
||||
|
||||
/// Detect available GPU(s) and return information.
|
||||
#[tauri::command]
|
||||
pub async fn detect_gpu(state: State<'_, AppState>) -> Result<GpuInfo, String> {
|
||||
// Check for cached GPU info
|
||||
if let Ok(training) = state.training.lock() {
|
||||
if let Some(ref info) = training.gpu_info {
|
||||
return Ok(info.clone());
|
||||
}
|
||||
}
|
||||
|
||||
// Detect GPU
|
||||
let info = detect_gpu_internal();
|
||||
|
||||
// Cache the result
|
||||
if let Ok(mut training) = state.training.lock() {
|
||||
training.gpu_info = Some(info.clone());
|
||||
}
|
||||
|
||||
Ok(info)
|
||||
}
|
||||
|
||||
fn detect_gpu_internal() -> GpuInfo {
|
||||
// Check for Metal on macOS
|
||||
#[cfg(target_os = "macos")]
|
||||
{
|
||||
// Check if system has Apple Silicon or discrete GPU
|
||||
let has_metal = std::process::Command::new("system_profiler")
|
||||
.args(["SPDisplaysDataType", "-json"])
|
||||
.output()
|
||||
.map(|o| {
|
||||
let output = String::from_utf8_lossy(&o.stdout);
|
||||
output.contains("Metal") || output.contains("Apple M")
|
||||
})
|
||||
.unwrap_or(false);
|
||||
|
||||
if has_metal {
|
||||
// Try to get GPU name
|
||||
let name = std::process::Command::new("system_profiler")
|
||||
.args(["SPDisplaysDataType"])
|
||||
.output()
|
||||
.ok()
|
||||
.and_then(|o| {
|
||||
let output = String::from_utf8_lossy(&o.stdout);
|
||||
// Parse chipset name
|
||||
for line in output.lines() {
|
||||
if line.contains("Chipset Model:") {
|
||||
return line.split(':').nth(1).map(|s| s.trim().to_string());
|
||||
}
|
||||
}
|
||||
None
|
||||
});
|
||||
|
||||
return GpuInfo {
|
||||
available: true,
|
||||
backend: GpuBackend::Metal,
|
||||
name,
|
||||
memory_mb: None, // Metal doesn't easily expose this
|
||||
cuda_version: None,
|
||||
metal_supported: true,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// Check for CUDA on Linux/Windows
|
||||
#[cfg(any(target_os = "linux", target_os = "windows"))]
|
||||
{
|
||||
// Try nvidia-smi for CUDA detection
|
||||
if let Ok(output) = std::process::Command::new("nvidia-smi")
|
||||
.args(["--query-gpu=name,memory.total", "--format=csv,noheader,nounits"])
|
||||
.output()
|
||||
{
|
||||
if output.status.success() {
|
||||
let stdout = String::from_utf8_lossy(&output.stdout);
|
||||
let parts: Vec<&str> = stdout.trim().split(',').collect();
|
||||
|
||||
let name = parts.first().map(|s| s.trim().to_string());
|
||||
let memory_mb = parts.get(1)
|
||||
.and_then(|s| s.trim().parse::<u64>().ok());
|
||||
|
||||
// Get CUDA version
|
||||
let cuda_version = std::process::Command::new("nvidia-smi")
|
||||
.output()
|
||||
.ok()
|
||||
.and_then(|o| {
|
||||
let output = String::from_utf8_lossy(&o.stdout);
|
||||
for line in output.lines() {
|
||||
if line.contains("CUDA Version:") {
|
||||
return line.split("CUDA Version:")
|
||||
.nth(1)
|
||||
.map(|s| s.split_whitespace().next().unwrap_or("").to_string());
|
||||
}
|
||||
}
|
||||
None
|
||||
});
|
||||
|
||||
return GpuInfo {
|
||||
available: true,
|
||||
backend: GpuBackend::Cuda,
|
||||
name,
|
||||
memory_mb,
|
||||
cuda_version,
|
||||
metal_supported: false,
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fall back to CPU
|
||||
GpuInfo {
|
||||
available: false,
|
||||
backend: GpuBackend::Cpu,
|
||||
name: None,
|
||||
memory_mb: None,
|
||||
cuda_version: None,
|
||||
metal_supported: false,
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Dataset Commands
|
||||
// ============================================================================
|
||||
|
||||
/// List available datasets (both standard and downloaded).
|
||||
#[tauri::command]
|
||||
pub async fn list_datasets(state: State<'_, AppState>) -> Result<Vec<String>, String> {
|
||||
let training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
|
||||
// Return IDs of downloaded datasets
|
||||
Ok(training.datasets.iter()
|
||||
.filter(|d| d.downloaded)
|
||||
.map(|d| d.id.clone())
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Get full dataset information.
|
||||
#[tauri::command]
|
||||
pub async fn get_datasets(state: State<'_, AppState>) -> Result<Vec<DatasetInfo>, String> {
|
||||
let mut training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
|
||||
// Initialize with standard datasets if empty
|
||||
if training.datasets.is_empty() {
|
||||
training.datasets = get_standard_datasets();
|
||||
}
|
||||
|
||||
Ok(training.datasets.clone())
|
||||
}
|
||||
|
||||
/// Download a dataset (placeholder - actual download would need async HTTP).
|
||||
#[tauri::command]
|
||||
pub async fn download_dataset(
|
||||
dataset_id: String,
|
||||
state: State<'_, AppState>,
|
||||
) -> Result<DatasetInfo, String> {
|
||||
let mut training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
|
||||
// Find the dataset
|
||||
let dataset = training.datasets.iter_mut()
|
||||
.find(|d| d.id == dataset_id)
|
||||
.ok_or_else(|| format!("Dataset not found: {}", dataset_id))?;
|
||||
|
||||
// Simulate download completion
|
||||
dataset.downloaded = true;
|
||||
dataset.path = Some(format!("~/.ruview/datasets/{}", dataset_id));
|
||||
|
||||
Ok(dataset.clone())
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Model/Checkpoint Commands
|
||||
// ============================================================================
|
||||
|
||||
/// List available model architectures.
|
||||
#[tauri::command]
|
||||
pub async fn list_models() -> Result<Vec<ModelInfo>, String> {
|
||||
Ok(get_standard_models())
|
||||
}
|
||||
|
||||
/// List saved checkpoints.
|
||||
#[tauri::command]
|
||||
pub async fn list_checkpoints(state: State<'_, AppState>) -> Result<Vec<CheckpointInfo>, String> {
|
||||
let training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
Ok(training.checkpoints.clone())
|
||||
}
|
||||
|
||||
/// Export a model checkpoint to ONNX or TorchScript.
|
||||
#[tauri::command]
|
||||
pub async fn export_model(
|
||||
checkpoint_id: String,
|
||||
format: String,
|
||||
state: State<'_, AppState>,
|
||||
) -> Result<String, String> {
|
||||
let training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
|
||||
let checkpoint = training.checkpoints.iter()
|
||||
.find(|c| c.id == checkpoint_id)
|
||||
.ok_or_else(|| format!("Checkpoint not found: {}", checkpoint_id))?;
|
||||
|
||||
let output_path = match format.as_str() {
|
||||
"onnx" => format!("{}.onnx", checkpoint.path.trim_end_matches(".pt")),
|
||||
"torchscript" => format!("{}.ts", checkpoint.path.trim_end_matches(".pt")),
|
||||
_ => return Err(format!("Unsupported format: {}", format)),
|
||||
};
|
||||
|
||||
// In a real implementation, this would call the actual export logic
|
||||
Ok(output_path)
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Training Job Commands
|
||||
// ============================================================================
|
||||
|
||||
/// Start a training job.
|
||||
#[tauri::command]
|
||||
pub async fn start_training(
|
||||
config: TrainingConfig,
|
||||
state: State<'_, AppState>,
|
||||
) -> Result<String, String> {
|
||||
let mut training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
|
||||
// Create a new job
|
||||
let job_id = uuid::Uuid::new_v4().to_string();
|
||||
let job = TrainingJob {
|
||||
id: job_id.clone(),
|
||||
config,
|
||||
status: TrainingStatus::Running,
|
||||
started_at: Some(chrono::Utc::now().to_rfc3339()),
|
||||
progress: TrainingProgress::default(),
|
||||
loss_history: Vec::new(),
|
||||
};
|
||||
|
||||
training.current_job = Some(job);
|
||||
|
||||
// In a real implementation, this would spawn a background training thread
|
||||
// and emit progress events via Tauri's event system
|
||||
|
||||
Ok(job_id)
|
||||
}
|
||||
|
||||
/// Stop the current training job.
|
||||
#[tauri::command]
|
||||
pub async fn stop_training(state: State<'_, AppState>) -> Result<(), String> {
|
||||
let mut training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
|
||||
if let Some(ref mut job) = training.current_job {
|
||||
job.status = TrainingStatus::Paused;
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Get current training progress.
|
||||
#[tauri::command]
|
||||
pub async fn training_progress(state: State<'_, AppState>) -> Result<Option<TrainingProgress>, String> {
|
||||
let training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
Ok(training.current_job.as_ref().map(|j| j.progress.clone()))
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// RuVector Configuration Commands
|
||||
// ============================================================================
|
||||
|
||||
/// Get current RuVector configuration.
|
||||
#[tauri::command]
|
||||
pub async fn get_ruvector_config(state: State<'_, AppState>) -> Result<RuVectorConfig, String> {
|
||||
let training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
Ok(training.ruvector_config.clone())
|
||||
}
|
||||
|
||||
/// Set RuVector configuration.
|
||||
#[tauri::command]
|
||||
pub async fn set_ruvector_config(
|
||||
config: RuVectorConfig,
|
||||
state: State<'_, AppState>,
|
||||
) -> Result<(), String> {
|
||||
let mut training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
training.ruvector_config = config;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Test RuVector modules on live CSI data.
|
||||
#[tauri::command]
|
||||
pub async fn test_ruvector_live(
|
||||
_state: State<'_, AppState>,
|
||||
) -> Result<LiveTestMetrics, String> {
|
||||
// In a real implementation, this would process live CSI data
|
||||
// through the RuVector pipeline and return metrics
|
||||
Ok(LiveTestMetrics {
|
||||
fps: 30.0,
|
||||
latency_ms: 15.0,
|
||||
persons_detected: 1,
|
||||
})
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Metrics Commands
|
||||
// ============================================================================
|
||||
|
||||
/// Get training history (loss/accuracy per epoch).
|
||||
#[tauri::command]
|
||||
pub async fn get_training_history(state: State<'_, AppState>) -> Result<Vec<EpochMetrics>, String> {
|
||||
let training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
Ok(training.training_history.clone())
|
||||
}
|
||||
|
||||
/// Get evaluation metrics.
|
||||
#[tauri::command]
|
||||
pub async fn get_evaluation_metrics(state: State<'_, AppState>) -> Result<Option<EvaluationMetrics>, String> {
|
||||
let training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
Ok(training.evaluation_metrics.clone())
|
||||
}
|
||||
|
||||
/// Get per-joint accuracy metrics.
|
||||
#[tauri::command]
|
||||
pub async fn get_joint_accuracies(state: State<'_, AppState>) -> Result<Vec<JointAccuracy>, String> {
|
||||
let training = state.training.lock().map_err(|e| e.to_string())?;
|
||||
Ok(training.joint_accuracies.clone())
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Tests
|
||||
// ============================================================================
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_standard_datasets() {
|
||||
let datasets = get_standard_datasets();
|
||||
assert_eq!(datasets.len(), 3);
|
||||
assert!(datasets.iter().any(|d| d.id == "mmfi"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_standard_models() {
|
||||
let models = get_standard_models();
|
||||
assert_eq!(models.len(), 6);
|
||||
assert!(models.iter().any(|m| m.id == "csi-encoder-cnn"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_gpu_internal() {
|
||||
let info = detect_gpu_internal();
|
||||
// Just verify it returns valid data
|
||||
assert!(matches!(info.backend, GpuBackend::Cpu | GpuBackend::Cuda | GpuBackend::Metal));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_ruvector_config_default() {
|
||||
let config = RuVectorConfig::default();
|
||||
assert!(config.mincut_enabled);
|
||||
assert_eq!(config.attention_heads, 4);
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
pub mod config;
|
||||
pub mod firmware;
|
||||
pub mod node;
|
||||
pub mod training;
|
||||
|
||||
@@ -0,0 +1,312 @@
|
||||
//! Training domain types for the desktop application.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// GPU backend type.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum GpuBackend {
|
||||
Cuda,
|
||||
Metal,
|
||||
#[default]
|
||||
Cpu,
|
||||
}
|
||||
|
||||
/// GPU information.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub struct GpuInfo {
|
||||
pub available: bool,
|
||||
pub backend: GpuBackend,
|
||||
pub name: Option<String>,
|
||||
pub memory_mb: Option<u64>,
|
||||
pub cuda_version: Option<String>,
|
||||
pub metal_supported: bool,
|
||||
}
|
||||
|
||||
/// Dataset format type.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum DatasetFormat {
|
||||
#[default]
|
||||
MmFi,
|
||||
WiPose,
|
||||
Wiar,
|
||||
Custom,
|
||||
}
|
||||
|
||||
/// Dataset information.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct DatasetInfo {
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
pub description: String,
|
||||
pub format: DatasetFormat,
|
||||
pub size_mb: f64,
|
||||
pub samples: u64,
|
||||
pub downloaded: bool,
|
||||
pub path: Option<String>,
|
||||
pub url: Option<String>,
|
||||
}
|
||||
|
||||
/// Model architecture type.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum ModelType {
|
||||
#[default]
|
||||
Encoder,
|
||||
Decoder,
|
||||
Embedding,
|
||||
Adaptor,
|
||||
}
|
||||
|
||||
/// Model architecture information.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ModelInfo {
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
pub model_type: ModelType,
|
||||
pub description: String,
|
||||
pub params_m: f64,
|
||||
pub memory_mb: u64,
|
||||
pub paper: Option<String>,
|
||||
}
|
||||
|
||||
/// Checkpoint information.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct CheckpointInfo {
|
||||
pub id: String,
|
||||
pub model_id: String,
|
||||
pub name: String,
|
||||
pub epoch: u32,
|
||||
pub val_loss: f64,
|
||||
pub created_at: String,
|
||||
pub path: String,
|
||||
pub size_mb: f64,
|
||||
}
|
||||
|
||||
/// Training configuration.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct TrainingConfig {
|
||||
pub dataset_id: String,
|
||||
pub model_id: String,
|
||||
pub epochs: u32,
|
||||
pub batch_size: u32,
|
||||
pub learning_rate: f64,
|
||||
pub optimizer: OptimizerType,
|
||||
pub weight_decay: f64,
|
||||
pub use_augmentation: bool,
|
||||
pub checkpoint_every: u32,
|
||||
}
|
||||
|
||||
impl Default for TrainingConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
dataset_id: "mmfi".into(),
|
||||
model_id: "csi-encoder-cnn".into(),
|
||||
epochs: 100,
|
||||
batch_size: 32,
|
||||
learning_rate: 0.001,
|
||||
optimizer: OptimizerType::Adam,
|
||||
weight_decay: 0.0001,
|
||||
use_augmentation: true,
|
||||
checkpoint_every: 10,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Optimizer type.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum OptimizerType {
|
||||
#[default]
|
||||
Adam,
|
||||
AdamW,
|
||||
Sgd,
|
||||
}
|
||||
|
||||
/// Training job status.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum TrainingStatus {
|
||||
#[default]
|
||||
Pending,
|
||||
Running,
|
||||
Paused,
|
||||
Completed,
|
||||
Failed,
|
||||
}
|
||||
|
||||
/// Training progress.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub struct TrainingProgress {
|
||||
pub epoch: u32,
|
||||
pub total_epochs: u32,
|
||||
pub batch: u32,
|
||||
pub total_batches: u32,
|
||||
pub train_loss: f64,
|
||||
pub val_loss: Option<f64>,
|
||||
pub learning_rate: f64,
|
||||
pub eta_secs: u64,
|
||||
pub gpu_memory_mb: Option<u64>,
|
||||
}
|
||||
|
||||
/// Training job.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct TrainingJob {
|
||||
pub id: String,
|
||||
pub config: TrainingConfig,
|
||||
pub status: TrainingStatus,
|
||||
pub started_at: Option<String>,
|
||||
pub progress: TrainingProgress,
|
||||
pub loss_history: Vec<EpochMetrics>,
|
||||
}
|
||||
|
||||
/// Metrics for a single epoch.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct EpochMetrics {
|
||||
pub epoch: u32,
|
||||
pub train_loss: f64,
|
||||
pub val_loss: f64,
|
||||
pub train_acc: f64,
|
||||
pub val_acc: f64,
|
||||
pub learning_rate: f64,
|
||||
pub timestamp: String,
|
||||
}
|
||||
|
||||
/// Evaluation metrics.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub struct EvaluationMetrics {
|
||||
pub pck_05: f64,
|
||||
pub pck_10: f64,
|
||||
pub pck_20: f64,
|
||||
pub map_50: f64,
|
||||
pub map_75: f64,
|
||||
pub iou: f64,
|
||||
}
|
||||
|
||||
/// Per-joint accuracy.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct JointAccuracy {
|
||||
pub joint: String,
|
||||
pub accuracy: f64,
|
||||
}
|
||||
|
||||
/// RuVector interpolation mode.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum InterpolationMode {
|
||||
Linear,
|
||||
Cubic,
|
||||
#[default]
|
||||
Sparse,
|
||||
}
|
||||
|
||||
/// RuVector module configuration.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct RuVectorConfig {
|
||||
// MinCut parameters
|
||||
pub mincut_enabled: bool,
|
||||
pub mincut_threshold: f64,
|
||||
pub mincut_max_persons: u32,
|
||||
|
||||
// Attention parameters
|
||||
pub attention_enabled: bool,
|
||||
pub attention_heads: u32,
|
||||
pub attention_dropout: f64,
|
||||
|
||||
// Temporal parameters
|
||||
pub temporal_enabled: bool,
|
||||
pub temporal_window_ms: u32,
|
||||
pub temporal_compression_ratio: u32,
|
||||
|
||||
// Solver parameters
|
||||
pub solver_enabled: bool,
|
||||
pub solver_interpolation: InterpolationMode,
|
||||
pub solver_subcarrier_count: u32,
|
||||
|
||||
// BVP parameters
|
||||
pub bvp_enabled: bool,
|
||||
pub bvp_filter_hz: (f64, f64),
|
||||
}
|
||||
|
||||
impl Default for RuVectorConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
mincut_enabled: true,
|
||||
mincut_threshold: 0.5,
|
||||
mincut_max_persons: 5,
|
||||
attention_enabled: true,
|
||||
attention_heads: 4,
|
||||
attention_dropout: 0.1,
|
||||
temporal_enabled: true,
|
||||
temporal_window_ms: 500,
|
||||
temporal_compression_ratio: 4,
|
||||
solver_enabled: true,
|
||||
solver_interpolation: InterpolationMode::Sparse,
|
||||
solver_subcarrier_count: 56,
|
||||
bvp_enabled: false,
|
||||
bvp_filter_hz: (0.7, 4.0),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Live test metrics from RuVector processing.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
|
||||
pub struct LiveTestMetrics {
|
||||
pub fps: f64,
|
||||
pub latency_ms: f64,
|
||||
pub persons_detected: u32,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_gpu_info_default() {
|
||||
let info = GpuInfo::default();
|
||||
assert!(!info.available);
|
||||
assert_eq!(info.backend, GpuBackend::Cpu);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_training_config_default() {
|
||||
let config = TrainingConfig::default();
|
||||
assert_eq!(config.epochs, 100);
|
||||
assert_eq!(config.batch_size, 32);
|
||||
assert_eq!(config.optimizer, OptimizerType::Adam);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_ruvector_config_default() {
|
||||
let config = RuVectorConfig::default();
|
||||
assert!(config.mincut_enabled);
|
||||
assert_eq!(config.mincut_threshold, 0.5);
|
||||
assert_eq!(config.attention_heads, 4);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_serialization() {
|
||||
let config = TrainingConfig::default();
|
||||
let json = serde_json::to_string(&config).unwrap();
|
||||
let parsed: TrainingConfig = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(parsed.epochs, config.epochs);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_dataset_info() {
|
||||
let dataset = DatasetInfo {
|
||||
id: "mmfi".into(),
|
||||
name: "MM-Fi Dataset".into(),
|
||||
description: "Multi-modal WiFi sensing".into(),
|
||||
format: DatasetFormat::MmFi,
|
||||
size_mb: 2400.0,
|
||||
samples: 320000,
|
||||
downloaded: false,
|
||||
path: None,
|
||||
url: Some("https://example.com/mmfi.zip".into()),
|
||||
};
|
||||
assert_eq!(dataset.id, "mmfi");
|
||||
assert!(!dataset.downloaded);
|
||||
}
|
||||
}
|
||||
@@ -2,7 +2,7 @@ pub mod commands;
|
||||
pub mod domain;
|
||||
pub mod state;
|
||||
|
||||
use commands::{discovery, flash, ota, provision, server, settings, wasm};
|
||||
use commands::{discovery, flash, ota, provision, server, settings, training, wasm};
|
||||
|
||||
pub fn run() {
|
||||
tauri::Builder::default()
|
||||
@@ -46,6 +46,23 @@ pub fn run() {
|
||||
// Settings
|
||||
settings::get_settings,
|
||||
settings::save_settings,
|
||||
// Training
|
||||
training::detect_gpu,
|
||||
training::list_datasets,
|
||||
training::get_datasets,
|
||||
training::download_dataset,
|
||||
training::list_models,
|
||||
training::list_checkpoints,
|
||||
training::export_model,
|
||||
training::start_training,
|
||||
training::stop_training,
|
||||
training::training_progress,
|
||||
training::get_ruvector_config,
|
||||
training::set_ruvector_config,
|
||||
training::test_ruvector_live,
|
||||
training::get_training_history,
|
||||
training::get_evaluation_metrics,
|
||||
training::get_joint_accuracies,
|
||||
])
|
||||
.run(tauri::generate_context!())
|
||||
.expect("error while running tauri application");
|
||||
|
||||
@@ -3,6 +3,10 @@ use std::sync::Mutex;
|
||||
use std::time::Instant;
|
||||
|
||||
use crate::domain::node::DiscoveredNode;
|
||||
use crate::domain::training::{
|
||||
CheckpointInfo, DatasetInfo, EpochMetrics, EvaluationMetrics,
|
||||
GpuInfo, JointAccuracy, RuVectorConfig, TrainingJob,
|
||||
};
|
||||
|
||||
/// Sub-state for discovered nodes.
|
||||
#[derive(Default)]
|
||||
@@ -87,6 +91,33 @@ impl Default for SettingsState {
|
||||
}
|
||||
}
|
||||
|
||||
/// Sub-state for training operations.
|
||||
pub struct TrainingState {
|
||||
pub gpu_info: Option<GpuInfo>,
|
||||
pub datasets: Vec<DatasetInfo>,
|
||||
pub checkpoints: Vec<CheckpointInfo>,
|
||||
pub current_job: Option<TrainingJob>,
|
||||
pub ruvector_config: RuVectorConfig,
|
||||
pub training_history: Vec<EpochMetrics>,
|
||||
pub evaluation_metrics: Option<EvaluationMetrics>,
|
||||
pub joint_accuracies: Vec<JointAccuracy>,
|
||||
}
|
||||
|
||||
impl Default for TrainingState {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
gpu_info: None,
|
||||
datasets: Vec::new(),
|
||||
checkpoints: Vec::new(),
|
||||
current_job: None,
|
||||
ruvector_config: RuVectorConfig::default(),
|
||||
training_history: Vec::new(),
|
||||
evaluation_metrics: None,
|
||||
joint_accuracies: Vec::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Top-level application state managed by Tauri.
|
||||
pub struct AppState {
|
||||
pub discovery: Mutex<DiscoveryState>,
|
||||
@@ -94,6 +125,7 @@ pub struct AppState {
|
||||
pub flash: Mutex<FlashState>,
|
||||
pub ota: Mutex<OtaState>,
|
||||
pub settings: Mutex<SettingsState>,
|
||||
pub training: Mutex<TrainingState>,
|
||||
}
|
||||
|
||||
impl Default for AppState {
|
||||
@@ -104,6 +136,7 @@ impl Default for AppState {
|
||||
flash: Mutex::new(FlashState::default()),
|
||||
ota: Mutex::new(OtaState::default()),
|
||||
settings: Mutex::new(SettingsState::default()),
|
||||
training: Mutex::new(TrainingState::default()),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -135,6 +168,9 @@ impl AppState {
|
||||
if let Ok(mut settings) = self.settings.lock() {
|
||||
*settings = SettingsState::default();
|
||||
}
|
||||
if let Ok(mut training) = self.training.lock() {
|
||||
*training = TrainingState::default();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"$schema": "https://raw.githubusercontent.com/tauri-apps/tauri/dev/crates/tauri-config-schema/schema.json",
|
||||
"productName": "RuView Desktop",
|
||||
"version": "0.4.4",
|
||||
"version": "0.5.0",
|
||||
"identifier": "net.ruv.ruview",
|
||||
"build": {
|
||||
"frontendDist": "ui/dist",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "ruview-desktop-ui",
|
||||
"private": true,
|
||||
"version": "0.4.4",
|
||||
"version": "0.5.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
|
||||
@@ -8,6 +8,7 @@ import { OtaUpdate } from "./pages/OtaUpdate";
|
||||
import { EdgeModules } from "./pages/EdgeModules";
|
||||
import { Sensing } from "./pages/Sensing";
|
||||
import { MeshView } from "./pages/MeshView";
|
||||
import Training from "./pages/Training";
|
||||
import { Settings } from "./pages/Settings";
|
||||
|
||||
type Page =
|
||||
@@ -19,6 +20,7 @@ type Page =
|
||||
| "wasm"
|
||||
| "sensing"
|
||||
| "mesh"
|
||||
| "training"
|
||||
| "settings";
|
||||
|
||||
interface NavItem {
|
||||
@@ -36,6 +38,7 @@ const NAV_ITEMS: NavItem[] = [
|
||||
{ id: "wasm", label: "Edge Modules", icon: "\u2B21" },
|
||||
{ id: "sensing", label: "Sensing", icon: "\u2248" },
|
||||
{ id: "mesh", label: "Mesh View", icon: "\u2B2F" },
|
||||
{ id: "training", label: "Training", icon: "\u2B50" },
|
||||
{ id: "settings", label: "Settings", icon: "\u2699" },
|
||||
];
|
||||
|
||||
@@ -99,6 +102,7 @@ const App: React.FC = () => {
|
||||
case "wasm": return <EdgeModules />;
|
||||
case "sensing": return <Sensing />;
|
||||
case "mesh": return <MeshView />;
|
||||
case "training": return <Training />;
|
||||
case "settings": return <Settings />;
|
||||
}
|
||||
};
|
||||
|
||||
+369
@@ -0,0 +1,369 @@
|
||||
import React, { useState, useEffect } from "react";
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
|
||||
interface Dataset {
|
||||
id: string;
|
||||
name: string;
|
||||
description: string;
|
||||
size_mb: number;
|
||||
samples: number;
|
||||
downloaded: boolean;
|
||||
path: string | null;
|
||||
}
|
||||
|
||||
const STANDARD_DATASETS: Omit<Dataset, "downloaded" | "path">[] = [
|
||||
{
|
||||
id: "mmfi",
|
||||
name: "MM-Fi Dataset",
|
||||
description: "Multi-modal WiFi sensing dataset with 40 subjects, 27 activities",
|
||||
size_mb: 2400,
|
||||
samples: 320000,
|
||||
},
|
||||
{
|
||||
id: "wipose",
|
||||
name: "Wi-Pose Dataset",
|
||||
description: "WiFi-based pose estimation with 3D skeleton annotations",
|
||||
size_mb: 1800,
|
||||
samples: 150000,
|
||||
},
|
||||
{
|
||||
id: "wiar",
|
||||
name: "WiAR Dataset",
|
||||
description: "WiFi activity recognition with CSI data",
|
||||
size_mb: 500,
|
||||
samples: 45000,
|
||||
},
|
||||
];
|
||||
|
||||
const DatasetsTab: React.FC = () => {
|
||||
const [datasets, setDatasets] = useState<Dataset[]>([]);
|
||||
const [downloading, setDownloading] = useState<string | null>(null);
|
||||
const [downloadProgress, setDownloadProgress] = useState<number>(0);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
loadDatasets();
|
||||
}, []);
|
||||
|
||||
const loadDatasets = async () => {
|
||||
try {
|
||||
const downloaded = await invoke<string[]>("list_datasets");
|
||||
const ds = STANDARD_DATASETS.map((d) => ({
|
||||
...d,
|
||||
downloaded: downloaded.includes(d.id),
|
||||
path: downloaded.includes(d.id) ? `~/.ruview/datasets/${d.id}` : null,
|
||||
}));
|
||||
setDatasets(ds);
|
||||
} catch (err) {
|
||||
// If command not implemented yet, show placeholders
|
||||
setDatasets(
|
||||
STANDARD_DATASETS.map((d) => ({
|
||||
...d,
|
||||
downloaded: false,
|
||||
path: null,
|
||||
}))
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
const handleDownload = async (datasetId: string) => {
|
||||
setDownloading(datasetId);
|
||||
setDownloadProgress(0);
|
||||
setError(null);
|
||||
|
||||
try {
|
||||
// Simulate download progress for now
|
||||
for (let i = 0; i <= 100; i += 10) {
|
||||
setDownloadProgress(i);
|
||||
await new Promise((r) => setTimeout(r, 500));
|
||||
}
|
||||
|
||||
// TODO: Call actual download command
|
||||
// await invoke("download_dataset", { datasetId });
|
||||
|
||||
setDatasets((prev) =>
|
||||
prev.map((d) =>
|
||||
d.id === datasetId
|
||||
? { ...d, downloaded: true, path: `~/.ruview/datasets/${d.id}` }
|
||||
: d
|
||||
)
|
||||
);
|
||||
} catch (err) {
|
||||
setError(`Download failed: ${err}`);
|
||||
} finally {
|
||||
setDownloading(null);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
<div>
|
||||
{/* Stats Row */}
|
||||
<div
|
||||
style={{
|
||||
display: "grid",
|
||||
gridTemplateColumns: "repeat(3, 1fr)",
|
||||
gap: "var(--space-4)",
|
||||
marginBottom: "var(--space-5)",
|
||||
}}
|
||||
>
|
||||
<StatCard
|
||||
label="Available Datasets"
|
||||
value={datasets.length}
|
||||
/>
|
||||
<StatCard
|
||||
label="Downloaded"
|
||||
value={datasets.filter((d) => d.downloaded).length}
|
||||
color="var(--status-online)"
|
||||
/>
|
||||
<StatCard
|
||||
label="Total Samples"
|
||||
value={`${(datasets.reduce((acc, d) => acc + (d.downloaded ? d.samples : 0), 0) / 1000).toFixed(0)}K`}
|
||||
/>
|
||||
</div>
|
||||
|
||||
{error && (
|
||||
<div
|
||||
style={{
|
||||
background: "rgba(248, 81, 73, 0.1)",
|
||||
border: "1px solid rgba(248, 81, 73, 0.3)",
|
||||
borderRadius: 6,
|
||||
padding: "var(--space-3)",
|
||||
marginBottom: "var(--space-4)",
|
||||
fontSize: 13,
|
||||
color: "var(--status-error)",
|
||||
}}
|
||||
>
|
||||
{error}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Dataset Cards */}
|
||||
<div
|
||||
style={{
|
||||
display: "grid",
|
||||
gridTemplateColumns: "repeat(auto-fill, minmax(340px, 1fr))",
|
||||
gap: "var(--space-4)",
|
||||
}}
|
||||
>
|
||||
{datasets.map((dataset) => (
|
||||
<div
|
||||
key={dataset.id}
|
||||
className="card"
|
||||
style={{
|
||||
padding: "var(--space-4)",
|
||||
opacity: dataset.downloaded ? 1 : 0.85,
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "start",
|
||||
marginBottom: "var(--space-3)",
|
||||
}}
|
||||
>
|
||||
<div>
|
||||
<h3 style={{ margin: 0, fontSize: 15, fontWeight: 600 }}>
|
||||
{dataset.name}
|
||||
</h3>
|
||||
<p
|
||||
style={{
|
||||
fontSize: 12,
|
||||
color: "var(--text-muted)",
|
||||
marginTop: 4,
|
||||
lineHeight: 1.4,
|
||||
}}
|
||||
>
|
||||
{dataset.description}
|
||||
</p>
|
||||
</div>
|
||||
{dataset.downloaded && (
|
||||
<span
|
||||
style={{
|
||||
background: "rgba(63, 185, 80, 0.15)",
|
||||
color: "var(--status-online)",
|
||||
padding: "2px 8px",
|
||||
borderRadius: 4,
|
||||
fontSize: 10,
|
||||
fontWeight: 600,
|
||||
}}
|
||||
>
|
||||
DOWNLOADED
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
gap: "var(--space-4)",
|
||||
fontSize: 12,
|
||||
color: "var(--text-secondary)",
|
||||
marginBottom: "var(--space-3)",
|
||||
}}
|
||||
>
|
||||
<span>📦 {(dataset.size_mb / 1024).toFixed(1)} GB</span>
|
||||
<span>📊 {(dataset.samples / 1000).toFixed(0)}K samples</span>
|
||||
</div>
|
||||
|
||||
{downloading === dataset.id ? (
|
||||
<div>
|
||||
<div
|
||||
style={{
|
||||
height: 4,
|
||||
background: "var(--border)",
|
||||
borderRadius: 2,
|
||||
overflow: "hidden",
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
width: `${downloadProgress}%`,
|
||||
height: "100%",
|
||||
background: "var(--accent)",
|
||||
transition: "width 0.3s",
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
fontSize: 11,
|
||||
color: "var(--text-muted)",
|
||||
marginTop: 4,
|
||||
textAlign: "center",
|
||||
}}
|
||||
>
|
||||
Downloading... {downloadProgress}%
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<div style={{ display: "flex", gap: "var(--space-2)" }}>
|
||||
{dataset.downloaded ? (
|
||||
<>
|
||||
<button
|
||||
style={{
|
||||
flex: 1,
|
||||
padding: "8px 12px",
|
||||
background: "rgba(56, 139, 253, 0.1)",
|
||||
border: "1px solid rgba(56, 139, 253, 0.3)",
|
||||
borderRadius: 6,
|
||||
color: "var(--accent)",
|
||||
fontSize: 12,
|
||||
fontWeight: 600,
|
||||
cursor: "pointer",
|
||||
}}
|
||||
>
|
||||
Preview
|
||||
</button>
|
||||
<button
|
||||
style={{
|
||||
padding: "8px 12px",
|
||||
background: "transparent",
|
||||
border: "1px solid var(--border)",
|
||||
borderRadius: 6,
|
||||
color: "var(--text-secondary)",
|
||||
fontSize: 12,
|
||||
fontWeight: 600,
|
||||
cursor: "pointer",
|
||||
}}
|
||||
>
|
||||
Delete
|
||||
</button>
|
||||
</>
|
||||
) : (
|
||||
<button
|
||||
onClick={() => handleDownload(dataset.id)}
|
||||
className="btn-gradient"
|
||||
style={{ flex: 1, fontSize: 12 }}
|
||||
>
|
||||
Download Dataset
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
{/* Import Custom Dataset */}
|
||||
<div
|
||||
className="card"
|
||||
style={{
|
||||
marginTop: "var(--space-5)",
|
||||
padding: "var(--space-4)",
|
||||
border: "2px dashed var(--border)",
|
||||
textAlign: "center",
|
||||
}}
|
||||
>
|
||||
<div style={{ fontSize: 32, marginBottom: "var(--space-2)" }}>📁</div>
|
||||
<h4 style={{ margin: 0, fontSize: 14, fontWeight: 600 }}>
|
||||
Import Custom Dataset
|
||||
</h4>
|
||||
<p
|
||||
style={{
|
||||
fontSize: 12,
|
||||
color: "var(--text-muted)",
|
||||
marginTop: 4,
|
||||
marginBottom: "var(--space-3)",
|
||||
}}
|
||||
>
|
||||
Import CSI recordings in CSV, NPZ, or HDF5 format
|
||||
</p>
|
||||
<button
|
||||
style={{
|
||||
padding: "8px 20px",
|
||||
background: "transparent",
|
||||
border: "1px solid var(--border)",
|
||||
borderRadius: 6,
|
||||
color: "var(--text-secondary)",
|
||||
fontSize: 12,
|
||||
fontWeight: 600,
|
||||
cursor: "pointer",
|
||||
}}
|
||||
>
|
||||
Browse Files
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
function StatCard({
|
||||
label,
|
||||
value,
|
||||
color,
|
||||
}: {
|
||||
label: string;
|
||||
value: number | string;
|
||||
color?: string;
|
||||
}) {
|
||||
return (
|
||||
<div className="card-glow" style={{ padding: "var(--space-4)" }}>
|
||||
<div
|
||||
style={{
|
||||
fontSize: 10,
|
||||
textTransform: "uppercase",
|
||||
letterSpacing: "0.06em",
|
||||
color: "var(--text-muted)",
|
||||
marginBottom: "var(--space-2)",
|
||||
fontWeight: 600,
|
||||
}}
|
||||
>
|
||||
{label}
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
fontFamily: "var(--font-mono)",
|
||||
fontSize: 28,
|
||||
fontWeight: 600,
|
||||
color: color || "var(--text-primary)",
|
||||
letterSpacing: "-0.02em",
|
||||
}}
|
||||
>
|
||||
{value}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default DatasetsTab;
|
||||
+609
@@ -0,0 +1,609 @@
|
||||
import React, { useState, useEffect } from "react";
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
|
||||
interface TrainingMetrics {
|
||||
epoch: number;
|
||||
train_loss: number;
|
||||
val_loss: number;
|
||||
train_acc: number;
|
||||
val_acc: number;
|
||||
learning_rate: number;
|
||||
timestamp: string;
|
||||
}
|
||||
|
||||
interface EvaluationMetrics {
|
||||
pck_05: number;
|
||||
pck_10: number;
|
||||
pck_20: number;
|
||||
map_50: number;
|
||||
map_75: number;
|
||||
iou: number;
|
||||
}
|
||||
|
||||
interface JointAccuracy {
|
||||
joint: string;
|
||||
accuracy: number;
|
||||
}
|
||||
|
||||
const JOINT_NAMES = [
|
||||
"nose",
|
||||
"left_eye",
|
||||
"right_eye",
|
||||
"left_ear",
|
||||
"right_ear",
|
||||
"left_shoulder",
|
||||
"right_shoulder",
|
||||
"left_elbow",
|
||||
"right_elbow",
|
||||
"left_wrist",
|
||||
"right_wrist",
|
||||
"left_hip",
|
||||
"right_hip",
|
||||
"left_knee",
|
||||
"right_knee",
|
||||
"left_ankle",
|
||||
"right_ankle",
|
||||
];
|
||||
|
||||
const MetricsTab: React.FC = () => {
|
||||
const [trainingHistory, setTrainingHistory] = useState<TrainingMetrics[]>([]);
|
||||
const [evaluation, setEvaluation] = useState<EvaluationMetrics | null>(null);
|
||||
const [jointAccuracies, setJointAccuracies] = useState<JointAccuracy[]>([]);
|
||||
const [selectedMetric, setSelectedMetric] = useState<"loss" | "accuracy">("loss");
|
||||
const [exporting, setExporting] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
loadMetrics();
|
||||
}, []);
|
||||
|
||||
const loadMetrics = async () => {
|
||||
try {
|
||||
const metrics = await invoke<TrainingMetrics[]>("get_training_history");
|
||||
setTrainingHistory(metrics);
|
||||
const evalMetrics = await invoke<EvaluationMetrics>("get_evaluation_metrics");
|
||||
setEvaluation(evalMetrics);
|
||||
const joints = await invoke<JointAccuracy[]>("get_joint_accuracies");
|
||||
setJointAccuracies(joints);
|
||||
} catch (err) {
|
||||
// Generate mock data for demonstration
|
||||
const mockHistory: TrainingMetrics[] = [];
|
||||
for (let i = 1; i <= 50; i++) {
|
||||
mockHistory.push({
|
||||
epoch: i,
|
||||
train_loss: 0.5 * Math.exp(-i / 20) + 0.02 + Math.random() * 0.01,
|
||||
val_loss: 0.55 * Math.exp(-i / 18) + 0.025 + Math.random() * 0.015,
|
||||
train_acc: 1 - 0.5 * Math.exp(-i / 15) - Math.random() * 0.02,
|
||||
val_acc: 1 - 0.55 * Math.exp(-i / 15) - Math.random() * 0.025,
|
||||
learning_rate: 0.001 * Math.pow(0.95, Math.floor(i / 10)),
|
||||
timestamp: new Date(Date.now() - (50 - i) * 60000).toISOString(),
|
||||
});
|
||||
}
|
||||
setTrainingHistory(mockHistory);
|
||||
|
||||
setEvaluation({
|
||||
pck_05: 0.72,
|
||||
pck_10: 0.89,
|
||||
pck_20: 0.96,
|
||||
map_50: 0.84,
|
||||
map_75: 0.71,
|
||||
iou: 0.78,
|
||||
});
|
||||
|
||||
setJointAccuracies(
|
||||
JOINT_NAMES.map((joint) => ({
|
||||
joint,
|
||||
accuracy: 0.7 + Math.random() * 0.25,
|
||||
}))
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
const exportMetrics = async (format: "csv" | "json" | "tensorboard") => {
|
||||
setExporting(true);
|
||||
try {
|
||||
if (format === "json") {
|
||||
const data = {
|
||||
training: trainingHistory,
|
||||
evaluation,
|
||||
joints: jointAccuracies,
|
||||
};
|
||||
const blob = new Blob([JSON.stringify(data, null, 2)], { type: "application/json" });
|
||||
downloadBlob(blob, "metrics.json");
|
||||
} else if (format === "csv") {
|
||||
const headers = "epoch,train_loss,val_loss,train_acc,val_acc,learning_rate\n";
|
||||
const rows = trainingHistory
|
||||
.map(
|
||||
(m) =>
|
||||
`${m.epoch},${m.train_loss.toFixed(6)},${m.val_loss.toFixed(6)},${m.train_acc.toFixed(4)},${m.val_acc.toFixed(4)},${m.learning_rate.toExponential(2)}`
|
||||
)
|
||||
.join("\n");
|
||||
const blob = new Blob([headers + rows], { type: "text/csv" });
|
||||
downloadBlob(blob, "training_history.csv");
|
||||
} else {
|
||||
// TensorBoard format would require server-side handling
|
||||
alert("TensorBoard export requires running the backend server");
|
||||
}
|
||||
} finally {
|
||||
setExporting(false);
|
||||
}
|
||||
};
|
||||
|
||||
const downloadBlob = (blob: Blob, filename: string) => {
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement("a");
|
||||
a.href = url;
|
||||
a.download = filename;
|
||||
a.click();
|
||||
URL.revokeObjectURL(url);
|
||||
};
|
||||
|
||||
const maxLoss = Math.max(
|
||||
...trainingHistory.map((m) => Math.max(m.train_loss, m.val_loss)),
|
||||
0.1
|
||||
);
|
||||
|
||||
return (
|
||||
<div>
|
||||
{/* Summary Stats */}
|
||||
<div
|
||||
style={{
|
||||
display: "grid",
|
||||
gridTemplateColumns: "repeat(4, 1fr)",
|
||||
gap: "var(--space-4)",
|
||||
marginBottom: "var(--space-5)",
|
||||
}}
|
||||
>
|
||||
<StatCard
|
||||
label="Epochs Trained"
|
||||
value={trainingHistory.length}
|
||||
/>
|
||||
<StatCard
|
||||
label="Best Val Loss"
|
||||
value={
|
||||
trainingHistory.length > 0
|
||||
? Math.min(...trainingHistory.map((m) => m.val_loss)).toFixed(4)
|
||||
: "—"
|
||||
}
|
||||
color="var(--status-online)"
|
||||
/>
|
||||
<StatCard
|
||||
label="Best Val Acc"
|
||||
value={
|
||||
trainingHistory.length > 0
|
||||
? `${(Math.max(...trainingHistory.map((m) => m.val_acc)) * 100).toFixed(1)}%`
|
||||
: "—"
|
||||
}
|
||||
color="var(--accent)"
|
||||
/>
|
||||
<StatCard
|
||||
label="PCK@0.1"
|
||||
value={evaluation ? `${(evaluation.pck_10 * 100).toFixed(1)}%` : "—"}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div style={{ display: "grid", gridTemplateColumns: "2fr 1fr", gap: "var(--space-5)" }}>
|
||||
{/* Loss/Accuracy Charts */}
|
||||
<div className="card" style={{ padding: "var(--space-4)" }}>
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "center",
|
||||
marginBottom: "var(--space-4)",
|
||||
}}
|
||||
>
|
||||
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600 }}>Training Curves</h3>
|
||||
<div style={{ display: "flex", gap: "var(--space-2)" }}>
|
||||
<button
|
||||
onClick={() => setSelectedMetric("loss")}
|
||||
style={{
|
||||
padding: "6px 12px",
|
||||
background: selectedMetric === "loss" ? "var(--accent)" : "transparent",
|
||||
border: `1px solid ${selectedMetric === "loss" ? "var(--accent)" : "var(--border)"}`,
|
||||
borderRadius: 4,
|
||||
color: selectedMetric === "loss" ? "white" : "var(--text-secondary)",
|
||||
fontSize: 11,
|
||||
fontWeight: 600,
|
||||
cursor: "pointer",
|
||||
}}
|
||||
>
|
||||
Loss
|
||||
</button>
|
||||
<button
|
||||
onClick={() => setSelectedMetric("accuracy")}
|
||||
style={{
|
||||
padding: "6px 12px",
|
||||
background: selectedMetric === "accuracy" ? "var(--accent)" : "transparent",
|
||||
border: `1px solid ${selectedMetric === "accuracy" ? "var(--accent)" : "var(--border)"}`,
|
||||
borderRadius: 4,
|
||||
color: selectedMetric === "accuracy" ? "white" : "var(--text-secondary)",
|
||||
fontSize: 11,
|
||||
fontWeight: 600,
|
||||
cursor: "pointer",
|
||||
}}
|
||||
>
|
||||
Accuracy
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Chart Area */}
|
||||
<div
|
||||
style={{
|
||||
height: 250,
|
||||
position: "relative",
|
||||
background: "var(--bg-secondary)",
|
||||
borderRadius: 8,
|
||||
padding: "var(--space-3)",
|
||||
}}
|
||||
>
|
||||
{trainingHistory.length === 0 ? (
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
alignItems: "center",
|
||||
justifyContent: "center",
|
||||
height: "100%",
|
||||
color: "var(--text-muted)",
|
||||
}}
|
||||
>
|
||||
<span style={{ fontSize: 32 }}>📊</span>
|
||||
<p style={{ fontSize: 13, marginTop: "var(--space-2)" }}>
|
||||
No training data yet
|
||||
</p>
|
||||
</div>
|
||||
) : (
|
||||
<svg width="100%" height="100%" viewBox="0 0 500 200" preserveAspectRatio="none">
|
||||
{/* Grid lines */}
|
||||
{[0, 0.25, 0.5, 0.75, 1].map((y) => (
|
||||
<line
|
||||
key={y}
|
||||
x1="0"
|
||||
y1={y * 180}
|
||||
x2="500"
|
||||
y2={y * 180}
|
||||
stroke="var(--border)"
|
||||
strokeWidth="0.5"
|
||||
strokeDasharray="4"
|
||||
/>
|
||||
))}
|
||||
|
||||
{/* Train line */}
|
||||
<polyline
|
||||
fill="none"
|
||||
stroke="var(--accent)"
|
||||
strokeWidth="2"
|
||||
points={trainingHistory
|
||||
.map((m, i) => {
|
||||
const x = (i / (trainingHistory.length - 1)) * 500;
|
||||
const value = selectedMetric === "loss" ? m.train_loss : m.train_acc;
|
||||
const y =
|
||||
selectedMetric === "loss"
|
||||
? (value / maxLoss) * 180
|
||||
: (1 - value) * 180;
|
||||
return `${x},${y}`;
|
||||
})
|
||||
.join(" ")}
|
||||
/>
|
||||
|
||||
{/* Val line */}
|
||||
<polyline
|
||||
fill="none"
|
||||
stroke="var(--status-online)"
|
||||
strokeWidth="2"
|
||||
points={trainingHistory
|
||||
.map((m, i) => {
|
||||
const x = (i / (trainingHistory.length - 1)) * 500;
|
||||
const value = selectedMetric === "loss" ? m.val_loss : m.val_acc;
|
||||
const y =
|
||||
selectedMetric === "loss"
|
||||
? (value / maxLoss) * 180
|
||||
: (1 - value) * 180;
|
||||
return `${x},${y}`;
|
||||
})
|
||||
.join(" ")}
|
||||
/>
|
||||
</svg>
|
||||
)}
|
||||
|
||||
{/* Legend */}
|
||||
<div
|
||||
style={{
|
||||
position: "absolute",
|
||||
top: "var(--space-2)",
|
||||
right: "var(--space-2)",
|
||||
display: "flex",
|
||||
gap: "var(--space-3)",
|
||||
fontSize: 11,
|
||||
}}
|
||||
>
|
||||
<span style={{ display: "flex", alignItems: "center", gap: 4 }}>
|
||||
<span
|
||||
style={{
|
||||
width: 12,
|
||||
height: 3,
|
||||
background: "var(--accent)",
|
||||
borderRadius: 2,
|
||||
}}
|
||||
/>
|
||||
Train
|
||||
</span>
|
||||
<span style={{ display: "flex", alignItems: "center", gap: 4 }}>
|
||||
<span
|
||||
style={{
|
||||
width: 12,
|
||||
height: 3,
|
||||
background: "var(--status-online)",
|
||||
borderRadius: 2,
|
||||
}}
|
||||
/>
|
||||
Validation
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Evaluation Metrics */}
|
||||
<div className="card" style={{ padding: "var(--space-4)" }}>
|
||||
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
|
||||
Evaluation Metrics
|
||||
</h3>
|
||||
|
||||
{!evaluation ? (
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
alignItems: "center",
|
||||
justifyContent: "center",
|
||||
height: 200,
|
||||
color: "var(--text-muted)",
|
||||
}}
|
||||
>
|
||||
<span style={{ fontSize: 32 }}>📏</span>
|
||||
<p style={{ fontSize: 13, marginTop: "var(--space-2)" }}>
|
||||
Run evaluation to see metrics
|
||||
</p>
|
||||
</div>
|
||||
) : (
|
||||
<div style={{ display: "flex", flexDirection: "column", gap: "var(--space-3)" }}>
|
||||
<MetricBar label="PCK@0.05" value={evaluation.pck_05} color="#f59e0b" />
|
||||
<MetricBar label="PCK@0.10" value={evaluation.pck_10} color="var(--accent)" />
|
||||
<MetricBar label="PCK@0.20" value={evaluation.pck_20} color="var(--status-online)" />
|
||||
<div style={{ height: 1, background: "var(--border)", margin: "var(--space-2) 0" }} />
|
||||
<MetricBar label="mAP@0.50" value={evaluation.map_50} color="#a855f7" />
|
||||
<MetricBar label="mAP@0.75" value={evaluation.map_75} color="#ec4899" />
|
||||
<MetricBar label="IoU" value={evaluation.iou} color="#06b6d4" />
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Joint-wise Accuracy */}
|
||||
<div className="card" style={{ marginTop: "var(--space-5)", padding: "var(--space-4)" }}>
|
||||
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
|
||||
Per-Joint Accuracy
|
||||
</h3>
|
||||
|
||||
{jointAccuracies.length === 0 ? (
|
||||
<div
|
||||
style={{
|
||||
textAlign: "center",
|
||||
padding: "var(--space-5)",
|
||||
color: "var(--text-muted)",
|
||||
}}
|
||||
>
|
||||
No joint accuracy data available
|
||||
</div>
|
||||
) : (
|
||||
<div
|
||||
style={{
|
||||
display: "grid",
|
||||
gridTemplateColumns: "repeat(auto-fill, minmax(140px, 1fr))",
|
||||
gap: "var(--space-3)",
|
||||
}}
|
||||
>
|
||||
{jointAccuracies.map((ja) => (
|
||||
<div
|
||||
key={ja.joint}
|
||||
style={{
|
||||
padding: "var(--space-3)",
|
||||
background: "var(--bg-secondary)",
|
||||
borderRadius: 6,
|
||||
textAlign: "center",
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
fontSize: 11,
|
||||
color: "var(--text-muted)",
|
||||
marginBottom: 4,
|
||||
textTransform: "capitalize",
|
||||
}}
|
||||
>
|
||||
{ja.joint.replace("_", " ")}
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
fontFamily: "var(--font-mono)",
|
||||
fontSize: 18,
|
||||
fontWeight: 600,
|
||||
color:
|
||||
ja.accuracy > 0.9
|
||||
? "var(--status-online)"
|
||||
: ja.accuracy > 0.8
|
||||
? "var(--accent)"
|
||||
: ja.accuracy > 0.7
|
||||
? "#f59e0b"
|
||||
: "var(--status-error)",
|
||||
}}
|
||||
>
|
||||
{(ja.accuracy * 100).toFixed(1)}%
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{/* Export Section */}
|
||||
<div
|
||||
className="card"
|
||||
style={{
|
||||
marginTop: "var(--space-5)",
|
||||
padding: "var(--space-4)",
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "center",
|
||||
}}
|
||||
>
|
||||
<div>
|
||||
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600 }}>Export Metrics</h3>
|
||||
<p style={{ fontSize: 12, color: "var(--text-muted)", marginTop: 4 }}>
|
||||
Download training history and evaluation results
|
||||
</p>
|
||||
</div>
|
||||
<div style={{ display: "flex", gap: "var(--space-2)" }}>
|
||||
<button
|
||||
onClick={() => exportMetrics("csv")}
|
||||
disabled={exporting || trainingHistory.length === 0}
|
||||
style={{
|
||||
padding: "8px 16px",
|
||||
background: "rgba(56, 139, 253, 0.1)",
|
||||
border: "1px solid rgba(56, 139, 253, 0.3)",
|
||||
borderRadius: 6,
|
||||
color: "var(--accent)",
|
||||
fontSize: 12,
|
||||
fontWeight: 600,
|
||||
cursor: trainingHistory.length === 0 ? "not-allowed" : "pointer",
|
||||
opacity: trainingHistory.length === 0 ? 0.5 : 1,
|
||||
}}
|
||||
>
|
||||
CSV
|
||||
</button>
|
||||
<button
|
||||
onClick={() => exportMetrics("json")}
|
||||
disabled={exporting || trainingHistory.length === 0}
|
||||
style={{
|
||||
padding: "8px 16px",
|
||||
background: "rgba(56, 139, 253, 0.1)",
|
||||
border: "1px solid rgba(56, 139, 253, 0.3)",
|
||||
borderRadius: 6,
|
||||
color: "var(--accent)",
|
||||
fontSize: 12,
|
||||
fontWeight: 600,
|
||||
cursor: trainingHistory.length === 0 ? "not-allowed" : "pointer",
|
||||
opacity: trainingHistory.length === 0 ? 0.5 : 1,
|
||||
}}
|
||||
>
|
||||
JSON
|
||||
</button>
|
||||
<button
|
||||
onClick={() => exportMetrics("tensorboard")}
|
||||
disabled={exporting || trainingHistory.length === 0}
|
||||
style={{
|
||||
padding: "8px 16px",
|
||||
background: "transparent",
|
||||
border: "1px solid var(--border)",
|
||||
borderRadius: 6,
|
||||
color: "var(--text-secondary)",
|
||||
fontSize: 12,
|
||||
fontWeight: 600,
|
||||
cursor: trainingHistory.length === 0 ? "not-allowed" : "pointer",
|
||||
opacity: trainingHistory.length === 0 ? 0.5 : 1,
|
||||
}}
|
||||
>
|
||||
TensorBoard
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
function StatCard({
|
||||
label,
|
||||
value,
|
||||
color,
|
||||
}: {
|
||||
label: string;
|
||||
value: number | string;
|
||||
color?: string;
|
||||
}) {
|
||||
return (
|
||||
<div className="card-glow" style={{ padding: "var(--space-4)" }}>
|
||||
<div
|
||||
style={{
|
||||
fontSize: 10,
|
||||
textTransform: "uppercase",
|
||||
letterSpacing: "0.06em",
|
||||
color: "var(--text-muted)",
|
||||
marginBottom: "var(--space-2)",
|
||||
fontWeight: 600,
|
||||
}}
|
||||
>
|
||||
{label}
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
fontFamily: "var(--font-mono)",
|
||||
fontSize: 28,
|
||||
fontWeight: 600,
|
||||
color: color || "var(--text-primary)",
|
||||
letterSpacing: "-0.02em",
|
||||
}}
|
||||
>
|
||||
{value}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function MetricBar({
|
||||
label,
|
||||
value,
|
||||
color,
|
||||
}: {
|
||||
label: string;
|
||||
value: number;
|
||||
color: string;
|
||||
}) {
|
||||
return (
|
||||
<div>
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
fontSize: 12,
|
||||
marginBottom: 4,
|
||||
}}
|
||||
>
|
||||
<span>{label}</span>
|
||||
<span style={{ fontFamily: "var(--font-mono)", fontWeight: 600 }}>
|
||||
{(value * 100).toFixed(1)}%
|
||||
</span>
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
height: 6,
|
||||
background: "var(--bg-secondary)",
|
||||
borderRadius: 3,
|
||||
overflow: "hidden",
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
width: `${value * 100}%`,
|
||||
height: "100%",
|
||||
background: color,
|
||||
borderRadius: 3,
|
||||
transition: "width 0.5s",
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default MetricsTab;
|
||||
+405
@@ -0,0 +1,405 @@
|
||||
import React, { useState, useEffect } from "react";
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
|
||||
interface ModelArchitecture {
|
||||
id: string;
|
||||
name: string;
|
||||
type: "encoder" | "decoder" | "embedding" | "adaptor";
|
||||
description: string;
|
||||
params_m: number;
|
||||
memory_mb: number;
|
||||
paper?: string;
|
||||
}
|
||||
|
||||
interface Checkpoint {
|
||||
id: string;
|
||||
model_id: string;
|
||||
name: string;
|
||||
epoch: number;
|
||||
val_loss: number;
|
||||
created_at: string;
|
||||
path: string;
|
||||
size_mb: number;
|
||||
}
|
||||
|
||||
const MODEL_ARCHITECTURES: ModelArchitecture[] = [
|
||||
{
|
||||
id: "csi-encoder-cnn",
|
||||
name: "CSI Encoder (CNN)",
|
||||
type: "encoder",
|
||||
description: "Convolutional encoder for CSI amplitude/phase features",
|
||||
params_m: 2.3,
|
||||
memory_mb: 128,
|
||||
},
|
||||
{
|
||||
id: "csi-encoder-transformer",
|
||||
name: "CSI Encoder (Transformer)",
|
||||
type: "encoder",
|
||||
description: "Self-attention based CSI feature extraction",
|
||||
params_m: 8.5,
|
||||
memory_mb: 384,
|
||||
paper: "WiFi-ViT 2024",
|
||||
},
|
||||
{
|
||||
id: "pose-decoder-lstm",
|
||||
name: "Pose Decoder (LSTM)",
|
||||
type: "decoder",
|
||||
description: "Recurrent decoder for temporal pose estimation",
|
||||
params_m: 1.8,
|
||||
memory_mb: 96,
|
||||
},
|
||||
{
|
||||
id: "pose-decoder-gru",
|
||||
name: "Pose Decoder (GRU)",
|
||||
type: "decoder",
|
||||
description: "Gated recurrent unit pose decoder (faster)",
|
||||
params_m: 1.2,
|
||||
memory_mb: 64,
|
||||
},
|
||||
{
|
||||
id: "aether-embedding",
|
||||
name: "AETHER Embedding",
|
||||
type: "embedding",
|
||||
description: "Contrastive CSI embedding for person re-identification (ADR-024)",
|
||||
params_m: 4.2,
|
||||
memory_mb: 192,
|
||||
paper: "AETHER 2025",
|
||||
},
|
||||
{
|
||||
id: "meridian-adaptor",
|
||||
name: "MERIDIAN Adaptor",
|
||||
type: "adaptor",
|
||||
description: "Cross-environment domain generalization module (ADR-027)",
|
||||
params_m: 3.1,
|
||||
memory_mb: 144,
|
||||
paper: "MERIDIAN 2025",
|
||||
},
|
||||
];
|
||||
|
||||
const ModelsTab: React.FC = () => {
|
||||
const [checkpoints, setCheckpoints] = useState<Checkpoint[]>([]);
|
||||
const [selectedModel, setSelectedModel] = useState<string | null>(null);
|
||||
const [exporting, setExporting] = useState<string | null>(null);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
loadCheckpoints();
|
||||
}, []);
|
||||
|
||||
const loadCheckpoints = async () => {
|
||||
try {
|
||||
const loaded = await invoke<Checkpoint[]>("list_checkpoints");
|
||||
setCheckpoints(loaded);
|
||||
} catch (err) {
|
||||
// Mock data if command not implemented
|
||||
setCheckpoints([
|
||||
{
|
||||
id: "ckpt-001",
|
||||
model_id: "csi-encoder-cnn",
|
||||
name: "CSI-CNN v1.2",
|
||||
epoch: 50,
|
||||
val_loss: 0.0234,
|
||||
created_at: "2026-03-08T14:30:00Z",
|
||||
path: "~/.ruview/models/csi-cnn-v1.2.pt",
|
||||
size_mb: 12.4,
|
||||
},
|
||||
{
|
||||
id: "ckpt-002",
|
||||
model_id: "pose-decoder-gru",
|
||||
name: "Pose-GRU v2.0",
|
||||
epoch: 100,
|
||||
val_loss: 0.0189,
|
||||
created_at: "2026-03-09T09:15:00Z",
|
||||
path: "~/.ruview/models/pose-gru-v2.pt",
|
||||
size_mb: 8.2,
|
||||
},
|
||||
]);
|
||||
}
|
||||
};
|
||||
|
||||
const handleExport = async (checkpointId: string, format: "onnx" | "torchscript") => {
|
||||
setExporting(checkpointId);
|
||||
setError(null);
|
||||
try {
|
||||
await invoke("export_model", { checkpointId, format });
|
||||
// Success notification would go here
|
||||
} catch (err) {
|
||||
setError(`Export failed: ${err}`);
|
||||
} finally {
|
||||
setExporting(null);
|
||||
}
|
||||
};
|
||||
|
||||
const getTypeColor = (type: ModelArchitecture["type"]) => {
|
||||
switch (type) {
|
||||
case "encoder":
|
||||
return "var(--accent)";
|
||||
case "decoder":
|
||||
return "var(--status-online)";
|
||||
case "embedding":
|
||||
return "#a855f7";
|
||||
case "adaptor":
|
||||
return "#f59e0b";
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
<div>
|
||||
{/* Stats Row */}
|
||||
<div
|
||||
style={{
|
||||
display: "grid",
|
||||
gridTemplateColumns: "repeat(4, 1fr)",
|
||||
gap: "var(--space-4)",
|
||||
marginBottom: "var(--space-5)",
|
||||
}}
|
||||
>
|
||||
<StatCard label="Architectures" value={MODEL_ARCHITECTURES.length} />
|
||||
<StatCard
|
||||
label="Checkpoints"
|
||||
value={checkpoints.length}
|
||||
color="var(--status-online)"
|
||||
/>
|
||||
<StatCard
|
||||
label="Total Params"
|
||||
value={`${MODEL_ARCHITECTURES.reduce((acc, m) => acc + m.params_m, 0).toFixed(1)}M`}
|
||||
/>
|
||||
<StatCard
|
||||
label="Storage Used"
|
||||
value={`${checkpoints.reduce((acc, c) => acc + c.size_mb, 0).toFixed(1)} MB`}
|
||||
/>
|
||||
</div>
|
||||
|
||||
{error && (
|
||||
<div
|
||||
style={{
|
||||
background: "rgba(248, 81, 73, 0.1)",
|
||||
border: "1px solid rgba(248, 81, 73, 0.3)",
|
||||
borderRadius: 6,
|
||||
padding: "var(--space-3)",
|
||||
marginBottom: "var(--space-4)",
|
||||
fontSize: 13,
|
||||
color: "var(--status-error)",
|
||||
}}
|
||||
>
|
||||
{error}
|
||||
</div>
|
||||
)}
|
||||
|
||||
{/* Model Architectures */}
|
||||
<h3 style={{ fontSize: 14, fontWeight: 600, marginBottom: "var(--space-3)" }}>
|
||||
Available Architectures
|
||||
</h3>
|
||||
<div
|
||||
style={{
|
||||
display: "grid",
|
||||
gridTemplateColumns: "repeat(auto-fill, minmax(300px, 1fr))",
|
||||
gap: "var(--space-3)",
|
||||
marginBottom: "var(--space-5)",
|
||||
}}
|
||||
>
|
||||
{MODEL_ARCHITECTURES.map((model) => (
|
||||
<div
|
||||
key={model.id}
|
||||
className="card"
|
||||
style={{
|
||||
padding: "var(--space-3)",
|
||||
cursor: "pointer",
|
||||
border:
|
||||
selectedModel === model.id
|
||||
? "1px solid var(--accent)"
|
||||
: "1px solid transparent",
|
||||
}}
|
||||
onClick={() => setSelectedModel(model.id)}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "start",
|
||||
marginBottom: "var(--space-2)",
|
||||
}}
|
||||
>
|
||||
<div>
|
||||
<h4 style={{ margin: 0, fontSize: 14, fontWeight: 600 }}>
|
||||
{model.name}
|
||||
</h4>
|
||||
<span
|
||||
style={{
|
||||
display: "inline-block",
|
||||
marginTop: 4,
|
||||
padding: "1px 6px",
|
||||
borderRadius: 3,
|
||||
fontSize: 10,
|
||||
fontWeight: 600,
|
||||
textTransform: "uppercase",
|
||||
background: `${getTypeColor(model.type)}20`,
|
||||
color: getTypeColor(model.type),
|
||||
}}
|
||||
>
|
||||
{model.type}
|
||||
</span>
|
||||
</div>
|
||||
{model.paper && (
|
||||
<span
|
||||
style={{
|
||||
fontSize: 10,
|
||||
color: "var(--text-muted)",
|
||||
fontStyle: "italic",
|
||||
}}
|
||||
>
|
||||
{model.paper}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
<p
|
||||
style={{
|
||||
fontSize: 11,
|
||||
color: "var(--text-muted)",
|
||||
margin: "var(--space-2) 0",
|
||||
lineHeight: 1.4,
|
||||
}}
|
||||
>
|
||||
{model.description}
|
||||
</p>
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
gap: "var(--space-3)",
|
||||
fontSize: 11,
|
||||
color: "var(--text-secondary)",
|
||||
}}
|
||||
>
|
||||
<span>🧮 {model.params_m}M params</span>
|
||||
<span>💾 {model.memory_mb} MB</span>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
{/* Checkpoints */}
|
||||
<h3 style={{ fontSize: 14, fontWeight: 600, marginBottom: "var(--space-3)" }}>
|
||||
Saved Checkpoints
|
||||
</h3>
|
||||
{checkpoints.length === 0 ? (
|
||||
<div
|
||||
className="card"
|
||||
style={{
|
||||
padding: "var(--space-5)",
|
||||
textAlign: "center",
|
||||
color: "var(--text-muted)",
|
||||
}}
|
||||
>
|
||||
<div style={{ fontSize: 32, marginBottom: "var(--space-2)" }}>📦</div>
|
||||
<p style={{ fontSize: 13 }}>No checkpoints saved yet</p>
|
||||
<p style={{ fontSize: 12 }}>Train a model to create checkpoints</p>
|
||||
</div>
|
||||
) : (
|
||||
<div style={{ display: "flex", flexDirection: "column", gap: "var(--space-2)" }}>
|
||||
{checkpoints.map((ckpt) => (
|
||||
<div
|
||||
key={ckpt.id}
|
||||
className="card"
|
||||
style={{
|
||||
padding: "var(--space-3)",
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "center",
|
||||
}}
|
||||
>
|
||||
<div>
|
||||
<div style={{ fontWeight: 600, fontSize: 13 }}>{ckpt.name}</div>
|
||||
<div
|
||||
style={{
|
||||
fontSize: 11,
|
||||
color: "var(--text-muted)",
|
||||
marginTop: 2,
|
||||
}}
|
||||
>
|
||||
Epoch {ckpt.epoch} • Val Loss: {ckpt.val_loss.toFixed(4)} •{" "}
|
||||
{ckpt.size_mb.toFixed(1)} MB
|
||||
</div>
|
||||
</div>
|
||||
<div style={{ display: "flex", gap: "var(--space-2)" }}>
|
||||
<button
|
||||
onClick={() => handleExport(ckpt.id, "onnx")}
|
||||
disabled={exporting === ckpt.id}
|
||||
style={{
|
||||
padding: "6px 12px",
|
||||
background: "rgba(56, 139, 253, 0.1)",
|
||||
border: "1px solid rgba(56, 139, 253, 0.3)",
|
||||
borderRadius: 4,
|
||||
color: "var(--accent)",
|
||||
fontSize: 11,
|
||||
fontWeight: 600,
|
||||
cursor: exporting === ckpt.id ? "wait" : "pointer",
|
||||
opacity: exporting === ckpt.id ? 0.6 : 1,
|
||||
}}
|
||||
>
|
||||
{exporting === ckpt.id ? "Exporting..." : "ONNX"}
|
||||
</button>
|
||||
<button
|
||||
onClick={() => handleExport(ckpt.id, "torchscript")}
|
||||
disabled={exporting === ckpt.id}
|
||||
style={{
|
||||
padding: "6px 12px",
|
||||
background: "transparent",
|
||||
border: "1px solid var(--border)",
|
||||
borderRadius: 4,
|
||||
color: "var(--text-secondary)",
|
||||
fontSize: 11,
|
||||
fontWeight: 600,
|
||||
cursor: exporting === ckpt.id ? "wait" : "pointer",
|
||||
opacity: exporting === ckpt.id ? 0.6 : 1,
|
||||
}}
|
||||
>
|
||||
TorchScript
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
function StatCard({
|
||||
label,
|
||||
value,
|
||||
color,
|
||||
}: {
|
||||
label: string;
|
||||
value: number | string;
|
||||
color?: string;
|
||||
}) {
|
||||
return (
|
||||
<div className="card-glow" style={{ padding: "var(--space-4)" }}>
|
||||
<div
|
||||
style={{
|
||||
fontSize: 10,
|
||||
textTransform: "uppercase",
|
||||
letterSpacing: "0.06em",
|
||||
color: "var(--text-muted)",
|
||||
marginBottom: "var(--space-2)",
|
||||
fontWeight: 600,
|
||||
}}
|
||||
>
|
||||
{label}
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
fontFamily: "var(--font-mono)",
|
||||
fontSize: 28,
|
||||
fontWeight: 600,
|
||||
color: color || "var(--text-primary)",
|
||||
letterSpacing: "-0.02em",
|
||||
}}
|
||||
>
|
||||
{value}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export default ModelsTab;
|
||||
+767
@@ -0,0 +1,767 @@
|
||||
import React, { useState, useEffect } from "react";
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
|
||||
interface RuVectorConfig {
|
||||
// MinCut Parameters
|
||||
mincut_enabled: boolean;
|
||||
mincut_threshold: number;
|
||||
mincut_max_persons: number;
|
||||
|
||||
// Attention Parameters
|
||||
attention_enabled: boolean;
|
||||
attention_heads: number;
|
||||
attention_dropout: number;
|
||||
|
||||
// Temporal Parameters
|
||||
temporal_enabled: boolean;
|
||||
temporal_window_ms: number;
|
||||
temporal_compression_ratio: number;
|
||||
|
||||
// Solver Parameters
|
||||
solver_enabled: boolean;
|
||||
solver_interpolation: "linear" | "cubic" | "sparse";
|
||||
solver_subcarrier_count: number;
|
||||
|
||||
// BVP Parameters
|
||||
bvp_enabled: boolean;
|
||||
bvp_filter_hz: [number, number];
|
||||
}
|
||||
|
||||
const DEFAULT_CONFIG: RuVectorConfig = {
|
||||
mincut_enabled: true,
|
||||
mincut_threshold: 0.5,
|
||||
mincut_max_persons: 5,
|
||||
attention_enabled: true,
|
||||
attention_heads: 4,
|
||||
attention_dropout: 0.1,
|
||||
temporal_enabled: true,
|
||||
temporal_window_ms: 500,
|
||||
temporal_compression_ratio: 4,
|
||||
solver_enabled: true,
|
||||
solver_interpolation: "sparse",
|
||||
solver_subcarrier_count: 56,
|
||||
bvp_enabled: false,
|
||||
bvp_filter_hz: [0.7, 4.0],
|
||||
};
|
||||
|
||||
const MODULES = [
|
||||
{
|
||||
id: "mincut",
|
||||
name: "MinCut Segmentation",
|
||||
crate: "ruvector-mincut",
|
||||
description: "Graph-based person segmentation using DynamicPersonMatcher",
|
||||
icon: "✂️",
|
||||
},
|
||||
{
|
||||
id: "attention",
|
||||
name: "Spatial Attention",
|
||||
crate: "ruvector-attention",
|
||||
description: "Attention-weighted antenna selection and BVP extraction",
|
||||
icon: "🎯",
|
||||
},
|
||||
{
|
||||
id: "temporal",
|
||||
name: "Temporal Tensor",
|
||||
crate: "ruvector-temporal-tensor",
|
||||
description: "Temporal CSI compression and breathing detection",
|
||||
icon: "⏱️",
|
||||
},
|
||||
{
|
||||
id: "solver",
|
||||
name: "Sparse Solver",
|
||||
crate: "ruvector-solver",
|
||||
description: "Sparse interpolation (114→56 subcarriers) and triangulation",
|
||||
icon: "🧮",
|
||||
},
|
||||
{
|
||||
id: "attn-mincut",
|
||||
name: "Attention MinCut",
|
||||
crate: "ruvector-attn-mincut",
|
||||
description: "Combined attention-weighted graph segmentation",
|
||||
icon: "🔀",
|
||||
},
|
||||
];
|
||||
|
||||
const RuVectorTab: React.FC = () => {
|
||||
const [config, setConfig] = useState<RuVectorConfig>(DEFAULT_CONFIG);
|
||||
const [testingLive, setTestingLive] = useState(false);
|
||||
const [liveMetrics, setLiveMetrics] = useState<{
|
||||
fps: number;
|
||||
latency_ms: number;
|
||||
persons_detected: number;
|
||||
} | null>(null);
|
||||
const [saved, setSaved] = useState(true);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
loadConfig();
|
||||
}, []);
|
||||
|
||||
const loadConfig = async () => {
|
||||
try {
|
||||
const loaded = await invoke<RuVectorConfig>("get_ruvector_config");
|
||||
setConfig(loaded);
|
||||
} catch (err) {
|
||||
// Use defaults if not implemented
|
||||
}
|
||||
};
|
||||
|
||||
const saveConfig = async () => {
|
||||
setError(null);
|
||||
try {
|
||||
await invoke("set_ruvector_config", { config });
|
||||
setSaved(true);
|
||||
} catch (err) {
|
||||
setError(`Failed to save: ${err}`);
|
||||
}
|
||||
};
|
||||
|
||||
const handleChange = <K extends keyof RuVectorConfig>(
|
||||
key: K,
|
||||
value: RuVectorConfig[K]
|
||||
) => {
|
||||
setConfig((prev) => ({ ...prev, [key]: value }));
|
||||
setSaved(false);
|
||||
};
|
||||
|
||||
const startLiveTest = async () => {
|
||||
setTestingLive(true);
|
||||
setError(null);
|
||||
try {
|
||||
// Simulate live testing metrics
|
||||
const interval = setInterval(() => {
|
||||
setLiveMetrics({
|
||||
fps: 25 + Math.random() * 10,
|
||||
latency_ms: 15 + Math.random() * 10,
|
||||
persons_detected: Math.floor(Math.random() * 3) + 1,
|
||||
});
|
||||
}, 500);
|
||||
|
||||
// Stop after 10 seconds for demo
|
||||
setTimeout(() => {
|
||||
clearInterval(interval);
|
||||
setTestingLive(false);
|
||||
setLiveMetrics(null);
|
||||
}, 10000);
|
||||
} catch (err) {
|
||||
setError(`Live test failed: ${err}`);
|
||||
setTestingLive(false);
|
||||
}
|
||||
};
|
||||
|
||||
const exportConfig = () => {
|
||||
const blob = new Blob([JSON.stringify(config, null, 2)], {
|
||||
type: "application/json",
|
||||
});
|
||||
const url = URL.createObjectURL(blob);
|
||||
const a = document.createElement("a");
|
||||
a.href = url;
|
||||
a.download = "ruvector-config.json";
|
||||
a.click();
|
||||
URL.revokeObjectURL(url);
|
||||
};
|
||||
|
||||
return (
|
||||
<div>
|
||||
{/* Module Cards */}
|
||||
<div
|
||||
style={{
|
||||
display: "grid",
|
||||
gridTemplateColumns: "repeat(auto-fill, minmax(200px, 1fr))",
|
||||
gap: "var(--space-3)",
|
||||
marginBottom: "var(--space-5)",
|
||||
}}
|
||||
>
|
||||
{MODULES.map((mod) => {
|
||||
const isEnabled =
|
||||
config[`${mod.id.replace("-", "_")}_enabled` as keyof RuVectorConfig] ?? true;
|
||||
return (
|
||||
<div
|
||||
key={mod.id}
|
||||
className="card"
|
||||
style={{
|
||||
padding: "var(--space-3)",
|
||||
opacity: isEnabled ? 1 : 0.5,
|
||||
transition: "opacity 0.2s",
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "start",
|
||||
}}
|
||||
>
|
||||
<span style={{ fontSize: 24 }}>{mod.icon}</span>
|
||||
<span
|
||||
style={{
|
||||
fontSize: 9,
|
||||
padding: "2px 6px",
|
||||
borderRadius: 3,
|
||||
background: isEnabled
|
||||
? "rgba(63, 185, 80, 0.15)"
|
||||
: "rgba(139, 148, 158, 0.15)",
|
||||
color: isEnabled ? "var(--status-online)" : "var(--text-muted)",
|
||||
fontWeight: 600,
|
||||
}}
|
||||
>
|
||||
{isEnabled ? "ON" : "OFF"}
|
||||
</span>
|
||||
</div>
|
||||
<h4 style={{ margin: "var(--space-2) 0 4px", fontSize: 13, fontWeight: 600 }}>
|
||||
{mod.name}
|
||||
</h4>
|
||||
<p
|
||||
style={{
|
||||
fontSize: 11,
|
||||
color: "var(--text-muted)",
|
||||
margin: 0,
|
||||
lineHeight: 1.4,
|
||||
}}
|
||||
>
|
||||
{mod.description}
|
||||
</p>
|
||||
<div
|
||||
style={{
|
||||
marginTop: "var(--space-2)",
|
||||
fontFamily: "var(--font-mono)",
|
||||
fontSize: 10,
|
||||
color: "var(--text-secondary)",
|
||||
}}
|
||||
>
|
||||
{mod.crate}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
|
||||
{error && (
|
||||
<div
|
||||
style={{
|
||||
background: "rgba(248, 81, 73, 0.1)",
|
||||
border: "1px solid rgba(248, 81, 73, 0.3)",
|
||||
borderRadius: 6,
|
||||
padding: "var(--space-3)",
|
||||
marginBottom: "var(--space-4)",
|
||||
fontSize: 13,
|
||||
color: "var(--status-error)",
|
||||
}}
|
||||
>
|
||||
{error}
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div style={{ display: "grid", gridTemplateColumns: "1fr 1fr", gap: "var(--space-5)" }}>
|
||||
{/* Configuration Panel */}
|
||||
<div className="card" style={{ padding: "var(--space-4)" }}>
|
||||
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
|
||||
Module Configuration
|
||||
</h3>
|
||||
|
||||
{/* MinCut Section */}
|
||||
<ConfigSection title="MinCut Segmentation">
|
||||
<ToggleRow
|
||||
label="Enable MinCut"
|
||||
checked={config.mincut_enabled}
|
||||
onChange={(v) => handleChange("mincut_enabled", v)}
|
||||
/>
|
||||
<SliderRow
|
||||
label="Threshold"
|
||||
value={config.mincut_threshold}
|
||||
min={0.1}
|
||||
max={1.0}
|
||||
step={0.05}
|
||||
onChange={(v) => handleChange("mincut_threshold", v)}
|
||||
disabled={!config.mincut_enabled}
|
||||
/>
|
||||
<NumberRow
|
||||
label="Max Persons"
|
||||
value={config.mincut_max_persons}
|
||||
min={1}
|
||||
max={10}
|
||||
onChange={(v) => handleChange("mincut_max_persons", v)}
|
||||
disabled={!config.mincut_enabled}
|
||||
/>
|
||||
</ConfigSection>
|
||||
|
||||
{/* Attention Section */}
|
||||
<ConfigSection title="Spatial Attention">
|
||||
<ToggleRow
|
||||
label="Enable Attention"
|
||||
checked={config.attention_enabled}
|
||||
onChange={(v) => handleChange("attention_enabled", v)}
|
||||
/>
|
||||
<NumberRow
|
||||
label="Attention Heads"
|
||||
value={config.attention_heads}
|
||||
min={1}
|
||||
max={16}
|
||||
onChange={(v) => handleChange("attention_heads", v)}
|
||||
disabled={!config.attention_enabled}
|
||||
/>
|
||||
<SliderRow
|
||||
label="Dropout"
|
||||
value={config.attention_dropout}
|
||||
min={0}
|
||||
max={0.5}
|
||||
step={0.05}
|
||||
onChange={(v) => handleChange("attention_dropout", v)}
|
||||
disabled={!config.attention_enabled}
|
||||
/>
|
||||
</ConfigSection>
|
||||
|
||||
{/* Temporal Section */}
|
||||
<ConfigSection title="Temporal Processing">
|
||||
<ToggleRow
|
||||
label="Enable Temporal"
|
||||
checked={config.temporal_enabled}
|
||||
onChange={(v) => handleChange("temporal_enabled", v)}
|
||||
/>
|
||||
<NumberRow
|
||||
label="Window (ms)"
|
||||
value={config.temporal_window_ms}
|
||||
min={100}
|
||||
max={2000}
|
||||
step={100}
|
||||
onChange={(v) => handleChange("temporal_window_ms", v)}
|
||||
disabled={!config.temporal_enabled}
|
||||
/>
|
||||
<NumberRow
|
||||
label="Compression Ratio"
|
||||
value={config.temporal_compression_ratio}
|
||||
min={1}
|
||||
max={16}
|
||||
onChange={(v) => handleChange("temporal_compression_ratio", v)}
|
||||
disabled={!config.temporal_enabled}
|
||||
/>
|
||||
</ConfigSection>
|
||||
|
||||
{/* Solver Section */}
|
||||
<ConfigSection title="Sparse Solver">
|
||||
<ToggleRow
|
||||
label="Enable Solver"
|
||||
checked={config.solver_enabled}
|
||||
onChange={(v) => handleChange("solver_enabled", v)}
|
||||
/>
|
||||
<div style={{ marginBottom: "var(--space-2)" }}>
|
||||
<label style={labelStyle}>Interpolation</label>
|
||||
<select
|
||||
value={config.solver_interpolation}
|
||||
onChange={(e) =>
|
||||
handleChange(
|
||||
"solver_interpolation",
|
||||
e.target.value as RuVectorConfig["solver_interpolation"]
|
||||
)
|
||||
}
|
||||
disabled={!config.solver_enabled}
|
||||
style={{
|
||||
...inputStyle,
|
||||
opacity: config.solver_enabled ? 1 : 0.5,
|
||||
}}
|
||||
>
|
||||
<option value="linear">Linear</option>
|
||||
<option value="cubic">Cubic</option>
|
||||
<option value="sparse">Sparse (L1)</option>
|
||||
</select>
|
||||
</div>
|
||||
<NumberRow
|
||||
label="Subcarrier Count"
|
||||
value={config.solver_subcarrier_count}
|
||||
min={28}
|
||||
max={114}
|
||||
onChange={(v) => handleChange("solver_subcarrier_count", v)}
|
||||
disabled={!config.solver_enabled}
|
||||
/>
|
||||
</ConfigSection>
|
||||
|
||||
{/* Action Buttons */}
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
gap: "var(--space-2)",
|
||||
marginTop: "var(--space-4)",
|
||||
}}
|
||||
>
|
||||
<button
|
||||
onClick={saveConfig}
|
||||
className="btn-gradient"
|
||||
style={{
|
||||
flex: 1,
|
||||
padding: "10px",
|
||||
fontSize: 12,
|
||||
opacity: saved ? 0.6 : 1,
|
||||
}}
|
||||
disabled={saved}
|
||||
>
|
||||
{saved ? "Saved" : "Save Configuration"}
|
||||
</button>
|
||||
<button
|
||||
onClick={exportConfig}
|
||||
style={{
|
||||
padding: "10px 16px",
|
||||
background: "transparent",
|
||||
border: "1px solid var(--border)",
|
||||
borderRadius: 6,
|
||||
color: "var(--text-secondary)",
|
||||
fontSize: 12,
|
||||
fontWeight: 600,
|
||||
cursor: "pointer",
|
||||
}}
|
||||
>
|
||||
Export
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Live Testing Panel */}
|
||||
<div className="card" style={{ padding: "var(--space-4)" }}>
|
||||
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
|
||||
Live Testing
|
||||
</h3>
|
||||
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
alignItems: "center",
|
||||
justifyContent: "center",
|
||||
minHeight: 200,
|
||||
background: "var(--bg-secondary)",
|
||||
borderRadius: 8,
|
||||
marginBottom: "var(--space-4)",
|
||||
}}
|
||||
>
|
||||
{testingLive ? (
|
||||
<>
|
||||
<div
|
||||
style={{
|
||||
fontSize: 48,
|
||||
animation: "pulse 1s infinite",
|
||||
}}
|
||||
>
|
||||
📡
|
||||
</div>
|
||||
<p style={{ fontSize: 13, color: "var(--text-secondary)", marginTop: "var(--space-2)" }}>
|
||||
Processing live CSI stream...
|
||||
</p>
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<div style={{ fontSize: 48, opacity: 0.5 }}>📡</div>
|
||||
<p style={{ fontSize: 13, color: "var(--text-muted)", marginTop: "var(--space-2)" }}>
|
||||
Start live test to apply config to real CSI data
|
||||
</p>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{liveMetrics && (
|
||||
<div
|
||||
style={{
|
||||
display: "grid",
|
||||
gridTemplateColumns: "repeat(3, 1fr)",
|
||||
gap: "var(--space-3)",
|
||||
marginBottom: "var(--space-4)",
|
||||
}}
|
||||
>
|
||||
<MetricCard label="FPS" value={liveMetrics.fps.toFixed(1)} />
|
||||
<MetricCard label="Latency" value={`${liveMetrics.latency_ms.toFixed(0)}ms`} />
|
||||
<MetricCard label="Persons" value={liveMetrics.persons_detected.toString()} />
|
||||
</div>
|
||||
)}
|
||||
|
||||
<button
|
||||
onClick={testingLive ? () => setTestingLive(false) : startLiveTest}
|
||||
style={{
|
||||
width: "100%",
|
||||
padding: "12px",
|
||||
background: testingLive
|
||||
? "rgba(248, 81, 73, 0.1)"
|
||||
: "rgba(56, 139, 253, 0.1)",
|
||||
border: `1px solid ${testingLive ? "rgba(248, 81, 73, 0.3)" : "rgba(56, 139, 253, 0.3)"}`,
|
||||
borderRadius: 6,
|
||||
color: testingLive ? "var(--status-error)" : "var(--accent)",
|
||||
fontSize: 13,
|
||||
fontWeight: 600,
|
||||
cursor: "pointer",
|
||||
}}
|
||||
>
|
||||
{testingLive ? "Stop Test" : "Start Live Test"}
|
||||
</button>
|
||||
|
||||
{/* Presets */}
|
||||
<div style={{ marginTop: "var(--space-5)" }}>
|
||||
<h4 style={{ fontSize: 12, fontWeight: 600, marginBottom: "var(--space-3)" }}>
|
||||
Quick Presets
|
||||
</h4>
|
||||
<div style={{ display: "flex", flexDirection: "column", gap: "var(--space-2)" }}>
|
||||
<PresetButton
|
||||
label="Low Latency"
|
||||
description="Minimal processing for real-time"
|
||||
onClick={() => {
|
||||
setConfig({
|
||||
...DEFAULT_CONFIG,
|
||||
attention_heads: 2,
|
||||
temporal_compression_ratio: 8,
|
||||
solver_subcarrier_count: 28,
|
||||
});
|
||||
setSaved(false);
|
||||
}}
|
||||
/>
|
||||
<PresetButton
|
||||
label="High Accuracy"
|
||||
description="Maximum quality, higher latency"
|
||||
onClick={() => {
|
||||
setConfig({
|
||||
...DEFAULT_CONFIG,
|
||||
attention_heads: 8,
|
||||
temporal_compression_ratio: 2,
|
||||
solver_subcarrier_count: 114,
|
||||
solver_interpolation: "cubic",
|
||||
});
|
||||
setSaved(false);
|
||||
}}
|
||||
/>
|
||||
<PresetButton
|
||||
label="Balanced"
|
||||
description="Default recommended settings"
|
||||
onClick={() => {
|
||||
setConfig(DEFAULT_CONFIG);
|
||||
setSaved(false);
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<style>{`
|
||||
@keyframes pulse {
|
||||
0%, 100% { transform: scale(1); }
|
||||
50% { transform: scale(1.1); }
|
||||
}
|
||||
`}</style>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
// Helper Components
|
||||
function ConfigSection({ title, children }: { title: string; children: React.ReactNode }) {
|
||||
return (
|
||||
<div style={{ marginBottom: "var(--space-4)" }}>
|
||||
<h4
|
||||
style={{
|
||||
fontSize: 11,
|
||||
fontWeight: 600,
|
||||
color: "var(--text-muted)",
|
||||
textTransform: "uppercase",
|
||||
letterSpacing: "0.04em",
|
||||
marginBottom: "var(--space-2)",
|
||||
}}
|
||||
>
|
||||
{title}
|
||||
</h4>
|
||||
{children}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function ToggleRow({
|
||||
label,
|
||||
checked,
|
||||
onChange,
|
||||
}: {
|
||||
label: string;
|
||||
checked: boolean;
|
||||
onChange: (v: boolean) => void;
|
||||
}) {
|
||||
return (
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "center",
|
||||
marginBottom: "var(--space-2)",
|
||||
}}
|
||||
>
|
||||
<span style={{ fontSize: 12 }}>{label}</span>
|
||||
<button
|
||||
onClick={() => onChange(!checked)}
|
||||
style={{
|
||||
width: 40,
|
||||
height: 22,
|
||||
borderRadius: 11,
|
||||
border: "none",
|
||||
background: checked ? "var(--accent)" : "var(--border)",
|
||||
position: "relative",
|
||||
cursor: "pointer",
|
||||
transition: "background 0.2s",
|
||||
}}
|
||||
>
|
||||
<span
|
||||
style={{
|
||||
position: "absolute",
|
||||
top: 2,
|
||||
left: checked ? 20 : 2,
|
||||
width: 18,
|
||||
height: 18,
|
||||
borderRadius: "50%",
|
||||
background: "white",
|
||||
transition: "left 0.2s",
|
||||
}}
|
||||
/>
|
||||
</button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function SliderRow({
|
||||
label,
|
||||
value,
|
||||
min,
|
||||
max,
|
||||
step,
|
||||
onChange,
|
||||
disabled,
|
||||
}: {
|
||||
label: string;
|
||||
value: number;
|
||||
min: number;
|
||||
max: number;
|
||||
step: number;
|
||||
onChange: (v: number) => void;
|
||||
disabled?: boolean;
|
||||
}) {
|
||||
return (
|
||||
<div style={{ marginBottom: "var(--space-2)", opacity: disabled ? 0.5 : 1 }}>
|
||||
<div style={{ display: "flex", justifyContent: "space-between", marginBottom: 4 }}>
|
||||
<span style={{ fontSize: 12 }}>{label}</span>
|
||||
<span style={{ fontSize: 11, fontFamily: "var(--font-mono)", color: "var(--text-muted)" }}>
|
||||
{value.toFixed(2)}
|
||||
</span>
|
||||
</div>
|
||||
<input
|
||||
type="range"
|
||||
value={value}
|
||||
min={min}
|
||||
max={max}
|
||||
step={step}
|
||||
onChange={(e) => onChange(parseFloat(e.target.value))}
|
||||
disabled={disabled}
|
||||
style={{ width: "100%", cursor: disabled ? "not-allowed" : "pointer" }}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function NumberRow({
|
||||
label,
|
||||
value,
|
||||
min,
|
||||
max,
|
||||
step = 1,
|
||||
onChange,
|
||||
disabled,
|
||||
}: {
|
||||
label: string;
|
||||
value: number;
|
||||
min: number;
|
||||
max: number;
|
||||
step?: number;
|
||||
onChange: (v: number) => void;
|
||||
disabled?: boolean;
|
||||
}) {
|
||||
return (
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "center",
|
||||
marginBottom: "var(--space-2)",
|
||||
opacity: disabled ? 0.5 : 1,
|
||||
}}
|
||||
>
|
||||
<span style={{ fontSize: 12 }}>{label}</span>
|
||||
<input
|
||||
type="number"
|
||||
value={value}
|
||||
min={min}
|
||||
max={max}
|
||||
step={step}
|
||||
onChange={(e) => onChange(parseInt(e.target.value) || min)}
|
||||
disabled={disabled}
|
||||
style={{
|
||||
width: 70,
|
||||
padding: "4px 8px",
|
||||
background: "var(--bg-secondary)",
|
||||
border: "1px solid var(--border)",
|
||||
borderRadius: 4,
|
||||
color: "var(--text-primary)",
|
||||
fontSize: 12,
|
||||
textAlign: "right",
|
||||
cursor: disabled ? "not-allowed" : "text",
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function MetricCard({ label, value }: { label: string; value: string }) {
|
||||
return (
|
||||
<div className="card" style={{ padding: "var(--space-3)", textAlign: "center" }}>
|
||||
<div style={{ fontSize: 10, color: "var(--text-muted)", marginBottom: 2 }}>{label}</div>
|
||||
<div style={{ fontFamily: "var(--font-mono)", fontSize: 18, fontWeight: 600 }}>{value}</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function PresetButton({
|
||||
label,
|
||||
description,
|
||||
onClick,
|
||||
}: {
|
||||
label: string;
|
||||
description: string;
|
||||
onClick: () => void;
|
||||
}) {
|
||||
return (
|
||||
<button
|
||||
onClick={onClick}
|
||||
style={{
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
alignItems: "start",
|
||||
padding: "var(--space-3)",
|
||||
background: "var(--bg-secondary)",
|
||||
border: "1px solid var(--border)",
|
||||
borderRadius: 6,
|
||||
cursor: "pointer",
|
||||
textAlign: "left",
|
||||
}}
|
||||
>
|
||||
<span style={{ fontSize: 12, fontWeight: 600, color: "var(--text-primary)" }}>{label}</span>
|
||||
<span style={{ fontSize: 11, color: "var(--text-muted)" }}>{description}</span>
|
||||
</button>
|
||||
);
|
||||
}
|
||||
|
||||
const labelStyle: React.CSSProperties = {
|
||||
display: "block",
|
||||
fontSize: 11,
|
||||
fontWeight: 600,
|
||||
color: "var(--text-muted)",
|
||||
marginBottom: 4,
|
||||
};
|
||||
|
||||
const inputStyle: React.CSSProperties = {
|
||||
width: "100%",
|
||||
padding: "8px 12px",
|
||||
background: "var(--bg-secondary)",
|
||||
border: "1px solid var(--border)",
|
||||
borderRadius: 6,
|
||||
color: "var(--text-primary)",
|
||||
fontSize: 13,
|
||||
};
|
||||
|
||||
export default RuVectorTab;
|
||||
+601
@@ -0,0 +1,601 @@
|
||||
import React, { useState, useEffect } from "react";
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
import { listen, UnlistenFn } from "@tauri-apps/api/event";
|
||||
|
||||
interface TrainingConfig {
|
||||
dataset_id: string;
|
||||
model_id: string;
|
||||
epochs: number;
|
||||
batch_size: number;
|
||||
learning_rate: number;
|
||||
optimizer: "adam" | "sgd" | "adamw";
|
||||
weight_decay: number;
|
||||
use_augmentation: boolean;
|
||||
checkpoint_every: number;
|
||||
}
|
||||
|
||||
interface TrainingProgress {
|
||||
epoch: number;
|
||||
total_epochs: number;
|
||||
batch: number;
|
||||
total_batches: number;
|
||||
train_loss: number;
|
||||
val_loss: number | null;
|
||||
learning_rate: number;
|
||||
eta_secs: number;
|
||||
gpu_memory_mb: number | null;
|
||||
}
|
||||
|
||||
interface TrainingJob {
|
||||
id: string;
|
||||
status: "running" | "paused" | "completed" | "failed";
|
||||
started_at: string;
|
||||
progress: TrainingProgress;
|
||||
}
|
||||
|
||||
const DEFAULT_CONFIG: TrainingConfig = {
|
||||
dataset_id: "mmfi",
|
||||
model_id: "csi-encoder-cnn",
|
||||
epochs: 100,
|
||||
batch_size: 32,
|
||||
learning_rate: 0.001,
|
||||
optimizer: "adam",
|
||||
weight_decay: 0.0001,
|
||||
use_augmentation: true,
|
||||
checkpoint_every: 10,
|
||||
};
|
||||
|
||||
interface TrainingTabProps {
|
||||
gpuAvailable: boolean;
|
||||
}
|
||||
|
||||
const TrainingTab: React.FC<TrainingTabProps> = ({ gpuAvailable }) => {
|
||||
const [config, setConfig] = useState<TrainingConfig>(DEFAULT_CONFIG);
|
||||
const [currentJob, setCurrentJob] = useState<TrainingJob | null>(null);
|
||||
const [lossHistory, setLossHistory] = useState<{ epoch: number; train: number; val: number }[]>(
|
||||
[]
|
||||
);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
let unlisten: UnlistenFn | undefined;
|
||||
|
||||
const setupListener = async () => {
|
||||
try {
|
||||
unlisten = await listen<TrainingProgress>("training:progress", (event) => {
|
||||
const progress = event.payload;
|
||||
setCurrentJob((prev) =>
|
||||
prev
|
||||
? { ...prev, progress }
|
||||
: {
|
||||
id: "job-1",
|
||||
status: "running",
|
||||
started_at: new Date().toISOString(),
|
||||
progress,
|
||||
}
|
||||
);
|
||||
|
||||
if (progress.val_loss !== null && progress.batch === progress.total_batches) {
|
||||
setLossHistory((prev) => [
|
||||
...prev,
|
||||
{
|
||||
epoch: progress.epoch,
|
||||
train: progress.train_loss,
|
||||
val: progress.val_loss!,
|
||||
},
|
||||
]);
|
||||
}
|
||||
});
|
||||
} catch (err) {
|
||||
console.error("Failed to setup training listener:", err);
|
||||
}
|
||||
};
|
||||
|
||||
setupListener();
|
||||
|
||||
return () => {
|
||||
if (unlisten) unlisten();
|
||||
};
|
||||
}, []);
|
||||
|
||||
const handleStartTraining = async () => {
|
||||
setError(null);
|
||||
try {
|
||||
await invoke("start_training", { config });
|
||||
setCurrentJob({
|
||||
id: `job-${Date.now()}`,
|
||||
status: "running",
|
||||
started_at: new Date().toISOString(),
|
||||
progress: {
|
||||
epoch: 0,
|
||||
total_epochs: config.epochs,
|
||||
batch: 0,
|
||||
total_batches: 0,
|
||||
train_loss: 0,
|
||||
val_loss: null,
|
||||
learning_rate: config.learning_rate,
|
||||
eta_secs: 0,
|
||||
gpu_memory_mb: null,
|
||||
},
|
||||
});
|
||||
} catch (err) {
|
||||
setError(`Failed to start training: ${err}`);
|
||||
}
|
||||
};
|
||||
|
||||
const handleStopTraining = async () => {
|
||||
try {
|
||||
await invoke("stop_training");
|
||||
setCurrentJob((prev) => (prev ? { ...prev, status: "paused" } : null));
|
||||
} catch (err) {
|
||||
setError(`Failed to stop training: ${err}`);
|
||||
}
|
||||
};
|
||||
|
||||
const formatEta = (seconds: number) => {
|
||||
if (seconds < 60) return `${seconds}s`;
|
||||
if (seconds < 3600) return `${Math.floor(seconds / 60)}m ${seconds % 60}s`;
|
||||
const hours = Math.floor(seconds / 3600);
|
||||
const mins = Math.floor((seconds % 3600) / 60);
|
||||
return `${hours}h ${mins}m`;
|
||||
};
|
||||
|
||||
const progress = currentJob?.progress;
|
||||
const epochProgress = progress ? (progress.epoch / progress.total_epochs) * 100 : 0;
|
||||
const batchProgress = progress && progress.total_batches > 0
|
||||
? (progress.batch / progress.total_batches) * 100
|
||||
: 0;
|
||||
|
||||
return (
|
||||
<div>
|
||||
{/* GPU Warning */}
|
||||
{!gpuAvailable && (
|
||||
<div
|
||||
style={{
|
||||
background: "rgba(245, 158, 11, 0.1)",
|
||||
border: "1px solid rgba(245, 158, 11, 0.3)",
|
||||
borderRadius: 6,
|
||||
padding: "var(--space-3)",
|
||||
marginBottom: "var(--space-4)",
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
gap: "var(--space-3)",
|
||||
}}
|
||||
>
|
||||
<span style={{ fontSize: 18 }}>⚠️</span>
|
||||
<div>
|
||||
<div style={{ fontWeight: 600, fontSize: 13, color: "#f59e0b" }}>
|
||||
GPU Not Available
|
||||
</div>
|
||||
<div style={{ fontSize: 12, color: "var(--text-muted)" }}>
|
||||
Training will use CPU, which is significantly slower. Consider using a
|
||||
machine with CUDA or Metal support.
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{error && (
|
||||
<div
|
||||
style={{
|
||||
background: "rgba(248, 81, 73, 0.1)",
|
||||
border: "1px solid rgba(248, 81, 73, 0.3)",
|
||||
borderRadius: 6,
|
||||
padding: "var(--space-3)",
|
||||
marginBottom: "var(--space-4)",
|
||||
fontSize: 13,
|
||||
color: "var(--status-error)",
|
||||
}}
|
||||
>
|
||||
{error}
|
||||
</div>
|
||||
)}
|
||||
|
||||
<div style={{ display: "grid", gridTemplateColumns: "1fr 1fr", gap: "var(--space-5)" }}>
|
||||
{/* Configuration Panel */}
|
||||
<div className="card" style={{ padding: "var(--space-4)" }}>
|
||||
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
|
||||
Training Configuration
|
||||
</h3>
|
||||
|
||||
<div style={{ display: "flex", flexDirection: "column", gap: "var(--space-3)" }}>
|
||||
<div>
|
||||
<label style={labelStyle}>Dataset</label>
|
||||
<select
|
||||
value={config.dataset_id}
|
||||
onChange={(e) => setConfig({ ...config, dataset_id: e.target.value })}
|
||||
style={inputStyle}
|
||||
>
|
||||
<option value="mmfi">MM-Fi Dataset</option>
|
||||
<option value="wipose">Wi-Pose Dataset</option>
|
||||
<option value="wiar">WiAR Dataset</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div>
|
||||
<label style={labelStyle}>Model Architecture</label>
|
||||
<select
|
||||
value={config.model_id}
|
||||
onChange={(e) => setConfig({ ...config, model_id: e.target.value })}
|
||||
style={inputStyle}
|
||||
>
|
||||
<option value="csi-encoder-cnn">CSI Encoder (CNN)</option>
|
||||
<option value="csi-encoder-transformer">CSI Encoder (Transformer)</option>
|
||||
<option value="pose-decoder-lstm">Pose Decoder (LSTM)</option>
|
||||
<option value="pose-decoder-gru">Pose Decoder (GRU)</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div style={{ display: "grid", gridTemplateColumns: "1fr 1fr", gap: "var(--space-3)" }}>
|
||||
<div>
|
||||
<label style={labelStyle}>Epochs</label>
|
||||
<input
|
||||
type="number"
|
||||
value={config.epochs}
|
||||
onChange={(e) => setConfig({ ...config, epochs: parseInt(e.target.value) || 1 })}
|
||||
min={1}
|
||||
max={1000}
|
||||
style={inputStyle}
|
||||
/>
|
||||
</div>
|
||||
<div>
|
||||
<label style={labelStyle}>Batch Size</label>
|
||||
<input
|
||||
type="number"
|
||||
value={config.batch_size}
|
||||
onChange={(e) =>
|
||||
setConfig({ ...config, batch_size: parseInt(e.target.value) || 1 })
|
||||
}
|
||||
min={1}
|
||||
max={512}
|
||||
style={inputStyle}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div style={{ display: "grid", gridTemplateColumns: "1fr 1fr", gap: "var(--space-3)" }}>
|
||||
<div>
|
||||
<label style={labelStyle}>Learning Rate</label>
|
||||
<input
|
||||
type="number"
|
||||
value={config.learning_rate}
|
||||
onChange={(e) =>
|
||||
setConfig({ ...config, learning_rate: parseFloat(e.target.value) || 0.001 })
|
||||
}
|
||||
step={0.0001}
|
||||
min={0.00001}
|
||||
max={1}
|
||||
style={inputStyle}
|
||||
/>
|
||||
</div>
|
||||
<div>
|
||||
<label style={labelStyle}>Optimizer</label>
|
||||
<select
|
||||
value={config.optimizer}
|
||||
onChange={(e) =>
|
||||
setConfig({ ...config, optimizer: e.target.value as TrainingConfig["optimizer"] })
|
||||
}
|
||||
style={inputStyle}
|
||||
>
|
||||
<option value="adam">Adam</option>
|
||||
<option value="adamw">AdamW</option>
|
||||
<option value="sgd">SGD</option>
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div style={{ display: "grid", gridTemplateColumns: "1fr 1fr", gap: "var(--space-3)" }}>
|
||||
<div>
|
||||
<label style={labelStyle}>Weight Decay</label>
|
||||
<input
|
||||
type="number"
|
||||
value={config.weight_decay}
|
||||
onChange={(e) =>
|
||||
setConfig({ ...config, weight_decay: parseFloat(e.target.value) || 0 })
|
||||
}
|
||||
step={0.0001}
|
||||
min={0}
|
||||
max={1}
|
||||
style={inputStyle}
|
||||
/>
|
||||
</div>
|
||||
<div>
|
||||
<label style={labelStyle}>Checkpoint Every</label>
|
||||
<input
|
||||
type="number"
|
||||
value={config.checkpoint_every}
|
||||
onChange={(e) =>
|
||||
setConfig({ ...config, checkpoint_every: parseInt(e.target.value) || 1 })
|
||||
}
|
||||
min={1}
|
||||
max={100}
|
||||
style={inputStyle}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div style={{ display: "flex", alignItems: "center", gap: "var(--space-2)" }}>
|
||||
<input
|
||||
type="checkbox"
|
||||
id="augmentation"
|
||||
checked={config.use_augmentation}
|
||||
onChange={(e) => setConfig({ ...config, use_augmentation: e.target.checked })}
|
||||
style={{ width: 16, height: 16 }}
|
||||
/>
|
||||
<label htmlFor="augmentation" style={{ fontSize: 13, cursor: "pointer" }}>
|
||||
Enable Data Augmentation
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div style={{ marginTop: "var(--space-3)" }}>
|
||||
{currentJob?.status === "running" ? (
|
||||
<button
|
||||
onClick={handleStopTraining}
|
||||
style={{
|
||||
width: "100%",
|
||||
padding: "12px",
|
||||
background: "rgba(248, 81, 73, 0.1)",
|
||||
border: "1px solid rgba(248, 81, 73, 0.3)",
|
||||
borderRadius: 6,
|
||||
color: "var(--status-error)",
|
||||
fontSize: 13,
|
||||
fontWeight: 600,
|
||||
cursor: "pointer",
|
||||
}}
|
||||
>
|
||||
Stop Training
|
||||
</button>
|
||||
) : (
|
||||
<button
|
||||
onClick={handleStartTraining}
|
||||
className="btn-gradient"
|
||||
style={{ width: "100%", padding: "12px", fontSize: 13 }}
|
||||
>
|
||||
Start Training
|
||||
</button>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Progress Panel */}
|
||||
<div className="card" style={{ padding: "var(--space-4)" }}>
|
||||
<h3 style={{ margin: 0, fontSize: 14, fontWeight: 600, marginBottom: "var(--space-4)" }}>
|
||||
Training Progress
|
||||
</h3>
|
||||
|
||||
{!currentJob ? (
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
alignItems: "center",
|
||||
justifyContent: "center",
|
||||
height: 300,
|
||||
color: "var(--text-muted)",
|
||||
}}
|
||||
>
|
||||
<div style={{ fontSize: 48, marginBottom: "var(--space-3)" }}>🎯</div>
|
||||
<p style={{ fontSize: 13 }}>No training job running</p>
|
||||
<p style={{ fontSize: 12 }}>Configure and start training to begin</p>
|
||||
</div>
|
||||
) : (
|
||||
<div style={{ display: "flex", flexDirection: "column", gap: "var(--space-4)" }}>
|
||||
{/* Status */}
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "center",
|
||||
}}
|
||||
>
|
||||
<div style={{ display: "flex", alignItems: "center", gap: "var(--space-2)" }}>
|
||||
<span
|
||||
style={{
|
||||
width: 8,
|
||||
height: 8,
|
||||
borderRadius: "50%",
|
||||
background:
|
||||
currentJob.status === "running"
|
||||
? "var(--status-online)"
|
||||
: currentJob.status === "paused"
|
||||
? "#f59e0b"
|
||||
: "var(--status-error)",
|
||||
animation: currentJob.status === "running" ? "pulse 1.5s infinite" : "none",
|
||||
}}
|
||||
/>
|
||||
<span style={{ fontSize: 13, fontWeight: 600, textTransform: "capitalize" }}>
|
||||
{currentJob.status}
|
||||
</span>
|
||||
</div>
|
||||
<span style={{ fontSize: 12, color: "var(--text-muted)" }}>
|
||||
ETA: {formatEta(progress?.eta_secs ?? 0)}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
{/* Epoch Progress */}
|
||||
<div>
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
fontSize: 12,
|
||||
marginBottom: 4,
|
||||
}}
|
||||
>
|
||||
<span>Epoch</span>
|
||||
<span>
|
||||
{progress?.epoch ?? 0} / {progress?.total_epochs ?? config.epochs}
|
||||
</span>
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
height: 6,
|
||||
background: "var(--border)",
|
||||
borderRadius: 3,
|
||||
overflow: "hidden",
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
width: `${epochProgress}%`,
|
||||
height: "100%",
|
||||
background: "var(--accent)",
|
||||
transition: "width 0.3s",
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Batch Progress */}
|
||||
<div>
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
fontSize: 12,
|
||||
marginBottom: 4,
|
||||
}}
|
||||
>
|
||||
<span>Batch</span>
|
||||
<span>
|
||||
{progress?.batch ?? 0} / {progress?.total_batches ?? 0}
|
||||
</span>
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
height: 4,
|
||||
background: "var(--border)",
|
||||
borderRadius: 2,
|
||||
overflow: "hidden",
|
||||
}}
|
||||
>
|
||||
<div
|
||||
style={{
|
||||
width: `${batchProgress}%`,
|
||||
height: "100%",
|
||||
background: "rgba(56, 139, 253, 0.5)",
|
||||
transition: "width 0.1s",
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Stats Grid */}
|
||||
<div
|
||||
style={{
|
||||
display: "grid",
|
||||
gridTemplateColumns: "repeat(2, 1fr)",
|
||||
gap: "var(--space-3)",
|
||||
}}
|
||||
>
|
||||
<div className="card" style={{ padding: "var(--space-3)" }}>
|
||||
<div style={{ fontSize: 10, color: "var(--text-muted)", marginBottom: 4 }}>
|
||||
Train Loss
|
||||
</div>
|
||||
<div style={{ fontFamily: "var(--font-mono)", fontSize: 20, fontWeight: 600 }}>
|
||||
{progress?.train_loss.toFixed(4) ?? "—"}
|
||||
</div>
|
||||
</div>
|
||||
<div className="card" style={{ padding: "var(--space-3)" }}>
|
||||
<div style={{ fontSize: 10, color: "var(--text-muted)", marginBottom: 4 }}>
|
||||
Val Loss
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
fontFamily: "var(--font-mono)",
|
||||
fontSize: 20,
|
||||
fontWeight: 600,
|
||||
color: "var(--status-online)",
|
||||
}}
|
||||
>
|
||||
{progress?.val_loss?.toFixed(4) ?? "—"}
|
||||
</div>
|
||||
</div>
|
||||
<div className="card" style={{ padding: "var(--space-3)" }}>
|
||||
<div style={{ fontSize: 10, color: "var(--text-muted)", marginBottom: 4 }}>
|
||||
Learning Rate
|
||||
</div>
|
||||
<div style={{ fontFamily: "var(--font-mono)", fontSize: 14, fontWeight: 600 }}>
|
||||
{progress?.learning_rate.toExponential(2) ?? "—"}
|
||||
</div>
|
||||
</div>
|
||||
<div className="card" style={{ padding: "var(--space-3)" }}>
|
||||
<div style={{ fontSize: 10, color: "var(--text-muted)", marginBottom: 4 }}>
|
||||
GPU Memory
|
||||
</div>
|
||||
<div style={{ fontFamily: "var(--font-mono)", fontSize: 14, fontWeight: 600 }}>
|
||||
{progress?.gpu_memory_mb ? `${progress.gpu_memory_mb} MB` : "N/A"}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Mini Loss Chart */}
|
||||
{lossHistory.length > 0 && (
|
||||
<div>
|
||||
<div style={{ fontSize: 12, fontWeight: 600, marginBottom: "var(--space-2)" }}>
|
||||
Loss History
|
||||
</div>
|
||||
<div
|
||||
style={{
|
||||
height: 80,
|
||||
display: "flex",
|
||||
alignItems: "flex-end",
|
||||
gap: 2,
|
||||
padding: "var(--space-2)",
|
||||
background: "var(--bg-secondary)",
|
||||
borderRadius: 4,
|
||||
}}
|
||||
>
|
||||
{lossHistory.slice(-20).map((h, i) => (
|
||||
<div
|
||||
key={i}
|
||||
style={{
|
||||
flex: 1,
|
||||
height: `${Math.max(5, Math.min(100, (1 - h.train) * 100))}%`,
|
||||
background: "var(--accent)",
|
||||
borderRadius: 2,
|
||||
opacity: 0.6 + (i / 20) * 0.4,
|
||||
}}
|
||||
title={`Epoch ${h.epoch}: Train=${h.train.toFixed(4)}, Val=${h.val.toFixed(4)}`}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<style>{`
|
||||
@keyframes pulse {
|
||||
0%, 100% { opacity: 1; }
|
||||
50% { opacity: 0.5; }
|
||||
}
|
||||
`}</style>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
const labelStyle: React.CSSProperties = {
|
||||
display: "block",
|
||||
fontSize: 11,
|
||||
fontWeight: 600,
|
||||
color: "var(--text-muted)",
|
||||
marginBottom: 4,
|
||||
textTransform: "uppercase",
|
||||
letterSpacing: "0.04em",
|
||||
};
|
||||
|
||||
const inputStyle: React.CSSProperties = {
|
||||
width: "100%",
|
||||
padding: "8px 12px",
|
||||
background: "var(--bg-secondary)",
|
||||
border: "1px solid var(--border)",
|
||||
borderRadius: 6,
|
||||
color: "var(--text-primary)",
|
||||
fontSize: 13,
|
||||
};
|
||||
|
||||
export default TrainingTab;
|
||||
+165
@@ -0,0 +1,165 @@
|
||||
import React, { useState, useEffect } from "react";
|
||||
import { invoke } from "@tauri-apps/api/core";
|
||||
import DatasetsTab from "./DatasetsTab";
|
||||
import ModelsTab from "./ModelsTab";
|
||||
import TrainingTab from "./TrainingTab";
|
||||
import RuVectorTab from "./RuVectorTab";
|
||||
import MetricsTab from "./MetricsTab";
|
||||
|
||||
type TrainingTabType = "datasets" | "models" | "training" | "ruvector" | "metrics";
|
||||
|
||||
interface GpuInfo {
|
||||
available: boolean;
|
||||
name: string | null;
|
||||
memory_mb: number | null;
|
||||
cuda_version: string | null;
|
||||
metal_supported: boolean;
|
||||
}
|
||||
|
||||
const Training: React.FC = () => {
|
||||
const [activeTab, setActiveTab] = useState<TrainingTabType>("datasets");
|
||||
const [gpuInfo, setGpuInfo] = useState<GpuInfo | null>(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
|
||||
useEffect(() => {
|
||||
detectGpu();
|
||||
}, []);
|
||||
|
||||
const detectGpu = async () => {
|
||||
try {
|
||||
const info = await invoke<GpuInfo>("detect_gpu");
|
||||
setGpuInfo(info);
|
||||
} catch (err) {
|
||||
console.error("GPU detection failed:", err);
|
||||
setGpuInfo({
|
||||
available: false,
|
||||
name: null,
|
||||
memory_mb: null,
|
||||
cuda_version: null,
|
||||
metal_supported: false,
|
||||
});
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
};
|
||||
|
||||
const tabs: { id: TrainingTabType; label: string; icon: string }[] = [
|
||||
{ id: "datasets", label: "Datasets", icon: "📊" },
|
||||
{ id: "models", label: "Models", icon: "🧠" },
|
||||
{ id: "training", label: "Training", icon: "⚡" },
|
||||
{ id: "ruvector", label: "RuVector", icon: "📡" },
|
||||
{ id: "metrics", label: "Metrics", icon: "📈" },
|
||||
];
|
||||
|
||||
return (
|
||||
<div style={{ padding: "var(--space-5)", maxWidth: 1400 }}>
|
||||
{/* Header */}
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
alignItems: "center",
|
||||
marginBottom: "var(--space-5)",
|
||||
}}
|
||||
>
|
||||
<div>
|
||||
<h1 className="heading-lg" style={{ margin: 0 }}>
|
||||
Training & Models
|
||||
</h1>
|
||||
<p
|
||||
style={{
|
||||
fontSize: 13,
|
||||
color: "var(--text-secondary)",
|
||||
marginTop: 4,
|
||||
}}
|
||||
>
|
||||
Train pose estimation models and configure RuVector signal processing
|
||||
</p>
|
||||
</div>
|
||||
|
||||
{/* GPU Status */}
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
gap: "var(--space-3)",
|
||||
padding: "var(--space-3) var(--space-4)",
|
||||
background: gpuInfo?.available
|
||||
? "rgba(63, 185, 80, 0.1)"
|
||||
: "rgba(139, 148, 158, 0.1)",
|
||||
border: `1px solid ${gpuInfo?.available ? "rgba(63, 185, 80, 0.3)" : "rgba(139, 148, 158, 0.3)"}`,
|
||||
borderRadius: 8,
|
||||
}}
|
||||
>
|
||||
<span style={{ fontSize: 18 }}>{gpuInfo?.available ? "🎮" : "💻"}</span>
|
||||
<div>
|
||||
<div style={{ fontSize: 12, fontWeight: 600, color: "var(--text-primary)" }}>
|
||||
{loading
|
||||
? "Detecting GPU..."
|
||||
: gpuInfo?.available
|
||||
? gpuInfo.name || "GPU Available"
|
||||
: "CPU Mode"}
|
||||
</div>
|
||||
<div style={{ fontSize: 11, color: "var(--text-muted)" }}>
|
||||
{gpuInfo?.cuda_version
|
||||
? `CUDA ${gpuInfo.cuda_version}`
|
||||
: gpuInfo?.metal_supported
|
||||
? "Metal Supported"
|
||||
: "No GPU acceleration"}
|
||||
{gpuInfo?.memory_mb && ` • ${Math.round(gpuInfo.memory_mb / 1024)}GB`}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Tabs */}
|
||||
<div
|
||||
style={{
|
||||
display: "flex",
|
||||
gap: "var(--space-1)",
|
||||
borderBottom: "1px solid var(--border)",
|
||||
marginBottom: "var(--space-5)",
|
||||
}}
|
||||
>
|
||||
{tabs.map((tab) => (
|
||||
<button
|
||||
key={tab.id}
|
||||
onClick={() => setActiveTab(tab.id)}
|
||||
style={{
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
gap: 6,
|
||||
padding: "12px 20px",
|
||||
border: "none",
|
||||
background: "transparent",
|
||||
color: activeTab === tab.id ? "var(--accent)" : "var(--text-secondary)",
|
||||
fontSize: 13,
|
||||
fontWeight: 600,
|
||||
cursor: "pointer",
|
||||
borderBottom:
|
||||
activeTab === tab.id
|
||||
? "2px solid var(--accent)"
|
||||
: "2px solid transparent",
|
||||
marginBottom: -1,
|
||||
transition: "color 0.15s, border-color 0.15s",
|
||||
}}
|
||||
>
|
||||
<span>{tab.icon}</span>
|
||||
{tab.label}
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
|
||||
{/* Tab Content */}
|
||||
<div>
|
||||
{activeTab === "datasets" && <DatasetsTab />}
|
||||
{activeTab === "models" && <ModelsTab />}
|
||||
{activeTab === "training" && <TrainingTab gpuAvailable={gpuInfo?.available ?? false} />}
|
||||
{activeTab === "ruvector" && <RuVectorTab />}
|
||||
{activeTab === "metrics" && <MetricsTab />}
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
export default Training;
|
||||
@@ -1,2 +1,2 @@
|
||||
// Application version - single source of truth
|
||||
export const APP_VERSION = "0.4.4";
|
||||
export const APP_VERSION = "0.5.0";
|
||||
|
||||
@@ -29,7 +29,6 @@
|
||||
<button class="nav-tab" data-tab="applications">Applications</button>
|
||||
<button class="nav-tab" data-tab="sensing">Sensing</button>
|
||||
<button class="nav-tab" data-tab="training">Training</button>
|
||||
<a href="pose-fusion.html" class="nav-tab" style="text-decoration:none">Pose Fusion</a>
|
||||
<a href="observatory.html" class="nav-tab" style="text-decoration:none">Observatory</a>
|
||||
</nav>
|
||||
|
||||
|
||||
@@ -1,160 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>WiFi-DensePose — Dual-Modal Pose Estimation</title>
|
||||
<link rel="stylesheet" href="pose-fusion/css/style.css">
|
||||
</head>
|
||||
<body>
|
||||
|
||||
<!-- Header -->
|
||||
<header class="header">
|
||||
<div class="header-left">
|
||||
<div class="logo"><span class="pi">π</span> DensePose</div>
|
||||
<div class="header-title">Dual-Modal Pose Estimation — Live Video + WiFi CSI Fusion</div>
|
||||
</div>
|
||||
<div class="header-right">
|
||||
<select id="mode-select" class="mode-select">
|
||||
<option value="dual">Dual Mode (Video + CSI)</option>
|
||||
<option value="video">Video Only</option>
|
||||
<option value="csi">CSI Only (WiFi)</option>
|
||||
</select>
|
||||
<div class="status-badge">
|
||||
<span id="status-dot" class="status-dot offline"></span>
|
||||
<span id="status-label">READY</span>
|
||||
</div>
|
||||
<span id="fps-display" class="fps-badge">-- FPS</span>
|
||||
<a href="index.html" class="back-link">← Dashboard</a>
|
||||
<a href="observatory.html" class="back-link">Observatory →</a>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<!-- Main Grid -->
|
||||
<div class="main-grid">
|
||||
|
||||
<!-- Video + Skeleton Panel -->
|
||||
<div class="video-panel">
|
||||
<video id="webcam" autoplay playsinline muted></video>
|
||||
<canvas id="skeleton-canvas"></canvas>
|
||||
<div class="video-overlay-label" id="mode-label">DUAL FUSION</div>
|
||||
|
||||
<div id="camera-prompt" class="camera-prompt">
|
||||
<p>Enable your webcam for live video pose estimation.<br>
|
||||
Or switch to <strong>CSI Only</strong> mode for WiFi-based sensing.</p>
|
||||
<button id="start-camera-btn">Enable Camera</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Side Panels -->
|
||||
<div class="side-panels">
|
||||
|
||||
<!-- Fusion Confidence -->
|
||||
<div class="panel">
|
||||
<div class="panel-title">◆ Fusion Confidence</div>
|
||||
<div class="fusion-bars">
|
||||
<div class="bar-row">
|
||||
<span class="bar-label">Video</span>
|
||||
<div class="bar-track"><div class="bar-fill video" id="video-bar" style="width:0%"></div></div>
|
||||
<span class="bar-value" id="video-bar-val">0%</span>
|
||||
</div>
|
||||
<div class="bar-row">
|
||||
<span class="bar-label">CSI</span>
|
||||
<div class="bar-track"><div class="bar-fill csi" id="csi-bar" style="width:0%"></div></div>
|
||||
<span class="bar-value" id="csi-bar-val">0%</span>
|
||||
</div>
|
||||
<div class="bar-row">
|
||||
<span class="bar-label">Fused</span>
|
||||
<div class="bar-track"><div class="bar-fill fused" id="fused-bar" style="width:0%"></div></div>
|
||||
<span class="bar-value" id="fused-bar-val">0%</span>
|
||||
</div>
|
||||
</div>
|
||||
<div style="margin-top:8px; font-size:10px; color:var(--text-label)">
|
||||
Cross-modal: <span id="cross-modal-sim" style="color:var(--green-glow)">0.000</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- CSI Heatmap -->
|
||||
<div class="panel">
|
||||
<div class="panel-title">◆ CSI Amplitude Heatmap</div>
|
||||
<div class="csi-canvas-wrapper">
|
||||
<canvas id="csi-canvas" width="320" height="120"></canvas>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Embedding Space -->
|
||||
<div class="panel">
|
||||
<div class="panel-title">◆ Embedding Space (2D Projection)</div>
|
||||
<div class="embedding-canvas-wrapper">
|
||||
<canvas id="embedding-canvas" width="320" height="140"></canvas>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Latency -->
|
||||
<div class="panel">
|
||||
<div class="panel-title">◆ Pipeline Latency</div>
|
||||
<div class="latency-grid">
|
||||
<div class="latency-item">
|
||||
<div class="latency-value" id="lat-video">--</div>
|
||||
<div class="latency-label">Video CNN</div>
|
||||
</div>
|
||||
<div class="latency-item">
|
||||
<div class="latency-value" id="lat-csi">--</div>
|
||||
<div class="latency-label">CSI CNN</div>
|
||||
</div>
|
||||
<div class="latency-item">
|
||||
<div class="latency-value" id="lat-fusion">--</div>
|
||||
<div class="latency-label">Fusion</div>
|
||||
</div>
|
||||
<div class="latency-item">
|
||||
<div class="latency-value" id="lat-total">--</div>
|
||||
<div class="latency-label">Total</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Controls -->
|
||||
<div class="panel">
|
||||
<div class="panel-title">◆ Controls</div>
|
||||
<div class="controls-row">
|
||||
<button class="btn" id="pause-btn">⏸ Pause</button>
|
||||
</div>
|
||||
|
||||
<div class="slider-row">
|
||||
<label>Confidence</label>
|
||||
<input type="range" id="confidence-slider" min="0" max="1" step="0.05" value="0.3">
|
||||
<span class="slider-val" id="confidence-value">0.30</span>
|
||||
</div>
|
||||
|
||||
<div style="margin-top:10px">
|
||||
<div class="panel-title" style="margin-bottom:6px">◆ Live CSI Source</div>
|
||||
<div style="display:flex;gap:6px">
|
||||
<input type="text" id="ws-url" placeholder="ws://localhost:3030/ws/csi"
|
||||
style="flex:1;background:rgba(255,255,255,0.05);border:1px solid var(--bg-panel-border);
|
||||
color:var(--text-primary);padding:5px 8px;border-radius:4px;font-size:11px;
|
||||
font-family:'JetBrains Mono',monospace">
|
||||
<button class="btn" id="connect-ws-btn">Connect</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div><!-- /side-panels -->
|
||||
|
||||
<!-- Bottom Bar -->
|
||||
<div class="bottom-bar">
|
||||
<div>
|
||||
WiFi-DensePose · Dual-Modal Pose Estimation ·
|
||||
Architecture: MobileNet-V3 × 2 → Attention Fusion → 17-Keypoint COCO
|
||||
</div>
|
||||
<div>
|
||||
<a href="https://github.com/ruvnet/wifi-densepose">GitHub</a> ·
|
||||
CNN: ruvector-cnn (JS fallback) ·
|
||||
<a href="observatory.html">Observatory</a>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
</div><!-- /main-grid -->
|
||||
|
||||
<script type="module" src="pose-fusion/js/main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -1,30 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Build WASM packages for the dual-modal pose estimation demo.
|
||||
# Requires: wasm-pack (cargo install wasm-pack)
|
||||
#
|
||||
# Usage: ./build.sh
|
||||
#
|
||||
# Output: pkg/ruvector_cnn_wasm/ — WASM CNN embedder for browser
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
VENDOR_DIR="$SCRIPT_DIR/../../vendor/ruvector"
|
||||
OUT_DIR="$SCRIPT_DIR/pkg/ruvector_cnn_wasm"
|
||||
|
||||
echo "Building ruvector-cnn-wasm..."
|
||||
wasm-pack build "$VENDOR_DIR/crates/ruvector-cnn-wasm" \
|
||||
--target web \
|
||||
--out-dir "$OUT_DIR" \
|
||||
--no-typescript
|
||||
|
||||
# Remove .gitignore so we can commit the build output for GitHub Pages
|
||||
rm -f "$OUT_DIR/.gitignore"
|
||||
|
||||
echo ""
|
||||
echo "Build complete!"
|
||||
echo " WASM: $(du -sh "$OUT_DIR/ruvector_cnn_wasm_bg.wasm" | cut -f1)"
|
||||
echo " JS: $(du -sh "$OUT_DIR/ruvector_cnn_wasm.js" | cut -f1)"
|
||||
echo ""
|
||||
echo "Serve the demo: cd $SCRIPT_DIR/.. && python3 -m http.server 8080"
|
||||
echo "Open: http://localhost:8080/pose-fusion.html"
|
||||
@@ -1,405 +0,0 @@
|
||||
/* WiFi-DensePose — Dual-Modal Pose Fusion Demo
|
||||
Dark theme matching Observatory */
|
||||
|
||||
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@300;400;600;700&family=JetBrains+Mono:wght@400;600&display=swap');
|
||||
|
||||
:root {
|
||||
--bg-deep: #080c14;
|
||||
--bg-panel: rgba(8, 16, 28, 0.92);
|
||||
--bg-panel-border: rgba(0, 210, 120, 0.25);
|
||||
--green-glow: #00d878;
|
||||
--green-bright:#3eff8a;
|
||||
--green-dim: #0a6b3a;
|
||||
--amber: #ffb020;
|
||||
--amber-dim: #a06800;
|
||||
--blue-signal: #2090ff;
|
||||
--blue-dim: #0a3060;
|
||||
--red-alert: #ff3040;
|
||||
--cyan: #00e5ff;
|
||||
--text-primary: #e8ece0;
|
||||
--text-secondary: rgba(232,236,224, 0.55);
|
||||
--text-label: rgba(232,236,224, 0.35);
|
||||
--radius: 8px;
|
||||
}
|
||||
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
|
||||
body {
|
||||
background: var(--bg-deep);
|
||||
font-family: 'Inter', -apple-system, sans-serif;
|
||||
color: var(--text-primary);
|
||||
-webkit-font-smoothing: antialiased;
|
||||
overflow-x: hidden;
|
||||
min-height: 100vh;
|
||||
}
|
||||
|
||||
/* === Header === */
|
||||
.header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 16px 24px;
|
||||
border-bottom: 1px solid var(--bg-panel-border);
|
||||
background: var(--bg-panel);
|
||||
backdrop-filter: blur(12px);
|
||||
}
|
||||
|
||||
.header-left {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
}
|
||||
|
||||
.logo {
|
||||
font-weight: 700;
|
||||
font-size: 24px;
|
||||
color: var(--green-glow);
|
||||
}
|
||||
|
||||
.logo .pi { font-style: normal; }
|
||||
|
||||
.header-title {
|
||||
font-size: 14px;
|
||||
color: var(--text-secondary);
|
||||
font-weight: 300;
|
||||
}
|
||||
|
||||
.header-right {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
}
|
||||
|
||||
.mode-select {
|
||||
background: rgba(0,210,120,0.1);
|
||||
border: 1px solid var(--bg-panel-border);
|
||||
color: var(--text-primary);
|
||||
padding: 6px 12px;
|
||||
border-radius: var(--radius);
|
||||
font-family: inherit;
|
||||
font-size: 13px;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.mode-select option { background: #0c1420; }
|
||||
|
||||
.status-badge {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
font-family: 'JetBrains Mono', monospace;
|
||||
font-size: 12px;
|
||||
padding: 4px 10px;
|
||||
border-radius: 12px;
|
||||
background: rgba(0,210,120,0.1);
|
||||
border: 1px solid var(--bg-panel-border);
|
||||
}
|
||||
|
||||
.status-dot {
|
||||
width: 8px; height: 8px;
|
||||
border-radius: 50%;
|
||||
background: var(--green-glow);
|
||||
box-shadow: 0 0 8px var(--green-glow);
|
||||
animation: pulse-dot 2s ease infinite;
|
||||
}
|
||||
|
||||
.status-dot.offline { background: #555; box-shadow: none; animation: none; }
|
||||
.status-dot.warning { background: var(--amber); box-shadow: 0 0 8px var(--amber); }
|
||||
|
||||
@keyframes pulse-dot {
|
||||
0%, 100% { opacity: 1; }
|
||||
50% { opacity: 0.5; }
|
||||
}
|
||||
|
||||
.fps-badge {
|
||||
font-family: 'JetBrains Mono', monospace;
|
||||
font-size: 12px;
|
||||
color: var(--green-glow);
|
||||
}
|
||||
|
||||
.back-link {
|
||||
color: var(--text-secondary);
|
||||
text-decoration: none;
|
||||
font-size: 13px;
|
||||
transition: color 0.2s;
|
||||
}
|
||||
.back-link:hover { color: var(--green-glow); }
|
||||
|
||||
/* === Main Layout === */
|
||||
.main-grid {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 360px;
|
||||
grid-template-rows: 1fr auto;
|
||||
gap: 16px;
|
||||
padding: 16px 24px;
|
||||
height: calc(100vh - 72px);
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
/* === Video Panel === */
|
||||
.video-panel {
|
||||
position: relative;
|
||||
background: #000;
|
||||
border-radius: var(--radius);
|
||||
border: 1px solid var(--bg-panel-border);
|
||||
overflow: hidden;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.video-panel video {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: cover;
|
||||
transform: scaleX(-1);
|
||||
}
|
||||
|
||||
.video-panel canvas {
|
||||
position: absolute;
|
||||
top: 0; left: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
transform: scaleX(-1);
|
||||
}
|
||||
|
||||
.video-overlay-label {
|
||||
position: absolute;
|
||||
top: 12px; left: 12px;
|
||||
font-family: 'JetBrains Mono', monospace;
|
||||
font-size: 11px;
|
||||
padding: 4px 8px;
|
||||
background: rgba(0,0,0,0.7);
|
||||
border-radius: 4px;
|
||||
color: var(--green-glow);
|
||||
z-index: 5;
|
||||
transform: scaleX(-1);
|
||||
}
|
||||
|
||||
.camera-prompt {
|
||||
position: absolute;
|
||||
top: 50%; left: 50%;
|
||||
transform: translate(-50%, -50%);
|
||||
text-align: center;
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.camera-prompt button {
|
||||
margin-top: 12px;
|
||||
padding: 10px 24px;
|
||||
background: var(--green-glow);
|
||||
color: #000;
|
||||
border: none;
|
||||
border-radius: var(--radius);
|
||||
font-family: inherit;
|
||||
font-weight: 600;
|
||||
font-size: 14px;
|
||||
cursor: pointer;
|
||||
transition: background 0.2s;
|
||||
}
|
||||
|
||||
.camera-prompt button:hover { background: var(--green-bright); }
|
||||
|
||||
/* === Side Panels === */
|
||||
.side-panels {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 12px;
|
||||
overflow-y: auto;
|
||||
min-height: 0;
|
||||
}
|
||||
|
||||
.panel {
|
||||
background: var(--bg-panel);
|
||||
border: 1px solid var(--bg-panel-border);
|
||||
border-radius: var(--radius);
|
||||
padding: 14px;
|
||||
}
|
||||
|
||||
.panel-title {
|
||||
font-size: 11px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 1.2px;
|
||||
color: var(--text-label);
|
||||
margin-bottom: 10px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
}
|
||||
|
||||
/* === CSI Heatmap === */
|
||||
.csi-canvas-wrapper {
|
||||
position: relative;
|
||||
border-radius: 4px;
|
||||
overflow: hidden;
|
||||
background: #000;
|
||||
}
|
||||
|
||||
.csi-canvas-wrapper canvas {
|
||||
width: 100%;
|
||||
display: block;
|
||||
}
|
||||
|
||||
/* === Fusion Bars === */
|
||||
.fusion-bars {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.bar-row {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.bar-label {
|
||||
font-family: 'JetBrains Mono', monospace;
|
||||
font-size: 11px;
|
||||
color: var(--text-secondary);
|
||||
width: 55px;
|
||||
text-align: right;
|
||||
}
|
||||
|
||||
.bar-track {
|
||||
flex: 1;
|
||||
height: 6px;
|
||||
background: rgba(255,255,255,0.06);
|
||||
border-radius: 3px;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.bar-fill {
|
||||
height: 100%;
|
||||
border-radius: 3px;
|
||||
transition: width 0.3s ease;
|
||||
}
|
||||
|
||||
.bar-fill.video { background: var(--cyan); }
|
||||
.bar-fill.csi { background: var(--amber); }
|
||||
.bar-fill.fused { background: var(--green-glow); box-shadow: 0 0 8px var(--green-glow); }
|
||||
|
||||
.bar-value {
|
||||
font-family: 'JetBrains Mono', monospace;
|
||||
font-size: 11px;
|
||||
color: var(--text-primary);
|
||||
width: 36px;
|
||||
}
|
||||
|
||||
/* === Embedding Space === */
|
||||
.embedding-canvas-wrapper {
|
||||
position: relative;
|
||||
background: #000;
|
||||
border-radius: 4px;
|
||||
overflow: hidden;
|
||||
}
|
||||
.embedding-canvas-wrapper canvas {
|
||||
width: 100%;
|
||||
display: block;
|
||||
}
|
||||
|
||||
/* === Latency Panel === */
|
||||
.latency-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(4, 1fr);
|
||||
gap: 6px;
|
||||
}
|
||||
|
||||
.latency-item {
|
||||
text-align: center;
|
||||
padding: 6px 0;
|
||||
}
|
||||
|
||||
.latency-value {
|
||||
font-family: 'JetBrains Mono', monospace;
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
color: var(--green-glow);
|
||||
}
|
||||
|
||||
.latency-label {
|
||||
font-size: 10px;
|
||||
color: var(--text-label);
|
||||
margin-top: 2px;
|
||||
}
|
||||
|
||||
/* === Controls === */
|
||||
.controls-row {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.btn {
|
||||
padding: 6px 14px;
|
||||
border: 1px solid var(--bg-panel-border);
|
||||
background: rgba(0,210,120,0.08);
|
||||
color: var(--text-primary);
|
||||
border-radius: var(--radius);
|
||||
font-family: inherit;
|
||||
font-size: 12px;
|
||||
cursor: pointer;
|
||||
transition: all 0.2s;
|
||||
}
|
||||
.btn:hover { background: rgba(0,210,120,0.2); }
|
||||
.btn.active { background: var(--green-glow); color: #000; font-weight: 600; }
|
||||
|
||||
.slider-row {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-top: 8px;
|
||||
}
|
||||
|
||||
.slider-row label {
|
||||
font-size: 11px;
|
||||
color: var(--text-secondary);
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.slider-row input[type=range] {
|
||||
flex: 1;
|
||||
accent-color: var(--green-glow);
|
||||
}
|
||||
|
||||
.slider-row .slider-val {
|
||||
font-family: 'JetBrains Mono', monospace;
|
||||
font-size: 11px;
|
||||
width: 32px;
|
||||
color: var(--green-glow);
|
||||
}
|
||||
|
||||
/* === Bottom Bar === */
|
||||
.bottom-bar {
|
||||
grid-column: 1 / -1;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
padding: 10px 16px;
|
||||
background: var(--bg-panel);
|
||||
border: 1px solid var(--bg-panel-border);
|
||||
border-radius: var(--radius);
|
||||
font-family: 'JetBrains Mono', monospace;
|
||||
font-size: 11px;
|
||||
color: var(--text-secondary);
|
||||
}
|
||||
|
||||
.bottom-bar a {
|
||||
color: var(--green-glow);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
/* === Skeleton colors === */
|
||||
.skeleton-joint { fill: var(--green-glow); }
|
||||
.skeleton-limb { stroke: var(--green-bright); }
|
||||
.skeleton-joint-csi { fill: var(--amber); }
|
||||
.skeleton-limb-csi { stroke: var(--amber); }
|
||||
|
||||
/* === Responsive === */
|
||||
@media (max-width: 900px) {
|
||||
.main-grid {
|
||||
grid-template-columns: 1fr;
|
||||
height: auto;
|
||||
overflow: auto;
|
||||
}
|
||||
.video-panel { aspect-ratio: 16/9; max-height: 50vh; }
|
||||
.side-panels { max-height: none; overflow: visible; }
|
||||
}
|
||||
@@ -1,247 +0,0 @@
|
||||
/**
|
||||
* CanvasRenderer — Renders skeleton overlay on video, CSI heatmap,
|
||||
* embedding space visualization, and fusion confidence bars.
|
||||
*/
|
||||
|
||||
import { SKELETON_CONNECTIONS } from './pose-decoder.js';
|
||||
|
||||
export class CanvasRenderer {
|
||||
constructor() {
|
||||
this.colors = {
|
||||
joint: '#00d878',
|
||||
jointGlow: 'rgba(0, 216, 120, 0.4)',
|
||||
limb: '#3eff8a',
|
||||
limbGlow: 'rgba(62, 255, 138, 0.15)',
|
||||
csiJoint: '#ffb020',
|
||||
csiLimb: '#ffc850',
|
||||
fused: '#00e5ff',
|
||||
confidence: 'rgba(255,255,255,0.3)',
|
||||
videoEmb: '#00e5ff',
|
||||
csiEmb: '#ffb020',
|
||||
fusedEmb: '#00d878',
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Draw skeleton overlay on the video canvas
|
||||
* @param {CanvasRenderingContext2D} ctx
|
||||
* @param {Array<{x,y,confidence}>} keypoints - Normalized [0,1] coordinates
|
||||
* @param {number} width - Canvas width
|
||||
* @param {number} height - Canvas height
|
||||
* @param {object} opts
|
||||
*/
|
||||
drawSkeleton(ctx, keypoints, width, height, opts = {}) {
|
||||
const minConf = opts.minConfidence || 0.3;
|
||||
const color = opts.color || 'green';
|
||||
const jointColor = color === 'amber' ? this.colors.csiJoint : this.colors.joint;
|
||||
const limbColor = color === 'amber' ? this.colors.csiLimb : this.colors.limb;
|
||||
const glowColor = color === 'amber' ? 'rgba(255,176,32,0.4)' : this.colors.jointGlow;
|
||||
|
||||
ctx.clearRect(0, 0, width, height);
|
||||
|
||||
if (!keypoints || keypoints.length === 0) return;
|
||||
|
||||
// Draw limbs first (behind joints)
|
||||
ctx.lineWidth = 3;
|
||||
ctx.lineCap = 'round';
|
||||
|
||||
for (const [i, j] of SKELETON_CONNECTIONS) {
|
||||
const kpA = keypoints[i];
|
||||
const kpB = keypoints[j];
|
||||
if (!kpA || !kpB || kpA.confidence < minConf || kpB.confidence < minConf) continue;
|
||||
|
||||
const ax = kpA.x * width, ay = kpA.y * height;
|
||||
const bx = kpB.x * width, by = kpB.y * height;
|
||||
const avgConf = (kpA.confidence + kpB.confidence) / 2;
|
||||
|
||||
// Glow
|
||||
ctx.strokeStyle = this.colors.limbGlow;
|
||||
ctx.lineWidth = 8;
|
||||
ctx.globalAlpha = avgConf * 0.4;
|
||||
ctx.beginPath();
|
||||
ctx.moveTo(ax, ay);
|
||||
ctx.lineTo(bx, by);
|
||||
ctx.stroke();
|
||||
|
||||
// Main line
|
||||
ctx.strokeStyle = limbColor;
|
||||
ctx.lineWidth = 2.5;
|
||||
ctx.globalAlpha = avgConf;
|
||||
ctx.beginPath();
|
||||
ctx.moveTo(ax, ay);
|
||||
ctx.lineTo(bx, by);
|
||||
ctx.stroke();
|
||||
}
|
||||
|
||||
// Draw joints
|
||||
ctx.globalAlpha = 1;
|
||||
for (const kp of keypoints) {
|
||||
if (!kp || kp.confidence < minConf) continue;
|
||||
|
||||
const x = kp.x * width;
|
||||
const y = kp.y * height;
|
||||
const r = 3 + kp.confidence * 3;
|
||||
|
||||
// Glow
|
||||
ctx.beginPath();
|
||||
ctx.arc(x, y, r + 4, 0, Math.PI * 2);
|
||||
ctx.fillStyle = glowColor;
|
||||
ctx.globalAlpha = kp.confidence * 0.6;
|
||||
ctx.fill();
|
||||
|
||||
// Joint dot
|
||||
ctx.beginPath();
|
||||
ctx.arc(x, y, r, 0, Math.PI * 2);
|
||||
ctx.fillStyle = jointColor;
|
||||
ctx.globalAlpha = kp.confidence;
|
||||
ctx.fill();
|
||||
|
||||
// White center
|
||||
ctx.beginPath();
|
||||
ctx.arc(x, y, r * 0.4, 0, Math.PI * 2);
|
||||
ctx.fillStyle = '#fff';
|
||||
ctx.globalAlpha = kp.confidence * 0.8;
|
||||
ctx.fill();
|
||||
}
|
||||
|
||||
ctx.globalAlpha = 1;
|
||||
|
||||
// Confidence label
|
||||
if (opts.label) {
|
||||
ctx.font = '11px "JetBrains Mono", monospace';
|
||||
ctx.fillStyle = jointColor;
|
||||
ctx.globalAlpha = 0.8;
|
||||
ctx.fillText(opts.label, 8, height - 8);
|
||||
ctx.globalAlpha = 1;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Draw CSI amplitude heatmap
|
||||
* @param {CanvasRenderingContext2D} ctx
|
||||
* @param {{ data: Float32Array, width: number, height: number }} heatmap
|
||||
* @param {number} canvasW
|
||||
* @param {number} canvasH
|
||||
*/
|
||||
drawCsiHeatmap(ctx, heatmap, canvasW, canvasH) {
|
||||
ctx.clearRect(0, 0, canvasW, canvasH);
|
||||
|
||||
if (!heatmap || !heatmap.data || heatmap.height < 2) {
|
||||
ctx.fillStyle = '#0a0e18';
|
||||
ctx.fillRect(0, 0, canvasW, canvasH);
|
||||
ctx.font = '11px "JetBrains Mono", monospace';
|
||||
ctx.fillStyle = 'rgba(255,255,255,0.3)';
|
||||
ctx.fillText('Waiting for CSI data...', 8, canvasH / 2);
|
||||
return;
|
||||
}
|
||||
|
||||
const { data, width: dw, height: dh } = heatmap;
|
||||
const cellW = canvasW / dw;
|
||||
const cellH = canvasH / dh;
|
||||
|
||||
for (let y = 0; y < dh; y++) {
|
||||
for (let x = 0; x < dw; x++) {
|
||||
const val = Math.min(1, Math.max(0, data[y * dw + x]));
|
||||
ctx.fillStyle = this._heatmapColor(val);
|
||||
ctx.fillRect(x * cellW, y * cellH, cellW + 0.5, cellH + 0.5);
|
||||
}
|
||||
}
|
||||
|
||||
// Axis labels
|
||||
ctx.font = '9px "JetBrains Mono", monospace';
|
||||
ctx.fillStyle = 'rgba(255,255,255,0.4)';
|
||||
ctx.fillText('Subcarrier →', 4, canvasH - 4);
|
||||
ctx.save();
|
||||
ctx.translate(canvasW - 4, canvasH - 4);
|
||||
ctx.rotate(-Math.PI / 2);
|
||||
ctx.fillText('Time ↑', 0, 0);
|
||||
ctx.restore();
|
||||
}
|
||||
|
||||
/**
|
||||
* Draw embedding space 2D projection
|
||||
* @param {CanvasRenderingContext2D} ctx
|
||||
* @param {{ video: Array, csi: Array, fused: Array }} points
|
||||
* @param {number} w
|
||||
* @param {number} h
|
||||
*/
|
||||
drawEmbeddingSpace(ctx, points, w, h) {
|
||||
ctx.fillStyle = '#050810';
|
||||
ctx.fillRect(0, 0, w, h);
|
||||
|
||||
// Grid
|
||||
ctx.strokeStyle = 'rgba(255,255,255,0.05)';
|
||||
ctx.lineWidth = 0.5;
|
||||
for (let i = 0; i <= 4; i++) {
|
||||
const x = (i / 4) * w;
|
||||
ctx.beginPath(); ctx.moveTo(x, 0); ctx.lineTo(x, h); ctx.stroke();
|
||||
const y = (i / 4) * h;
|
||||
ctx.beginPath(); ctx.moveTo(0, y); ctx.lineTo(w, y); ctx.stroke();
|
||||
}
|
||||
|
||||
// Axes
|
||||
ctx.strokeStyle = 'rgba(255,255,255,0.1)';
|
||||
ctx.lineWidth = 1;
|
||||
ctx.beginPath(); ctx.moveTo(w / 2, 0); ctx.lineTo(w / 2, h); ctx.stroke();
|
||||
ctx.beginPath(); ctx.moveTo(0, h / 2); ctx.lineTo(w, h / 2); ctx.stroke();
|
||||
|
||||
const drawPoints = (pts, color, size) => {
|
||||
if (!pts || pts.length === 0) return;
|
||||
const len = pts.length;
|
||||
for (let i = 0; i < len; i++) {
|
||||
const p = pts[i];
|
||||
if (!p) continue;
|
||||
const age = 1 - (i / len) * 0.7; // Fade older points
|
||||
const px = w / 2 + p[0] * w * 0.35;
|
||||
const py = h / 2 + p[1] * h * 0.35;
|
||||
|
||||
if (px < 0 || px > w || py < 0 || py > h) continue;
|
||||
|
||||
ctx.beginPath();
|
||||
ctx.arc(px, py, size, 0, Math.PI * 2);
|
||||
ctx.fillStyle = color;
|
||||
ctx.globalAlpha = age * 0.7;
|
||||
ctx.fill();
|
||||
}
|
||||
};
|
||||
|
||||
drawPoints(points.video, this.colors.videoEmb, 3);
|
||||
drawPoints(points.csi, this.colors.csiEmb, 3);
|
||||
drawPoints(points.fused, this.colors.fusedEmb, 4);
|
||||
ctx.globalAlpha = 1;
|
||||
|
||||
// Legend
|
||||
ctx.font = '9px "JetBrains Mono", monospace';
|
||||
const legends = [
|
||||
{ color: this.colors.videoEmb, label: 'Video' },
|
||||
{ color: this.colors.csiEmb, label: 'CSI' },
|
||||
{ color: this.colors.fusedEmb, label: 'Fused' },
|
||||
];
|
||||
legends.forEach((l, i) => {
|
||||
const ly = 12 + i * 14;
|
||||
ctx.fillStyle = l.color;
|
||||
ctx.beginPath();
|
||||
ctx.arc(10, ly - 3, 3, 0, Math.PI * 2);
|
||||
ctx.fill();
|
||||
ctx.fillStyle = 'rgba(255,255,255,0.5)';
|
||||
ctx.fillText(l.label, 18, ly);
|
||||
});
|
||||
}
|
||||
|
||||
_heatmapColor(val) {
|
||||
// Dark blue → cyan → green → yellow → red
|
||||
if (val < 0.25) {
|
||||
const t = val / 0.25;
|
||||
return `rgb(${Math.floor(t * 20)}, ${Math.floor(20 + t * 60)}, ${Math.floor(60 + t * 100)})`;
|
||||
} else if (val < 0.5) {
|
||||
const t = (val - 0.25) / 0.25;
|
||||
return `rgb(${Math.floor(20 + t * 20)}, ${Math.floor(80 + t * 100)}, ${Math.floor(160 - t * 60)})`;
|
||||
} else if (val < 0.75) {
|
||||
const t = (val - 0.5) / 0.25;
|
||||
return `rgb(${Math.floor(40 + t * 180)}, ${Math.floor(180 + t * 75)}, ${Math.floor(100 - t * 80)})`;
|
||||
} else {
|
||||
const t = (val - 0.75) / 0.25;
|
||||
return `rgb(${Math.floor(220 + t * 35)}, ${Math.floor(255 - t * 120)}, ${Math.floor(20 - t * 20)})`;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,226 +0,0 @@
|
||||
/**
|
||||
* CNN Embedder — Lightweight MobileNet-V3-style feature extractor.
|
||||
*
|
||||
* Architecture mirrors ruvector-cnn: Conv2D → BatchNorm → ReLU → Pool → Project → L2 Normalize
|
||||
* Uses pre-seeded random weights (deterministic). When ruvector-cnn-wasm is available,
|
||||
* transparently delegates to the WASM implementation.
|
||||
*
|
||||
* Two instances are created: one for video frames, one for CSI pseudo-images.
|
||||
*/
|
||||
|
||||
// Seeded PRNG for deterministic weight initialization
|
||||
function mulberry32(seed) {
|
||||
return function() {
|
||||
let t = (seed += 0x6D2B79F5);
|
||||
t = Math.imul(t ^ (t >>> 15), t | 1);
|
||||
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
|
||||
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
|
||||
};
|
||||
}
|
||||
|
||||
export class CnnEmbedder {
|
||||
/**
|
||||
* @param {object} opts
|
||||
* @param {number} opts.inputSize - Square input dimension (default 56 for speed)
|
||||
* @param {number} opts.embeddingDim - Output embedding dimension (default 128)
|
||||
* @param {boolean} opts.normalize - L2 normalize output
|
||||
* @param {number} opts.seed - PRNG seed for weight init
|
||||
*/
|
||||
constructor(opts = {}) {
|
||||
this.inputSize = opts.inputSize || 56;
|
||||
this.embeddingDim = opts.embeddingDim || 128;
|
||||
this.normalize = opts.normalize !== false;
|
||||
this.wasmEmbedder = null;
|
||||
|
||||
// Initialize weights with deterministic PRNG
|
||||
const rng = mulberry32(opts.seed || 42);
|
||||
const randRange = (lo, hi) => lo + rng() * (hi - lo);
|
||||
|
||||
// Conv 3x3: 3 input channels → 16 output channels
|
||||
this.convWeights = new Float32Array(3 * 3 * 3 * 16);
|
||||
for (let i = 0; i < this.convWeights.length; i++) {
|
||||
this.convWeights[i] = randRange(-0.15, 0.15);
|
||||
}
|
||||
|
||||
// BatchNorm params (16 channels)
|
||||
this.bnGamma = new Float32Array(16).fill(1.0);
|
||||
this.bnBeta = new Float32Array(16).fill(0.0);
|
||||
this.bnMean = new Float32Array(16).fill(0.0);
|
||||
this.bnVar = new Float32Array(16).fill(1.0);
|
||||
|
||||
// Projection: 16 → embeddingDim
|
||||
this.projWeights = new Float32Array(16 * this.embeddingDim);
|
||||
for (let i = 0; i < this.projWeights.length; i++) {
|
||||
this.projWeights[i] = randRange(-0.1, 0.1);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Try to load WASM embedder from ruvector-cnn-wasm package
|
||||
* @param {string} wasmPath - Path to the WASM package directory
|
||||
*/
|
||||
async tryLoadWasm(wasmPath) {
|
||||
try {
|
||||
const mod = await import(`${wasmPath}/ruvector_cnn_wasm.js`);
|
||||
await mod.default();
|
||||
const config = new mod.EmbedderConfig();
|
||||
config.input_size = this.inputSize;
|
||||
config.embedding_dim = this.embeddingDim;
|
||||
config.normalize = this.normalize;
|
||||
this.wasmEmbedder = new mod.WasmCnnEmbedder(config);
|
||||
console.log('[CNN] WASM embedder loaded successfully');
|
||||
return true;
|
||||
} catch (e) {
|
||||
console.log('[CNN] WASM not available, using JS fallback:', e.message);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract embedding from RGB image data
|
||||
* @param {Uint8Array} rgbData - RGB pixel data (H*W*3)
|
||||
* @param {number} width
|
||||
* @param {number} height
|
||||
* @returns {Float32Array} embedding vector
|
||||
*/
|
||||
extract(rgbData, width, height) {
|
||||
if (this.wasmEmbedder) {
|
||||
try {
|
||||
const result = this.wasmEmbedder.extract(rgbData, width, height);
|
||||
return new Float32Array(result);
|
||||
} catch (_) { /* fallback to JS */ }
|
||||
}
|
||||
return this._extractJS(rgbData, width, height);
|
||||
}
|
||||
|
||||
_extractJS(rgbData, width, height) {
|
||||
// 1. Resize to inputSize × inputSize if needed
|
||||
const sz = this.inputSize;
|
||||
let input;
|
||||
if (width === sz && height === sz) {
|
||||
input = new Float32Array(rgbData.length);
|
||||
for (let i = 0; i < rgbData.length; i++) input[i] = rgbData[i] / 255.0;
|
||||
} else {
|
||||
input = this._resize(rgbData, width, height, sz, sz);
|
||||
}
|
||||
|
||||
// 2. ImageNet normalization
|
||||
const mean = [0.485, 0.456, 0.406];
|
||||
const std = [0.229, 0.224, 0.225];
|
||||
const pixels = sz * sz;
|
||||
for (let i = 0; i < pixels; i++) {
|
||||
input[i * 3] = (input[i * 3] - mean[0]) / std[0];
|
||||
input[i * 3 + 1] = (input[i * 3 + 1] - mean[1]) / std[1];
|
||||
input[i * 3 + 2] = (input[i * 3 + 2] - mean[2]) / std[2];
|
||||
}
|
||||
|
||||
// 3. Conv2D 3x3 (3 → 16 channels)
|
||||
const convOut = this._conv2d3x3(input, sz, sz, 3, 16);
|
||||
|
||||
// 4. BatchNorm
|
||||
this._batchNorm(convOut, 16);
|
||||
|
||||
// 5. ReLU
|
||||
for (let i = 0; i < convOut.length; i++) {
|
||||
if (convOut[i] < 0) convOut[i] = 0;
|
||||
}
|
||||
|
||||
// 6. Global average pooling → 16-dim
|
||||
const outH = sz - 2, outW = sz - 2;
|
||||
const pooled = new Float32Array(16);
|
||||
const spatial = outH * outW;
|
||||
for (let i = 0; i < spatial; i++) {
|
||||
for (let c = 0; c < 16; c++) {
|
||||
pooled[c] += convOut[i * 16 + c];
|
||||
}
|
||||
}
|
||||
for (let c = 0; c < 16; c++) pooled[c] /= spatial;
|
||||
|
||||
// 7. Linear projection → embeddingDim
|
||||
const emb = new Float32Array(this.embeddingDim);
|
||||
for (let o = 0; o < this.embeddingDim; o++) {
|
||||
let sum = 0;
|
||||
for (let i = 0; i < 16; i++) {
|
||||
sum += pooled[i] * this.projWeights[i * this.embeddingDim + o];
|
||||
}
|
||||
emb[o] = sum;
|
||||
}
|
||||
|
||||
// 8. L2 normalize
|
||||
if (this.normalize) {
|
||||
let norm = 0;
|
||||
for (let i = 0; i < emb.length; i++) norm += emb[i] * emb[i];
|
||||
norm = Math.sqrt(norm);
|
||||
if (norm > 1e-8) {
|
||||
for (let i = 0; i < emb.length; i++) emb[i] /= norm;
|
||||
}
|
||||
}
|
||||
|
||||
return emb;
|
||||
}
|
||||
|
||||
_conv2d3x3(input, H, W, Cin, Cout) {
|
||||
const outH = H - 2, outW = W - 2;
|
||||
const output = new Float32Array(outH * outW * Cout);
|
||||
for (let y = 0; y < outH; y++) {
|
||||
for (let x = 0; x < outW; x++) {
|
||||
for (let co = 0; co < Cout; co++) {
|
||||
let sum = 0;
|
||||
for (let ky = 0; ky < 3; ky++) {
|
||||
for (let kx = 0; kx < 3; kx++) {
|
||||
for (let ci = 0; ci < Cin; ci++) {
|
||||
const px = ((y + ky) * W + (x + kx)) * Cin + ci;
|
||||
const wt = (((ky * 3 + kx) * Cin) + ci) * Cout + co;
|
||||
sum += input[px] * this.convWeights[wt];
|
||||
}
|
||||
}
|
||||
}
|
||||
output[(y * outW + x) * Cout + co] = sum;
|
||||
}
|
||||
}
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
_batchNorm(data, channels) {
|
||||
const spatial = data.length / channels;
|
||||
for (let i = 0; i < spatial; i++) {
|
||||
for (let c = 0; c < channels; c++) {
|
||||
const idx = i * channels + c;
|
||||
data[idx] = this.bnGamma[c] * (data[idx] - this.bnMean[c]) / Math.sqrt(this.bnVar[c] + 1e-5) + this.bnBeta[c];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
_resize(rgbData, srcW, srcH, dstW, dstH) {
|
||||
const output = new Float32Array(dstW * dstH * 3);
|
||||
const xRatio = srcW / dstW;
|
||||
const yRatio = srcH / dstH;
|
||||
for (let y = 0; y < dstH; y++) {
|
||||
for (let x = 0; x < dstW; x++) {
|
||||
const sx = Math.min(Math.floor(x * xRatio), srcW - 1);
|
||||
const sy = Math.min(Math.floor(y * yRatio), srcH - 1);
|
||||
const srcIdx = (sy * srcW + sx) * 3;
|
||||
const dstIdx = (y * dstW + x) * 3;
|
||||
output[dstIdx] = rgbData[srcIdx] / 255.0;
|
||||
output[dstIdx + 1] = rgbData[srcIdx + 1] / 255.0;
|
||||
output[dstIdx + 2] = rgbData[srcIdx + 2] / 255.0;
|
||||
}
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
/** Cosine similarity between two embeddings */
|
||||
static cosineSimilarity(a, b) {
|
||||
let dot = 0, normA = 0, normB = 0;
|
||||
for (let i = 0; i < a.length; i++) {
|
||||
dot += a[i] * b[i];
|
||||
normA += a[i] * a[i];
|
||||
normB += b[i] * b[i];
|
||||
}
|
||||
normA = Math.sqrt(normA);
|
||||
normB = Math.sqrt(normB);
|
||||
if (normA < 1e-8 || normB < 1e-8) return 0;
|
||||
return dot / (normA * normB);
|
||||
}
|
||||
}
|
||||
@@ -1,267 +0,0 @@
|
||||
/**
|
||||
* CSI Simulator — Generates realistic WiFi Channel State Information data.
|
||||
*
|
||||
* In live mode, connects to the sensing server via WebSocket.
|
||||
* In demo mode, generates synthetic CSI that correlates with detected motion.
|
||||
*
|
||||
* Outputs: 3-channel pseudo-image (amplitude, phase, temporal diff)
|
||||
* matching the ADR-018 frame format expectations.
|
||||
*/
|
||||
|
||||
export class CsiSimulator {
|
||||
constructor(opts = {}) {
|
||||
this.subcarriers = opts.subcarriers || 52; // 802.11n HT20
|
||||
this.timeWindow = opts.timeWindow || 56; // frames in sliding window
|
||||
this.mode = 'demo'; // 'demo' | 'live'
|
||||
this.ws = null;
|
||||
|
||||
// Circular buffer for CSI frames
|
||||
this.amplitudeBuffer = [];
|
||||
this.phaseBuffer = [];
|
||||
this.frameCount = 0;
|
||||
|
||||
// Noise parameters
|
||||
this._rng = this._mulberry32(opts.seed || 7);
|
||||
this._noiseState = new Float32Array(this.subcarriers);
|
||||
this._baseAmplitude = new Float32Array(this.subcarriers);
|
||||
this._basePhase = new Float32Array(this.subcarriers);
|
||||
|
||||
// Initialize base CSI profile (empty room)
|
||||
for (let i = 0; i < this.subcarriers; i++) {
|
||||
this._baseAmplitude[i] = 0.5 + 0.3 * Math.sin(i * 0.12);
|
||||
this._basePhase[i] = (i / this.subcarriers) * Math.PI * 2;
|
||||
}
|
||||
|
||||
// Person influence (updated from video motion)
|
||||
this.personPresence = 0;
|
||||
this.personX = 0.5;
|
||||
this.personY = 0.5;
|
||||
this.personMotion = 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Connect to live sensing server WebSocket
|
||||
* @param {string} url - WebSocket URL (e.g. ws://localhost:3030/ws/csi)
|
||||
*/
|
||||
async connectLive(url) {
|
||||
return new Promise((resolve) => {
|
||||
try {
|
||||
this.ws = new WebSocket(url);
|
||||
this.ws.binaryType = 'arraybuffer';
|
||||
this.ws.onmessage = (evt) => this._handleLiveFrame(evt.data);
|
||||
this.ws.onopen = () => { this.mode = 'live'; resolve(true); };
|
||||
this.ws.onerror = () => resolve(false);
|
||||
this.ws.onclose = () => { this.mode = 'demo'; };
|
||||
// Timeout after 3s
|
||||
setTimeout(() => { if (this.mode !== 'live') resolve(false); }, 3000);
|
||||
} catch {
|
||||
resolve(false);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
disconnect() {
|
||||
if (this.ws) { this.ws.close(); this.ws = null; }
|
||||
this.mode = 'demo';
|
||||
}
|
||||
|
||||
get isLive() { return this.mode === 'live'; }
|
||||
|
||||
/**
|
||||
* Update person state from video detection (for correlated demo data).
|
||||
* When person exits frame, CSI maintains presence with slow decay
|
||||
* (simulating through-wall sensing capability).
|
||||
*/
|
||||
updatePersonState(presence, x, y, motion) {
|
||||
if (presence > 0.1) {
|
||||
// Person detected in video — update CSI state directly
|
||||
this.personPresence = presence;
|
||||
this.personX = x;
|
||||
this.personY = y;
|
||||
this.personMotion = motion;
|
||||
this._lastSeenTime = performance.now();
|
||||
this._lastSeenX = x;
|
||||
this._lastSeenY = y;
|
||||
} else if (this._lastSeenTime) {
|
||||
// Person NOT in video — CSI "through-wall" persistence
|
||||
const elapsed = (performance.now() - this._lastSeenTime) / 1000;
|
||||
// CSI can sense through walls for ~10 seconds with decaying confidence
|
||||
const decayRate = 0.15; // Lose ~15% per second
|
||||
this.personPresence = Math.max(0, 1.0 - elapsed * decayRate);
|
||||
// Position slowly drifts (person walking behind wall)
|
||||
this.personX = this._lastSeenX;
|
||||
this.personY = this._lastSeenY;
|
||||
this.personMotion = Math.max(0, motion * 0.5 + this.personPresence * 0.2);
|
||||
|
||||
if (this.personPresence < 0.05) {
|
||||
this._lastSeenTime = null;
|
||||
}
|
||||
} else {
|
||||
this.personPresence = 0;
|
||||
this.personMotion = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate next CSI frame (demo mode) or return latest live frame
|
||||
* @param {number} elapsed - Time in seconds
|
||||
* @returns {{ amplitude: Float32Array, phase: Float32Array, snr: number }}
|
||||
*/
|
||||
nextFrame(elapsed) {
|
||||
const amp = new Float32Array(this.subcarriers);
|
||||
const phase = new Float32Array(this.subcarriers);
|
||||
|
||||
if (this.mode === 'live' && this._liveAmplitude) {
|
||||
amp.set(this._liveAmplitude);
|
||||
phase.set(this._livePhase);
|
||||
} else {
|
||||
this._generateDemoFrame(amp, phase, elapsed);
|
||||
}
|
||||
|
||||
// Push to circular buffer
|
||||
this.amplitudeBuffer.push(new Float32Array(amp));
|
||||
this.phaseBuffer.push(new Float32Array(phase));
|
||||
if (this.amplitudeBuffer.length > this.timeWindow) {
|
||||
this.amplitudeBuffer.shift();
|
||||
this.phaseBuffer.shift();
|
||||
}
|
||||
|
||||
// SNR estimate
|
||||
let signalPower = 0, noisePower = 0;
|
||||
for (let i = 0; i < this.subcarriers; i++) {
|
||||
signalPower += amp[i] * amp[i];
|
||||
noisePower += this._noiseState[i] * this._noiseState[i];
|
||||
}
|
||||
const snr = noisePower > 0 ? 10 * Math.log10(signalPower / noisePower) : 30;
|
||||
|
||||
this.frameCount++;
|
||||
return { amplitude: amp, phase, snr: Math.max(0, Math.min(40, snr)) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Build 3-channel pseudo-image for CNN input
|
||||
* @param {number} targetSize - Output image dimension (square)
|
||||
* @returns {Uint8Array} RGB data (targetSize * targetSize * 3)
|
||||
*/
|
||||
buildPseudoImage(targetSize = 56) {
|
||||
const buf = this.amplitudeBuffer;
|
||||
const pBuf = this.phaseBuffer;
|
||||
const frames = buf.length;
|
||||
if (frames < 2) {
|
||||
return new Uint8Array(targetSize * targetSize * 3);
|
||||
}
|
||||
|
||||
const rgb = new Uint8Array(targetSize * targetSize * 3);
|
||||
|
||||
for (let y = 0; y < targetSize; y++) {
|
||||
const fi = Math.min(Math.floor(y / targetSize * frames), frames - 1);
|
||||
for (let x = 0; x < targetSize; x++) {
|
||||
const si = Math.min(Math.floor(x / targetSize * this.subcarriers), this.subcarriers - 1);
|
||||
const idx = (y * targetSize + x) * 3;
|
||||
|
||||
// R: Amplitude (normalized to 0-255)
|
||||
const ampVal = buf[fi][si];
|
||||
rgb[idx] = Math.min(255, Math.max(0, Math.floor(ampVal * 255)));
|
||||
|
||||
// G: Phase (wrapped to 0-255)
|
||||
const phaseVal = (pBuf[fi][si] % (2 * Math.PI) + 2 * Math.PI) % (2 * Math.PI);
|
||||
rgb[idx + 1] = Math.floor(phaseVal / (2 * Math.PI) * 255);
|
||||
|
||||
// B: Temporal difference
|
||||
if (fi > 0) {
|
||||
const diff = Math.abs(buf[fi][si] - buf[fi - 1][si]);
|
||||
rgb[idx + 2] = Math.min(255, Math.floor(diff * 500));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return rgb;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get heatmap data for visualization
|
||||
* @returns {{ data: Float32Array, width: number, height: number }}
|
||||
*/
|
||||
getHeatmapData() {
|
||||
const frames = this.amplitudeBuffer.length;
|
||||
const w = this.subcarriers;
|
||||
const h = Math.min(frames, this.timeWindow);
|
||||
const data = new Float32Array(w * h);
|
||||
for (let y = 0; y < h; y++) {
|
||||
const fi = frames - h + y;
|
||||
if (fi >= 0 && fi < frames) {
|
||||
for (let x = 0; x < w; x++) {
|
||||
data[y * w + x] = this.amplitudeBuffer[fi][x];
|
||||
}
|
||||
}
|
||||
}
|
||||
return { data, width: w, height: h };
|
||||
}
|
||||
|
||||
// === Private ===
|
||||
|
||||
_generateDemoFrame(amp, phase, elapsed) {
|
||||
const rng = this._rng;
|
||||
const presence = this.personPresence;
|
||||
const motion = this.personMotion;
|
||||
const px = this.personX;
|
||||
|
||||
for (let i = 0; i < this.subcarriers; i++) {
|
||||
// Base CSI profile (frequency-selective channel)
|
||||
let a = this._baseAmplitude[i];
|
||||
let p = this._basePhase[i] + elapsed * 0.05;
|
||||
|
||||
// Environmental noise (correlated across subcarriers)
|
||||
this._noiseState[i] = 0.95 * this._noiseState[i] + 0.05 * (rng() * 2 - 1) * 0.03;
|
||||
a += this._noiseState[i];
|
||||
|
||||
// Person-induced CSI perturbation
|
||||
if (presence > 0.1) {
|
||||
// Subcarrier-dependent body reflection (Fresnel zone model)
|
||||
const freqOffset = (i - this.subcarriers * px) / (this.subcarriers * 0.3);
|
||||
const bodyReflection = presence * 0.25 * Math.exp(-freqOffset * freqOffset);
|
||||
|
||||
// Motion causes amplitude fluctuation
|
||||
const motionEffect = motion * 0.15 * Math.sin(elapsed * 3.5 + i * 0.3);
|
||||
|
||||
// Breathing modulation (0.2-0.3 Hz)
|
||||
const breathing = presence * 0.02 * Math.sin(elapsed * 1.5 + i * 0.05);
|
||||
|
||||
a += bodyReflection + motionEffect + breathing;
|
||||
p += presence * 0.4 * Math.sin(elapsed * 2.1 + i * 0.15);
|
||||
}
|
||||
|
||||
amp[i] = Math.max(0, Math.min(1, a));
|
||||
phase[i] = p;
|
||||
}
|
||||
}
|
||||
|
||||
_handleLiveFrame(data) {
|
||||
const view = new DataView(data);
|
||||
// Check ADR-018 magic: 0xC5110001
|
||||
if (data.byteLength < 20) return;
|
||||
const magic = view.getUint32(0, true);
|
||||
if (magic !== 0xC5110001) return;
|
||||
|
||||
const numSub = Math.min(view.getUint16(8, true), this.subcarriers);
|
||||
this._liveAmplitude = new Float32Array(this.subcarriers);
|
||||
this._livePhase = new Float32Array(this.subcarriers);
|
||||
|
||||
const headerSize = 20;
|
||||
for (let i = 0; i < numSub && (headerSize + i * 4 + 3) < data.byteLength; i++) {
|
||||
const real = view.getInt16(headerSize + i * 4, true);
|
||||
const imag = view.getInt16(headerSize + i * 4 + 2, true);
|
||||
this._liveAmplitude[i] = Math.sqrt(real * real + imag * imag) / 2048;
|
||||
this._livePhase[i] = Math.atan2(imag, real);
|
||||
}
|
||||
}
|
||||
|
||||
_mulberry32(seed) {
|
||||
return function() {
|
||||
let t = (seed += 0x6D2B79F5);
|
||||
t = Math.imul(t ^ (t >>> 15), t | 1);
|
||||
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
|
||||
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -1,166 +0,0 @@
|
||||
/**
|
||||
* FusionEngine — Attention-weighted dual-modal embedding fusion.
|
||||
*
|
||||
* Combines visual (camera) and CSI (WiFi) embeddings with dynamic
|
||||
* confidence gating based on signal quality.
|
||||
*/
|
||||
|
||||
export class FusionEngine {
|
||||
/**
|
||||
* @param {number} embeddingDim
|
||||
*/
|
||||
constructor(embeddingDim = 128) {
|
||||
this.embeddingDim = embeddingDim;
|
||||
|
||||
// Learnable attention weights (initialized to balanced 0.5)
|
||||
// In production, these would be loaded from trained JSON
|
||||
this.attentionWeights = new Float32Array(embeddingDim).fill(0.5);
|
||||
|
||||
// Dynamic modality confidence [0, 1]
|
||||
this.videoConfidence = 1.0;
|
||||
this.csiConfidence = 0.0;
|
||||
this.fusedConfidence = 0.5;
|
||||
|
||||
// Smoothing for confidence transitions
|
||||
this._smoothAlpha = 0.85;
|
||||
|
||||
// Embedding history for visualization
|
||||
this.recentVideoEmbeddings = [];
|
||||
this.recentCsiEmbeddings = [];
|
||||
this.recentFusedEmbeddings = [];
|
||||
this.maxHistory = 50;
|
||||
}
|
||||
|
||||
/**
|
||||
* Update quality-based confidence scores
|
||||
* @param {number} videoBrightness - [0,1] video brightness quality
|
||||
* @param {number} videoMotion - [0,1] motion detected
|
||||
* @param {number} csiSnr - CSI signal-to-noise ratio in dB
|
||||
* @param {boolean} csiActive - Whether CSI source is connected
|
||||
*/
|
||||
updateConfidence(videoBrightness, videoMotion, csiSnr, csiActive) {
|
||||
// Video confidence: drops with low brightness, boosted by motion
|
||||
let vc = 0;
|
||||
if (videoBrightness > 0.05) {
|
||||
vc = Math.min(1, videoBrightness * 1.5) * 0.7 + Math.min(1, videoMotion * 3) * 0.3;
|
||||
}
|
||||
|
||||
// CSI confidence: based on SNR and connection status
|
||||
let cc = 0;
|
||||
if (csiActive) {
|
||||
cc = Math.min(1, csiSnr / 25); // 25dB = full confidence
|
||||
}
|
||||
|
||||
// Smooth transitions
|
||||
this.videoConfidence = this._smoothAlpha * this.videoConfidence + (1 - this._smoothAlpha) * vc;
|
||||
this.csiConfidence = this._smoothAlpha * this.csiConfidence + (1 - this._smoothAlpha) * cc;
|
||||
|
||||
// Fused confidence is the max of either (fusion can only help)
|
||||
this.fusedConfidence = Math.min(1, Math.sqrt(
|
||||
this.videoConfidence * this.videoConfidence + this.csiConfidence * this.csiConfidence
|
||||
));
|
||||
}
|
||||
|
||||
/**
|
||||
* Fuse video and CSI embeddings
|
||||
* @param {Float32Array|null} videoEmb - Visual embedding (or null if video-off)
|
||||
* @param {Float32Array|null} csiEmb - CSI embedding (or null if CSI-off)
|
||||
* @param {string} mode - 'dual' | 'video' | 'csi'
|
||||
* @returns {Float32Array} Fused embedding
|
||||
*/
|
||||
fuse(videoEmb, csiEmb, mode = 'dual') {
|
||||
const dim = this.embeddingDim;
|
||||
const fused = new Float32Array(dim);
|
||||
|
||||
if (mode === 'video' || !csiEmb) {
|
||||
if (videoEmb) fused.set(videoEmb);
|
||||
this._recordEmbedding(videoEmb, null, fused);
|
||||
return fused;
|
||||
}
|
||||
|
||||
if (mode === 'csi' || !videoEmb) {
|
||||
if (csiEmb) fused.set(csiEmb);
|
||||
this._recordEmbedding(null, csiEmb, fused);
|
||||
return fused;
|
||||
}
|
||||
|
||||
// Dual mode: attention-weighted fusion with confidence gating
|
||||
const totalConf = this.videoConfidence + this.csiConfidence;
|
||||
const videoWeight = totalConf > 0 ? this.videoConfidence / totalConf : 0.5;
|
||||
|
||||
for (let i = 0; i < dim; i++) {
|
||||
const alpha = this.attentionWeights[i] * videoWeight +
|
||||
(1 - this.attentionWeights[i]) * (1 - videoWeight);
|
||||
fused[i] = alpha * videoEmb[i] + (1 - alpha) * csiEmb[i];
|
||||
}
|
||||
|
||||
// Re-normalize
|
||||
let norm = 0;
|
||||
for (let i = 0; i < dim; i++) norm += fused[i] * fused[i];
|
||||
norm = Math.sqrt(norm);
|
||||
if (norm > 1e-8) {
|
||||
for (let i = 0; i < dim; i++) fused[i] /= norm;
|
||||
}
|
||||
|
||||
this._recordEmbedding(videoEmb, csiEmb, fused);
|
||||
return fused;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get embedding pairs for 2D visualization (PCA projection)
|
||||
* @returns {{ video: Array, csi: Array, fused: Array }}
|
||||
*/
|
||||
getEmbeddingPoints() {
|
||||
// Simple 2D projection using first two principal components (approximated)
|
||||
const project = (emb) => {
|
||||
if (!emb || emb.length < 4) return null;
|
||||
// Use pairs of dimensions as crude 2D projection
|
||||
let x = 0, y = 0;
|
||||
for (let i = 0; i < emb.length; i += 2) {
|
||||
x += emb[i] * (i % 4 < 2 ? 1 : -1);
|
||||
if (i + 1 < emb.length) {
|
||||
y += emb[i + 1] * (i % 4 < 2 ? 1 : -1);
|
||||
}
|
||||
}
|
||||
return [x * 2, y * 2]; // Scale for visibility
|
||||
};
|
||||
|
||||
return {
|
||||
video: this.recentVideoEmbeddings.map(project).filter(Boolean),
|
||||
csi: this.recentCsiEmbeddings.map(project).filter(Boolean),
|
||||
fused: this.recentFusedEmbeddings.map(project).filter(Boolean)
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Cross-modal similarity score
|
||||
* @returns {number} Cosine similarity between latest video and CSI embeddings
|
||||
*/
|
||||
getCrossModalSimilarity() {
|
||||
const v = this.recentVideoEmbeddings[this.recentVideoEmbeddings.length - 1];
|
||||
const c = this.recentCsiEmbeddings[this.recentCsiEmbeddings.length - 1];
|
||||
if (!v || !c) return 0;
|
||||
|
||||
let dot = 0, na = 0, nb = 0;
|
||||
for (let i = 0; i < v.length; i++) {
|
||||
dot += v[i] * c[i];
|
||||
na += v[i] * v[i];
|
||||
nb += c[i] * c[i];
|
||||
}
|
||||
na = Math.sqrt(na); nb = Math.sqrt(nb);
|
||||
return (na > 1e-8 && nb > 1e-8) ? dot / (na * nb) : 0;
|
||||
}
|
||||
|
||||
_recordEmbedding(video, csi, fused) {
|
||||
if (video) {
|
||||
this.recentVideoEmbeddings.push(new Float32Array(video));
|
||||
if (this.recentVideoEmbeddings.length > this.maxHistory) this.recentVideoEmbeddings.shift();
|
||||
}
|
||||
if (csi) {
|
||||
this.recentCsiEmbeddings.push(new Float32Array(csi));
|
||||
if (this.recentCsiEmbeddings.length > this.maxHistory) this.recentCsiEmbeddings.shift();
|
||||
}
|
||||
this.recentFusedEmbeddings.push(new Float32Array(fused));
|
||||
if (this.recentFusedEmbeddings.length > this.maxHistory) this.recentFusedEmbeddings.shift();
|
||||
}
|
||||
}
|
||||
@@ -1,315 +0,0 @@
|
||||
/**
|
||||
* WiFi-DensePose — Dual-Modal Pose Estimation Demo
|
||||
*
|
||||
* Main orchestration: video capture → CNN embedding → CSI processing → fusion → rendering
|
||||
*/
|
||||
|
||||
import { VideoCapture } from './video-capture.js';
|
||||
import { CsiSimulator } from './csi-simulator.js';
|
||||
import { CnnEmbedder } from './cnn-embedder.js';
|
||||
import { FusionEngine } from './fusion-engine.js';
|
||||
import { PoseDecoder } from './pose-decoder.js';
|
||||
import { CanvasRenderer } from './canvas-renderer.js';
|
||||
|
||||
// === State ===
|
||||
let mode = 'dual'; // 'dual' | 'video' | 'csi'
|
||||
let isRunning = false;
|
||||
let isPaused = false;
|
||||
let startTime = 0;
|
||||
let frameCount = 0;
|
||||
let fps = 0;
|
||||
let lastFpsTime = 0;
|
||||
let confidenceThreshold = 0.3;
|
||||
|
||||
// Latency tracking
|
||||
const latency = { video: 0, csi: 0, fusion: 0, total: 0 };
|
||||
|
||||
// === Components ===
|
||||
const videoCapture = new VideoCapture(document.getElementById('webcam'));
|
||||
const csiSimulator = new CsiSimulator({ subcarriers: 52, timeWindow: 56 });
|
||||
const visualCnn = new CnnEmbedder({ inputSize: 56, embeddingDim: 128, seed: 42 });
|
||||
const csiCnn = new CnnEmbedder({ inputSize: 56, embeddingDim: 128, seed: 137 });
|
||||
const fusionEngine = new FusionEngine(128);
|
||||
const poseDecoder = new PoseDecoder(128);
|
||||
const renderer = new CanvasRenderer();
|
||||
|
||||
// === Canvas Elements ===
|
||||
const skeletonCanvas = document.getElementById('skeleton-canvas');
|
||||
const skeletonCtx = skeletonCanvas.getContext('2d');
|
||||
const csiCanvas = document.getElementById('csi-canvas');
|
||||
const csiCtx = csiCanvas.getContext('2d');
|
||||
const embeddingCanvas = document.getElementById('embedding-canvas');
|
||||
const embeddingCtx = embeddingCanvas.getContext('2d');
|
||||
|
||||
// === UI Elements ===
|
||||
const modeSelect = document.getElementById('mode-select');
|
||||
const statusDot = document.getElementById('status-dot');
|
||||
const statusLabel = document.getElementById('status-label');
|
||||
const fpsDisplay = document.getElementById('fps-display');
|
||||
const cameraPrompt = document.getElementById('camera-prompt');
|
||||
const startCameraBtn = document.getElementById('start-camera-btn');
|
||||
const pauseBtn = document.getElementById('pause-btn');
|
||||
const confSlider = document.getElementById('confidence-slider');
|
||||
const confValue = document.getElementById('confidence-value');
|
||||
const wsUrlInput = document.getElementById('ws-url');
|
||||
const connectWsBtn = document.getElementById('connect-ws-btn');
|
||||
|
||||
// Fusion bar elements
|
||||
const videoBar = document.getElementById('video-bar');
|
||||
const csiBar = document.getElementById('csi-bar');
|
||||
const fusedBar = document.getElementById('fused-bar');
|
||||
const videoBarVal = document.getElementById('video-bar-val');
|
||||
const csiBarVal = document.getElementById('csi-bar-val');
|
||||
const fusedBarVal = document.getElementById('fused-bar-val');
|
||||
|
||||
// Latency elements
|
||||
const latVideoEl = document.getElementById('lat-video');
|
||||
const latCsiEl = document.getElementById('lat-csi');
|
||||
const latFusionEl = document.getElementById('lat-fusion');
|
||||
const latTotalEl = document.getElementById('lat-total');
|
||||
|
||||
// Cross-modal similarity
|
||||
const crossModalEl = document.getElementById('cross-modal-sim');
|
||||
|
||||
// === Initialize ===
|
||||
function init() {
|
||||
resizeCanvases();
|
||||
window.addEventListener('resize', resizeCanvases);
|
||||
|
||||
// Mode change
|
||||
modeSelect.addEventListener('change', (e) => {
|
||||
mode = e.target.value;
|
||||
updateModeUI();
|
||||
});
|
||||
|
||||
// Camera start
|
||||
startCameraBtn.addEventListener('click', startCamera);
|
||||
|
||||
// Pause
|
||||
pauseBtn.addEventListener('click', () => {
|
||||
isPaused = !isPaused;
|
||||
pauseBtn.textContent = isPaused ? '▶ Resume' : '⏸ Pause';
|
||||
pauseBtn.classList.toggle('active', isPaused);
|
||||
});
|
||||
|
||||
// Confidence slider
|
||||
confSlider.addEventListener('input', (e) => {
|
||||
confidenceThreshold = parseFloat(e.target.value);
|
||||
confValue.textContent = confidenceThreshold.toFixed(2);
|
||||
});
|
||||
|
||||
// WebSocket connect
|
||||
connectWsBtn.addEventListener('click', async () => {
|
||||
const url = wsUrlInput.value.trim();
|
||||
if (!url) return;
|
||||
connectWsBtn.textContent = 'Connecting...';
|
||||
const ok = await csiSimulator.connectLive(url);
|
||||
connectWsBtn.textContent = ok ? '✓ Connected' : 'Connect';
|
||||
if (ok) {
|
||||
connectWsBtn.classList.add('active');
|
||||
}
|
||||
});
|
||||
|
||||
// Try to load WASM embedders (non-blocking)
|
||||
// Resolve relative to this JS module file (in pose-fusion/js/) → ../pkg/
|
||||
const wasmBase = new URL('../pkg/ruvector_cnn_wasm', import.meta.url).href;
|
||||
visualCnn.tryLoadWasm(wasmBase);
|
||||
csiCnn.tryLoadWasm(wasmBase);
|
||||
|
||||
// Auto-connect to local sensing server WebSocket if available
|
||||
const defaultWsUrl = 'ws://localhost:8765/ws/sensing';
|
||||
if (wsUrlInput) wsUrlInput.value = defaultWsUrl;
|
||||
csiSimulator.connectLive(defaultWsUrl).then(ok => {
|
||||
if (ok && connectWsBtn) {
|
||||
connectWsBtn.textContent = '✓ Live ESP32';
|
||||
connectWsBtn.classList.add('active');
|
||||
statusLabel.textContent = 'LIVE CSI';
|
||||
statusDot.classList.remove('offline');
|
||||
}
|
||||
});
|
||||
|
||||
// Auto-start camera for video/dual modes
|
||||
updateModeUI();
|
||||
startTime = performance.now() / 1000;
|
||||
isRunning = true;
|
||||
requestAnimationFrame(mainLoop);
|
||||
}
|
||||
|
||||
async function startCamera() {
|
||||
cameraPrompt.style.display = 'none';
|
||||
const ok = await videoCapture.start();
|
||||
if (ok) {
|
||||
statusDot.classList.remove('offline');
|
||||
statusLabel.textContent = 'LIVE';
|
||||
resizeCanvases();
|
||||
} else {
|
||||
cameraPrompt.style.display = 'flex';
|
||||
cameraPrompt.querySelector('p').textContent = 'Camera access denied. Try CSI-only mode.';
|
||||
}
|
||||
}
|
||||
|
||||
function updateModeUI() {
|
||||
const needsVideo = mode !== 'csi';
|
||||
const needsCsi = mode !== 'video';
|
||||
|
||||
// Show/hide camera prompt
|
||||
if (needsVideo && !videoCapture.isActive) {
|
||||
cameraPrompt.style.display = 'flex';
|
||||
} else {
|
||||
cameraPrompt.style.display = 'none';
|
||||
}
|
||||
}
|
||||
|
||||
function resizeCanvases() {
|
||||
const videoPanel = document.querySelector('.video-panel');
|
||||
if (videoPanel) {
|
||||
const rect = videoPanel.getBoundingClientRect();
|
||||
skeletonCanvas.width = rect.width;
|
||||
skeletonCanvas.height = rect.height;
|
||||
}
|
||||
|
||||
// CSI canvas
|
||||
csiCanvas.width = csiCanvas.parentElement.clientWidth;
|
||||
csiCanvas.height = 120;
|
||||
|
||||
// Embedding canvas
|
||||
embeddingCanvas.width = embeddingCanvas.parentElement.clientWidth;
|
||||
embeddingCanvas.height = 140;
|
||||
}
|
||||
|
||||
// === Main Loop ===
|
||||
function mainLoop(timestamp) {
|
||||
if (!isRunning) return;
|
||||
requestAnimationFrame(mainLoop);
|
||||
|
||||
if (isPaused) return;
|
||||
|
||||
const elapsed = performance.now() / 1000 - startTime;
|
||||
const totalStart = performance.now();
|
||||
|
||||
// --- Video Pipeline ---
|
||||
let videoEmb = null;
|
||||
let motionRegion = null;
|
||||
if (mode !== 'csi' && videoCapture.isActive) {
|
||||
const t0 = performance.now();
|
||||
const frame = videoCapture.captureFrame(56, 56);
|
||||
if (frame) {
|
||||
videoEmb = visualCnn.extract(frame.rgb, frame.width, frame.height);
|
||||
motionRegion = videoCapture.detectMotionRegion(56, 56);
|
||||
|
||||
// Feed motion to CSI simulator for correlated demo data
|
||||
// When detected=false, CSI simulator handles through-wall persistence
|
||||
csiSimulator.updatePersonState(
|
||||
motionRegion.detected ? 1.0 : 0,
|
||||
motionRegion.detected ? motionRegion.x + motionRegion.w / 2 : 0.5,
|
||||
motionRegion.detected ? motionRegion.y + motionRegion.h / 2 : 0.5,
|
||||
frame.motion
|
||||
);
|
||||
|
||||
fusionEngine.updateConfidence(
|
||||
frame.brightness, frame.motion,
|
||||
0, csiSimulator.isLive || mode === 'dual'
|
||||
);
|
||||
}
|
||||
latency.video = performance.now() - t0;
|
||||
}
|
||||
|
||||
// --- CSI Pipeline ---
|
||||
let csiEmb = null;
|
||||
if (mode !== 'video') {
|
||||
const t0 = performance.now();
|
||||
const csiFrame = csiSimulator.nextFrame(elapsed);
|
||||
const pseudoImage = csiSimulator.buildPseudoImage(56);
|
||||
csiEmb = csiCnn.extract(pseudoImage, 56, 56);
|
||||
|
||||
fusionEngine.updateConfidence(
|
||||
videoCapture.brightnessScore,
|
||||
videoCapture.motionScore,
|
||||
csiFrame.snr,
|
||||
true
|
||||
);
|
||||
|
||||
// Draw CSI heatmap
|
||||
const heatmap = csiSimulator.getHeatmapData();
|
||||
renderer.drawCsiHeatmap(csiCtx, heatmap, csiCanvas.width, csiCanvas.height);
|
||||
|
||||
latency.csi = performance.now() - t0;
|
||||
}
|
||||
|
||||
// --- Fusion ---
|
||||
const t0f = performance.now();
|
||||
const fusedEmb = fusionEngine.fuse(videoEmb, csiEmb, mode);
|
||||
latency.fusion = performance.now() - t0f;
|
||||
|
||||
// --- Pose Decode ---
|
||||
// For CSI-only mode, generate a synthetic motion region from CSI energy
|
||||
if (mode === 'csi' && (!motionRegion || !motionRegion.detected)) {
|
||||
const csiPresence = csiSimulator.personPresence;
|
||||
if (csiPresence > 0.1) {
|
||||
motionRegion = {
|
||||
detected: true,
|
||||
x: 0.25, y: 0.15, w: 0.5, h: 0.7,
|
||||
coverage: csiPresence,
|
||||
motionGrid: null,
|
||||
gridCols: 10,
|
||||
gridRows: 8
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// CSI state for through-wall tracking
|
||||
const csiState = {
|
||||
csiPresence: csiSimulator.personPresence,
|
||||
isLive: csiSimulator.isLive
|
||||
};
|
||||
|
||||
const keypoints = poseDecoder.decode(fusedEmb, motionRegion, elapsed, csiState);
|
||||
|
||||
// --- Render Skeleton ---
|
||||
const labelMap = { dual: 'DUAL FUSION', video: 'VIDEO ONLY', csi: 'CSI ONLY' };
|
||||
renderer.drawSkeleton(skeletonCtx, keypoints, skeletonCanvas.width, skeletonCanvas.height, {
|
||||
minConfidence: confidenceThreshold,
|
||||
color: mode === 'csi' ? 'amber' : 'green',
|
||||
label: labelMap[mode]
|
||||
});
|
||||
|
||||
// --- Render Embedding Space ---
|
||||
const embPoints = fusionEngine.getEmbeddingPoints();
|
||||
renderer.drawEmbeddingSpace(embeddingCtx, embPoints, embeddingCanvas.width, embeddingCanvas.height);
|
||||
|
||||
// --- Update UI ---
|
||||
latency.total = performance.now() - totalStart;
|
||||
|
||||
// FPS
|
||||
frameCount++;
|
||||
if (timestamp - lastFpsTime > 500) {
|
||||
fps = Math.round(frameCount * 1000 / (timestamp - lastFpsTime));
|
||||
lastFpsTime = timestamp;
|
||||
frameCount = 0;
|
||||
fpsDisplay.textContent = `${fps} FPS`;
|
||||
}
|
||||
|
||||
// Fusion bars
|
||||
const vc = fusionEngine.videoConfidence;
|
||||
const cc = fusionEngine.csiConfidence;
|
||||
const fc = fusionEngine.fusedConfidence;
|
||||
videoBar.style.width = `${vc * 100}%`;
|
||||
csiBar.style.width = `${cc * 100}%`;
|
||||
fusedBar.style.width = `${fc * 100}%`;
|
||||
videoBarVal.textContent = `${Math.round(vc * 100)}%`;
|
||||
csiBarVal.textContent = `${Math.round(cc * 100)}%`;
|
||||
fusedBarVal.textContent = `${Math.round(fc * 100)}%`;
|
||||
|
||||
// Latency
|
||||
latVideoEl.textContent = `${latency.video.toFixed(1)}ms`;
|
||||
latCsiEl.textContent = `${latency.csi.toFixed(1)}ms`;
|
||||
latFusionEl.textContent = `${latency.fusion.toFixed(1)}ms`;
|
||||
latTotalEl.textContent = `${latency.total.toFixed(1)}ms`;
|
||||
|
||||
// Cross-modal similarity
|
||||
const sim = fusionEngine.getCrossModalSimilarity();
|
||||
crossModalEl.textContent = sim.toFixed(3);
|
||||
}
|
||||
|
||||
// Boot
|
||||
document.addEventListener('DOMContentLoaded', init);
|
||||
@@ -1,373 +0,0 @@
|
||||
/**
|
||||
* PoseDecoder — Maps motion detection grid → 17 COCO keypoints.
|
||||
*
|
||||
* Uses per-cell motion intensity to track actual body part positions:
|
||||
* - Head: top-center motion cluster
|
||||
* - Shoulders/Elbows/Wrists: lateral motion in upper body zone
|
||||
* - Hips/Knees/Ankles: lower body motion distribution
|
||||
*
|
||||
* When person exits frame, CSI data continues tracking (through-wall mode).
|
||||
*/
|
||||
|
||||
// COCO keypoint definitions
|
||||
export const KEYPOINT_NAMES = [
|
||||
'nose', 'left_eye', 'right_eye', 'left_ear', 'right_ear',
|
||||
'left_shoulder', 'right_shoulder', 'left_elbow', 'right_elbow',
|
||||
'left_wrist', 'right_wrist', 'left_hip', 'right_hip',
|
||||
'left_knee', 'right_knee', 'left_ankle', 'right_ankle'
|
||||
];
|
||||
|
||||
// Skeleton connections (pairs of keypoint indices)
|
||||
export const SKELETON_CONNECTIONS = [
|
||||
[0, 1], [0, 2], [1, 3], [2, 4], // Head
|
||||
[5, 6], // Shoulders
|
||||
[5, 7], [7, 9], // Left arm
|
||||
[6, 8], [8, 10], // Right arm
|
||||
[5, 11], [6, 12], // Torso
|
||||
[11, 12], // Hips
|
||||
[11, 13], [13, 15], // Left leg
|
||||
[12, 14], [14, 16], // Right leg
|
||||
];
|
||||
|
||||
// Standard body proportions (relative to body height)
|
||||
const PROPORTIONS = {
|
||||
headToShoulder: 0.15,
|
||||
shoulderWidth: 0.25,
|
||||
shoulderToElbow: 0.18,
|
||||
elbowToWrist: 0.16,
|
||||
shoulderToHip: 0.30,
|
||||
hipWidth: 0.18,
|
||||
hipToKnee: 0.24,
|
||||
kneeToAnkle: 0.24,
|
||||
eyeSpacing: 0.04,
|
||||
earSpacing: 0.07,
|
||||
};
|
||||
|
||||
export class PoseDecoder {
|
||||
constructor(embeddingDim = 128) {
|
||||
this.embeddingDim = embeddingDim;
|
||||
this.smoothedKeypoints = null;
|
||||
this.smoothingFactor = 0.45; // Lower = more responsive to movement
|
||||
this._time = 0;
|
||||
|
||||
// Through-wall tracking state
|
||||
this._lastBodyState = null;
|
||||
this._ghostState = null;
|
||||
this._ghostConfidence = 0;
|
||||
this._ghostVelocity = { x: 0, y: 0 };
|
||||
|
||||
// Arm tracking history (smoothed positions)
|
||||
this._leftArmY = 0.5;
|
||||
this._rightArmY = 0.5;
|
||||
this._leftArmX = 0;
|
||||
this._rightArmX = 0;
|
||||
this._headOffsetX = 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode motion data into 17 keypoints
|
||||
* @param {Float32Array} embedding - Fused embedding vector
|
||||
* @param {{ detected, x, y, w, h, motionGrid, gridCols, gridRows, motionCx, motionCy, exitDirection }} motionRegion
|
||||
* @param {number} elapsed - Time in seconds
|
||||
* @param {{ csiPresence: number }} csiState - CSI sensing state for through-wall
|
||||
* @returns {Array<{x: number, y: number, confidence: number, name: string}>}
|
||||
*/
|
||||
decode(embedding, motionRegion, elapsed, csiState = {}) {
|
||||
this._time = elapsed;
|
||||
|
||||
const hasMotion = motionRegion && motionRegion.detected;
|
||||
const hasCsi = csiState && csiState.csiPresence > 0.1;
|
||||
|
||||
if (hasMotion) {
|
||||
// Active tracking from video motion grid
|
||||
this._ghostConfidence = 0;
|
||||
const rawKeypoints = this._trackFromMotionGrid(motionRegion, embedding, elapsed);
|
||||
this._lastBodyState = { keypoints: rawKeypoints.map(kp => ({...kp})), time: elapsed };
|
||||
|
||||
// Track exit velocity
|
||||
if (motionRegion.exitDirection) {
|
||||
const speed = 0.008;
|
||||
this._ghostVelocity = {
|
||||
x: motionRegion.exitDirection === 'left' ? -speed : motionRegion.exitDirection === 'right' ? speed : 0,
|
||||
y: motionRegion.exitDirection === 'up' ? -speed : motionRegion.exitDirection === 'down' ? speed : 0
|
||||
};
|
||||
}
|
||||
|
||||
// Apply temporal smoothing
|
||||
if (this.smoothedKeypoints && this.smoothedKeypoints.length === rawKeypoints.length) {
|
||||
const alpha = this.smoothingFactor;
|
||||
for (let i = 0; i < rawKeypoints.length; i++) {
|
||||
rawKeypoints[i].x = alpha * this.smoothedKeypoints[i].x + (1 - alpha) * rawKeypoints[i].x;
|
||||
rawKeypoints[i].y = alpha * this.smoothedKeypoints[i].y + (1 - alpha) * rawKeypoints[i].y;
|
||||
}
|
||||
}
|
||||
|
||||
this.smoothedKeypoints = rawKeypoints;
|
||||
return rawKeypoints;
|
||||
|
||||
} else if (this._lastBodyState && (hasCsi || this._ghostConfidence > 0.05)) {
|
||||
// Through-wall mode: person left frame but CSI still senses them
|
||||
return this._trackThroughWall(elapsed, csiState);
|
||||
|
||||
} else if (this.smoothedKeypoints) {
|
||||
// Fade out
|
||||
const faded = this.smoothedKeypoints.map(kp => ({
|
||||
...kp,
|
||||
confidence: kp.confidence * 0.88
|
||||
})).filter(kp => kp.confidence > 0.05);
|
||||
if (faded.length === 0) this.smoothedKeypoints = null;
|
||||
else this.smoothedKeypoints = faded;
|
||||
return faded;
|
||||
}
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Track body parts from the motion grid.
|
||||
* The grid tells us WHERE motion is happening → we map that to joint positions.
|
||||
*/
|
||||
_trackFromMotionGrid(region, embedding, elapsed) {
|
||||
const grid = region.motionGrid;
|
||||
const cols = region.gridCols || 10;
|
||||
const rows = region.gridRows || 8;
|
||||
|
||||
// Body bounding box
|
||||
const cx = region.x + region.w / 2;
|
||||
const cy = region.y + region.h / 2;
|
||||
const bodyH = Math.max(region.h, 0.3);
|
||||
const bodyW = Math.max(region.w, 0.15);
|
||||
|
||||
// Analyze the motion grid to find arm positions
|
||||
// Divide body into zones: head (top 20%), arms (top 60% sides), torso (center), legs (bottom 40%)
|
||||
if (grid) {
|
||||
const armAnalysis = this._analyzeArmMotion(grid, cols, rows, region);
|
||||
// Smooth arm tracking
|
||||
this._leftArmY = 0.6 * this._leftArmY + 0.4 * armAnalysis.leftArmHeight;
|
||||
this._rightArmY = 0.6 * this._rightArmY + 0.4 * armAnalysis.rightArmHeight;
|
||||
this._leftArmX = 0.6 * this._leftArmX + 0.4 * armAnalysis.leftArmSpread;
|
||||
this._rightArmX = 0.6 * this._rightArmX + 0.4 * armAnalysis.rightArmSpread;
|
||||
this._headOffsetX = 0.7 * this._headOffsetX + 0.3 * armAnalysis.headOffsetX;
|
||||
}
|
||||
|
||||
const P = PROPORTIONS;
|
||||
const halfW = P.shoulderWidth * bodyH / 2;
|
||||
const hipHalfW = P.hipWidth * bodyH / 2;
|
||||
|
||||
// Breathing (subtle)
|
||||
const breathe = Math.sin(elapsed * 1.5) * 0.002;
|
||||
|
||||
// Core body positions from detection center
|
||||
const hipY = cy + bodyH * 0.15;
|
||||
const shoulderY = hipY - P.shoulderToHip * bodyH + breathe;
|
||||
const headY = shoulderY - P.headToShoulder * bodyH;
|
||||
const kneeY = hipY + P.hipToKnee * bodyH;
|
||||
const ankleY = kneeY + P.kneeToAnkle * bodyH;
|
||||
|
||||
// HEAD follows motion centroid
|
||||
const headX = cx + this._headOffsetX * bodyW * 0.3;
|
||||
|
||||
// ARM POSITIONS driven by motion grid analysis
|
||||
// leftArmY: 0 = arm down at side, 1 = arm fully raised
|
||||
// leftArmSpread: how far out the arm extends
|
||||
const leftArmRaise = this._leftArmY; // 0-1
|
||||
const rightArmRaise = this._rightArmY;
|
||||
const leftSpread = 0.02 + this._leftArmX * 0.12;
|
||||
const rightSpread = 0.02 + this._rightArmX * 0.12;
|
||||
|
||||
// Elbow: interpolate between "at side" and "raised"
|
||||
const lElbowY = shoulderY + P.shoulderToElbow * bodyH * (1 - leftArmRaise * 0.9);
|
||||
const rElbowY = shoulderY + P.shoulderToElbow * bodyH * (1 - rightArmRaise * 0.9);
|
||||
const lElbowX = cx - halfW - leftSpread;
|
||||
const rElbowX = cx + halfW + rightSpread;
|
||||
|
||||
// Wrist: extends further when raised
|
||||
const lWristY = lElbowY + P.elbowToWrist * bodyH * (1 - leftArmRaise * 1.1);
|
||||
const rWristY = rElbowY + P.elbowToWrist * bodyH * (1 - rightArmRaise * 1.1);
|
||||
const lWristX = lElbowX - leftSpread * 0.6;
|
||||
const rWristX = rElbowX + rightSpread * 0.6;
|
||||
|
||||
// Leg motion from lower grid cells
|
||||
const legMotion = grid ? this._analyzeLegMotion(grid, cols, rows) : { left: 0, right: 0 };
|
||||
const legSwing = 0.015;
|
||||
|
||||
const keypoints = [
|
||||
// 0: nose
|
||||
{ x: headX, y: headY + 0.01, confidence: 0.92 },
|
||||
// 1: left_eye
|
||||
{ x: headX - P.eyeSpacing * bodyH, y: headY - 0.005, confidence: 0.88 },
|
||||
// 2: right_eye
|
||||
{ x: headX + P.eyeSpacing * bodyH, y: headY - 0.005, confidence: 0.88 },
|
||||
// 3: left_ear
|
||||
{ x: headX - P.earSpacing * bodyH, y: headY + 0.005, confidence: 0.72 },
|
||||
// 4: right_ear
|
||||
{ x: headX + P.earSpacing * bodyH, y: headY + 0.005, confidence: 0.72 },
|
||||
// 5: left_shoulder
|
||||
{ x: cx - halfW, y: shoulderY, confidence: 0.94 },
|
||||
// 6: right_shoulder
|
||||
{ x: cx + halfW, y: shoulderY, confidence: 0.94 },
|
||||
// 7: left_elbow
|
||||
{ x: lElbowX, y: lElbowY, confidence: 0.87 },
|
||||
// 8: right_elbow
|
||||
{ x: rElbowX, y: rElbowY, confidence: 0.87 },
|
||||
// 9: left_wrist
|
||||
{ x: lWristX, y: lWristY, confidence: 0.82 },
|
||||
// 10: right_wrist
|
||||
{ x: rWristX, y: rWristY, confidence: 0.82 },
|
||||
// 11: left_hip
|
||||
{ x: cx - hipHalfW, y: hipY, confidence: 0.91 },
|
||||
// 12: right_hip
|
||||
{ x: cx + hipHalfW, y: hipY, confidence: 0.91 },
|
||||
// 13: left_knee
|
||||
{ x: cx - hipHalfW + legMotion.left * legSwing, y: kneeY, confidence: 0.88 },
|
||||
// 14: right_knee
|
||||
{ x: cx + hipHalfW + legMotion.right * legSwing, y: kneeY, confidence: 0.88 },
|
||||
// 15: left_ankle
|
||||
{ x: cx - hipHalfW + legMotion.left * legSwing * 1.3, y: ankleY, confidence: 0.83 },
|
||||
// 16: right_ankle
|
||||
{ x: cx + hipHalfW + legMotion.right * legSwing * 1.3, y: ankleY, confidence: 0.83 },
|
||||
];
|
||||
|
||||
for (let i = 0; i < keypoints.length; i++) {
|
||||
keypoints[i].name = KEYPOINT_NAMES[i];
|
||||
}
|
||||
|
||||
return keypoints;
|
||||
}
|
||||
|
||||
/**
|
||||
* Analyze the motion grid to determine arm positions.
|
||||
* Left side of grid = left side of body, etc.
|
||||
*/
|
||||
_analyzeArmMotion(grid, cols, rows, region) {
|
||||
// Body center column
|
||||
const centerCol = Math.floor(cols / 2);
|
||||
|
||||
// Upper body rows (top 60% of detected region)
|
||||
const upperEnd = Math.floor(rows * 0.6);
|
||||
|
||||
// Compute motion intensity for left vs right, at different heights
|
||||
let leftUpperMotion = 0, leftMidMotion = 0;
|
||||
let rightUpperMotion = 0, rightMidMotion = 0;
|
||||
let leftCount = 0, rightCount = 0;
|
||||
let headMotionX = 0, headMotionWeight = 0;
|
||||
|
||||
for (let r = 0; r < upperEnd; r++) {
|
||||
const heightWeight = 1.0 - (r / upperEnd) * 0.3; // Upper rows weighted more
|
||||
|
||||
// Head zone: top 25%, center 40% of width
|
||||
if (r < Math.floor(rows * 0.25)) {
|
||||
const headLeft = Math.floor(cols * 0.3);
|
||||
const headRight = Math.floor(cols * 0.7);
|
||||
for (let c = headLeft; c <= headRight; c++) {
|
||||
const val = grid[r][c];
|
||||
headMotionX += (c / cols - 0.5) * val;
|
||||
headMotionWeight += val;
|
||||
}
|
||||
}
|
||||
|
||||
// Left arm zone: left 40% of grid
|
||||
for (let c = 0; c < Math.floor(cols * 0.4); c++) {
|
||||
const val = grid[r][c];
|
||||
if (r < rows * 0.3) leftUpperMotion += val * heightWeight;
|
||||
else leftMidMotion += val * heightWeight;
|
||||
leftCount++;
|
||||
}
|
||||
|
||||
// Right arm zone: right 40% of grid
|
||||
for (let c = Math.floor(cols * 0.6); c < cols; c++) {
|
||||
const val = grid[r][c];
|
||||
if (r < rows * 0.3) rightUpperMotion += val * heightWeight;
|
||||
else rightMidMotion += val * heightWeight;
|
||||
rightCount++;
|
||||
}
|
||||
}
|
||||
|
||||
// Normalize
|
||||
const leftTotal = leftUpperMotion + leftMidMotion;
|
||||
const rightTotal = rightUpperMotion + rightMidMotion;
|
||||
const maxMotion = 0.15; // Calibration threshold
|
||||
|
||||
// Arm height: 0 = at side, 1 = raised
|
||||
// High motion in upper-left → left arm is raised
|
||||
const leftArmHeight = Math.min(1, (leftUpperMotion / maxMotion) * 2);
|
||||
const rightArmHeight = Math.min(1, (rightUpperMotion / maxMotion) * 2);
|
||||
|
||||
// Arm spread: how far out from body
|
||||
const leftArmSpread = Math.min(1, leftTotal / maxMotion);
|
||||
const rightArmSpread = Math.min(1, rightTotal / maxMotion);
|
||||
|
||||
// Head offset
|
||||
const headOffsetX = headMotionWeight > 0.01 ? headMotionX / headMotionWeight : 0;
|
||||
|
||||
return { leftArmHeight, rightArmHeight, leftArmSpread, rightArmSpread, headOffsetX };
|
||||
}
|
||||
|
||||
/**
|
||||
* Analyze lower grid for leg motion.
|
||||
*/
|
||||
_analyzeLegMotion(grid, cols, rows) {
|
||||
const lowerStart = Math.floor(rows * 0.6);
|
||||
let leftMotion = 0, rightMotion = 0;
|
||||
|
||||
for (let r = lowerStart; r < rows; r++) {
|
||||
for (let c = 0; c < Math.floor(cols / 2); c++) {
|
||||
leftMotion += grid[r][c];
|
||||
}
|
||||
for (let c = Math.floor(cols / 2); c < cols; c++) {
|
||||
rightMotion += grid[r][c];
|
||||
}
|
||||
}
|
||||
|
||||
// Return as -1 to 1 range (asymmetry indicates which leg is moving)
|
||||
const total = leftMotion + rightMotion + 0.001;
|
||||
return {
|
||||
left: (leftMotion - rightMotion) / total,
|
||||
right: (rightMotion - leftMotion) / total
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Through-wall tracking: continue showing pose via CSI when person left video frame.
|
||||
* The skeleton drifts in the exit direction with decreasing confidence.
|
||||
*/
|
||||
_trackThroughWall(elapsed, csiState) {
|
||||
if (!this._lastBodyState) return [];
|
||||
|
||||
const dt = elapsed - this._lastBodyState.time;
|
||||
const csiPresence = csiState.csiPresence || 0;
|
||||
|
||||
// Initialize ghost on first call
|
||||
if (this._ghostConfidence <= 0.05) {
|
||||
this._ghostConfidence = 0.8;
|
||||
this._ghostState = this._lastBodyState.keypoints.map(kp => ({...kp}));
|
||||
}
|
||||
|
||||
// Ghost confidence decays, but CSI presence sustains it
|
||||
const csiBoost = Math.min(0.7, csiPresence * 0.8);
|
||||
this._ghostConfidence = Math.max(0.05, this._ghostConfidence * 0.995 - 0.001 + csiBoost * 0.002);
|
||||
|
||||
// Drift the ghost in exit direction
|
||||
const vx = this._ghostVelocity.x;
|
||||
const vy = this._ghostVelocity.y;
|
||||
|
||||
// Breathing continues via CSI
|
||||
const breathe = Math.sin(elapsed * 1.5) * 0.003 * csiPresence;
|
||||
|
||||
const keypoints = this._ghostState.map((kp, i) => {
|
||||
return {
|
||||
x: kp.x + vx * dt * 0.3,
|
||||
y: kp.y + vy * dt * 0.3 + (i >= 5 && i <= 6 ? breathe : 0),
|
||||
confidence: kp.confidence * this._ghostConfidence * (0.5 + csiPresence * 0.5),
|
||||
name: kp.name
|
||||
};
|
||||
});
|
||||
|
||||
// Slow down drift over time
|
||||
this._ghostVelocity.x *= 0.998;
|
||||
this._ghostVelocity.y *= 0.998;
|
||||
|
||||
this.smoothedKeypoints = keypoints;
|
||||
return keypoints;
|
||||
}
|
||||
}
|
||||
@@ -1,235 +0,0 @@
|
||||
/**
|
||||
* VideoCapture — getUserMedia webcam capture with frame extraction.
|
||||
* Provides quality metrics (brightness, motion) for fusion confidence gating.
|
||||
*/
|
||||
|
||||
export class VideoCapture {
|
||||
constructor(videoElement) {
|
||||
this.video = videoElement;
|
||||
this.stream = null;
|
||||
this.offscreen = document.createElement('canvas');
|
||||
this.offCtx = this.offscreen.getContext('2d', { willReadFrequently: true });
|
||||
this.prevFrame = null;
|
||||
this.motionScore = 0;
|
||||
this.brightnessScore = 0;
|
||||
}
|
||||
|
||||
async start(constraints = {}) {
|
||||
const defaultConstraints = {
|
||||
video: {
|
||||
width: { ideal: 640 },
|
||||
height: { ideal: 480 },
|
||||
facingMode: 'user',
|
||||
frameRate: { ideal: 30 }
|
||||
},
|
||||
audio: false
|
||||
};
|
||||
|
||||
try {
|
||||
this.stream = await navigator.mediaDevices.getUserMedia(
|
||||
Object.keys(constraints).length ? constraints : defaultConstraints
|
||||
);
|
||||
this.video.srcObject = this.stream;
|
||||
await this.video.play();
|
||||
|
||||
this.offscreen.width = this.video.videoWidth;
|
||||
this.offscreen.height = this.video.videoHeight;
|
||||
|
||||
return true;
|
||||
} catch (err) {
|
||||
console.error('[Video] Camera access failed:', err.message);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
stop() {
|
||||
if (this.stream) {
|
||||
this.stream.getTracks().forEach(t => t.stop());
|
||||
this.stream = null;
|
||||
}
|
||||
this.video.srcObject = null;
|
||||
}
|
||||
|
||||
get isActive() {
|
||||
return this.stream !== null && this.video.readyState >= 2;
|
||||
}
|
||||
|
||||
get width() { return this.video.videoWidth || 640; }
|
||||
get height() { return this.video.videoHeight || 480; }
|
||||
|
||||
/**
|
||||
* Capture current frame as RGB Uint8Array + compute quality metrics.
|
||||
* @param {number} targetW - Target width for CNN input
|
||||
* @param {number} targetH - Target height for CNN input
|
||||
* @returns {{ rgb: Uint8Array, width: number, height: number, motion: number, brightness: number }}
|
||||
*/
|
||||
captureFrame(targetW = 56, targetH = 56) {
|
||||
if (!this.isActive) return null;
|
||||
|
||||
// Draw to offscreen at target resolution
|
||||
this.offscreen.width = targetW;
|
||||
this.offscreen.height = targetH;
|
||||
this.offCtx.drawImage(this.video, 0, 0, targetW, targetH);
|
||||
const imageData = this.offCtx.getImageData(0, 0, targetW, targetH);
|
||||
const rgba = imageData.data;
|
||||
|
||||
// Convert RGBA → RGB
|
||||
const pixels = targetW * targetH;
|
||||
const rgb = new Uint8Array(pixels * 3);
|
||||
let brightnessSum = 0;
|
||||
let motionSum = 0;
|
||||
|
||||
for (let i = 0; i < pixels; i++) {
|
||||
const r = rgba[i * 4];
|
||||
const g = rgba[i * 4 + 1];
|
||||
const b = rgba[i * 4 + 2];
|
||||
rgb[i * 3] = r;
|
||||
rgb[i * 3 + 1] = g;
|
||||
rgb[i * 3 + 2] = b;
|
||||
|
||||
// Luminance for brightness
|
||||
const lum = 0.299 * r + 0.587 * g + 0.114 * b;
|
||||
brightnessSum += lum;
|
||||
|
||||
// Motion: diff from previous frame
|
||||
if (this.prevFrame) {
|
||||
const pr = this.prevFrame[i * 3];
|
||||
const pg = this.prevFrame[i * 3 + 1];
|
||||
const pb = this.prevFrame[i * 3 + 2];
|
||||
motionSum += Math.abs(r - pr) + Math.abs(g - pg) + Math.abs(b - pb);
|
||||
}
|
||||
}
|
||||
|
||||
this.brightnessScore = brightnessSum / (pixels * 255);
|
||||
this.motionScore = this.prevFrame ? Math.min(1, motionSum / (pixels * 100)) : 0;
|
||||
this.prevFrame = new Uint8Array(rgb);
|
||||
|
||||
return {
|
||||
rgb,
|
||||
width: targetW,
|
||||
height: targetH,
|
||||
motion: this.motionScore,
|
||||
brightness: this.brightnessScore
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Capture full-resolution RGBA for overlay rendering
|
||||
* @returns {ImageData|null}
|
||||
*/
|
||||
captureFullFrame() {
|
||||
if (!this.isActive) return null;
|
||||
this.offscreen.width = this.width;
|
||||
this.offscreen.height = this.height;
|
||||
this.offCtx.drawImage(this.video, 0, 0);
|
||||
return this.offCtx.getImageData(0, 0, this.width, this.height);
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect motion region + detailed motion grid for body-part tracking.
|
||||
* Returns bounding box + a grid showing WHERE motion is concentrated.
|
||||
* @returns {{ x, y, w, h, detected: boolean, motionGrid: number[][], gridCols: number, gridRows: number, exitDirection: string|null }}
|
||||
*/
|
||||
detectMotionRegion(targetW = 56, targetH = 56) {
|
||||
if (!this.isActive || !this.prevFrame) return { detected: false, motionGrid: null };
|
||||
|
||||
this.offscreen.width = targetW;
|
||||
this.offscreen.height = targetH;
|
||||
this.offCtx.drawImage(this.video, 0, 0, targetW, targetH);
|
||||
const rgba = this.offCtx.getImageData(0, 0, targetW, targetH).data;
|
||||
|
||||
let minX = targetW, minY = targetH, maxX = 0, maxY = 0;
|
||||
let motionPixels = 0;
|
||||
const threshold = 25;
|
||||
|
||||
// Motion grid: divide frame into cells and track motion intensity per cell
|
||||
const gridCols = 10;
|
||||
const gridRows = 8;
|
||||
const cellW = targetW / gridCols;
|
||||
const cellH = targetH / gridRows;
|
||||
const motionGrid = Array.from({ length: gridRows }, () => new Float32Array(gridCols));
|
||||
const cellPixels = cellW * cellH;
|
||||
|
||||
// Also track motion centroid weighted by intensity
|
||||
let motionCxSum = 0, motionCySum = 0, motionWeightSum = 0;
|
||||
|
||||
for (let y = 0; y < targetH; y++) {
|
||||
for (let x = 0; x < targetW; x++) {
|
||||
const i = y * targetW + x;
|
||||
const r = rgba[i * 4], g = rgba[i * 4 + 1], b = rgba[i * 4 + 2];
|
||||
const pr = this.prevFrame[i * 3], pg = this.prevFrame[i * 3 + 1], pb = this.prevFrame[i * 3 + 2];
|
||||
const diff = Math.abs(r - pr) + Math.abs(g - pg) + Math.abs(b - pb);
|
||||
|
||||
if (diff > threshold * 3) {
|
||||
motionPixels++;
|
||||
if (x < minX) minX = x;
|
||||
if (y < minY) minY = y;
|
||||
if (x > maxX) maxX = x;
|
||||
if (y > maxY) maxY = y;
|
||||
}
|
||||
|
||||
// Accumulate per-cell motion intensity
|
||||
const gc = Math.min(Math.floor(x / cellW), gridCols - 1);
|
||||
const gr = Math.min(Math.floor(y / cellH), gridRows - 1);
|
||||
const intensity = diff / (3 * 255); // Normalize 0-1
|
||||
motionGrid[gr][gc] += intensity / cellPixels;
|
||||
|
||||
// Weighted centroid
|
||||
if (diff > threshold) {
|
||||
motionCxSum += x * diff;
|
||||
motionCySum += y * diff;
|
||||
motionWeightSum += diff;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const detected = motionPixels > (targetW * targetH * 0.02);
|
||||
|
||||
// Motion centroid (normalized 0-1)
|
||||
const motionCx = motionWeightSum > 0 ? motionCxSum / (motionWeightSum * targetW) : 0.5;
|
||||
const motionCy = motionWeightSum > 0 ? motionCySum / (motionWeightSum * targetH) : 0.5;
|
||||
|
||||
// Detect exit direction: if centroid is near edges
|
||||
let exitDirection = null;
|
||||
if (detected && motionCx < 0.1) exitDirection = 'left';
|
||||
else if (detected && motionCx > 0.9) exitDirection = 'right';
|
||||
else if (detected && motionCy < 0.1) exitDirection = 'up';
|
||||
else if (detected && motionCy > 0.9) exitDirection = 'down';
|
||||
|
||||
// Track last known position for through-wall persistence
|
||||
if (detected) {
|
||||
this._lastDetected = {
|
||||
x: minX / targetW,
|
||||
y: minY / targetH,
|
||||
w: (maxX - minX) / targetW,
|
||||
h: (maxY - minY) / targetH,
|
||||
cx: motionCx,
|
||||
cy: motionCy,
|
||||
exitDirection,
|
||||
time: performance.now()
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
detected,
|
||||
x: minX / targetW,
|
||||
y: minY / targetH,
|
||||
w: (maxX - minX) / targetW,
|
||||
h: (maxY - minY) / targetH,
|
||||
coverage: motionPixels / (targetW * targetH),
|
||||
motionGrid,
|
||||
gridCols,
|
||||
gridRows,
|
||||
motionCx,
|
||||
motionCy,
|
||||
exitDirection
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the last known detection info (for through-wall persistence)
|
||||
*/
|
||||
get lastDetection() {
|
||||
return this._lastDetected || null;
|
||||
}
|
||||
}
|
||||
@@ -1,26 +0,0 @@
|
||||
{
|
||||
"name": "ruvector-cnn-wasm",
|
||||
"type": "module",
|
||||
"description": "WASM bindings for ruvector-cnn - CNN feature extraction for image embeddings",
|
||||
"version": "0.1.0",
|
||||
"license": "MIT OR Apache-2.0",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/ruvnet/ruvector"
|
||||
},
|
||||
"files": [
|
||||
"ruvector_cnn_wasm_bg.wasm",
|
||||
"ruvector_cnn_wasm.js"
|
||||
],
|
||||
"main": "ruvector_cnn_wasm.js",
|
||||
"sideEffects": [
|
||||
"./snippets/*"
|
||||
],
|
||||
"keywords": [
|
||||
"cnn",
|
||||
"embeddings",
|
||||
"wasm",
|
||||
"simd",
|
||||
"machine-learning"
|
||||
]
|
||||
}
|
||||
@@ -1,802 +0,0 @@
|
||||
/**
|
||||
* Configuration for CNN embedder
|
||||
*/
|
||||
export class EmbedderConfig {
|
||||
__destroy_into_raw() {
|
||||
const ptr = this.__wbg_ptr;
|
||||
this.__wbg_ptr = 0;
|
||||
EmbedderConfigFinalization.unregister(this);
|
||||
return ptr;
|
||||
}
|
||||
free() {
|
||||
const ptr = this.__destroy_into_raw();
|
||||
wasm.__wbg_embedderconfig_free(ptr, 0);
|
||||
}
|
||||
constructor() {
|
||||
const ret = wasm.embedderconfig_new();
|
||||
this.__wbg_ptr = ret >>> 0;
|
||||
EmbedderConfigFinalization.register(this, this.__wbg_ptr, this);
|
||||
return this;
|
||||
}
|
||||
/**
|
||||
* Output embedding dimension
|
||||
* @returns {number}
|
||||
*/
|
||||
get embedding_dim() {
|
||||
const ret = wasm.__wbg_get_embedderconfig_embedding_dim(this.__wbg_ptr);
|
||||
return ret >>> 0;
|
||||
}
|
||||
/**
|
||||
* Input image size (square)
|
||||
* @returns {number}
|
||||
*/
|
||||
get input_size() {
|
||||
const ret = wasm.__wbg_get_embedderconfig_input_size(this.__wbg_ptr);
|
||||
return ret >>> 0;
|
||||
}
|
||||
/**
|
||||
* Whether to L2 normalize embeddings
|
||||
* @returns {boolean}
|
||||
*/
|
||||
get normalize() {
|
||||
const ret = wasm.__wbg_get_embedderconfig_normalize(this.__wbg_ptr);
|
||||
return ret !== 0;
|
||||
}
|
||||
/**
|
||||
* Output embedding dimension
|
||||
* @param {number} arg0
|
||||
*/
|
||||
set embedding_dim(arg0) {
|
||||
wasm.__wbg_set_embedderconfig_embedding_dim(this.__wbg_ptr, arg0);
|
||||
}
|
||||
/**
|
||||
* Input image size (square)
|
||||
* @param {number} arg0
|
||||
*/
|
||||
set input_size(arg0) {
|
||||
wasm.__wbg_set_embedderconfig_input_size(this.__wbg_ptr, arg0);
|
||||
}
|
||||
/**
|
||||
* Whether to L2 normalize embeddings
|
||||
* @param {boolean} arg0
|
||||
*/
|
||||
set normalize(arg0) {
|
||||
wasm.__wbg_set_embedderconfig_normalize(this.__wbg_ptr, arg0);
|
||||
}
|
||||
}
|
||||
if (Symbol.dispose) EmbedderConfig.prototype[Symbol.dispose] = EmbedderConfig.prototype.free;
|
||||
|
||||
/**
|
||||
* Layer operations for building custom networks
|
||||
*/
|
||||
export class LayerOps {
|
||||
__destroy_into_raw() {
|
||||
const ptr = this.__wbg_ptr;
|
||||
this.__wbg_ptr = 0;
|
||||
LayerOpsFinalization.unregister(this);
|
||||
return ptr;
|
||||
}
|
||||
free() {
|
||||
const ptr = this.__destroy_into_raw();
|
||||
wasm.__wbg_layerops_free(ptr, 0);
|
||||
}
|
||||
/**
|
||||
* Apply batch normalization (returns new array)
|
||||
* @param {Float32Array} input
|
||||
* @param {Float32Array} gamma
|
||||
* @param {Float32Array} beta
|
||||
* @param {Float32Array} mean
|
||||
* @param {Float32Array} _var
|
||||
* @param {number} epsilon
|
||||
* @returns {Float32Array}
|
||||
*/
|
||||
static batch_norm(input, gamma, beta, mean, _var, epsilon) {
|
||||
try {
|
||||
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
|
||||
const ptr0 = passArrayF32ToWasm0(input, wasm.__wbindgen_export2);
|
||||
const len0 = WASM_VECTOR_LEN;
|
||||
const ptr1 = passArrayF32ToWasm0(gamma, wasm.__wbindgen_export2);
|
||||
const len1 = WASM_VECTOR_LEN;
|
||||
const ptr2 = passArrayF32ToWasm0(beta, wasm.__wbindgen_export2);
|
||||
const len2 = WASM_VECTOR_LEN;
|
||||
const ptr3 = passArrayF32ToWasm0(mean, wasm.__wbindgen_export2);
|
||||
const len3 = WASM_VECTOR_LEN;
|
||||
const ptr4 = passArrayF32ToWasm0(_var, wasm.__wbindgen_export2);
|
||||
const len4 = WASM_VECTOR_LEN;
|
||||
wasm.layerops_batch_norm(retptr, ptr0, len0, ptr1, len1, ptr2, len2, ptr3, len3, ptr4, len4, epsilon);
|
||||
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
|
||||
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
|
||||
var v6 = getArrayF32FromWasm0(r0, r1).slice();
|
||||
wasm.__wbindgen_export(r0, r1 * 4, 4);
|
||||
return v6;
|
||||
} finally {
|
||||
wasm.__wbindgen_add_to_stack_pointer(16);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Apply global average pooling
|
||||
* Returns one value per channel
|
||||
* @param {Float32Array} input
|
||||
* @param {number} height
|
||||
* @param {number} width
|
||||
* @param {number} channels
|
||||
* @returns {Float32Array}
|
||||
*/
|
||||
static global_avg_pool(input, height, width, channels) {
|
||||
try {
|
||||
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
|
||||
const ptr0 = passArrayF32ToWasm0(input, wasm.__wbindgen_export2);
|
||||
const len0 = WASM_VECTOR_LEN;
|
||||
wasm.layerops_global_avg_pool(retptr, ptr0, len0, height, width, channels);
|
||||
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
|
||||
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
|
||||
var v2 = getArrayF32FromWasm0(r0, r1).slice();
|
||||
wasm.__wbindgen_export(r0, r1 * 4, 4);
|
||||
return v2;
|
||||
} finally {
|
||||
wasm.__wbindgen_add_to_stack_pointer(16);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (Symbol.dispose) LayerOps.prototype[Symbol.dispose] = LayerOps.prototype.free;
|
||||
|
||||
/**
|
||||
* SIMD-optimized operations
|
||||
*/
|
||||
export class SimdOps {
|
||||
__destroy_into_raw() {
|
||||
const ptr = this.__wbg_ptr;
|
||||
this.__wbg_ptr = 0;
|
||||
SimdOpsFinalization.unregister(this);
|
||||
return ptr;
|
||||
}
|
||||
free() {
|
||||
const ptr = this.__destroy_into_raw();
|
||||
wasm.__wbg_simdops_free(ptr, 0);
|
||||
}
|
||||
/**
|
||||
* Dot product of two vectors
|
||||
* @param {Float32Array} a
|
||||
* @param {Float32Array} b
|
||||
* @returns {number}
|
||||
*/
|
||||
static dot_product(a, b) {
|
||||
const ptr0 = passArrayF32ToWasm0(a, wasm.__wbindgen_export2);
|
||||
const len0 = WASM_VECTOR_LEN;
|
||||
const ptr1 = passArrayF32ToWasm0(b, wasm.__wbindgen_export2);
|
||||
const len1 = WASM_VECTOR_LEN;
|
||||
const ret = wasm.simdops_dot_product(ptr0, len0, ptr1, len1);
|
||||
return ret;
|
||||
}
|
||||
/**
|
||||
* L2 normalize a vector (returns new array)
|
||||
* @param {Float32Array} data
|
||||
* @returns {Float32Array}
|
||||
*/
|
||||
static l2_normalize(data) {
|
||||
try {
|
||||
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
|
||||
const ptr0 = passArrayF32ToWasm0(data, wasm.__wbindgen_export2);
|
||||
const len0 = WASM_VECTOR_LEN;
|
||||
wasm.simdops_l2_normalize(retptr, ptr0, len0);
|
||||
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
|
||||
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
|
||||
var v2 = getArrayF32FromWasm0(r0, r1).slice();
|
||||
wasm.__wbindgen_export(r0, r1 * 4, 4);
|
||||
return v2;
|
||||
} finally {
|
||||
wasm.__wbindgen_add_to_stack_pointer(16);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* ReLU activation (returns new array)
|
||||
* @param {Float32Array} data
|
||||
* @returns {Float32Array}
|
||||
*/
|
||||
static relu(data) {
|
||||
try {
|
||||
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
|
||||
const ptr0 = passArrayF32ToWasm0(data, wasm.__wbindgen_export2);
|
||||
const len0 = WASM_VECTOR_LEN;
|
||||
wasm.simdops_relu(retptr, ptr0, len0);
|
||||
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
|
||||
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
|
||||
var v2 = getArrayF32FromWasm0(r0, r1).slice();
|
||||
wasm.__wbindgen_export(r0, r1 * 4, 4);
|
||||
return v2;
|
||||
} finally {
|
||||
wasm.__wbindgen_add_to_stack_pointer(16);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* ReLU6 activation (returns new array)
|
||||
* @param {Float32Array} data
|
||||
* @returns {Float32Array}
|
||||
*/
|
||||
static relu6(data) {
|
||||
try {
|
||||
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
|
||||
const ptr0 = passArrayF32ToWasm0(data, wasm.__wbindgen_export2);
|
||||
const len0 = WASM_VECTOR_LEN;
|
||||
wasm.simdops_relu6(retptr, ptr0, len0);
|
||||
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
|
||||
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
|
||||
var v2 = getArrayF32FromWasm0(r0, r1).slice();
|
||||
wasm.__wbindgen_export(r0, r1 * 4, 4);
|
||||
return v2;
|
||||
} finally {
|
||||
wasm.__wbindgen_add_to_stack_pointer(16);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (Symbol.dispose) SimdOps.prototype[Symbol.dispose] = SimdOps.prototype.free;
|
||||
|
||||
/**
|
||||
* WASM CNN Embedder for image feature extraction
|
||||
*/
|
||||
export class WasmCnnEmbedder {
|
||||
__destroy_into_raw() {
|
||||
const ptr = this.__wbg_ptr;
|
||||
this.__wbg_ptr = 0;
|
||||
WasmCnnEmbedderFinalization.unregister(this);
|
||||
return ptr;
|
||||
}
|
||||
free() {
|
||||
const ptr = this.__destroy_into_raw();
|
||||
wasm.__wbg_wasmcnnembedder_free(ptr, 0);
|
||||
}
|
||||
/**
|
||||
* Compute cosine similarity between two embeddings
|
||||
* @param {Float32Array} a
|
||||
* @param {Float32Array} b
|
||||
* @returns {number}
|
||||
*/
|
||||
cosine_similarity(a, b) {
|
||||
try {
|
||||
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
|
||||
const ptr0 = passArrayF32ToWasm0(a, wasm.__wbindgen_export2);
|
||||
const len0 = WASM_VECTOR_LEN;
|
||||
const ptr1 = passArrayF32ToWasm0(b, wasm.__wbindgen_export2);
|
||||
const len1 = WASM_VECTOR_LEN;
|
||||
wasm.wasmcnnembedder_cosine_similarity(retptr, this.__wbg_ptr, ptr0, len0, ptr1, len1);
|
||||
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
|
||||
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
|
||||
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
|
||||
if (r2) {
|
||||
throw takeObject(r1);
|
||||
}
|
||||
return r0;
|
||||
} finally {
|
||||
wasm.__wbindgen_add_to_stack_pointer(16);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Get the embedding dimension
|
||||
* @returns {number}
|
||||
*/
|
||||
get embedding_dim() {
|
||||
const ret = wasm.wasmcnnembedder_embedding_dim(this.__wbg_ptr);
|
||||
return ret >>> 0;
|
||||
}
|
||||
/**
|
||||
* Extract embedding from image data (RGB format, row-major)
|
||||
* @param {Uint8Array} image_data
|
||||
* @param {number} width
|
||||
* @param {number} height
|
||||
* @returns {Float32Array}
|
||||
*/
|
||||
extract(image_data, width, height) {
|
||||
try {
|
||||
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
|
||||
const ptr0 = passArray8ToWasm0(image_data, wasm.__wbindgen_export2);
|
||||
const len0 = WASM_VECTOR_LEN;
|
||||
wasm.wasmcnnembedder_extract(retptr, this.__wbg_ptr, ptr0, len0, width, height);
|
||||
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
|
||||
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
|
||||
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
|
||||
var r3 = getDataViewMemory0().getInt32(retptr + 4 * 3, true);
|
||||
if (r3) {
|
||||
throw takeObject(r2);
|
||||
}
|
||||
var v2 = getArrayF32FromWasm0(r0, r1).slice();
|
||||
wasm.__wbindgen_export(r0, r1 * 4, 4);
|
||||
return v2;
|
||||
} finally {
|
||||
wasm.__wbindgen_add_to_stack_pointer(16);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Create a new CNN embedder
|
||||
* @param {EmbedderConfig | null} [config]
|
||||
*/
|
||||
constructor(config) {
|
||||
try {
|
||||
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
|
||||
let ptr0 = 0;
|
||||
if (!isLikeNone(config)) {
|
||||
_assertClass(config, EmbedderConfig);
|
||||
ptr0 = config.__destroy_into_raw();
|
||||
}
|
||||
wasm.wasmcnnembedder_new(retptr, ptr0);
|
||||
var r0 = getDataViewMemory0().getInt32(retptr + 4 * 0, true);
|
||||
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
|
||||
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
|
||||
if (r2) {
|
||||
throw takeObject(r1);
|
||||
}
|
||||
this.__wbg_ptr = r0 >>> 0;
|
||||
WasmCnnEmbedderFinalization.register(this, this.__wbg_ptr, this);
|
||||
return this;
|
||||
} finally {
|
||||
wasm.__wbindgen_add_to_stack_pointer(16);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (Symbol.dispose) WasmCnnEmbedder.prototype[Symbol.dispose] = WasmCnnEmbedder.prototype.free;
|
||||
|
||||
/**
|
||||
* InfoNCE loss for contrastive learning (SimCLR style)
|
||||
*/
|
||||
export class WasmInfoNCELoss {
|
||||
__destroy_into_raw() {
|
||||
const ptr = this.__wbg_ptr;
|
||||
this.__wbg_ptr = 0;
|
||||
WasmInfoNCELossFinalization.unregister(this);
|
||||
return ptr;
|
||||
}
|
||||
free() {
|
||||
const ptr = this.__destroy_into_raw();
|
||||
wasm.__wbg_wasminfonceloss_free(ptr, 0);
|
||||
}
|
||||
/**
|
||||
* Compute loss for a batch of embedding pairs
|
||||
* embeddings: [2N, D] flattened where (i, i+N) are positive pairs
|
||||
* @param {Float32Array} embeddings
|
||||
* @param {number} batch_size
|
||||
* @param {number} dim
|
||||
* @returns {number}
|
||||
*/
|
||||
forward(embeddings, batch_size, dim) {
|
||||
try {
|
||||
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
|
||||
const ptr0 = passArrayF32ToWasm0(embeddings, wasm.__wbindgen_export2);
|
||||
const len0 = WASM_VECTOR_LEN;
|
||||
wasm.wasminfonceloss_forward(retptr, this.__wbg_ptr, ptr0, len0, batch_size, dim);
|
||||
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
|
||||
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
|
||||
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
|
||||
if (r2) {
|
||||
throw takeObject(r1);
|
||||
}
|
||||
return r0;
|
||||
} finally {
|
||||
wasm.__wbindgen_add_to_stack_pointer(16);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Create new InfoNCE loss with temperature parameter
|
||||
* @param {number} temperature
|
||||
*/
|
||||
constructor(temperature) {
|
||||
const ret = wasm.wasminfonceloss_new(temperature);
|
||||
this.__wbg_ptr = ret >>> 0;
|
||||
WasmInfoNCELossFinalization.register(this, this.__wbg_ptr, this);
|
||||
return this;
|
||||
}
|
||||
/**
|
||||
* Get the temperature parameter
|
||||
* @returns {number}
|
||||
*/
|
||||
get temperature() {
|
||||
const ret = wasm.wasminfonceloss_temperature(this.__wbg_ptr);
|
||||
return ret;
|
||||
}
|
||||
}
|
||||
if (Symbol.dispose) WasmInfoNCELoss.prototype[Symbol.dispose] = WasmInfoNCELoss.prototype.free;
|
||||
|
||||
/**
|
||||
* Triplet loss for metric learning
|
||||
*/
|
||||
export class WasmTripletLoss {
|
||||
__destroy_into_raw() {
|
||||
const ptr = this.__wbg_ptr;
|
||||
this.__wbg_ptr = 0;
|
||||
WasmTripletLossFinalization.unregister(this);
|
||||
return ptr;
|
||||
}
|
||||
free() {
|
||||
const ptr = this.__destroy_into_raw();
|
||||
wasm.__wbg_wasmtripletloss_free(ptr, 0);
|
||||
}
|
||||
/**
|
||||
* Compute loss for a batch of triplets
|
||||
* @param {Float32Array} anchors
|
||||
* @param {Float32Array} positives
|
||||
* @param {Float32Array} negatives
|
||||
* @param {number} dim
|
||||
* @returns {number}
|
||||
*/
|
||||
forward(anchors, positives, negatives, dim) {
|
||||
try {
|
||||
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
|
||||
const ptr0 = passArrayF32ToWasm0(anchors, wasm.__wbindgen_export2);
|
||||
const len0 = WASM_VECTOR_LEN;
|
||||
const ptr1 = passArrayF32ToWasm0(positives, wasm.__wbindgen_export2);
|
||||
const len1 = WASM_VECTOR_LEN;
|
||||
const ptr2 = passArrayF32ToWasm0(negatives, wasm.__wbindgen_export2);
|
||||
const len2 = WASM_VECTOR_LEN;
|
||||
wasm.wasmtripletloss_forward(retptr, this.__wbg_ptr, ptr0, len0, ptr1, len1, ptr2, len2, dim);
|
||||
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
|
||||
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
|
||||
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
|
||||
if (r2) {
|
||||
throw takeObject(r1);
|
||||
}
|
||||
return r0;
|
||||
} finally {
|
||||
wasm.__wbindgen_add_to_stack_pointer(16);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Compute loss for a single triplet
|
||||
* @param {Float32Array} anchor
|
||||
* @param {Float32Array} positive
|
||||
* @param {Float32Array} negative
|
||||
* @returns {number}
|
||||
*/
|
||||
forward_single(anchor, positive, negative) {
|
||||
try {
|
||||
const retptr = wasm.__wbindgen_add_to_stack_pointer(-16);
|
||||
const ptr0 = passArrayF32ToWasm0(anchor, wasm.__wbindgen_export2);
|
||||
const len0 = WASM_VECTOR_LEN;
|
||||
const ptr1 = passArrayF32ToWasm0(positive, wasm.__wbindgen_export2);
|
||||
const len1 = WASM_VECTOR_LEN;
|
||||
const ptr2 = passArrayF32ToWasm0(negative, wasm.__wbindgen_export2);
|
||||
const len2 = WASM_VECTOR_LEN;
|
||||
wasm.wasmtripletloss_forward_single(retptr, this.__wbg_ptr, ptr0, len0, ptr1, len1, ptr2, len2);
|
||||
var r0 = getDataViewMemory0().getFloat32(retptr + 4 * 0, true);
|
||||
var r1 = getDataViewMemory0().getInt32(retptr + 4 * 1, true);
|
||||
var r2 = getDataViewMemory0().getInt32(retptr + 4 * 2, true);
|
||||
if (r2) {
|
||||
throw takeObject(r1);
|
||||
}
|
||||
return r0;
|
||||
} finally {
|
||||
wasm.__wbindgen_add_to_stack_pointer(16);
|
||||
}
|
||||
}
|
||||
/**
|
||||
* Get the margin parameter
|
||||
* @returns {number}
|
||||
*/
|
||||
get margin() {
|
||||
const ret = wasm.wasmtripletloss_margin(this.__wbg_ptr);
|
||||
return ret;
|
||||
}
|
||||
/**
|
||||
* Create new triplet loss with margin
|
||||
* @param {number} margin
|
||||
*/
|
||||
constructor(margin) {
|
||||
const ret = wasm.wasmtripletloss_new(margin);
|
||||
this.__wbg_ptr = ret >>> 0;
|
||||
WasmTripletLossFinalization.register(this, this.__wbg_ptr, this);
|
||||
return this;
|
||||
}
|
||||
}
|
||||
if (Symbol.dispose) WasmTripletLoss.prototype[Symbol.dispose] = WasmTripletLoss.prototype.free;
|
||||
|
||||
/**
|
||||
* Initialize panic hook for better error messages
|
||||
*/
|
||||
export function init() {
|
||||
wasm.init();
|
||||
}
|
||||
|
||||
function __wbg_get_imports() {
|
||||
const import0 = {
|
||||
__proto__: null,
|
||||
__wbg___wbindgen_throw_39bc967c0e5a9b58: function(arg0, arg1) {
|
||||
throw new Error(getStringFromWasm0(arg0, arg1));
|
||||
},
|
||||
__wbg_error_a6fa202b58aa1cd3: function(arg0, arg1) {
|
||||
let deferred0_0;
|
||||
let deferred0_1;
|
||||
try {
|
||||
deferred0_0 = arg0;
|
||||
deferred0_1 = arg1;
|
||||
console.error(getStringFromWasm0(arg0, arg1));
|
||||
} finally {
|
||||
wasm.__wbindgen_export(deferred0_0, deferred0_1, 1);
|
||||
}
|
||||
},
|
||||
__wbg_new_227d7c05414eb861: function() {
|
||||
const ret = new Error();
|
||||
return addHeapObject(ret);
|
||||
},
|
||||
__wbg_stack_3b0d974bbf31e44f: function(arg0, arg1) {
|
||||
const ret = getObject(arg1).stack;
|
||||
const ptr1 = passStringToWasm0(ret, wasm.__wbindgen_export2, wasm.__wbindgen_export3);
|
||||
const len1 = WASM_VECTOR_LEN;
|
||||
getDataViewMemory0().setInt32(arg0 + 4 * 1, len1, true);
|
||||
getDataViewMemory0().setInt32(arg0 + 4 * 0, ptr1, true);
|
||||
},
|
||||
__wbindgen_cast_0000000000000001: function(arg0, arg1) {
|
||||
// Cast intrinsic for `Ref(String) -> Externref`.
|
||||
const ret = getStringFromWasm0(arg0, arg1);
|
||||
return addHeapObject(ret);
|
||||
},
|
||||
__wbindgen_object_drop_ref: function(arg0) {
|
||||
takeObject(arg0);
|
||||
},
|
||||
};
|
||||
return {
|
||||
__proto__: null,
|
||||
"./ruvector_cnn_wasm_bg.js": import0,
|
||||
};
|
||||
}
|
||||
|
||||
const EmbedderConfigFinalization = (typeof FinalizationRegistry === 'undefined')
|
||||
? { register: () => {}, unregister: () => {} }
|
||||
: new FinalizationRegistry(ptr => wasm.__wbg_embedderconfig_free(ptr >>> 0, 1));
|
||||
const LayerOpsFinalization = (typeof FinalizationRegistry === 'undefined')
|
||||
? { register: () => {}, unregister: () => {} }
|
||||
: new FinalizationRegistry(ptr => wasm.__wbg_layerops_free(ptr >>> 0, 1));
|
||||
const SimdOpsFinalization = (typeof FinalizationRegistry === 'undefined')
|
||||
? { register: () => {}, unregister: () => {} }
|
||||
: new FinalizationRegistry(ptr => wasm.__wbg_simdops_free(ptr >>> 0, 1));
|
||||
const WasmCnnEmbedderFinalization = (typeof FinalizationRegistry === 'undefined')
|
||||
? { register: () => {}, unregister: () => {} }
|
||||
: new FinalizationRegistry(ptr => wasm.__wbg_wasmcnnembedder_free(ptr >>> 0, 1));
|
||||
const WasmInfoNCELossFinalization = (typeof FinalizationRegistry === 'undefined')
|
||||
? { register: () => {}, unregister: () => {} }
|
||||
: new FinalizationRegistry(ptr => wasm.__wbg_wasminfonceloss_free(ptr >>> 0, 1));
|
||||
const WasmTripletLossFinalization = (typeof FinalizationRegistry === 'undefined')
|
||||
? { register: () => {}, unregister: () => {} }
|
||||
: new FinalizationRegistry(ptr => wasm.__wbg_wasmtripletloss_free(ptr >>> 0, 1));
|
||||
|
||||
function addHeapObject(obj) {
|
||||
if (heap_next === heap.length) heap.push(heap.length + 1);
|
||||
const idx = heap_next;
|
||||
heap_next = heap[idx];
|
||||
|
||||
heap[idx] = obj;
|
||||
return idx;
|
||||
}
|
||||
|
||||
function _assertClass(instance, klass) {
|
||||
if (!(instance instanceof klass)) {
|
||||
throw new Error(`expected instance of ${klass.name}`);
|
||||
}
|
||||
}
|
||||
|
||||
function dropObject(idx) {
|
||||
if (idx < 1028) return;
|
||||
heap[idx] = heap_next;
|
||||
heap_next = idx;
|
||||
}
|
||||
|
||||
function getArrayF32FromWasm0(ptr, len) {
|
||||
ptr = ptr >>> 0;
|
||||
return getFloat32ArrayMemory0().subarray(ptr / 4, ptr / 4 + len);
|
||||
}
|
||||
|
||||
let cachedDataViewMemory0 = null;
|
||||
function getDataViewMemory0() {
|
||||
if (cachedDataViewMemory0 === null || cachedDataViewMemory0.buffer.detached === true || (cachedDataViewMemory0.buffer.detached === undefined && cachedDataViewMemory0.buffer !== wasm.memory.buffer)) {
|
||||
cachedDataViewMemory0 = new DataView(wasm.memory.buffer);
|
||||
}
|
||||
return cachedDataViewMemory0;
|
||||
}
|
||||
|
||||
let cachedFloat32ArrayMemory0 = null;
|
||||
function getFloat32ArrayMemory0() {
|
||||
if (cachedFloat32ArrayMemory0 === null || cachedFloat32ArrayMemory0.byteLength === 0) {
|
||||
cachedFloat32ArrayMemory0 = new Float32Array(wasm.memory.buffer);
|
||||
}
|
||||
return cachedFloat32ArrayMemory0;
|
||||
}
|
||||
|
||||
function getStringFromWasm0(ptr, len) {
|
||||
ptr = ptr >>> 0;
|
||||
return decodeText(ptr, len);
|
||||
}
|
||||
|
||||
let cachedUint8ArrayMemory0 = null;
|
||||
function getUint8ArrayMemory0() {
|
||||
if (cachedUint8ArrayMemory0 === null || cachedUint8ArrayMemory0.byteLength === 0) {
|
||||
cachedUint8ArrayMemory0 = new Uint8Array(wasm.memory.buffer);
|
||||
}
|
||||
return cachedUint8ArrayMemory0;
|
||||
}
|
||||
|
||||
function getObject(idx) { return heap[idx]; }
|
||||
|
||||
let heap = new Array(1024).fill(undefined);
|
||||
heap.push(undefined, null, true, false);
|
||||
|
||||
let heap_next = heap.length;
|
||||
|
||||
function isLikeNone(x) {
|
||||
return x === undefined || x === null;
|
||||
}
|
||||
|
||||
function passArray8ToWasm0(arg, malloc) {
|
||||
const ptr = malloc(arg.length * 1, 1) >>> 0;
|
||||
getUint8ArrayMemory0().set(arg, ptr / 1);
|
||||
WASM_VECTOR_LEN = arg.length;
|
||||
return ptr;
|
||||
}
|
||||
|
||||
function passArrayF32ToWasm0(arg, malloc) {
|
||||
const ptr = malloc(arg.length * 4, 4) >>> 0;
|
||||
getFloat32ArrayMemory0().set(arg, ptr / 4);
|
||||
WASM_VECTOR_LEN = arg.length;
|
||||
return ptr;
|
||||
}
|
||||
|
||||
function passStringToWasm0(arg, malloc, realloc) {
|
||||
if (realloc === undefined) {
|
||||
const buf = cachedTextEncoder.encode(arg);
|
||||
const ptr = malloc(buf.length, 1) >>> 0;
|
||||
getUint8ArrayMemory0().subarray(ptr, ptr + buf.length).set(buf);
|
||||
WASM_VECTOR_LEN = buf.length;
|
||||
return ptr;
|
||||
}
|
||||
|
||||
let len = arg.length;
|
||||
let ptr = malloc(len, 1) >>> 0;
|
||||
|
||||
const mem = getUint8ArrayMemory0();
|
||||
|
||||
let offset = 0;
|
||||
|
||||
for (; offset < len; offset++) {
|
||||
const code = arg.charCodeAt(offset);
|
||||
if (code > 0x7F) break;
|
||||
mem[ptr + offset] = code;
|
||||
}
|
||||
if (offset !== len) {
|
||||
if (offset !== 0) {
|
||||
arg = arg.slice(offset);
|
||||
}
|
||||
ptr = realloc(ptr, len, len = offset + arg.length * 3, 1) >>> 0;
|
||||
const view = getUint8ArrayMemory0().subarray(ptr + offset, ptr + len);
|
||||
const ret = cachedTextEncoder.encodeInto(arg, view);
|
||||
|
||||
offset += ret.written;
|
||||
ptr = realloc(ptr, len, offset, 1) >>> 0;
|
||||
}
|
||||
|
||||
WASM_VECTOR_LEN = offset;
|
||||
return ptr;
|
||||
}
|
||||
|
||||
function takeObject(idx) {
|
||||
const ret = getObject(idx);
|
||||
dropObject(idx);
|
||||
return ret;
|
||||
}
|
||||
|
||||
let cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });
|
||||
cachedTextDecoder.decode();
|
||||
const MAX_SAFARI_DECODE_BYTES = 2146435072;
|
||||
let numBytesDecoded = 0;
|
||||
function decodeText(ptr, len) {
|
||||
numBytesDecoded += len;
|
||||
if (numBytesDecoded >= MAX_SAFARI_DECODE_BYTES) {
|
||||
cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });
|
||||
cachedTextDecoder.decode();
|
||||
numBytesDecoded = len;
|
||||
}
|
||||
return cachedTextDecoder.decode(getUint8ArrayMemory0().subarray(ptr, ptr + len));
|
||||
}
|
||||
|
||||
const cachedTextEncoder = new TextEncoder();
|
||||
|
||||
if (!('encodeInto' in cachedTextEncoder)) {
|
||||
cachedTextEncoder.encodeInto = function (arg, view) {
|
||||
const buf = cachedTextEncoder.encode(arg);
|
||||
view.set(buf);
|
||||
return {
|
||||
read: arg.length,
|
||||
written: buf.length
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
let WASM_VECTOR_LEN = 0;
|
||||
|
||||
let wasmModule, wasm;
|
||||
function __wbg_finalize_init(instance, module) {
|
||||
wasm = instance.exports;
|
||||
wasmModule = module;
|
||||
cachedDataViewMemory0 = null;
|
||||
cachedFloat32ArrayMemory0 = null;
|
||||
cachedUint8ArrayMemory0 = null;
|
||||
wasm.__wbindgen_start();
|
||||
return wasm;
|
||||
}
|
||||
|
||||
async function __wbg_load(module, imports) {
|
||||
if (typeof Response === 'function' && module instanceof Response) {
|
||||
if (typeof WebAssembly.instantiateStreaming === 'function') {
|
||||
try {
|
||||
return await WebAssembly.instantiateStreaming(module, imports);
|
||||
} catch (e) {
|
||||
const validResponse = module.ok && expectedResponseType(module.type);
|
||||
|
||||
if (validResponse && module.headers.get('Content-Type') !== 'application/wasm') {
|
||||
console.warn("`WebAssembly.instantiateStreaming` failed because your server does not serve Wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n", e);
|
||||
|
||||
} else { throw e; }
|
||||
}
|
||||
}
|
||||
|
||||
const bytes = await module.arrayBuffer();
|
||||
return await WebAssembly.instantiate(bytes, imports);
|
||||
} else {
|
||||
const instance = await WebAssembly.instantiate(module, imports);
|
||||
|
||||
if (instance instanceof WebAssembly.Instance) {
|
||||
return { instance, module };
|
||||
} else {
|
||||
return instance;
|
||||
}
|
||||
}
|
||||
|
||||
function expectedResponseType(type) {
|
||||
switch (type) {
|
||||
case 'basic': case 'cors': case 'default': return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function initSync(module) {
|
||||
if (wasm !== undefined) return wasm;
|
||||
|
||||
|
||||
if (module !== undefined) {
|
||||
if (Object.getPrototypeOf(module) === Object.prototype) {
|
||||
({module} = module)
|
||||
} else {
|
||||
console.warn('using deprecated parameters for `initSync()`; pass a single object instead')
|
||||
}
|
||||
}
|
||||
|
||||
const imports = __wbg_get_imports();
|
||||
if (!(module instanceof WebAssembly.Module)) {
|
||||
module = new WebAssembly.Module(module);
|
||||
}
|
||||
const instance = new WebAssembly.Instance(module, imports);
|
||||
return __wbg_finalize_init(instance, module);
|
||||
}
|
||||
|
||||
async function __wbg_init(module_or_path) {
|
||||
if (wasm !== undefined) return wasm;
|
||||
|
||||
|
||||
if (module_or_path !== undefined) {
|
||||
if (Object.getPrototypeOf(module_or_path) === Object.prototype) {
|
||||
({module_or_path} = module_or_path)
|
||||
} else {
|
||||
console.warn('using deprecated parameters for the initialization function; pass a single object instead')
|
||||
}
|
||||
}
|
||||
|
||||
if (module_or_path === undefined) {
|
||||
module_or_path = new URL('ruvector_cnn_wasm_bg.wasm', import.meta.url);
|
||||
}
|
||||
const imports = __wbg_get_imports();
|
||||
|
||||
if (typeof module_or_path === 'string' || (typeof Request === 'function' && module_or_path instanceof Request) || (typeof URL === 'function' && module_or_path instanceof URL)) {
|
||||
module_or_path = fetch(module_or_path);
|
||||
}
|
||||
|
||||
const { instance, module } = await __wbg_load(await module_or_path, imports);
|
||||
|
||||
return __wbg_finalize_init(instance, module);
|
||||
}
|
||||
|
||||
export { initSync, __wbg_init as default };
|
||||
Binary file not shown.
Reference in New Issue
Block a user