Compare commits
21 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 51140f599f | |||
| 47d0640c49 | |||
| 6959668e21 | |||
| 6a408b30e8 | |||
| 64dae5b1c1 | |||
| 8e487c54ea | |||
| 135d7d3d8c | |||
| 9dd61bdbfa | |||
| 8166d8d822 | |||
| fdc7142dfa | |||
| 02192b0232 | |||
| 67f1fc162e | |||
| 4e925dba50 | |||
| 46d718d62f | |||
| 88d39e2639 | |||
| 7c2e7e2b27 | |||
| 0aab555821 | |||
| df394019cc | |||
| 47861de821 | |||
| 779bf8ff43 | |||
| fbd7d837c7 |
@@ -1,4 +1,4 @@
|
||||
# WiFi DensePose
|
||||
# π RuView: WiFi DensePose:
|
||||
|
||||
**See through walls with WiFi.** No cameras. No wearables. Just radio waves.
|
||||
|
||||
@@ -12,6 +12,7 @@ WiFi DensePose turns commodity WiFi signals into real-time human pose estimation
|
||||
[](#esp32-s3-hardware-pipeline)
|
||||
[](https://crates.io/crates/wifi-densepose-ruvector)
|
||||
|
||||
|
||||
> | What | How | Speed |
|
||||
> |------|-----|-------|
|
||||
> | **Pose estimation** | CSI subcarrier amplitude/phase → DensePose UV maps | 54K fps (Rust) |
|
||||
@@ -54,6 +55,12 @@ docker run -p 3000:3000 ruvnet/wifi-densepose:latest
|
||||
|
||||
---
|
||||
|
||||
|
||||
<img src="assets/screen.png" alt="WiFi DensePose — Live pose detection with setup guide" width="800">
|
||||
<br>
|
||||
<em>Real-time pose skeleton from WiFi CSI signals — no cameras, no wearables</em>
|
||||
|
||||
|
||||
## 🚀 Key Features
|
||||
|
||||
### Sensing
|
||||
@@ -80,7 +87,8 @@ The system learns on its own and gets smarter over time — no hand-tuning, no l
|
||||
| 🎯 | **AI Signal Processing** | Attention networks, graph algorithms, and smart compression replace hand-tuned thresholds — adapts to each room automatically ([RuVector](https://github.com/ruvnet/ruvector)) |
|
||||
| 🌍 | **Works Everywhere** | Train once, deploy in any room — adversarial domain generalization strips environment bias so models transfer across rooms, buildings, and hardware ([ADR-027](docs/adr/ADR-027-cross-environment-domain-generalization.md)) |
|
||||
| 👁️ | **Cross-Viewpoint Fusion** | Learned attention fuses multiple viewpoints with geometric bias — reduces body occlusion and depth ambiguity that physics prevents any single sensor from solving ([ADR-031](docs/adr/ADR-031-ruview-sensing-first-rf-mode.md)) |
|
||||
| 🔮 | **Signal-Line Protocol** | `ruvector-crv` 6-stage CRV pipeline maps CSI sensing to Poincare ball embeddings, GNN topology, SNN temporal encoding, and MinCut partitioning | -- |
|
||||
| 🔮 | **Signal-Line Protocol** | `ruvector-crv` 6-stage CRV pipeline maps CSI sensing to Poincare ball embeddings, GNN topology, SNN temporal encoding, and MinCut partitioning ([ADR-033](docs/adr/ADR-033-crv-signal-line-sensing-integration.md)) |
|
||||
| 🔒 | **QUIC Mesh Security** | `midstreamer-quic` TLS 1.3 AEAD transport with HMAC-authenticated beacons, SipHash frame integrity, replay protection, and connection migration ([ADR-032](docs/adr/ADR-032-multistatic-mesh-security-hardening.md)) |
|
||||
|
||||
### Performance & Deployment
|
||||
|
||||
@@ -1585,9 +1593,13 @@ Multistatic sensing, persistent field model, and cross-viewpoint fusion — the
|
||||
- **Channel-Hopping Firmware** — ESP32 firmware extended with hop table, timer-driven channel switching, NDP injection stub; NVS config for all TDM parameters; fully backward-compatible
|
||||
- **DDD Domain Model** — 6 bounded contexts, ubiquitous language, aggregate roots, domain events, full event bus specification
|
||||
- **`ruvector-crv` 6-stage CRV signal-line integration (ADR-033)** — Maps Coordinate Remote Viewing methodology to WiFi CSI: gestalt classification, sensory encoding, GNN topology, SNN coherence gating, differentiable search, MinCut partitioning; cross-session convergence for multi-room identity continuity
|
||||
- **ADR-032 multistatic mesh security hardening** — Bounded calibration buffers, atomic counters, division-by-zero guards, NaN-safe normalization across all multistatic modules
|
||||
- **ADR-032 multistatic mesh security hardening** — HMAC-SHA256 beacon auth, SipHash-2-4 frame integrity, NDP rate limiter, coherence gate timeout, bounded buffers, NVS credential zeroing, atomic firmware state
|
||||
- **ADR-032a QUIC transport layer** — `midstreamer-quic` TLS 1.3 AEAD for aggregator nodes, dual-mode security (ManualCrypto/QuicTransport), QUIC stream mapping, connection migration, congestion control
|
||||
- **ADR-033 CRV signal-line sensing integration** — Architecture decision record for the 6-stage CRV pipeline mapping to ruvector components
|
||||
- **9,000+ lines of new Rust code** across 17 modules with 300+ tests
|
||||
- **Temporal gesture matching** — `midstreamer-temporal-compare` DTW/LCS/edit-distance gesture classification with quantized feature comparison
|
||||
- **Attractor drift analysis** — `midstreamer-attractor` Takens' theorem phase-space embedding with Lyapunov exponent regime detection (Stable/Periodic/Chaotic)
|
||||
- **v0.3.0 published** — All 15 workspace crates published to [crates.io](https://crates.io/crates/wifi-densepose-core) with updated dependencies
|
||||
- **28,000+ lines of new Rust code** across 26 modules with 400+ tests
|
||||
- **Security hardened** — Bounded buffers, NaN guards, no panics in public APIs, input validation at all boundaries
|
||||
|
||||
### v3.0.0 — 2026-03-01
|
||||
|
||||
|
After Width: | Height: | Size: 270 KiB |
@@ -1,369 +0,0 @@
|
||||
# Claude Code Configuration — WiFi-DensePose + Claude Flow V3
|
||||
|
||||
## Project: wifi-densepose
|
||||
|
||||
WiFi-based human pose estimation using Channel State Information (CSI).
|
||||
Dual codebase: Python v1 (`v1/`) and Rust port (`rust-port/wifi-densepose-rs/`).
|
||||
### Key Rust Crates
|
||||
| Crate | Description |
|
||||
|-------|-------------|
|
||||
| `wifi-densepose-core` | Core types, traits, error types, CSI frame primitives |
|
||||
| `wifi-densepose-signal` | SOTA signal processing + RuvSense multistatic sensing (14 modules) |
|
||||
| `wifi-densepose-nn` | Neural network inference (ONNX, PyTorch, Candle backends) |
|
||||
| `wifi-densepose-train` | Training pipeline with ruvector integration + ruview_metrics |
|
||||
| `wifi-densepose-mat` | Mass Casualty Assessment Tool — disaster survivor detection |
|
||||
| `wifi-densepose-hardware` | ESP32 aggregator, TDM protocol, channel hopping firmware |
|
||||
| `wifi-densepose-ruvector` | RuVector v2.0.4 integration + cross-viewpoint fusion (5 modules) |
|
||||
| `wifi-densepose-api` | REST API (Axum) |
|
||||
| `wifi-densepose-db` | Database layer (Postgres, SQLite, Redis) |
|
||||
| `wifi-densepose-config` | Configuration management |
|
||||
| `wifi-densepose-wasm` | WebAssembly bindings for browser deployment |
|
||||
| `wifi-densepose-cli` | CLI tool (`wifi-densepose` binary) |
|
||||
| `wifi-densepose-sensing-server` | Lightweight Axum server for WiFi sensing UI |
|
||||
| `wifi-densepose-wifiscan` | Multi-BSSID WiFi scanning (ADR-022) |
|
||||
| `wifi-densepose-vitals` | ESP32 CSI-grade vital sign extraction (ADR-021) |
|
||||
|
||||
### RuvSense Modules (`signal/src/ruvsense/`)
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `multiband.rs` | Multi-band CSI frame fusion, cross-channel coherence |
|
||||
| `phase_align.rs` | Iterative LO phase offset estimation, circular mean |
|
||||
| `multistatic.rs` | Attention-weighted fusion, geometric diversity |
|
||||
| `coherence.rs` | Z-score coherence scoring, DriftProfile |
|
||||
| `coherence_gate.rs` | Accept/PredictOnly/Reject/Recalibrate gate decisions |
|
||||
| `pose_tracker.rs` | 17-keypoint Kalman tracker with AETHER re-ID embeddings |
|
||||
| `field_model.rs` | SVD room eigenstructure, perturbation extraction |
|
||||
| `tomography.rs` | RF tomography, ISTA L1 solver, voxel grid |
|
||||
| `longitudinal.rs` | Welford stats, biomechanics drift detection |
|
||||
| `intention.rs` | Pre-movement lead signals (200-500ms) |
|
||||
| `cross_room.rs` | Environment fingerprinting, transition graph |
|
||||
| `gesture.rs` | DTW template matching gesture classifier |
|
||||
| `adversarial.rs` | Physically impossible signal detection, multi-link consistency |
|
||||
|
||||
### Cross-Viewpoint Fusion (`ruvector/src/viewpoint/`)
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `attention.rs` | CrossViewpointAttention, GeometricBias, softmax with G_bias |
|
||||
| `geometry.rs` | GeometricDiversityIndex, Cramer-Rao bounds, Fisher Information |
|
||||
| `coherence.rs` | Phase phasor coherence, hysteresis gate |
|
||||
| `fusion.rs` | MultistaticArray aggregate root, domain events |
|
||||
|
||||
### RuVector v2.0.4 Integration (ADR-016 complete, ADR-017 proposed)
|
||||
All 5 ruvector crates integrated in workspace:
|
||||
- `ruvector-mincut` → `metrics.rs` (DynamicPersonMatcher) + `subcarrier_selection.rs`
|
||||
- `ruvector-attn-mincut` → `model.rs` (apply_antenna_attention) + `spectrogram.rs`
|
||||
- `ruvector-temporal-tensor` → `dataset.rs` (CompressedCsiBuffer) + `breathing.rs`
|
||||
- `ruvector-solver` → `subcarrier.rs` (sparse interpolation 114→56) + `triangulation.rs`
|
||||
- `ruvector-attention` → `model.rs` (apply_spatial_attention) + `bvp.rs`
|
||||
|
||||
### Architecture Decisions
|
||||
32 ADRs in `docs/adr/` (ADR-001 through ADR-032). Key ones:
|
||||
- ADR-014: SOTA signal processing (Accepted)
|
||||
- ADR-015: MM-Fi + Wi-Pose training datasets (Accepted)
|
||||
- ADR-016: RuVector training pipeline integration (Accepted — complete)
|
||||
- ADR-017: RuVector signal + MAT integration (Proposed — next target)
|
||||
- ADR-024: Contrastive CSI embedding / AETHER (Accepted)
|
||||
- ADR-027: Cross-environment domain generalization / MERIDIAN (Accepted)
|
||||
- ADR-028: ESP32 capability audit + witness verification (Accepted)
|
||||
- ADR-029: RuvSense multistatic sensing mode (Proposed)
|
||||
- ADR-030: RuvSense persistent field model (Proposed)
|
||||
- ADR-031: RuView sensing-first RF mode (Proposed)
|
||||
- ADR-032: Multistatic mesh security hardening (Proposed)
|
||||
|
||||
### Build & Test Commands (this repo)
|
||||
```bash
|
||||
# Rust — full workspace tests (1,031+ tests, ~2 min)
|
||||
cd rust-port/wifi-densepose-rs
|
||||
cargo test --workspace --no-default-features
|
||||
|
||||
# Rust — single crate check (no GPU needed)
|
||||
cargo check -p wifi-densepose-train --no-default-features
|
||||
|
||||
# Rust — publish crates (dependency order)
|
||||
cargo publish -p wifi-densepose-core --no-default-features
|
||||
cargo publish -p wifi-densepose-signal --no-default-features
|
||||
# ... see crate publishing order below
|
||||
|
||||
# Python — deterministic proof verification (SHA-256)
|
||||
python v1/data/proof/verify.py
|
||||
|
||||
# Python — test suite
|
||||
cd v1 && python -m pytest tests/ -x -q
|
||||
```
|
||||
|
||||
### Crate Publishing Order
|
||||
Crates must be published in dependency order:
|
||||
1. `wifi-densepose-core` (no internal deps)
|
||||
2. `wifi-densepose-vitals` (no internal deps)
|
||||
3. `wifi-densepose-wifiscan` (no internal deps)
|
||||
4. `wifi-densepose-hardware` (no internal deps)
|
||||
5. `wifi-densepose-config` (no internal deps)
|
||||
6. `wifi-densepose-db` (no internal deps)
|
||||
7. `wifi-densepose-signal` (depends on core)
|
||||
8. `wifi-densepose-nn` (no internal deps, workspace only)
|
||||
9. `wifi-densepose-ruvector` (no internal deps, workspace only)
|
||||
10. `wifi-densepose-train` (depends on signal, nn)
|
||||
11. `wifi-densepose-mat` (depends on core, signal, nn)
|
||||
12. `wifi-densepose-api` (no internal deps)
|
||||
13. `wifi-densepose-wasm` (depends on mat)
|
||||
14. `wifi-densepose-sensing-server` (depends on wifiscan)
|
||||
15. `wifi-densepose-cli` (depends on mat)
|
||||
|
||||
### Validation & Witness Verification (ADR-028)
|
||||
|
||||
**After any significant code change, run the full validation:**
|
||||
|
||||
```bash
|
||||
# 1. Rust tests — must be 1,031+ passed, 0 failed
|
||||
cd rust-port/wifi-densepose-rs
|
||||
cargo test --workspace --no-default-features
|
||||
|
||||
# 2. Python proof — must print VERDICT: PASS
|
||||
cd ../..
|
||||
python v1/data/proof/verify.py
|
||||
|
||||
# 3. Generate witness bundle (includes both above + firmware hashes)
|
||||
bash scripts/generate-witness-bundle.sh
|
||||
|
||||
# 4. Self-verify the bundle — must be 7/7 PASS
|
||||
cd dist/witness-bundle-ADR028-*/
|
||||
bash VERIFY.sh
|
||||
```
|
||||
|
||||
**If the Python proof hash changes** (e.g., numpy/scipy version update):
|
||||
```bash
|
||||
# Regenerate the expected hash, then verify it passes
|
||||
python v1/data/proof/verify.py --generate-hash
|
||||
python v1/data/proof/verify.py
|
||||
```
|
||||
|
||||
**Witness bundle contents** (`dist/witness-bundle-ADR028-<sha>.tar.gz`):
|
||||
- `WITNESS-LOG-028.md` — 33-row attestation matrix with evidence per capability
|
||||
- `ADR-028-esp32-capability-audit.md` — Full audit findings
|
||||
- `proof/verify.py` + `expected_features.sha256` — Deterministic pipeline proof
|
||||
- `test-results/rust-workspace-tests.log` — Full cargo test output
|
||||
- `firmware-manifest/source-hashes.txt` — SHA-256 of all 7 ESP32 firmware files
|
||||
- `crate-manifest/versions.txt` — All 15 crates with versions
|
||||
- `VERIFY.sh` — One-command self-verification for recipients
|
||||
|
||||
**Key proof artifacts:**
|
||||
- `v1/data/proof/verify.py` — Trust Kill Switch: feeds reference signal through production pipeline, hashes output
|
||||
- `v1/data/proof/expected_features.sha256` — Published expected hash
|
||||
- `v1/data/proof/sample_csi_data.json` — 1,000 synthetic CSI frames (seed=42)
|
||||
- `docs/WITNESS-LOG-028.md` — 11-step reproducible verification procedure
|
||||
- `docs/adr/ADR-028-esp32-capability-audit.md` — Complete audit record
|
||||
|
||||
### Branch
|
||||
Default branch: `main`
|
||||
Active feature branch: `ruvsense-full-implementation` (PR #77)
|
||||
|
||||
---
|
||||
|
||||
## Behavioral Rules (Always Enforced)
|
||||
|
||||
- Do what has been asked; nothing more, nothing less
|
||||
- NEVER create files unless they're absolutely necessary for achieving your goal
|
||||
- ALWAYS prefer editing an existing file to creating a new one
|
||||
- NEVER proactively create documentation files (*.md) or README files unless explicitly requested
|
||||
- NEVER save working files, text/mds, or tests to the root folder
|
||||
- Never continuously check status after spawning a swarm — wait for results
|
||||
- ALWAYS read a file before editing it
|
||||
- NEVER commit secrets, credentials, or .env files
|
||||
|
||||
## File Organization
|
||||
|
||||
- NEVER save to root folder — use the directories below
|
||||
- `docs/adr/` — Architecture Decision Records (32 ADRs)
|
||||
- `docs/ddd/` — Domain-Driven Design models
|
||||
- `rust-port/wifi-densepose-rs/crates/` — Rust workspace crates (15 crates)
|
||||
- `rust-port/wifi-densepose-rs/crates/wifi-densepose-signal/src/ruvsense/` — RuvSense multistatic modules (14 files)
|
||||
- `rust-port/wifi-densepose-rs/crates/wifi-densepose-ruvector/src/viewpoint/` — Cross-viewpoint fusion (5 files)
|
||||
- `rust-port/wifi-densepose-rs/crates/wifi-densepose-hardware/src/esp32/` — ESP32 TDM protocol
|
||||
- `firmware/esp32-csi-node/main/` — ESP32 C firmware (channel hopping, NVS config, TDM)
|
||||
- `v1/src/` — Python source (core, hardware, services, api)
|
||||
- `v1/data/proof/` — Deterministic CSI proof bundles
|
||||
- `.claude-flow/` — Claude Flow coordination state (committed for team sharing)
|
||||
- `.claude/` — Claude Code settings, agents, memory (committed for team sharing)
|
||||
|
||||
## Project Architecture
|
||||
|
||||
- Follow Domain-Driven Design with bounded contexts
|
||||
- Keep files under 500 lines
|
||||
- Use typed interfaces for all public APIs
|
||||
- Prefer TDD London School (mock-first) for new code
|
||||
- Use event sourcing for state changes
|
||||
- Ensure input validation at system boundaries
|
||||
|
||||
### Project Config
|
||||
|
||||
- **Topology**: hierarchical-mesh
|
||||
- **Max Agents**: 15
|
||||
- **Memory**: hybrid
|
||||
- **HNSW**: Enabled
|
||||
- **Neural**: Enabled
|
||||
|
||||
## Pre-Merge Checklist
|
||||
|
||||
Before merging any PR, verify each item applies and is addressed:
|
||||
|
||||
1. **Rust tests pass** — `cargo test --workspace --no-default-features` (1,031+ passed, 0 failed)
|
||||
2. **Python proof passes** — `python v1/data/proof/verify.py` (VERDICT: PASS)
|
||||
3. **README.md** — Update platform tables, crate descriptions, hardware tables, feature summaries if scope changed
|
||||
4. **CLAUDE.md** — Update crate table, ADR list, module tables, version if scope changed
|
||||
5. **CHANGELOG.md** — Add entry under `[Unreleased]` with what was added/fixed/changed
|
||||
6. **User guide** (`docs/user-guide.md`) — Update if new data sources, CLI flags, or setup steps were added
|
||||
7. **ADR index** — Update ADR count in README docs table if a new ADR was created
|
||||
8. **Witness bundle** — Regenerate if tests or proof hash changed: `bash scripts/generate-witness-bundle.sh`
|
||||
9. **Docker Hub image** — Only rebuild if Dockerfile, dependencies, or runtime behavior changed
|
||||
10. **Crate publishing** — Only needed if a crate is published to crates.io and its public API changed
|
||||
11. **`.gitignore`** — Add any new build artifacts or binaries
|
||||
12. **Security audit** — Run security review for new modules touching hardware/network boundaries
|
||||
|
||||
## Build & Test
|
||||
|
||||
```bash
|
||||
# Build
|
||||
npm run build
|
||||
|
||||
# Test
|
||||
npm test
|
||||
|
||||
# Lint
|
||||
npm run lint
|
||||
```
|
||||
|
||||
- ALWAYS run tests after making code changes
|
||||
- ALWAYS verify build succeeds before committing
|
||||
|
||||
## Security Rules
|
||||
|
||||
- NEVER hardcode API keys, secrets, or credentials in source files
|
||||
- NEVER commit .env files or any file containing secrets
|
||||
- Always validate user input at system boundaries
|
||||
- Always sanitize file paths to prevent directory traversal
|
||||
- Run `npx @claude-flow/cli@latest security scan` after security-related changes
|
||||
|
||||
## Concurrency: 1 MESSAGE = ALL RELATED OPERATIONS
|
||||
|
||||
- All operations MUST be concurrent/parallel in a single message
|
||||
- Use Claude Code's Task tool for spawning agents, not just MCP
|
||||
- ALWAYS batch ALL todos in ONE TodoWrite call (5-10+ minimum)
|
||||
- ALWAYS spawn ALL agents in ONE message with full instructions via Task tool
|
||||
- ALWAYS batch ALL file reads/writes/edits in ONE message
|
||||
- ALWAYS batch ALL Bash commands in ONE message
|
||||
|
||||
## Swarm Orchestration
|
||||
|
||||
- MUST initialize the swarm using CLI tools when starting complex tasks
|
||||
- MUST spawn concurrent agents using Claude Code's Task tool
|
||||
- Never use CLI tools alone for execution — Task tool agents do the actual work
|
||||
- MUST call CLI tools AND Task tool in ONE message for complex work
|
||||
|
||||
### 3-Tier Model Routing (ADR-026)
|
||||
|
||||
| Tier | Handler | Latency | Cost | Use Cases |
|
||||
|------|---------|---------|------|-----------|
|
||||
| **1** | Agent Booster (WASM) | <1ms | $0 | Simple transforms (var→const, add types) — Skip LLM |
|
||||
| **2** | Haiku | ~500ms | $0.0002 | Simple tasks, low complexity (<30%) |
|
||||
| **3** | Sonnet/Opus | 2-5s | $0.003-0.015 | Complex reasoning, architecture, security (>30%) |
|
||||
|
||||
- Always check for `[AGENT_BOOSTER_AVAILABLE]` or `[TASK_MODEL_RECOMMENDATION]` before spawning agents
|
||||
- Use Edit tool directly when `[AGENT_BOOSTER_AVAILABLE]`
|
||||
|
||||
## Swarm Configuration & Anti-Drift
|
||||
|
||||
- ALWAYS use hierarchical topology for coding swarms
|
||||
- Keep maxAgents at 6-8 for tight coordination
|
||||
- Use specialized strategy for clear role boundaries
|
||||
- Use `raft` consensus for hive-mind (leader maintains authoritative state)
|
||||
- Run frequent checkpoints via `post-task` hooks
|
||||
- Keep shared memory namespace for all agents
|
||||
|
||||
```bash
|
||||
npx @claude-flow/cli@latest swarm init --topology hierarchical --max-agents 8 --strategy specialized
|
||||
```
|
||||
|
||||
## Swarm Execution Rules
|
||||
|
||||
- ALWAYS use `run_in_background: true` for all agent Task calls
|
||||
- ALWAYS put ALL agent Task calls in ONE message for parallel execution
|
||||
- After spawning, STOP — do NOT add more tool calls or check status
|
||||
- Never poll TaskOutput or check swarm status — trust agents to return
|
||||
- When agent results arrive, review ALL results before proceeding
|
||||
|
||||
## V3 CLI Commands
|
||||
|
||||
### Core Commands
|
||||
|
||||
| Command | Subcommands | Description |
|
||||
|---------|-------------|-------------|
|
||||
| `init` | 4 | Project initialization |
|
||||
| `agent` | 8 | Agent lifecycle management |
|
||||
| `swarm` | 6 | Multi-agent swarm coordination |
|
||||
| `memory` | 11 | AgentDB memory with HNSW search |
|
||||
| `task` | 6 | Task creation and lifecycle |
|
||||
| `session` | 7 | Session state management |
|
||||
| `hooks` | 17 | Self-learning hooks + 12 workers |
|
||||
| `hive-mind` | 6 | Byzantine fault-tolerant consensus |
|
||||
|
||||
### Quick CLI Examples
|
||||
|
||||
```bash
|
||||
npx @claude-flow/cli@latest init --wizard
|
||||
npx @claude-flow/cli@latest agent spawn -t coder --name my-coder
|
||||
npx @claude-flow/cli@latest swarm init --v3-mode
|
||||
npx @claude-flow/cli@latest memory search --query "authentication patterns"
|
||||
npx @claude-flow/cli@latest doctor --fix
|
||||
```
|
||||
|
||||
## Available Agents (60+ Types)
|
||||
|
||||
### Core Development
|
||||
`coder`, `reviewer`, `tester`, `planner`, `researcher`
|
||||
|
||||
### Specialized
|
||||
`security-architect`, `security-auditor`, `memory-specialist`, `performance-engineer`
|
||||
|
||||
### Swarm Coordination
|
||||
`hierarchical-coordinator`, `mesh-coordinator`, `adaptive-coordinator`
|
||||
|
||||
### GitHub & Repository
|
||||
`pr-manager`, `code-review-swarm`, `issue-tracker`, `release-manager`
|
||||
|
||||
### SPARC Methodology
|
||||
`sparc-coord`, `sparc-coder`, `specification`, `pseudocode`, `architecture`
|
||||
|
||||
## Memory Commands Reference
|
||||
|
||||
```bash
|
||||
# Store (REQUIRED: --key, --value; OPTIONAL: --namespace, --ttl, --tags)
|
||||
npx @claude-flow/cli@latest memory store --key "pattern-auth" --value "JWT with refresh" --namespace patterns
|
||||
|
||||
# Search (REQUIRED: --query; OPTIONAL: --namespace, --limit, --threshold)
|
||||
npx @claude-flow/cli@latest memory search --query "authentication patterns"
|
||||
|
||||
# List (OPTIONAL: --namespace, --limit)
|
||||
npx @claude-flow/cli@latest memory list --namespace patterns --limit 10
|
||||
|
||||
# Retrieve (REQUIRED: --key; OPTIONAL: --namespace)
|
||||
npx @claude-flow/cli@latest memory retrieve --key "pattern-auth" --namespace patterns
|
||||
```
|
||||
|
||||
## Quick Setup
|
||||
|
||||
```bash
|
||||
claude mcp add claude-flow -- npx -y @claude-flow/cli@latest
|
||||
npx @claude-flow/cli@latest daemon start
|
||||
npx @claude-flow/cli@latest doctor --fix
|
||||
```
|
||||
|
||||
## Claude Code vs CLI Tools
|
||||
|
||||
- Claude Code's Task tool handles ALL execution: agents, file ops, code generation, git
|
||||
- CLI tools handle coordination via Bash: swarm init, memory, hooks, routing
|
||||
- NEVER use CLI tools as a substitute for Task tool agents
|
||||
|
||||
## Support
|
||||
|
||||
- Documentation: https://github.com/ruvnet/claude-flow
|
||||
- Issues: https://github.com/ruvnet/claude-flow/issues
|
||||
@@ -42,5 +42,15 @@ EXPOSE 5005/udp
|
||||
|
||||
ENV RUST_LOG=info
|
||||
|
||||
ENTRYPOINT ["/app/sensing-server"]
|
||||
CMD ["--source", "simulated", "--tick-ms", "100", "--ui-path", "/app/ui", "--http-port", "3000", "--ws-port", "3001"]
|
||||
# CSI_SOURCE controls which data source the sensing server uses at startup.
|
||||
# auto — probe UDP port 5005 for an ESP32 first; fall back to simulation (default)
|
||||
# esp32 — receive real CSI frames from an ESP32 device over UDP port 5005
|
||||
# wifi — use host Wi-Fi RSSI/scan data (Windows netsh; not available in containers)
|
||||
# simulated — generate synthetic CSI frames (no hardware required)
|
||||
# Override at runtime: docker run -e CSI_SOURCE=esp32 ...
|
||||
ENV CSI_SOURCE=auto
|
||||
|
||||
ENTRYPOINT ["/bin/sh", "-c"]
|
||||
# Shell-form CMD allows $CSI_SOURCE to be substituted at container start.
|
||||
# The ENV default above (CSI_SOURCE=auto) applies when the variable is unset.
|
||||
CMD ["/app/sensing-server --source ${CSI_SOURCE} --tick-ms 100 --ui-path /app/ui --http-port 3000 --ws-port 3001"]
|
||||
|
||||
@@ -12,7 +12,14 @@ services:
|
||||
- "5005:5005/udp" # ESP32 UDP
|
||||
environment:
|
||||
- RUST_LOG=info
|
||||
command: ["--source", "simulated", "--tick-ms", "100", "--ui-path", "/app/ui", "--http-port", "3000", "--ws-port", "3001"]
|
||||
# CSI_SOURCE controls the data source for the sensing server.
|
||||
# Options: auto (default) — probe for ESP32 UDP then fall back to simulation
|
||||
# esp32 — receive real CSI frames from an ESP32 on UDP port 5005
|
||||
# wifi — use host Wi-Fi RSSI/scan data (Windows netsh)
|
||||
# simulated — generate synthetic CSI data (no hardware required)
|
||||
- CSI_SOURCE=${CSI_SOURCE:-auto}
|
||||
# command is passed as arguments to ENTRYPOINT (/bin/sh -c), so $CSI_SOURCE is expanded by the shell.
|
||||
command: ["/app/sensing-server --source ${CSI_SOURCE:-auto} --tick-ms 100 --ui-path /app/ui --http-port 3000 --ws-port 3001"]
|
||||
|
||||
python-sensing:
|
||||
build:
|
||||
|
||||
@@ -0,0 +1,688 @@
|
||||
# ADR-034: Expo React Native Mobile Application
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| **Status** | Accepted |
|
||||
| **Date** | 2026-03-02 |
|
||||
| **Deciders** | MaTriXy, rUv |
|
||||
| **Codename** | **FieldView** -- Mobile Companion for WiFi-DensePose Field Deployment |
|
||||
| **Relates to** | ADR-019 (Sensing-Only UI Mode), ADR-021 (Vital Sign Detection), ADR-026 (Survivor Track Lifecycle), ADR-029 (RuvSense Multistatic), ADR-031 (RuView Sensing-First RF), ADR-032 (Mesh Security) |
|
||||
|
||||
---
|
||||
|
||||
## 1. Context
|
||||
|
||||
### 1.1 Need for a Mobile Companion
|
||||
|
||||
WiFi-DensePose is a WiFi-based human pose estimation system using Channel State Information (CSI) from ESP32 mesh nodes. The existing web UI (`ui/`) serves desktop browsers but is not optimized for mobile form factors. Three deployment scenarios demand a purpose-built mobile application:
|
||||
|
||||
1. **Disaster response (WiFi-MAT)**: First responders deploying ESP32 mesh nodes in collapsed structures need a portable device to visualize survivor detections, breathing/heart rate vitals, and zone maps in real time. A laptop is impractical in rubble fields.
|
||||
2. **Building security**: Security operators patrolling a facility need a handheld display showing occupancy by zone, movement alerts, and historical patterns. The phone in their pocket is the natural form factor.
|
||||
3. **Healthcare monitoring**: Clinical staff monitoring patients via CSI-based contactless vitals need a tablet view at the bedside or nurse station, with gauges for breathing rate and heart rate that update in real time.
|
||||
|
||||
In all three scenarios, the mobile device does not communicate with ESP32 nodes directly. Instead, a Rust sensing server (`wifi-densepose-sensing-server`, ADR-031) aggregates ESP32 UDP streams and exposes a WebSocket API. The mobile app connects to this server over local WiFi.
|
||||
|
||||
### 1.2 Technology Selection Rationale
|
||||
|
||||
| Requirement | Decision | Rationale |
|
||||
|-------------|----------|-----------|
|
||||
| Cross-platform (iOS + Android + Web) | Expo SDK 55 + React Native 0.83 | Single codebase, managed workflow, OTA updates |
|
||||
| Real-time streaming | WebSocket (ws://host:3001/ws/sensing) | Sub-100ms latency from CSI capture to mobile display |
|
||||
| 3D visualization | Three.js Gaussian splat via WebView | Reuses existing `ui/` Three.js splat renderer; avoids native OpenGL binding |
|
||||
| State management | Zustand | Minimal boilerplate, React-concurrent safe, selector-based re-renders |
|
||||
| Persistence | AsyncStorage | Built into Expo, sufficient for settings and small cached state |
|
||||
| Navigation | react-navigation v7 (bottom tabs) | Standard React Native navigation; 5-tab layout fits mobile ergonomics |
|
||||
| WiFi RSSI scanning | Platform-specific (Android: react-native-wifi-reborn, iOS: CoreWLAN stub, Web: synthetic) | No cross-platform WiFi scanning API exists; platform modules are required |
|
||||
| E2E testing | Maestro YAML specs | Declarative, no Detox native build dependency, runs on CI |
|
||||
| Design system | Dark theme (#0D1117 bg, #32B8C6 accent) | Matches existing `ui/` sensing dashboard aesthetic; reduces eye strain in field conditions |
|
||||
|
||||
### 1.3 Relationship to Existing UI
|
||||
|
||||
The desktop web UI (`ui/`) and the mobile app share no code at the component level, but they consume the same backend APIs:
|
||||
|
||||
- **WebSocket**: `ws://host:3001/ws/sensing` -- streaming SensingFrame JSON
|
||||
- **REST**: `http://host:3000/api/v1/...` -- configuration, history, health
|
||||
|
||||
The mobile app's Three.js Gaussian splat viewer (LiveScreen) loads the same splat HTML bundle used by the desktop UI, rendered inside a WebView (native) or iframe (web).
|
||||
|
||||
---
|
||||
|
||||
## 2. Decision
|
||||
|
||||
Build an Expo React Native mobile application at `ui/mobile/` that provides five primary screens for field operators, connected to the Rust sensing server via WebSocket streaming. The app automatically falls back to simulated data when the sensing server is unreachable, enabling demos and offline testing.
|
||||
|
||||
### 2.1 Screen Architecture
|
||||
|
||||
```
|
||||
+---------------------------------------------------------------+
|
||||
| MainTabs (Bottom Tab Navigator) |
|
||||
+---------------------------------------------------------------+
|
||||
| |
|
||||
| +----------+ +----------+ +----------+ +--------+ +-----+ |
|
||||
| | Live | | Vitals | | Zones | | MAT | | Cog | |
|
||||
| | (3D splat| |(breathing| |(floor | |(disaster| |(set-| |
|
||||
| | + HUD) | | + heart) | | plan SVG)| |response)| |tings| |
|
||||
| +----------+ +----------+ +----------+ +--------+ +-----+ |
|
||||
| |
|
||||
+---------------------------------------------------------------+
|
||||
| ConnectionBanner (Connected / Simulated / Disconnected) |
|
||||
+---------------------------------------------------------------+
|
||||
```
|
||||
|
||||
**Screen responsibilities:**
|
||||
|
||||
| Screen | Primary View | Data Source | Key Components |
|
||||
|--------|-------------|-------------|----------------|
|
||||
| **Live** | 3D Gaussian splat with 17 COCO keypoints + HUD overlay | `poseStore.latestFrame` | `GaussianSplatWebView`, `LiveHUD`, `HudOverlay` |
|
||||
| **Vitals** | Breathing BPM gauge, heart rate BPM gauge, sparkline history | `poseStore.latestFrame.vital_signs` | `BreathingGauge`, `HeartRateGauge`, `MetricCard`, `SparklineChart` |
|
||||
| **Zones** | Floor plan SVG with occupancy heat overlay, zone legend | `poseStore.latestFrame.persons` | `FloorPlanSvg`, `OccupancyGrid`, `ZoneLegend` |
|
||||
| **MAT** | Survivor counter, zone map WebView, alert list | `matStore.survivors`, `matStore.alerts` | `SurvivorCounter`, `MatWebView`, `AlertList`, `AlertCard` |
|
||||
| **Settings** | Server URL input, theme picker, RSSI toggle | `settingsStore` | `ServerUrlInput`, `ThemePicker`, `RssiToggle` |
|
||||
|
||||
### 2.2 State Architecture
|
||||
|
||||
Three Zustand stores separate concerns and prevent unnecessary re-renders:
|
||||
|
||||
```
|
||||
+------------------------------------------------------------+
|
||||
| Zustand Stores |
|
||||
+------------------------------------------------------------+
|
||||
| |
|
||||
| poseStore |
|
||||
| +--------------------------------------------------------+ |
|
||||
| | connectionStatus: 'connected' | 'simulated' | 'error' | |
|
||||
| | latestFrame: SensingFrame | null | |
|
||||
| | frameHistory: RingBuffer<SensingFrame> | |
|
||||
| | features: FeatureVector | null | |
|
||||
| | persons: Person[] | |
|
||||
| | vitalSigns: VitalSigns | null | |
|
||||
| +--------------------------------------------------------+ |
|
||||
| |
|
||||
| matStore |
|
||||
| +--------------------------------------------------------+ |
|
||||
| | survivors: Survivor[] | |
|
||||
| | alerts: MatAlert[] | |
|
||||
| | events: MatEvent[] | |
|
||||
| | zoneMap: ZoneMap | null | |
|
||||
| +--------------------------------------------------------+ |
|
||||
| |
|
||||
| settingsStore (persisted via AsyncStorage) |
|
||||
| +--------------------------------------------------------+ |
|
||||
| | serverUrl: string (default: 'http://localhost:3000') | |
|
||||
| | wsUrl: string (default: 'ws://localhost:3001') | |
|
||||
| | theme: 'dark' | 'light' | |
|
||||
| | rssiEnabled: boolean | |
|
||||
| | simulationMode: boolean | |
|
||||
| +--------------------------------------------------------+ |
|
||||
| |
|
||||
+------------------------------------------------------------+
|
||||
```
|
||||
|
||||
### 2.3 Service Layer
|
||||
|
||||
Four services encapsulate external communication and data generation:
|
||||
|
||||
| Service | File | Responsibility |
|
||||
|---------|------|----------------|
|
||||
| `ws.service` | `src/services/ws.service.ts` | WebSocket connection lifecycle, reconnection with exponential backoff, SensingFrame parsing, dispatches to `poseStore` |
|
||||
| `api.service` | `src/services/api.service.ts` | REST calls to sensing server (health check, configuration, history endpoints) |
|
||||
| `rssi.service` | `src/services/rssi.service.ts` (+ platform variants) | Platform-specific WiFi RSSI scanning. Android uses `react-native-wifi-reborn`, iOS provides a CoreWLAN stub, Web generates synthetic RSSI values |
|
||||
| `simulation.service` | `src/services/simulation.service.ts` | Generates synthetic SensingFrame data when the real server is unreachable. Produces realistic amplitude, phase, vital signs, and person data on a configurable tick interval |
|
||||
|
||||
**Platform-specific RSSI service files:**
|
||||
|
||||
| File | Platform | Implementation |
|
||||
|------|----------|----------------|
|
||||
| `rssi.service.android.ts` | Android | `react-native-wifi-reborn` native module, requires `ACCESS_FINE_LOCATION` permission |
|
||||
| `rssi.service.ios.ts` | iOS | CoreWLAN stub (returns empty scan results; Apple restricts WiFi scanning to system apps) |
|
||||
| `rssi.service.web.ts` | Web | Synthetic RSSI values generated from noise model |
|
||||
| `rssi.service.ts` | Default | Re-exports platform-appropriate module via React Native file resolution |
|
||||
|
||||
### 2.4 Data Flow
|
||||
|
||||
```
|
||||
ESP32 Mesh Nodes
|
||||
|
|
||||
| UDP CSI frames (ADR-029 TDM protocol)
|
||||
v
|
||||
+---------------------------+
|
||||
| Rust Sensing Server |
|
||||
| (wifi-densepose-sensing- |
|
||||
| server, ADR-031) |
|
||||
| |
|
||||
| Aggregates ESP32 streams |
|
||||
| Runs RuvSense pipeline |
|
||||
| Exposes WS + REST APIs |
|
||||
+---------------------------+
|
||||
| |
|
||||
| WebSocket | REST
|
||||
| ws://host:3001 | http://host:3000
|
||||
| /ws/sensing | /api/v1/...
|
||||
v v
|
||||
+---------------------------+
|
||||
| Expo Mobile App |
|
||||
| |
|
||||
| ws.service |
|
||||
| -> poseStore |
|
||||
| -> matStore |
|
||||
| |
|
||||
| Screens subscribe to |
|
||||
| stores via Zustand |
|
||||
| selectors |
|
||||
+---------------------------+
|
||||
```
|
||||
|
||||
**Connection lifecycle:**
|
||||
|
||||
1. App boots. `settingsStore` loads persisted server URL from AsyncStorage.
|
||||
2. `ws.service` opens WebSocket to `wsUrl/ws/sensing`.
|
||||
3. On each message, `ws.service` parses the `SensingFrame` JSON and dispatches to `poseStore`.
|
||||
4. If the WebSocket fails, `ws.service` retries with exponential backoff (1s, 2s, 4s, 8s, 16s max).
|
||||
5. After `MAX_RECONNECT_ATTEMPTS` (5) consecutive failures, `ws.service` switches to `simulation.service`, which generates synthetic frames at 10 Hz.
|
||||
6. `poseStore.connectionStatus` transitions: `connected` -> `error` -> `simulated`.
|
||||
7. `ConnectionBanner` component reflects the current status on all screens.
|
||||
8. If the server becomes reachable again, `ws.service` reconnects and resumes live data.
|
||||
|
||||
### 2.5 SensingFrame JSON Schema
|
||||
|
||||
The WebSocket stream delivers JSON frames matching the Rust `SensingFrame` struct:
|
||||
|
||||
```typescript
|
||||
interface SensingFrame {
|
||||
timestamp: number; // Unix epoch ms
|
||||
amplitude: number[]; // Per-subcarrier amplitude (52 or 114 values)
|
||||
phase: number[]; // Per-subcarrier phase (radians)
|
||||
features: {
|
||||
mean_amplitude: number;
|
||||
std_amplitude: number;
|
||||
phase_slope: number;
|
||||
doppler_shift: number;
|
||||
delay_spread: number;
|
||||
};
|
||||
classification: string; // "empty" | "single_person" | "multi_person" | "motion"
|
||||
confidence: number; // 0.0 - 1.0
|
||||
persons: Array<{
|
||||
id: number;
|
||||
keypoints: Array<[number, number, number]>; // 17 COCO keypoints [x, y, confidence]
|
||||
bbox: [number, number, number, number]; // [x, y, width, height]
|
||||
track_id: number;
|
||||
}>;
|
||||
vital_signs?: {
|
||||
breathing_rate_bpm: number;
|
||||
heart_rate_bpm: number;
|
||||
breathing_confidence: number;
|
||||
heart_confidence: number;
|
||||
};
|
||||
rssi?: number;
|
||||
node_id?: number;
|
||||
}
|
||||
```
|
||||
|
||||
### 2.6 Three.js Gaussian Splat Rendering
|
||||
|
||||
The LiveScreen uses a WebView (native) or iframe (web) to render a Three.js Gaussian splat scene. This avoids native OpenGL bindings while reusing the existing splat renderer from the desktop UI.
|
||||
|
||||
**Native path (iOS/Android):**
|
||||
- `GaussianSplatWebView.tsx` renders a `<WebView>` loading a bundled HTML page.
|
||||
- The HTML page initializes a Three.js scene with Gaussian splat shaders.
|
||||
- Communication between React Native and the WebView uses `postMessage` / `onMessage` bridge.
|
||||
- `useGaussianBridge.ts` hook manages the bridge, sending skeleton keypoint updates as JSON.
|
||||
|
||||
**Web path:**
|
||||
- `GaussianSplatWebView.web.tsx` (platform-specific file) renders an `<iframe>` with the same HTML bundle.
|
||||
- Communication uses `window.postMessage` with origin checks.
|
||||
|
||||
### 2.7 Design System
|
||||
|
||||
| Token | Value | Usage |
|
||||
|-------|-------|-------|
|
||||
| `colors.background` | `#0D1117` | Primary background (dark theme) |
|
||||
| `colors.surface` | `#161B22` | Card/panel backgrounds |
|
||||
| `colors.border` | `#30363D` | Borders, dividers |
|
||||
| `colors.accent` | `#32B8C6` | Primary accent, active tab, gauge fill |
|
||||
| `colors.danger` | `#F85149` | Alerts, errors, critical vitals |
|
||||
| `colors.warning` | `#D29922` | Warnings, degraded state |
|
||||
| `colors.success` | `#3FB950` | Connected status, normal vitals |
|
||||
| `colors.text` | `#E6EDF3` | Primary text |
|
||||
| `colors.textSecondary` | `#8B949E` | Secondary/muted text |
|
||||
| `typography.mono` | `Courier New` | Monospace for data values, HUD |
|
||||
| `spacing.xs` | `4` | Tight spacing |
|
||||
| `spacing.sm` | `8` | Small spacing |
|
||||
| `spacing.md` | `16` | Medium spacing |
|
||||
| `spacing.lg` | `24` | Large spacing |
|
||||
| `spacing.xl` | `32` | Extra-large spacing |
|
||||
|
||||
The dark theme is the default and primary design target, optimized for field conditions (low ambient light, glare reduction). A light theme variant is available via the Settings screen.
|
||||
|
||||
### 2.8 ESP32 Integration Model
|
||||
|
||||
The mobile app does not communicate with ESP32 nodes directly. The architecture is:
|
||||
|
||||
```
|
||||
ESP32 Node A ---\
|
||||
ESP32 Node B ----+---> Sensing Server (Raspberry Pi / Laptop) <---> Mobile App
|
||||
ESP32 Node C ---/ (local WiFi) (local WiFi)
|
||||
```
|
||||
|
||||
- **Field deployment**: The sensing server runs on a Raspberry Pi 4 or operator laptop. All devices (ESP32 nodes, server, mobile app) connect to the same local WiFi network or a portable router.
|
||||
- **Server URL**: Configurable in Settings screen. Default: `http://localhost:3000` (server) and `ws://localhost:3001/ws/sensing` (WebSocket). In field use, the operator sets this to the server's LAN IP (e.g., `http://192.168.1.100:3000`).
|
||||
- **No BLE/direct connection**: ESP32 nodes use UDP broadcast for CSI frames (ADR-029). The mobile app has no UDP listener; it consumes the server's processed output.
|
||||
|
||||
---
|
||||
|
||||
## 3. Directory Structure
|
||||
|
||||
```
|
||||
ui/mobile/
|
||||
|-- App.tsx # Root component, ThemeProvider + NavigationContainer
|
||||
|-- app.config.ts # Expo config (SDK 55, app name, icons, splash)
|
||||
|-- app.json # Expo static config
|
||||
|-- babel.config.js # Babel config (expo-router preset)
|
||||
|-- eas.json # EAS Build profiles (dev, preview, production)
|
||||
|-- index.ts # Entry point (registerRootComponent)
|
||||
|-- jest.config.js # Jest config for unit tests
|
||||
|-- jest.setup.ts # Jest setup (mock AsyncStorage, react-native modules)
|
||||
|-- metro.config.js # Metro bundler config
|
||||
|-- package.json # Dependencies and scripts
|
||||
|-- tsconfig.json # TypeScript config (strict mode)
|
||||
|
|
||||
|-- assets/
|
||||
| |-- android-icon-background.png # Android adaptive icon background
|
||||
| |-- android-icon-foreground.png # Android adaptive icon foreground
|
||||
| |-- android-icon-monochrome.png # Android monochrome icon
|
||||
| |-- favicon.png # Web favicon
|
||||
| |-- icon.png # App icon (1024x1024)
|
||||
| |-- splash-icon.png # Splash screen icon
|
||||
|
|
||||
|-- e2e/ # Maestro E2E test specs
|
||||
| |-- live_screen.yaml # LiveScreen: splat renders, HUD shows data
|
||||
| |-- vitals_screen.yaml # VitalsScreen: gauges animate, sparklines update
|
||||
| |-- zones_screen.yaml # ZonesScreen: floor plan renders, legend visible
|
||||
| |-- mat_screen.yaml # MATScreen: survivor count, alerts list
|
||||
| |-- settings_screen.yaml # SettingsScreen: URL input, theme toggle
|
||||
| |-- offline_fallback.yaml # Simulated mode activates on server disconnect
|
||||
|
|
||||
|-- src/
|
||||
| |-- components/ # Shared UI components (12 components)
|
||||
| | |-- ConnectionBanner.tsx # Status banner: Connected/Simulated/Disconnected
|
||||
| | |-- ErrorBoundary.tsx # React error boundary with fallback UI
|
||||
| | |-- GaugeArc.tsx # SVG arc gauge (used by vitals)
|
||||
| | |-- HudOverlay.tsx # Translucent HUD overlay for LiveScreen
|
||||
| | |-- LoadingSpinner.tsx # Animated loading indicator
|
||||
| | |-- ModeBadge.tsx # Badge showing current mode (Live/Sim)
|
||||
| | |-- OccupancyGrid.tsx # Grid overlay for zone occupancy
|
||||
| | |-- SignalBar.tsx # WiFi signal strength bar
|
||||
| | |-- SparklineChart.tsx # Inline sparkline chart (SVG)
|
||||
| | |-- StatusDot.tsx # Colored status dot indicator
|
||||
| | |-- ThemedText.tsx # Text component with theme support
|
||||
| | |-- ThemedView.tsx # View component with theme support
|
||||
| |
|
||||
| |-- constants/ # App-wide constants
|
||||
| | |-- api.ts # REST API endpoint paths, timeouts
|
||||
| | |-- simulation.ts # Simulation tick rate, data ranges
|
||||
| | |-- websocket.ts # WS reconnect config, max attempts
|
||||
| |
|
||||
| |-- hooks/ # Custom React hooks (5 hooks)
|
||||
| | |-- usePoseStream.ts # Subscribe to poseStore, manage WS lifecycle
|
||||
| | |-- useRssiScanner.ts # Platform RSSI scanning with permission handling
|
||||
| | |-- useServerReachability.ts # Periodic health check, reachability state
|
||||
| | |-- useTheme.ts # Theme context consumer
|
||||
| | |-- useWebViewBridge.ts # WebView <-> RN message bridge
|
||||
| |
|
||||
| |-- navigation/ # React Navigation setup
|
||||
| | |-- MainTabs.tsx # Bottom tab navigator (5 tabs)
|
||||
| | |-- RootNavigator.tsx # Root stack (splash -> MainTabs)
|
||||
| | |-- types.ts # Navigation type definitions
|
||||
| |
|
||||
| |-- screens/ # Screen modules (5 screens)
|
||||
| | |-- LiveScreen/
|
||||
| | | |-- index.tsx # LiveScreen container
|
||||
| | | |-- GaussianSplatWebView.tsx # Native: WebView 3D splat
|
||||
| | | |-- GaussianSplatWebView.web.tsx # Web: iframe 3D splat
|
||||
| | | |-- LiveHUD.tsx # Heads-up display overlay
|
||||
| | | |-- useGaussianBridge.ts # Bridge hook for splat WebView
|
||||
| | |
|
||||
| | |-- VitalsScreen/
|
||||
| | | |-- index.tsx # VitalsScreen container
|
||||
| | | |-- BreathingGauge.tsx # Breathing rate arc gauge
|
||||
| | | |-- HeartRateGauge.tsx # Heart rate arc gauge
|
||||
| | | |-- MetricCard.tsx # Metric display card
|
||||
| | |
|
||||
| | |-- ZonesScreen/
|
||||
| | | |-- index.tsx # ZonesScreen container
|
||||
| | | |-- FloorPlanSvg.tsx # SVG floor plan with occupancy overlay
|
||||
| | | |-- useOccupancyGrid.ts # Occupancy grid computation hook
|
||||
| | | |-- ZoneLegend.tsx # Zone color legend
|
||||
| | |
|
||||
| | |-- MATScreen/
|
||||
| | | |-- index.tsx # MATScreen container
|
||||
| | | |-- SurvivorCounter.tsx # Large survivor count display
|
||||
| | | |-- MatWebView.tsx # WebView for MAT zone map
|
||||
| | | |-- AlertList.tsx # Scrollable alert list
|
||||
| | | |-- AlertCard.tsx # Individual alert card
|
||||
| | | |-- useMatBridge.ts # Bridge hook for MAT WebView
|
||||
| | |
|
||||
| | |-- SettingsScreen/
|
||||
| | |-- index.tsx # SettingsScreen container
|
||||
| | |-- ServerUrlInput.tsx # Server URL text input with validation
|
||||
| | |-- ThemePicker.tsx # Dark/light theme toggle
|
||||
| | |-- RssiToggle.tsx # RSSI scanning enable/disable
|
||||
| |
|
||||
| |-- services/ # External communication services (4 services)
|
||||
| | |-- ws.service.ts # WebSocket client with reconnection
|
||||
| | |-- api.service.ts # REST API client (fetch-based)
|
||||
| | |-- rssi.service.ts # Default RSSI service (platform re-export)
|
||||
| | |-- rssi.service.android.ts # Android RSSI via react-native-wifi-reborn
|
||||
| | |-- rssi.service.ios.ts # iOS CoreWLAN stub
|
||||
| | |-- rssi.service.web.ts # Web synthetic RSSI
|
||||
| | |-- simulation.service.ts # Synthetic SensingFrame generator
|
||||
| |
|
||||
| |-- stores/ # Zustand state stores (3 stores)
|
||||
| | |-- poseStore.ts # Connection state, frames, features, persons
|
||||
| | |-- matStore.ts # Survivors, alerts, events, zone map
|
||||
| | |-- settingsStore.ts # Server URL, theme, RSSI toggle (persisted)
|
||||
| |
|
||||
| |-- theme/ # Design system tokens
|
||||
| | |-- index.ts # Theme re-exports
|
||||
| | |-- colors.ts # Color palette (dark + light)
|
||||
| | |-- spacing.ts # Spacing scale
|
||||
| | |-- typography.ts # Font families and sizes
|
||||
| | |-- ThemeContext.tsx # React context for theme
|
||||
| |
|
||||
| |-- types/ # TypeScript type definitions
|
||||
| | |-- api.ts # REST API response types
|
||||
| | |-- html.d.ts # HTML asset module declaration
|
||||
| | |-- mat.ts # MAT domain types (Survivor, Alert, Event)
|
||||
| | |-- navigation.ts # Navigation param list types
|
||||
| | |-- react-native-wifi-reborn.d.ts # Type stubs for wifi-reborn
|
||||
| | |-- sensing.ts # SensingFrame, Person, VitalSigns types
|
||||
| |
|
||||
| |-- utils/ # Utility functions
|
||||
| | |-- colorMap.ts # Value-to-color mapping for gauges
|
||||
| | |-- formatters.ts # Number/date formatting helpers
|
||||
| | |-- ringBuffer.ts # Fixed-size ring buffer for frame history
|
||||
| | |-- urlValidator.ts # Server URL validation
|
||||
| |
|
||||
| |-- __tests__/ # Unit tests (mirroring src/ structure)
|
||||
| |-- test-utils.tsx # Test utilities, render helpers, mocks
|
||||
| |-- components/ # Component unit tests (7 test files)
|
||||
| |-- hooks/ # Hook unit tests (3 test files)
|
||||
| |-- screens/ # Screen unit tests (5 test files)
|
||||
| |-- services/ # Service unit tests (4 test files)
|
||||
| |-- stores/ # Store unit tests (3 test files)
|
||||
| |-- utils/ # Utility unit tests (3 test files)
|
||||
```
|
||||
|
||||
**File count summary:**
|
||||
|
||||
| Category | Files |
|
||||
|----------|-------|
|
||||
| Source (components, screens, services, stores, hooks, utils, types, theme, navigation) | 63 `.ts`/`.tsx` files |
|
||||
| Unit tests | 25 test files |
|
||||
| E2E tests (Maestro) | 6 YAML specs |
|
||||
| Config (babel, metro, jest, tsconfig, eas, app) | 7 config files |
|
||||
| Assets | 6 image files |
|
||||
| **Total** | **107 files** |
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation Plan (File-Level)
|
||||
|
||||
### 4.1 Phase 1: Core Infrastructure
|
||||
|
||||
| File | Purpose | Priority |
|
||||
|------|---------|----------|
|
||||
| `App.tsx` | Root component with ThemeProvider and NavigationContainer | P0 |
|
||||
| `index.ts` | Expo entry point | P0 |
|
||||
| `app.config.ts` | Expo SDK 55 configuration | P0 |
|
||||
| `src/theme/colors.ts` | Dark and light color palettes | P0 |
|
||||
| `src/theme/spacing.ts` | Spacing scale | P0 |
|
||||
| `src/theme/typography.ts` | Font definitions | P0 |
|
||||
| `src/theme/ThemeContext.tsx` | React context provider for theme | P0 |
|
||||
| `src/navigation/MainTabs.tsx` | Bottom tab navigator with 5 tabs | P0 |
|
||||
| `src/navigation/RootNavigator.tsx` | Root stack navigator | P0 |
|
||||
| `src/types/sensing.ts` | SensingFrame, Person, VitalSigns type definitions | P0 |
|
||||
|
||||
### 4.2 Phase 2: State and Services
|
||||
|
||||
| File | Purpose | Priority |
|
||||
|------|---------|----------|
|
||||
| `src/stores/poseStore.ts` | Zustand store for connection state, frames, persons | P0 |
|
||||
| `src/stores/matStore.ts` | Zustand store for MAT survivors, alerts, events | P0 |
|
||||
| `src/stores/settingsStore.ts` | Zustand store with AsyncStorage persistence | P0 |
|
||||
| `src/services/ws.service.ts` | WebSocket client with reconnection and dispatch | P0 |
|
||||
| `src/services/api.service.ts` | REST API client | P1 |
|
||||
| `src/services/simulation.service.ts` | Synthetic SensingFrame generator for fallback | P0 |
|
||||
| `src/services/rssi.service.ts` | Platform RSSI re-export | P1 |
|
||||
| `src/services/rssi.service.android.ts` | Android react-native-wifi-reborn integration | P1 |
|
||||
| `src/services/rssi.service.ios.ts` | iOS CoreWLAN stub | P2 |
|
||||
| `src/services/rssi.service.web.ts` | Web synthetic RSSI | P1 |
|
||||
| `src/utils/ringBuffer.ts` | Fixed-size ring buffer for frame history | P0 |
|
||||
| `src/utils/urlValidator.ts` | Server URL validation | P1 |
|
||||
|
||||
### 4.3 Phase 3: Shared Components
|
||||
|
||||
| File | Purpose | Priority |
|
||||
|------|---------|----------|
|
||||
| `src/components/ConnectionBanner.tsx` | Status banner across all screens | P0 |
|
||||
| `src/components/GaugeArc.tsx` | SVG arc gauge for vitals | P0 |
|
||||
| `src/components/SparklineChart.tsx` | Inline sparkline for history | P0 |
|
||||
| `src/components/OccupancyGrid.tsx` | Grid overlay for zones | P1 |
|
||||
| `src/components/StatusDot.tsx` | Colored status indicator | P1 |
|
||||
| `src/components/SignalBar.tsx` | WiFi signal strength display | P1 |
|
||||
| `src/components/ModeBadge.tsx` | Live/Sim mode badge | P1 |
|
||||
| `src/components/ErrorBoundary.tsx` | React error boundary | P0 |
|
||||
| `src/components/LoadingSpinner.tsx` | Loading state indicator | P1 |
|
||||
| `src/components/ThemedText.tsx` | Themed text component | P0 |
|
||||
| `src/components/ThemedView.tsx` | Themed view component | P0 |
|
||||
| `src/components/HudOverlay.tsx` | Translucent HUD for Live screen | P1 |
|
||||
|
||||
### 4.4 Phase 4: Screens
|
||||
|
||||
| File | Purpose | Priority |
|
||||
|------|---------|----------|
|
||||
| `src/screens/LiveScreen/index.tsx` | Live 3D splat + HUD | P0 |
|
||||
| `src/screens/LiveScreen/GaussianSplatWebView.tsx` | Native WebView for splat | P0 |
|
||||
| `src/screens/LiveScreen/GaussianSplatWebView.web.tsx` | Web iframe for splat | P1 |
|
||||
| `src/screens/LiveScreen/LiveHUD.tsx` | HUD overlay with metrics | P1 |
|
||||
| `src/screens/LiveScreen/useGaussianBridge.ts` | WebView bridge hook | P0 |
|
||||
| `src/screens/VitalsScreen/index.tsx` | Vitals gauges and sparklines | P0 |
|
||||
| `src/screens/VitalsScreen/BreathingGauge.tsx` | Breathing rate gauge | P0 |
|
||||
| `src/screens/VitalsScreen/HeartRateGauge.tsx` | Heart rate gauge | P0 |
|
||||
| `src/screens/VitalsScreen/MetricCard.tsx` | Vitals metric card | P1 |
|
||||
| `src/screens/ZonesScreen/index.tsx` | Floor plan with occupancy | P1 |
|
||||
| `src/screens/ZonesScreen/FloorPlanSvg.tsx` | SVG floor plan renderer | P1 |
|
||||
| `src/screens/ZonesScreen/useOccupancyGrid.ts` | Occupancy computation | P1 |
|
||||
| `src/screens/ZonesScreen/ZoneLegend.tsx` | Zone legend | P2 |
|
||||
| `src/screens/MATScreen/index.tsx` | MAT dashboard | P1 |
|
||||
| `src/screens/MATScreen/SurvivorCounter.tsx` | Survivor count display | P1 |
|
||||
| `src/screens/MATScreen/MatWebView.tsx` | MAT zone map WebView | P1 |
|
||||
| `src/screens/MATScreen/AlertList.tsx` | Alert list | P1 |
|
||||
| `src/screens/MATScreen/AlertCard.tsx` | Alert card | P2 |
|
||||
| `src/screens/MATScreen/useMatBridge.ts` | MAT WebView bridge | P1 |
|
||||
| `src/screens/SettingsScreen/index.tsx` | Settings form | P0 |
|
||||
| `src/screens/SettingsScreen/ServerUrlInput.tsx` | Server URL input | P0 |
|
||||
| `src/screens/SettingsScreen/ThemePicker.tsx` | Theme toggle | P2 |
|
||||
| `src/screens/SettingsScreen/RssiToggle.tsx` | RSSI toggle | P2 |
|
||||
|
||||
### 4.5 Phase 5: Testing
|
||||
|
||||
| File | Purpose | Priority |
|
||||
|------|---------|----------|
|
||||
| `src/__tests__/stores/poseStore.test.ts` | Store state transitions, frame processing | P0 |
|
||||
| `src/__tests__/stores/matStore.test.ts` | MAT store state management | P1 |
|
||||
| `src/__tests__/stores/settingsStore.test.ts` | Persistence, defaults | P1 |
|
||||
| `src/__tests__/services/ws.service.test.ts` | WS connection, reconnection, fallback | P0 |
|
||||
| `src/__tests__/services/simulation.service.test.ts` | Synthetic frame generation | P1 |
|
||||
| `src/__tests__/services/api.service.test.ts` | REST client mocking | P1 |
|
||||
| `src/__tests__/services/rssi.service.test.ts` | Platform RSSI mocking | P2 |
|
||||
| `src/__tests__/components/*.test.tsx` | Component render tests (7 files) | P1 |
|
||||
| `src/__tests__/hooks/*.test.ts` | Hook behavior tests (3 files) | P1 |
|
||||
| `src/__tests__/screens/*.test.tsx` | Screen integration tests (5 files) | P1 |
|
||||
| `src/__tests__/utils/*.test.ts` | Utility function tests (3 files) | P1 |
|
||||
| `e2e/*.yaml` | Maestro E2E specs (6 files) | P2 |
|
||||
|
||||
---
|
||||
|
||||
## 5. Acceptance Criteria
|
||||
|
||||
### 5.1 Build and Platform Support
|
||||
|
||||
| ID | Criterion | Test Method |
|
||||
|----|-----------|-------------|
|
||||
| B-1 | App builds successfully with `npx expo start` for iOS, Android, and Web | CI build matrix: `expo start --ios`, `--android`, `--web` |
|
||||
| B-2 | App runs on iOS Simulator (iPhone 15 Pro, iOS 17+) | Manual verification on Simulator |
|
||||
| B-3 | App runs on Android Emulator (API 34+) | Manual verification on Emulator |
|
||||
| B-4 | App runs in web browser (Chrome 120+, Safari 17+, Firefox 120+) | Manual verification in browsers |
|
||||
| B-5 | TypeScript compiles with zero errors in strict mode | `npx tsc --noEmit` in CI |
|
||||
|
||||
### 5.2 WebSocket and Data Streaming
|
||||
|
||||
| ID | Criterion | Test Method |
|
||||
|----|-----------|-------------|
|
||||
| W-1 | WebSocket connects to sensing server and receives SensingFrame JSON | Integration test: start server, verify `poseStore.connectionStatus === 'connected'` |
|
||||
| W-2 | `poseStore.latestFrame` updates within 100ms of WebSocket message receipt | Unit test: mock WS, measure dispatch latency |
|
||||
| W-3 | WebSocket reconnects with exponential backoff after connection loss | Unit test: simulate WS close, verify retry intervals (1s, 2s, 4s, 8s, 16s) |
|
||||
| W-4 | Automatic fallback to simulated data within 5 seconds of connection failure | Unit test: fail WS 5 times, verify `connectionStatus === 'simulated'` within 5s |
|
||||
| W-5 | App recovers gracefully from sensing server restart (reconnects without crash) | Integration test: kill server, restart, verify reconnection and `connectionStatus === 'connected'` |
|
||||
|
||||
### 5.3 Screen Rendering
|
||||
|
||||
| ID | Criterion | Test Method |
|
||||
|----|-----------|-------------|
|
||||
| S-1 | All 5 screens render correctly with live data from sensing server | Integration test: connect to server, navigate all tabs, verify content |
|
||||
| S-2 | All 5 screens render correctly with simulated data | Unit test: set `connectionStatus = 'simulated'`, verify all screens render |
|
||||
| S-3 | Vital signs gauges animate smoothly (breathing BPM, heart rate BPM) | Visual inspection: gauges update at frame rate without jank |
|
||||
| S-4 | 3D Gaussian splat viewer shows skeleton with 17 COCO keypoints | Integration test: verify WebView loads, bridge sends keypoints, splat renders |
|
||||
| S-5 | Floor plan SVG updates with occupancy data when persons are detected | Unit test: inject 3 persons into poseStore, verify 3 markers on FloorPlanSvg |
|
||||
| S-6 | MAT dashboard shows survivor count, zone map, and alert list | Unit test: inject matStore data, verify SurvivorCounter and AlertList render |
|
||||
| S-7 | Connection banner shows correct status text and color for all 3 states | Unit test: cycle through `connected`/`simulated`/`error`, verify banner text and color |
|
||||
|
||||
### 5.4 Persistence and Settings
|
||||
|
||||
| ID | Criterion | Test Method |
|
||||
|----|-----------|-------------|
|
||||
| P-1 | Settings persist across app restarts (server URL, theme, RSSI toggle) | Integration test: set values, kill app, restart, verify values restored |
|
||||
| P-2 | Default server URL is `http://localhost:3000` when no persisted value exists | Unit test: clear AsyncStorage, verify default |
|
||||
| P-3 | Server URL input validates format before saving | Unit test: submit `not-a-url`, verify rejection; submit `http://192.168.1.1:3000`, verify acceptance |
|
||||
|
||||
### 5.5 Navigation and UX
|
||||
|
||||
| ID | Criterion | Test Method |
|
||||
|----|-----------|-------------|
|
||||
| N-1 | Bottom tab navigation works with correct icons for all 5 tabs | E2E: Maestro navigates all tabs, verifies active state |
|
||||
| N-2 | Dark theme renders correctly on all platforms (background #0D1117, accent #32B8C6) | Visual inspection on iOS, Android, Web |
|
||||
| N-3 | No infinite render loops or memory leaks in stores | Unit test: mount all screens, process 1000 frames, verify no memory growth beyond ring buffer size |
|
||||
| N-4 | ErrorBoundary catches and displays fallback UI for component errors | Unit test: throw in child component, verify fallback renders |
|
||||
|
||||
### 5.6 Platform-Specific Features
|
||||
|
||||
| ID | Criterion | Test Method |
|
||||
|----|-----------|-------------|
|
||||
| R-1 | RSSI scanning works on Android with react-native-wifi-reborn | Manual test on Android device with location permission granted |
|
||||
| R-2 | iOS RSSI service returns empty results without crashing | Unit test: call `scanNetworks()` on iOS, verify empty array returned |
|
||||
| R-3 | Web RSSI service generates synthetic RSSI values | Unit test: call `scanNetworks()` on web, verify synthetic data returned |
|
||||
|
||||
### 5.7 Testing
|
||||
|
||||
| ID | Criterion | Test Method |
|
||||
|----|-----------|-------------|
|
||||
| T-1 | All unit tests pass (`npm test` exits 0) | CI: `cd ui/mobile && npm test` |
|
||||
| T-2 | E2E Maestro tests pass for all 5 screens | CI: `maestro test e2e/` |
|
||||
| T-3 | E2E offline fallback test passes (simulated mode activates on disconnect) | CI: `maestro test e2e/offline_fallback.yaml` |
|
||||
| T-4 | No TypeScript type errors | CI: `npx tsc --noEmit` |
|
||||
|
||||
---
|
||||
|
||||
## 6. Consequences
|
||||
|
||||
### 6.1 Positive
|
||||
|
||||
- **Single codebase for three platforms**: Expo SDK 55 with React Native 0.83 builds iOS, Android, and Web from the same TypeScript source, reducing development and maintenance cost by approximately 60% compared to separate native apps.
|
||||
- **Instant field deployment**: Operators can install the app via Expo Go (development) or EAS Build (production) and connect to a local sensing server within minutes. No server-side mobile infrastructure required.
|
||||
- **Sub-100ms display latency**: WebSocket streaming from the Rust sensing server to the mobile app introduces less than 100ms additional latency beyond the CSI processing pipeline, providing near-real-time visualization.
|
||||
- **Offline-capable demos**: The simulation service generates realistic synthetic SensingFrame data, enabling demonstrations to stakeholders and testing without ESP32 hardware or a running sensing server.
|
||||
- **Operator-friendly UX**: Five purpose-built screens cover the primary use cases (live view, vitals, zones, MAT, settings) with a bottom-tab navigation pattern familiar to mobile users.
|
||||
- **Testable architecture**: Zustand stores with selector-based subscriptions, service-layer abstraction, and Maestro E2E specs provide a comprehensive testing strategy from unit to integration to end-to-end.
|
||||
- **Reuses existing infrastructure**: The app consumes the same WebSocket and REST APIs as the desktop UI, requiring no backend changes. The Three.js splat renderer is reused via WebView.
|
||||
|
||||
### 6.2 Negative
|
||||
|
||||
- **WebView-based 3D rendering has lower performance than native OpenGL**: The Gaussian splat viewer runs inside a WebView (native) or iframe (web), adding a JavaScript-to-native bridge hop and limiting frame rate to approximately 30 FPS on mid-range devices. Native OpenGL or Metal/Vulkan rendering would achieve 60 FPS but requires platform-specific code.
|
||||
- **react-native-wifi-reborn requires native module linking for Android RSSI**: This breaks the pure Expo managed workflow for Android builds. EAS Build with a custom development client is required. iOS RSSI scanning is not possible at all due to Apple restrictions.
|
||||
- **Expo managed workflow limits some native module access**: Certain native APIs (background location, Bluetooth LE, raw WiFi frames) are not available without ejecting to a bare workflow. This constrains future features like Bluetooth mesh fallback.
|
||||
- **WebView bridge latency**: Communication between React Native and the Three.js WebView via `postMessage` adds 5-15ms per message, reducing effective update rate for the 3D splat view. This is acceptable for 10-20 Hz sensing frame rates but would become a bottleneck at higher rates.
|
||||
- **AsyncStorage has no encryption**: Settings (including server URL) are stored in plaintext AsyncStorage. For security-sensitive deployments, expo-secure-store should replace AsyncStorage for credential storage.
|
||||
|
||||
### 6.3 Risks
|
||||
|
||||
| Risk | Probability | Impact | Mitigation |
|
||||
|------|-------------|--------|------------|
|
||||
| Expo SDK 55 breaking changes in future updates | Medium | Build failures, API deprecations | Pin SDK version in `app.config.ts`; test upgrades in preview branch |
|
||||
| WebView memory pressure on low-end Android devices | Medium | OOM crash during Three.js splat rendering | Implement splat LOD (level of detail) fallback; monitor WebView memory via `onContentProcessDidTerminate` |
|
||||
| react-native-wifi-reborn unmaintained or incompatible with RN 0.83 | Low | Android RSSI scanning broken | Fork and patch if needed; RSSI scanning is a secondary feature |
|
||||
| Sensing server WebSocket protocol changes | Medium | Frame parsing errors, broken display | Version the WebSocket protocol; add `protocol_version` field to SensingFrame |
|
||||
| Battery drain from continuous WebSocket connection on mobile | Medium | Poor user experience in extended field use | Implement configurable update rate throttling in settings; pause WS when app is backgrounded |
|
||||
| Three.js Gaussian splat HTML bundle size exceeds WebView limits | Low | Slow initial load, white screen | Lazy-load splat bundle; show placeholder skeleton during load; cache bundle in AsyncStorage |
|
||||
|
||||
---
|
||||
|
||||
## 7. Future Work
|
||||
|
||||
### 7.1 Offline Model Inference
|
||||
|
||||
Run a quantized ONNX pose estimation model directly on the mobile device using `onnxruntime-react-native`. This would allow the app to process raw CSI data (received via a local UDP relay or Bluetooth) without a sensing server, enabling fully disconnected field operation.
|
||||
|
||||
**Prerequisites:** Export the trained WiFi-DensePose model (ADR-023) to ONNX format; quantize to INT8 for mobile; benchmark inference latency on iPhone 15 and Pixel 8.
|
||||
|
||||
### 7.2 Push Notifications for MAT Alerts
|
||||
|
||||
Integrate Firebase Cloud Messaging (Android) and APNs (iOS) to deliver push notifications when the sensing server detects new survivors or critical vital sign alerts. This allows operators to be alerted even when the app is backgrounded.
|
||||
|
||||
**Prerequisites:** Add a push notification endpoint to the Rust sensing server; implement Expo Notifications integration in the mobile app.
|
||||
|
||||
### 7.3 Apple Watch Companion
|
||||
|
||||
Build a watchOS companion app using Expo's experimental watch support or a native SwiftUI module. The watch would display a minimal vitals view (breathing rate, heart rate, alert count) on the operator's wrist, with haptic feedback for critical MAT alerts.
|
||||
|
||||
**Prerequisites:** Evaluate Expo watch support maturity; define minimal watch screen set; implement WatchConnectivity bridge.
|
||||
|
||||
### 7.4 Bluetooth Mesh Fallback
|
||||
|
||||
When WiFi is unavailable (collapsed building, power outage), use Bluetooth Low Energy (BLE) mesh to relay aggregated CSI summaries from ESP32 nodes to the mobile device. This requires ejecting from Expo managed workflow to bare workflow for BLE native module access.
|
||||
|
||||
**Prerequisites:** Implement BLE GATT service on ESP32 firmware (ADR-018); integrate `react-native-ble-plx` in bare Expo workflow; define BLE CSI summary protocol (compressed, lower bandwidth than WiFi).
|
||||
|
||||
### 7.5 Multi-Server Dashboard
|
||||
|
||||
Support connecting to multiple sensing servers simultaneously (e.g., one per floor or building wing). The app would aggregate data from all servers into a unified zone map and MAT dashboard with per-server status indicators.
|
||||
|
||||
**Prerequisites:** Extend `settingsStore` to support server list; modify `ws.service` to manage multiple WebSocket connections; merge `poseStore` frames from multiple sources with server-id tags.
|
||||
|
||||
---
|
||||
|
||||
## 8. Related ADRs
|
||||
|
||||
| ADR | Relationship |
|
||||
|-----|-------------|
|
||||
| ADR-019 (Sensing-Only UI Mode) | **Extended**: The mobile app is the field-optimized evolution of the sensing-only UI mode, adding native mobile capabilities (push, RSSI, offline) |
|
||||
| ADR-021 (Vital Sign Detection) | **Consumed**: VitalsScreen displays breathing_rate_bpm and heart_rate_bpm extracted by the ADR-021 pipeline |
|
||||
| ADR-026 (Survivor Track Lifecycle) | **Consumed**: MATScreen displays survivor tracks with lifecycle states (detected, confirmed, rescued, lost) from ADR-026 |
|
||||
| ADR-029 (RuvSense Multistatic) | **Consumed**: The sensing server aggregates ESP32 TDM frames (ADR-029) and streams processed results to the mobile app |
|
||||
| ADR-031 (RuView Sensing-First RF) | **Consumed**: The WebSocket and REST APIs exposed by `wifi-densepose-sensing-server` (ADR-031) are the mobile app's data source |
|
||||
| ADR-032 (Mesh Security) | **Consumed**: Authenticated CSI frames (ADR-032) ensure the mobile app displays trustworthy data, not spoofed sensor readings |
|
||||
|
||||
---
|
||||
|
||||
## 9. References
|
||||
|
||||
1. Expo SDK 55 Documentation. https://docs.expo.dev/
|
||||
2. React Native 0.83 Release Notes. https://reactnative.dev/
|
||||
3. Zustand v5. https://github.com/pmndrs/zustand
|
||||
4. React Navigation v7. https://reactnavigation.org/
|
||||
5. Maestro Mobile Testing Framework. https://maestro.mobile.dev/
|
||||
6. react-native-wifi-reborn. https://github.com/JuanSeBestworker/react-native-wifi-reborn
|
||||
7. Three.js Gaussian Splatting. https://github.com/mrdoob/three.js
|
||||
8. AsyncStorage. https://react-native-async-storage.github.io/async-storage/
|
||||
9. Geng, J. et al. (2023). "DensePose From WiFi." arXiv:2301.00250.
|
||||
10. ADR-019 through ADR-032 (internal).
|
||||
@@ -0,0 +1,98 @@
|
||||
# ADR-035: Live Sensing UI Accuracy & Data Source Transparency
|
||||
|
||||
## Status
|
||||
Accepted
|
||||
|
||||
## Date
|
||||
2026-03-02
|
||||
|
||||
## Context
|
||||
|
||||
Issue #86 reported that the live demo shows a static/barely-animated stick figure and the sensing page displays inaccurate data, despite a working ESP32 sending real CSI frames. Investigation revealed three root causes:
|
||||
|
||||
1. **Docker defaults to `--source simulated`** — even with a real ESP32 connected, the server generates synthetic sine-wave data instead of reading UDP frames.
|
||||
2. **Live demo pose is analytically computed** — `derive_pose_from_sensing()` generates keypoints using `sin(tick)` math unrelated to actual signal content. No trained `.rvf` model is loaded by default.
|
||||
3. **Sensing feature extraction is oversimplified** — the server uses single-frame thresholds for motion detection and has no temporal analysis (breathing FFT, sliding window variance, frame history).
|
||||
4. **No data source indicator** — users cannot tell whether they are seeing real or simulated data.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Docker: Auto-detect data source
|
||||
- Default `CSI_SOURCE` changed from `simulated` to `auto`.
|
||||
- `auto` probes UDP port 5005 for an ESP32; falls back to simulation if none found.
|
||||
- Users override via `CSI_SOURCE=esp32 docker-compose up`.
|
||||
|
||||
### 2. Signal-responsive pose derivation
|
||||
- `derive_pose_from_sensing()` now reads actual sensing features:
|
||||
- `motion_band_power` drives limb splay and walking gait detection (> 0.55).
|
||||
- `breathing_band_power` drives torso expansion/contraction phased to breathing rate.
|
||||
- `variance` seeds per-joint noise so the skeleton moves independently.
|
||||
- `dominant_freq_hz` drives lateral torso lean.
|
||||
- `change_points` add burst jitter to extremity keypoints.
|
||||
- Tick rate reduced from 500ms to 100ms (2 fps → 10 fps).
|
||||
- `pose_source` field (`signal_derived` | `model_inference`) added to every WebSocket frame.
|
||||
|
||||
### 3. Temporal feature extraction
|
||||
- 100-frame circular buffer (`VecDeque`) added to `AppStateInner`.
|
||||
- Per-subcarrier temporal variance via Welford-style accumulation.
|
||||
- Breathing rate estimation via 9-candidate Goertzel filter bank (0.1–0.5 Hz) with 3x SNR gate.
|
||||
- Frame-to-frame L2 motion score replaces single-frame amplitude thresholds.
|
||||
- Signal quality metric: SNR-based (RSSI − noise floor) blended with temporal stability.
|
||||
- Signal field driven by subcarrier variance spatial mapping instead of fixed animation.
|
||||
|
||||
### 4. Data source transparency in UI
|
||||
- **Sensing tab**: Banner showing "LIVE - ESP32" (green), "RECONNECTING..." (yellow), or "SIMULATED DATA" (red).
|
||||
- **Live Demo tab**: "Estimation Mode" badge showing "Signal-Derived" (green) or "Model Inference" (blue).
|
||||
- **Setup Guide** panel explaining what each ESP32 count provides (1x: presence/breathing, 3x: localization, 4x+: full pose with trained model).
|
||||
- Simulation fallback delayed from immediate to 5 failed reconnect attempts (~30s).
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Users with real ESP32 hardware get real data by default (auto-detect).
|
||||
- Simulated data is clearly labeled — no more confusion about data authenticity.
|
||||
- Pose skeleton visually responds to actual signal changes (motion, breathing, variance).
|
||||
- Feature extraction produces physiologically meaningful metrics (breathing rate via Goertzel, temporal motion detection).
|
||||
- Setup guide manages expectations about what each hardware configuration provides.
|
||||
|
||||
### Negative
|
||||
- Signal-derived pose is still an approximation, not neural network inference. Per-limb tracking requires a trained `.rvf` model + 4+ ESP32 nodes.
|
||||
- Goertzel filter bank adds ~O(9×N) computation per frame (negligible at 100 frames).
|
||||
- Users with only 1 ESP32 may still be disappointed that arm tracking doesn't work — but the UI now explains why.
|
||||
|
||||
### 5. Dark mode consistency
|
||||
- Live Demo tab converted from light theme to dark mode matching the rest of the UI.
|
||||
- All sidebar panels, badges, buttons, dropdowns use dark backgrounds with muted text.
|
||||
|
||||
### 6. Render mode implementations
|
||||
All four render modes in the pose visualization dropdown now produce distinct visual output:
|
||||
|
||||
| Mode | Rendering |
|
||||
|------|-----------|
|
||||
| **Skeleton** | Green lines connecting joints + red keypoint dots |
|
||||
| **Keypoints** | Large colored dots with glow and labels, no connecting lines |
|
||||
| **Heatmap** | Gaussian radial blobs per keypoint (hue per person), faint skeleton overlay at 25% opacity |
|
||||
| **Dense** | Body region segmentation with colored filled polygons — head (red), torso (blue), left arm (green), right arm (orange), left leg (purple), right leg (yellow) |
|
||||
|
||||
Previously heatmap and dense were stubs that fell back to skeleton mode.
|
||||
|
||||
### 7. pose_source passthrough fix
|
||||
The `pose_source` field from the WebSocket message was being dropped in `convertZoneDataToRestFormat()` in `pose.service.js`. Now passed through so the Estimation Mode badge displays correctly.
|
||||
|
||||
## Files Changed
|
||||
- `docker/Dockerfile.rust` — `CSI_SOURCE=auto` env, shell entrypoint for variable expansion
|
||||
- `docker/docker-compose.yml` — `CSI_SOURCE=${CSI_SOURCE:-auto}`, shell command string
|
||||
- `wifi-densepose-sensing-server/src/main.rs` — frame history buffer, Goertzel breathing estimation, temporal motion score, signal-driven pose derivation, pose_source field, 100ms tick default
|
||||
- `ui/services/sensing.service.js` — `dataSource` state, delayed simulation fallback, `_simulated` marker
|
||||
- `ui/services/pose.service.js` — `pose_source` passthrough in data conversion
|
||||
- `ui/components/SensingTab.js` — data source banner, "About This Data" card
|
||||
- `ui/components/LiveDemoTab.js` — estimation mode badge, setup guide panel, dark mode theme
|
||||
- `ui/utils/pose-renderer.js` — heatmap (Gaussian blobs) and dense (body region segmentation) render modes
|
||||
- `ui/style.css` — banner, badge, guide panel, and about-text styles
|
||||
- `README.md` — live pose detection screenshot
|
||||
- `assets/screen.png` — screenshot asset
|
||||
|
||||
## References
|
||||
- Issue: https://github.com/ruvnet/wifi-densepose/issues/86
|
||||
- ADR-029: RuvSense multistatic sensing mode (proposed — full pipeline integration)
|
||||
- ADR-014: SOTA signal processing
|
||||
@@ -71,8 +71,8 @@ struct Args {
|
||||
#[arg(long, default_value = "../../ui")]
|
||||
ui_path: PathBuf,
|
||||
|
||||
/// Tick interval in milliseconds
|
||||
#[arg(long, default_value = "500")]
|
||||
/// Tick interval in milliseconds (default 100 ms = 10 fps for smooth pose animation)
|
||||
#[arg(long, default_value = "100")]
|
||||
tick_ms: u64,
|
||||
|
||||
/// Data source: auto, wifi, esp32, simulate
|
||||
@@ -266,6 +266,10 @@ struct BoundingBox {
|
||||
struct AppStateInner {
|
||||
latest_update: Option<SensingUpdate>,
|
||||
rssi_history: VecDeque<f64>,
|
||||
/// Circular buffer of recent CSI amplitude vectors for temporal analysis.
|
||||
/// Each entry is the full subcarrier amplitude vector for one frame.
|
||||
/// Capacity: FRAME_HISTORY_CAPACITY frames.
|
||||
frame_history: VecDeque<Vec<f64>>,
|
||||
tick: u64,
|
||||
source: String,
|
||||
tx: broadcast::Sender<String>,
|
||||
@@ -287,6 +291,10 @@ struct AppStateInner {
|
||||
model_loaded: bool,
|
||||
}
|
||||
|
||||
/// Number of frames retained in `frame_history` for temporal analysis.
|
||||
/// At 500 ms ticks this covers ~50 seconds; at 100 ms ticks ~10 seconds.
|
||||
const FRAME_HISTORY_CAPACITY: usize = 100;
|
||||
|
||||
type SharedState = Arc<RwLock<AppStateInner>>;
|
||||
|
||||
// ── ESP32 UDP frame parser ───────────────────────────────────────────────────
|
||||
@@ -343,43 +351,96 @@ fn parse_esp32_frame(buf: &[u8]) -> Option<Esp32Frame> {
|
||||
|
||||
// ── Signal field generation ──────────────────────────────────────────────────
|
||||
|
||||
/// Generate a signal field that reflects where motion and signal changes are occurring.
|
||||
///
|
||||
/// Instead of a fixed-animation circle, this function uses the actual sensing data:
|
||||
/// - `subcarrier_variances`: per-subcarrier variance computed from the frame history.
|
||||
/// High-variance subcarriers indicate spatial directions where the signal is disrupted.
|
||||
/// - `motion_score`: overall motion intensity [0, 1].
|
||||
/// - `breathing_rate_hz`: estimated breathing rate in Hz; if > 0, adds a breathing ring.
|
||||
/// - `signal_quality`: overall quality metric [0, 1] modulates field brightness.
|
||||
///
|
||||
/// The field grid is 20×20 cells representing a top-down view of the room.
|
||||
/// Hotspots are derived from the subcarrier index (treated as an angular bin) so that
|
||||
/// subcarriers with the highest variance produce peaks at the corresponding directions.
|
||||
fn generate_signal_field(
|
||||
_mean_rssi: f64,
|
||||
variance: f64,
|
||||
motion_score: f64,
|
||||
tick: u64,
|
||||
breathing_rate_hz: f64,
|
||||
signal_quality: f64,
|
||||
subcarrier_variances: &[f64],
|
||||
) -> SignalField {
|
||||
let grid = 20;
|
||||
let grid = 20usize;
|
||||
let mut values = vec![0.0f64; grid * grid];
|
||||
let center = grid as f64 / 2.0;
|
||||
let tick_f = tick as f64;
|
||||
let center = (grid as f64 - 1.0) / 2.0;
|
||||
|
||||
// Normalise subcarrier variances to [0, 1].
|
||||
let max_var = subcarrier_variances.iter().cloned().fold(0.0f64, f64::max);
|
||||
let norm_factor = if max_var > 1e-9 { max_var } else { 1.0 };
|
||||
|
||||
// For each cell, accumulate contributions from all subcarriers.
|
||||
// Each subcarrier k is assigned an angular direction proportional to its index
|
||||
// so that different subcarriers illuminate different regions of the room.
|
||||
let n_sub = subcarrier_variances.len().max(1);
|
||||
for (k, &var) in subcarrier_variances.iter().enumerate() {
|
||||
let weight = (var / norm_factor) * motion_score;
|
||||
if weight < 1e-6 {
|
||||
continue;
|
||||
}
|
||||
// Map subcarrier index to an angle across the full 2π sweep.
|
||||
let angle = (k as f64 / n_sub as f64) * 2.0 * std::f64::consts::PI;
|
||||
// Place the hotspot at a distance proportional to the weight, capped at 40% of
|
||||
// the grid radius so it stays within the room model.
|
||||
let radius = center * 0.8 * weight.sqrt();
|
||||
let hx = center + radius * angle.cos();
|
||||
let hz = center + radius * angle.sin();
|
||||
|
||||
for z in 0..grid {
|
||||
for x in 0..grid {
|
||||
let dx = x as f64 - hx;
|
||||
let dz = z as f64 - hz;
|
||||
let dist2 = dx * dx + dz * dz;
|
||||
// Gaussian blob centred on the hotspot; spread scales with weight.
|
||||
let spread = (0.5 + weight * 2.0).max(0.5);
|
||||
values[z * grid + x] += weight * (-dist2 / (2.0 * spread * spread)).exp();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Base radial attenuation from the router assumed at grid centre.
|
||||
for z in 0..grid {
|
||||
for x in 0..grid {
|
||||
let dx = x as f64 - center;
|
||||
let dz = z as f64 - center;
|
||||
let dist = (dx * dx + dz * dz).sqrt();
|
||||
|
||||
// Base radial attenuation from router at center
|
||||
let base = (-dist * 0.15).exp();
|
||||
|
||||
// Body disruption blob
|
||||
let body_x = center + 3.0 * (tick_f * 0.02).sin();
|
||||
let body_z = center + 2.0 * (tick_f * 0.015).cos();
|
||||
let body_dist = ((x as f64 - body_x).powi(2) + (z as f64 - body_z).powi(2)).sqrt();
|
||||
let disruption = motion_score * 0.6 * (-body_dist * 0.4).exp();
|
||||
|
||||
// Breathing ring modulation
|
||||
let breath_ring = if variance > 1.0 {
|
||||
0.1 * (tick_f * 0.3).sin() * (-((dist - 5.0).powi(2)) * 0.1).exp()
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
|
||||
values[z * grid + x] = (base + disruption + breath_ring).clamp(0.0, 1.0);
|
||||
let base = signal_quality * (-dist * 0.12).exp();
|
||||
values[z * grid + x] += base * 0.3;
|
||||
}
|
||||
}
|
||||
|
||||
// Breathing ring: if a breathing rate was estimated add a faint annular highlight
|
||||
// at a radius corresponding to typical chest-wall displacement range.
|
||||
if breathing_rate_hz > 0.05 {
|
||||
let ring_r = center * 0.55;
|
||||
let ring_width = 1.8f64;
|
||||
for z in 0..grid {
|
||||
for x in 0..grid {
|
||||
let dx = x as f64 - center;
|
||||
let dz = z as f64 - center;
|
||||
let dist = (dx * dx + dz * dz).sqrt();
|
||||
let ring_val = 0.08 * (-(dist - ring_r).powi(2) / (2.0 * ring_width * ring_width)).exp();
|
||||
values[z * grid + x] += ring_val;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Clamp and normalise to [0, 1].
|
||||
let field_max = values.iter().cloned().fold(0.0f64, f64::max);
|
||||
let scale = if field_max > 1e-9 { 1.0 / field_max } else { 1.0 };
|
||||
for v in &mut values {
|
||||
*v = (*v * scale).clamp(0.0, 1.0);
|
||||
}
|
||||
|
||||
SignalField {
|
||||
grid_size: [grid, 1, grid],
|
||||
values,
|
||||
@@ -388,21 +449,163 @@ fn generate_signal_field(
|
||||
|
||||
// ── Feature extraction from ESP32 frame ──────────────────────────────────────
|
||||
|
||||
fn extract_features_from_frame(frame: &Esp32Frame) -> (FeatureInfo, ClassificationInfo) {
|
||||
let n = frame.amplitudes.len().max(1) as f64;
|
||||
/// Estimate breathing rate in Hz from the amplitude time series stored in `frame_history`.
|
||||
///
|
||||
/// Approach:
|
||||
/// 1. Build a scalar time series by computing the mean amplitude of each historical frame.
|
||||
/// 2. Run a peak-detection pass: count rising-edge zero-crossings of the de-meaned signal.
|
||||
/// 3. Convert the crossing rate to Hz, clipped to the physiological range 0.1–0.5 Hz
|
||||
/// (12–30 breaths/min).
|
||||
///
|
||||
/// For accuracy the function additionally applies a simple 3-tap Goertzel-style power
|
||||
/// estimate at evenly-spaced candidate frequencies in the breathing band and returns
|
||||
/// the candidate with the highest energy.
|
||||
fn estimate_breathing_rate_hz(frame_history: &VecDeque<Vec<f64>>, sample_rate_hz: f64) -> f64 {
|
||||
let n = frame_history.len();
|
||||
if n < 6 {
|
||||
return 0.0;
|
||||
}
|
||||
|
||||
// Build scalar time series: mean amplitude per frame.
|
||||
let series: Vec<f64> = frame_history.iter()
|
||||
.map(|amps| {
|
||||
if amps.is_empty() { 0.0 } else { amps.iter().sum::<f64>() / amps.len() as f64 }
|
||||
})
|
||||
.collect();
|
||||
|
||||
let mean_s = series.iter().sum::<f64>() / n as f64;
|
||||
// De-mean.
|
||||
let detrended: Vec<f64> = series.iter().map(|x| x - mean_s).collect();
|
||||
|
||||
// Goertzel power at candidate frequencies in the breathing band [0.1, 0.5] Hz.
|
||||
// We evaluate 9 candidate frequencies uniformly spaced in that band.
|
||||
let n_candidates = 9usize;
|
||||
let f_low = 0.1f64;
|
||||
let f_high = 0.5f64;
|
||||
let mut best_freq = 0.0f64;
|
||||
let mut best_power = 0.0f64;
|
||||
|
||||
for i in 0..n_candidates {
|
||||
let freq = f_low + (f_high - f_low) * i as f64 / (n_candidates - 1).max(1) as f64;
|
||||
let omega = 2.0 * std::f64::consts::PI * freq / sample_rate_hz;
|
||||
let coeff = 2.0 * omega.cos();
|
||||
let mut s_prev2 = 0.0f64;
|
||||
let mut s_prev1 = 0.0f64;
|
||||
for &x in &detrended {
|
||||
let s = x + coeff * s_prev1 - s_prev2;
|
||||
s_prev2 = s_prev1;
|
||||
s_prev1 = s;
|
||||
}
|
||||
// Goertzel magnitude squared.
|
||||
let power = s_prev2 * s_prev2 + s_prev1 * s_prev1 - coeff * s_prev1 * s_prev2;
|
||||
if power > best_power {
|
||||
best_power = power;
|
||||
best_freq = freq;
|
||||
}
|
||||
}
|
||||
|
||||
// Only report a breathing rate if the Goertzel energy is meaningfully above noise.
|
||||
// Threshold: power must exceed 10× the average power across all candidates.
|
||||
let avg_power = {
|
||||
let mut total = 0.0f64;
|
||||
for i in 0..n_candidates {
|
||||
let freq = f_low + (f_high - f_low) * i as f64 / (n_candidates - 1).max(1) as f64;
|
||||
let omega = 2.0 * std::f64::consts::PI * freq / sample_rate_hz;
|
||||
let coeff = 2.0 * omega.cos();
|
||||
let mut s_prev2 = 0.0f64;
|
||||
let mut s_prev1 = 0.0f64;
|
||||
for &x in &detrended {
|
||||
let s = x + coeff * s_prev1 - s_prev2;
|
||||
s_prev2 = s_prev1;
|
||||
s_prev1 = s;
|
||||
}
|
||||
total += s_prev2 * s_prev2 + s_prev1 * s_prev1 - coeff * s_prev1 * s_prev2;
|
||||
}
|
||||
total / n_candidates as f64
|
||||
};
|
||||
|
||||
if best_power > avg_power * 3.0 {
|
||||
best_freq.clamp(f_low, f_high)
|
||||
} else {
|
||||
0.0
|
||||
}
|
||||
}
|
||||
|
||||
/// Compute per-subcarrier variance across the sliding window of `frame_history`.
|
||||
///
|
||||
/// For each subcarrier index `k`, returns `Var[A_k]` over all stored frames.
|
||||
/// This captures spatial signal variation; subcarriers whose amplitude fluctuates
|
||||
/// heavily across time correspond to directions with motion.
|
||||
fn compute_subcarrier_variances(frame_history: &VecDeque<Vec<f64>>, n_sub: usize) -> Vec<f64> {
|
||||
if frame_history.is_empty() || n_sub == 0 {
|
||||
return vec![0.0; n_sub];
|
||||
}
|
||||
|
||||
let n_frames = frame_history.len() as f64;
|
||||
let mut means = vec![0.0f64; n_sub];
|
||||
let mut sq_means = vec![0.0f64; n_sub];
|
||||
|
||||
for frame in frame_history.iter() {
|
||||
for k in 0..n_sub {
|
||||
let a = if k < frame.len() { frame[k] } else { 0.0 };
|
||||
means[k] += a;
|
||||
sq_means[k] += a * a;
|
||||
}
|
||||
}
|
||||
|
||||
(0..n_sub)
|
||||
.map(|k| {
|
||||
let mean = means[k] / n_frames;
|
||||
let sq_mean = sq_means[k] / n_frames;
|
||||
(sq_mean - mean * mean).max(0.0)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Extract features from the current ESP32 frame, enhanced with temporal context from
|
||||
/// `frame_history`.
|
||||
///
|
||||
/// Improvements over the previous single-frame approach:
|
||||
///
|
||||
/// - **Variance**: computed as the mean of per-subcarrier temporal variance across the
|
||||
/// sliding window, not just the intra-frame spatial variance.
|
||||
/// - **Motion detection**: uses frame-to-frame temporal difference (mean L2 change
|
||||
/// between the current frame and the previous frame) normalised by signal amplitude,
|
||||
/// so that actual changes are detected rather than just a threshold on the current frame.
|
||||
/// - **Breathing rate**: estimated via Goertzel filter bank on the 0.1–0.5 Hz band of
|
||||
/// the amplitude time series.
|
||||
/// - **Signal quality**: based on SNR estimate (RSSI – noise floor) and subcarrier
|
||||
/// variance stability.
|
||||
fn extract_features_from_frame(
|
||||
frame: &Esp32Frame,
|
||||
frame_history: &VecDeque<Vec<f64>>,
|
||||
sample_rate_hz: f64,
|
||||
) -> (FeatureInfo, ClassificationInfo, f64, Vec<f64>) {
|
||||
let n_sub = frame.amplitudes.len().max(1);
|
||||
let n = n_sub as f64;
|
||||
let mean_amp: f64 = frame.amplitudes.iter().sum::<f64>() / n;
|
||||
let mean_rssi = frame.rssi as f64;
|
||||
|
||||
let variance: f64 = frame.amplitudes.iter()
|
||||
// ── Intra-frame subcarrier variance (spatial spread across subcarriers) ──
|
||||
let intra_variance: f64 = frame.amplitudes.iter()
|
||||
.map(|a| (a - mean_amp).powi(2))
|
||||
.sum::<f64>() / n;
|
||||
|
||||
// Simple spectral analysis on amplitude vector
|
||||
let spectral_power: f64 = frame.amplitudes.iter()
|
||||
.map(|a| a * a)
|
||||
.sum::<f64>() / n;
|
||||
// ── Temporal (sliding-window) per-subcarrier variance ──
|
||||
let sub_variances = compute_subcarrier_variances(frame_history, n_sub);
|
||||
let temporal_variance: f64 = if sub_variances.is_empty() {
|
||||
intra_variance
|
||||
} else {
|
||||
sub_variances.iter().sum::<f64>() / sub_variances.len() as f64
|
||||
};
|
||||
|
||||
// Motion band: high-frequency subcarrier variance
|
||||
// Use the larger of intra-frame and temporal variance as the reported variance.
|
||||
let variance = intra_variance.max(temporal_variance);
|
||||
|
||||
// ── Spectral power ──
|
||||
let spectral_power: f64 = frame.amplitudes.iter().map(|a| a * a).sum::<f64>() / n;
|
||||
|
||||
// ── Motion band power (upper half of subcarriers, high spatial frequency) ──
|
||||
let half = frame.amplitudes.len() / 2;
|
||||
let motion_band_power = if half > 0 {
|
||||
frame.amplitudes[half..].iter()
|
||||
@@ -412,7 +615,7 @@ fn extract_features_from_frame(frame: &Esp32Frame) -> (FeatureInfo, Classificati
|
||||
0.0
|
||||
};
|
||||
|
||||
// Breathing band: low-frequency variance
|
||||
// ── Breathing band power (lower half of subcarriers, low spatial frequency) ──
|
||||
let breathing_band_power = if half > 0 {
|
||||
frame.amplitudes[..half].iter()
|
||||
.map(|a| (a - mean_amp).powi(2))
|
||||
@@ -421,7 +624,7 @@ fn extract_features_from_frame(frame: &Esp32Frame) -> (FeatureInfo, Classificati
|
||||
0.0
|
||||
};
|
||||
|
||||
// Dominant frequency estimate (peak subcarrier index → Hz)
|
||||
// ── Dominant frequency via peak subcarrier index ──
|
||||
let peak_idx = frame.amplitudes.iter()
|
||||
.enumerate()
|
||||
.max_by(|a, b| a.1.partial_cmp(b.1).unwrap_or(std::cmp::Ordering::Equal))
|
||||
@@ -429,12 +632,47 @@ fn extract_features_from_frame(frame: &Esp32Frame) -> (FeatureInfo, Classificati
|
||||
.unwrap_or(0);
|
||||
let dominant_freq_hz = peak_idx as f64 * 0.05;
|
||||
|
||||
// Change point detection (simple threshold crossing count)
|
||||
// ── Change point detection (threshold-crossing count in current frame) ──
|
||||
let threshold = mean_amp * 1.2;
|
||||
let change_points = frame.amplitudes.windows(2)
|
||||
.filter(|w| (w[0] < threshold) != (w[1] < threshold))
|
||||
.count();
|
||||
|
||||
// ── Motion score: sliding-window temporal difference ──
|
||||
// Compare current frame against the most recent historical frame.
|
||||
// The difference is normalised by the mean amplitude to be scale-invariant.
|
||||
let temporal_motion_score = if let Some(prev_frame) = frame_history.back() {
|
||||
let n_cmp = n_sub.min(prev_frame.len());
|
||||
if n_cmp > 0 {
|
||||
let diff_energy: f64 = (0..n_cmp)
|
||||
.map(|k| (frame.amplitudes[k] - prev_frame[k]).powi(2))
|
||||
.sum::<f64>() / n_cmp as f64;
|
||||
// Normalise by mean squared amplitude to get a dimensionless ratio.
|
||||
let ref_energy = mean_amp * mean_amp + 1e-9;
|
||||
(diff_energy / ref_energy).sqrt().clamp(0.0, 1.0)
|
||||
} else {
|
||||
0.0
|
||||
}
|
||||
} else {
|
||||
// No history yet — fall back to intra-frame variance-based estimate.
|
||||
(intra_variance / (mean_amp * mean_amp + 1e-9)).sqrt().clamp(0.0, 1.0)
|
||||
};
|
||||
|
||||
// Blend temporal motion with variance-based motion for robustness.
|
||||
let variance_motion = (temporal_variance / 10.0).clamp(0.0, 1.0);
|
||||
let motion_score = (temporal_motion_score * 0.7 + variance_motion * 0.3).clamp(0.0, 1.0);
|
||||
|
||||
// ── Signal quality metric ──
|
||||
// Based on estimated SNR (RSSI relative to noise floor) and subcarrier consistency.
|
||||
let snr_db = (frame.rssi as f64 - frame.noise_floor as f64).max(0.0);
|
||||
let snr_quality = (snr_db / 40.0).clamp(0.0, 1.0); // 40 dB → quality = 1.0
|
||||
// Penalise quality when temporal variance is very high (unstable signal).
|
||||
let stability = (1.0 - (temporal_variance / (mean_amp * mean_amp + 1e-9)).clamp(0.0, 1.0)).max(0.0);
|
||||
let signal_quality = (snr_quality * 0.6 + stability * 0.4).clamp(0.0, 1.0);
|
||||
|
||||
// ── Breathing rate estimation ──
|
||||
let breathing_rate_hz = estimate_breathing_rate_hz(frame_history, sample_rate_hz);
|
||||
|
||||
let features = FeatureInfo {
|
||||
mean_rssi,
|
||||
variance,
|
||||
@@ -445,23 +683,24 @@ fn extract_features_from_frame(frame: &Esp32Frame) -> (FeatureInfo, Classificati
|
||||
spectral_power,
|
||||
};
|
||||
|
||||
// Classification
|
||||
let motion_score = (variance / 10.0).clamp(0.0, 1.0);
|
||||
let (motion_level, presence) = if motion_score > 0.5 {
|
||||
// ── Classification ──
|
||||
let (motion_level, presence) = if motion_score > 0.4 {
|
||||
("active".to_string(), true)
|
||||
} else if motion_score > 0.1 {
|
||||
} else if motion_score > 0.08 {
|
||||
("present_still".to_string(), true)
|
||||
} else {
|
||||
("absent".to_string(), false)
|
||||
};
|
||||
|
||||
let confidence = (0.4 + signal_quality * 0.3 + motion_score * 0.3).clamp(0.0, 1.0);
|
||||
|
||||
let classification = ClassificationInfo {
|
||||
motion_level,
|
||||
presence,
|
||||
confidence: 0.5 + motion_score * 0.5,
|
||||
confidence,
|
||||
};
|
||||
|
||||
(features, classification)
|
||||
(features, classification, breathing_rate_hz, sub_variances)
|
||||
}
|
||||
|
||||
// ── Windows WiFi RSSI collector ──────────────────────────────────────────────
|
||||
@@ -596,7 +835,16 @@ async fn windows_wifi_task(state: SharedState, tick_ms: u64) {
|
||||
phases: multi_ap_frame.phases.clone(),
|
||||
};
|
||||
|
||||
let (features, classification) = extract_features_from_frame(&frame);
|
||||
// ── Step 4b: Update frame history and extract features ───────
|
||||
let mut s_write_pre = state.write().await;
|
||||
s_write_pre.frame_history.push_back(frame.amplitudes.clone());
|
||||
if s_write_pre.frame_history.len() > FRAME_HISTORY_CAPACITY {
|
||||
s_write_pre.frame_history.pop_front();
|
||||
}
|
||||
let sample_rate_hz = 1000.0 / tick_ms as f64;
|
||||
let (features, classification, breathing_rate_hz, sub_variances) =
|
||||
extract_features_from_frame(&frame, &s_write_pre.frame_history, sample_rate_hz);
|
||||
drop(s_write_pre);
|
||||
|
||||
// ── Step 5: Build enhanced fields from pipeline result ───────
|
||||
let enhanced_motion = Some(serde_json::json!({
|
||||
@@ -640,6 +888,7 @@ async fn windows_wifi_task(state: SharedState, tick_ms: u64) {
|
||||
let vitals = s.vital_detector.process_frame(&frame.amplitudes, &frame.phases);
|
||||
s.latest_vitals = vitals.clone();
|
||||
|
||||
let feat_variance = features.variance;
|
||||
let update = SensingUpdate {
|
||||
msg_type: "sensing_update".to_string(),
|
||||
timestamp: chrono::Utc::now().timestamp_millis() as f64 / 1000.0,
|
||||
@@ -654,7 +903,10 @@ async fn windows_wifi_task(state: SharedState, tick_ms: u64) {
|
||||
}],
|
||||
features,
|
||||
classification,
|
||||
signal_field: generate_signal_field(first_rssi, 1.0, motion_score, tick),
|
||||
signal_field: generate_signal_field(
|
||||
first_rssi, motion_score, breathing_rate_hz,
|
||||
feat_variance.min(1.0), &sub_variances,
|
||||
),
|
||||
vital_signs: Some(vitals),
|
||||
enhanced_motion,
|
||||
enhanced_breathing,
|
||||
@@ -715,9 +967,16 @@ async fn windows_wifi_fallback_tick(state: &SharedState, seq: u32) {
|
||||
phases: vec![0.0],
|
||||
};
|
||||
|
||||
let (features, classification) = extract_features_from_frame(&frame);
|
||||
|
||||
let mut s = state.write().await;
|
||||
// Update frame history before extracting features.
|
||||
s.frame_history.push_back(frame.amplitudes.clone());
|
||||
if s.frame_history.len() > FRAME_HISTORY_CAPACITY {
|
||||
s.frame_history.pop_front();
|
||||
}
|
||||
let sample_rate_hz = 2.0_f64; // fallback tick ~ 500 ms => 2 Hz
|
||||
let (features, classification, breathing_rate_hz, sub_variances) =
|
||||
extract_features_from_frame(&frame, &s.frame_history, sample_rate_hz);
|
||||
|
||||
s.source = format!("wifi:{ssid}");
|
||||
s.rssi_history.push_back(rssi_dbm);
|
||||
if s.rssi_history.len() > 60 {
|
||||
@@ -738,6 +997,7 @@ async fn windows_wifi_fallback_tick(state: &SharedState, seq: u32) {
|
||||
let vitals = s.vital_detector.process_frame(&frame.amplitudes, &frame.phases);
|
||||
s.latest_vitals = vitals.clone();
|
||||
|
||||
let feat_variance = features.variance;
|
||||
let update = SensingUpdate {
|
||||
msg_type: "sensing_update".to_string(),
|
||||
timestamp: chrono::Utc::now().timestamp_millis() as f64 / 1000.0,
|
||||
@@ -752,7 +1012,10 @@ async fn windows_wifi_fallback_tick(state: &SharedState, seq: u32) {
|
||||
}],
|
||||
features,
|
||||
classification,
|
||||
signal_field: generate_signal_field(rssi_dbm, 1.0, motion_score, tick),
|
||||
signal_field: generate_signal_field(
|
||||
rssi_dbm, motion_score, breathing_rate_hz,
|
||||
feat_variance.min(1.0), &sub_variances,
|
||||
),
|
||||
vital_signs: Some(vitals),
|
||||
enhanced_motion: None,
|
||||
enhanced_breathing: None,
|
||||
@@ -902,7 +1165,47 @@ async fn handle_ws_pose_client(mut socket: WebSocket, state: SharedState) {
|
||||
// Parse the sensing update and convert to pose format
|
||||
if let Ok(sensing) = serde_json::from_str::<SensingUpdate>(&json) {
|
||||
if sensing.msg_type == "sensing_update" {
|
||||
let persons = derive_pose_from_sensing(&sensing);
|
||||
// Determine pose estimation mode for the UI indicator.
|
||||
// "model_inference" — a trained RVF model is loaded.
|
||||
// "signal_derived" — keypoints estimated from raw CSI features.
|
||||
let model_loaded = {
|
||||
let s = state.read().await;
|
||||
s.model_loaded
|
||||
};
|
||||
let pose_source = if model_loaded {
|
||||
"model_inference"
|
||||
} else {
|
||||
"signal_derived"
|
||||
};
|
||||
|
||||
let persons = if model_loaded {
|
||||
// When a trained model is loaded, prefer its keypoints if present.
|
||||
sensing.pose_keypoints.as_ref().map(|kps| {
|
||||
let kp_names = [
|
||||
"nose","left_eye","right_eye","left_ear","right_ear",
|
||||
"left_shoulder","right_shoulder","left_elbow","right_elbow",
|
||||
"left_wrist","right_wrist","left_hip","right_hip",
|
||||
"left_knee","right_knee","left_ankle","right_ankle",
|
||||
];
|
||||
let keypoints: Vec<PoseKeypoint> = kps.iter()
|
||||
.enumerate()
|
||||
.map(|(i, kp)| PoseKeypoint {
|
||||
name: kp_names.get(i).unwrap_or(&"unknown").to_string(),
|
||||
x: kp[0], y: kp[1], z: kp[2], confidence: kp[3],
|
||||
})
|
||||
.collect();
|
||||
vec![PersonDetection {
|
||||
id: 1,
|
||||
confidence: sensing.classification.confidence,
|
||||
bbox: BoundingBox { x: 260.0, y: 150.0, width: 120.0, height: 220.0 },
|
||||
keypoints,
|
||||
zone: "zone_1".into(),
|
||||
}]
|
||||
}).unwrap_or_else(|| derive_pose_from_sensing(&sensing))
|
||||
} else {
|
||||
derive_pose_from_sensing(&sensing)
|
||||
};
|
||||
|
||||
let pose_msg = serde_json::json!({
|
||||
"type": "pose_data",
|
||||
"zone_id": "zone_1",
|
||||
@@ -913,12 +1216,16 @@ async fn handle_ws_pose_client(mut socket: WebSocket, state: SharedState) {
|
||||
},
|
||||
"confidence": if sensing.classification.presence { sensing.classification.confidence } else { 0.0 },
|
||||
"activity": sensing.classification.motion_level,
|
||||
// pose_source tells the UI which estimation mode is active.
|
||||
"pose_source": pose_source,
|
||||
"metadata": {
|
||||
"frame_id": format!("rust_frame_{}", sensing.tick),
|
||||
"processing_time_ms": 1,
|
||||
"source": sensing.source,
|
||||
"tick": sensing.tick,
|
||||
"signal_strength": sensing.features.mean_rssi,
|
||||
"motion_band_power": sensing.features.motion_band_power,
|
||||
"breathing_band_power": sensing.features.breathing_band_power,
|
||||
}
|
||||
}
|
||||
});
|
||||
@@ -972,21 +1279,86 @@ async fn latest(State(state): State<SharedState>) -> Json<serde_json::Value> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Generate WiFi-derived pose keypoints from sensing data
|
||||
/// Generate WiFi-derived pose keypoints from sensing data.
|
||||
///
|
||||
/// Keypoint positions are modulated by real signal features rather than a pure
|
||||
/// time-based sine/cosine loop:
|
||||
///
|
||||
/// - `motion_band_power` drives whole-body translation and limb splay
|
||||
/// - `variance` seeds per-frame noise so the skeleton never freezes
|
||||
/// - `breathing_band_power` expands/contracts torso keypoints (shoulders, hips)
|
||||
/// - `dominant_freq_hz` tilts the upper body laterally (lean direction)
|
||||
/// - `change_points` adds burst jitter to extremities (wrists, ankles)
|
||||
///
|
||||
/// When `presence == false` no persons are returned (empty room).
|
||||
/// When walking is detected (`motion_score > 0.55`) the figure shifts laterally
|
||||
/// with a stride-swing pattern applied to arms and legs.
|
||||
fn derive_pose_from_sensing(update: &SensingUpdate) -> Vec<PersonDetection> {
|
||||
let cls = &update.classification;
|
||||
if !cls.presence {
|
||||
return vec![];
|
||||
}
|
||||
|
||||
let t = update.tick as f64 * 0.05;
|
||||
let motion = if cls.motion_level == "active" { 1.0 }
|
||||
else if cls.motion_level == "present_still" { 0.3 }
|
||||
else { 0.0 };
|
||||
let feat = &update.features;
|
||||
|
||||
// COCO 17-keypoint skeleton, positions derived from signal field
|
||||
let base_x = 320.0 + 30.0 * t.sin() * motion;
|
||||
let base_y = 240.0 + 15.0 * (t * 0.7).cos() * motion;
|
||||
// ── Signal-derived scalars ────────────────────────────────────────────────
|
||||
|
||||
// Continuous motion score from motion_band_power (0..1).
|
||||
// motion_band_power is the high-frequency subcarrier variance — it is high
|
||||
// when a body is actively moving through the RF field.
|
||||
let motion_score = (feat.motion_band_power / 15.0).clamp(0.0, 1.0);
|
||||
let is_walking = motion_score > 0.55;
|
||||
|
||||
// Breathing expansion: torso keypoints shift ±breath_amp pixels per cycle.
|
||||
// breathing_band_power comes from low-frequency subcarrier variance.
|
||||
let breath_amp = (feat.breathing_band_power * 4.0).clamp(0.0, 12.0);
|
||||
|
||||
// Breathing phase: use the vital-sign estimate if available, otherwise
|
||||
// derive a proxy from breathing_band_power and the tick counter.
|
||||
let breath_phase = if let Some(ref vs) = update.vital_signs {
|
||||
// breathing_rate_bpm is Option<f64>; fall back to 15 BPM if not yet estimated.
|
||||
// 15 BPM -> 0.25 Hz, which sits comfortably in the breathing band.
|
||||
let bpm = vs.breathing_rate_bpm.unwrap_or(15.0);
|
||||
let freq = (bpm / 60.0).clamp(0.1, 0.5);
|
||||
(update.tick as f64 * freq * 0.1 * std::f64::consts::TAU).sin()
|
||||
} else {
|
||||
(update.tick as f64 * 0.08 + feat.breathing_band_power).sin()
|
||||
};
|
||||
|
||||
// Lateral lean derived from dominant_freq_hz (peak subcarrier index -> Hz).
|
||||
// Maps 0..10 Hz range to ±18 px horizontal shift of the torso center.
|
||||
let lean_x = (feat.dominant_freq_hz / 5.0 - 1.0).clamp(-1.0, 1.0) * 18.0;
|
||||
|
||||
// Walking stride: lateral body displacement oscillating with motion_band_power.
|
||||
// Amplitude is zero when the person is stationary.
|
||||
let stride_x = if is_walking {
|
||||
let stride_phase = (feat.motion_band_power * 0.7 + update.tick as f64 * 0.12).sin();
|
||||
stride_phase * 45.0 * motion_score
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
|
||||
// Burst jitter from change_points: rapid threshold crossings in the
|
||||
// amplitude vector indicate fast movement or sudden signal disturbance.
|
||||
let burst = (feat.change_points as f64 / 8.0).clamp(0.0, 1.0);
|
||||
|
||||
// Deterministic per-frame noise seeded by variance and tick.
|
||||
// Uses the fractional part of a large sine to get a tick-dependent value
|
||||
// in (-1, 1) without needing a PRNG.
|
||||
let noise_seed = feat.variance * 31.7 + update.tick as f64 * 17.3;
|
||||
let noise_val = (noise_seed.sin() * 43758.545).fract();
|
||||
|
||||
// Scale base confidence by SNR proxy (high variance = better signal quality).
|
||||
let snr_factor = ((feat.variance - 0.5) / 10.0).clamp(0.0, 1.0);
|
||||
let base_confidence = cls.confidence * (0.6 + 0.4 * snr_factor);
|
||||
|
||||
// ── Skeleton base position ────────────────────────────────────────────────
|
||||
|
||||
// Center figure on a 640x480 canvas.
|
||||
let base_x = 320.0 + stride_x + lean_x * 0.5;
|
||||
let base_y = 240.0 - motion_score * 8.0;
|
||||
|
||||
// ── COCO 17-keypoint offsets from hip-center ──────────────────────────────
|
||||
|
||||
let kp_names = [
|
||||
"nose", "left_eye", "right_eye", "left_ear", "right_ear",
|
||||
@@ -994,49 +1366,130 @@ fn derive_pose_from_sensing(update: &SensingUpdate) -> Vec<PersonDetection> {
|
||||
"left_wrist", "right_wrist", "left_hip", "right_hip",
|
||||
"left_knee", "right_knee", "left_ankle", "right_ankle",
|
||||
];
|
||||
|
||||
// Nominal (dx, dy) offsets from hip-center in pixels.
|
||||
let kp_offsets: [(f64, f64); 17] = [
|
||||
(0.0, -80.0), // nose
|
||||
(-8.0, -88.0), // left_eye
|
||||
(8.0, -88.0), // right_eye
|
||||
(-16.0, -82.0), // left_ear
|
||||
(16.0, -82.0), // right_ear
|
||||
(-30.0, -50.0), // left_shoulder
|
||||
(30.0, -50.0), // right_shoulder
|
||||
(-45.0, -15.0), // left_elbow
|
||||
(45.0, -15.0), // right_elbow
|
||||
(-50.0, 20.0), // left_wrist
|
||||
(50.0, 20.0), // right_wrist
|
||||
(-20.0, 20.0), // left_hip
|
||||
(20.0, 20.0), // right_hip
|
||||
(-22.0, 70.0), // left_knee
|
||||
(22.0, 70.0), // right_knee
|
||||
(-24.0, 120.0), // left_ankle
|
||||
(24.0, 120.0), // right_ankle
|
||||
( 0.0, -80.0), // 0 nose
|
||||
( -8.0, -88.0), // 1 left_eye
|
||||
( 8.0, -88.0), // 2 right_eye
|
||||
(-16.0, -82.0), // 3 left_ear
|
||||
( 16.0, -82.0), // 4 right_ear
|
||||
(-30.0, -50.0), // 5 left_shoulder
|
||||
( 30.0, -50.0), // 6 right_shoulder
|
||||
(-45.0, -15.0), // 7 left_elbow
|
||||
( 45.0, -15.0), // 8 right_elbow
|
||||
(-50.0, 20.0), // 9 left_wrist
|
||||
( 50.0, 20.0), // 10 right_wrist
|
||||
(-20.0, 20.0), // 11 left_hip
|
||||
( 20.0, 20.0), // 12 right_hip
|
||||
(-22.0, 70.0), // 13 left_knee
|
||||
( 22.0, 70.0), // 14 right_knee
|
||||
(-24.0, 120.0), // 15 left_ankle
|
||||
( 24.0, 120.0), // 16 right_ankle
|
||||
];
|
||||
|
||||
// Torso keypoints: left_shoulder(5), right_shoulder(6), left_hip(11), right_hip(12).
|
||||
// These respond to the breathing expansion signal.
|
||||
const TORSO_KP: [usize; 4] = [5, 6, 11, 12];
|
||||
|
||||
// Extremity keypoints: left_wrist(9), right_wrist(10), left_ankle(15), right_ankle(16).
|
||||
// These pick up burst jitter from high change_points counts.
|
||||
const EXTREMITY_KP: [usize; 4] = [9, 10, 15, 16];
|
||||
|
||||
let keypoints: Vec<PoseKeypoint> = kp_names.iter().zip(kp_offsets.iter())
|
||||
.enumerate()
|
||||
.map(|(i, (name, (dx, dy)))| {
|
||||
let jitter = motion * 3.0 * (t * 2.0 + i as f64).sin();
|
||||
// ── Breathing expansion (torso only) ─────────────────────────
|
||||
let breath_dx = if TORSO_KP.contains(&i) {
|
||||
// Shoulders spread outward; hips compress inward on inhale.
|
||||
let sign = if *dx < 0.0 { -1.0 } else { 1.0 };
|
||||
sign * breath_amp * breath_phase * 0.5
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
let breath_dy = if TORSO_KP.contains(&i) {
|
||||
// Shoulders rise slightly; hips descend slightly on inhale.
|
||||
let sign = if *dy < 0.0 { -1.0 } else { 1.0 };
|
||||
sign * breath_amp * breath_phase * 0.3
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
|
||||
// ── Extremity burst jitter ────────────────────────────────────
|
||||
let extremity_jitter = if EXTREMITY_KP.contains(&i) {
|
||||
// Each extremity gets an independent phase offset.
|
||||
let phase = noise_seed + i as f64 * 2.399; // golden-angle spacing
|
||||
(
|
||||
phase.sin() * burst * motion_score * 12.0,
|
||||
(phase * 1.31).cos() * burst * motion_score * 8.0,
|
||||
)
|
||||
} else {
|
||||
(0.0, 0.0)
|
||||
};
|
||||
|
||||
// ── Per-joint motion noise (scales with signal variance) ──────
|
||||
// Different seed per keypoint so every joint moves independently.
|
||||
let kp_noise_x = ((noise_seed + i as f64 * 1.618).sin() * 43758.545).fract()
|
||||
* feat.variance.sqrt().clamp(0.0, 3.0) * motion_score;
|
||||
let kp_noise_y = ((noise_seed + i as f64 * 2.718).cos() * 31415.926).fract()
|
||||
* feat.variance.sqrt().clamp(0.0, 3.0) * motion_score * 0.6;
|
||||
|
||||
// ── Walking arm/leg swing (contralateral gait pattern) ────────
|
||||
let swing_dy = if is_walking {
|
||||
let stride_phase =
|
||||
(feat.motion_band_power * 0.7 + update.tick as f64 * 0.12).sin();
|
||||
match i {
|
||||
7 | 9 => -stride_phase * 20.0 * motion_score, // left elbow/wrist
|
||||
8 | 10 => stride_phase * 20.0 * motion_score, // right elbow/wrist
|
||||
13 | 15 => stride_phase * 25.0 * motion_score, // left knee/ankle
|
||||
14 | 16 => -stride_phase * 25.0 * motion_score, // right knee/ankle
|
||||
_ => 0.0,
|
||||
}
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
|
||||
// ── Compose final position ────────────────────────────────────
|
||||
let final_x =
|
||||
base_x + dx + breath_dx + extremity_jitter.0 + kp_noise_x;
|
||||
let final_y =
|
||||
base_y + dy + breath_dy + extremity_jitter.1 + kp_noise_y + swing_dy;
|
||||
|
||||
// Extremity confidence is lower when signal variance is low.
|
||||
let kp_conf = if EXTREMITY_KP.contains(&i) {
|
||||
base_confidence * (0.7 + 0.3 * snr_factor) * (0.85 + 0.15 * noise_val)
|
||||
} else {
|
||||
base_confidence
|
||||
* (0.88 + 0.12 * ((i as f64 * 0.7 + noise_seed).cos()))
|
||||
};
|
||||
|
||||
PoseKeypoint {
|
||||
name: name.to_string(),
|
||||
x: base_x + dx + jitter,
|
||||
y: base_y + dy + jitter * 0.5,
|
||||
z: 0.0,
|
||||
confidence: cls.confidence * (0.85 + 0.15 * (i as f64 * 0.3).cos()),
|
||||
x: final_x,
|
||||
y: final_y,
|
||||
z: lean_x * 0.02, // slight Z depth from lean direction
|
||||
confidence: kp_conf.clamp(0.1, 1.0),
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Bounding box derived from actual keypoint extents with padding.
|
||||
let xs: Vec<f64> = keypoints.iter().map(|k| k.x).collect();
|
||||
let ys: Vec<f64> = keypoints.iter().map(|k| k.y).collect();
|
||||
let min_x = xs.iter().cloned().fold(f64::MAX, f64::min) - 10.0;
|
||||
let min_y = ys.iter().cloned().fold(f64::MAX, f64::min) - 10.0;
|
||||
let max_x = xs.iter().cloned().fold(f64::MIN, f64::max) + 10.0;
|
||||
let max_y = ys.iter().cloned().fold(f64::MIN, f64::max) + 10.0;
|
||||
|
||||
vec![PersonDetection {
|
||||
id: 1,
|
||||
confidence: cls.confidence,
|
||||
keypoints,
|
||||
bbox: BoundingBox {
|
||||
x: base_x - 60.0,
|
||||
y: base_y - 90.0,
|
||||
width: 120.0,
|
||||
height: 220.0,
|
||||
x: min_x,
|
||||
y: min_y,
|
||||
width: (max_x - min_x).max(80.0),
|
||||
height: (max_y - min_y).max(160.0),
|
||||
},
|
||||
zone: "zone_1".into(),
|
||||
}]
|
||||
@@ -1161,7 +1614,7 @@ async fn stream_status(State(state): State<SharedState>) -> Json<serde_json::Val
|
||||
Json(serde_json::json!({
|
||||
"active": true,
|
||||
"clients": s.tx.receiver_count(),
|
||||
"fps": 2,
|
||||
"fps": if s.tick > 1 { 10u64 } else { 0u64 },
|
||||
"source": s.source,
|
||||
}))
|
||||
}
|
||||
@@ -1311,10 +1764,20 @@ async fn udp_receiver_task(state: SharedState, udp_port: u16) {
|
||||
debug!("ESP32 frame from {src}: node={}, subs={}, seq={}",
|
||||
frame.node_id, frame.n_subcarriers, frame.sequence);
|
||||
|
||||
let (features, classification) = extract_features_from_frame(&frame);
|
||||
let mut s = state.write().await;
|
||||
s.source = "esp32".to_string();
|
||||
|
||||
// Append current amplitudes to history before extracting features so
|
||||
// that temporal analysis includes the most recent frame.
|
||||
s.frame_history.push_back(frame.amplitudes.clone());
|
||||
if s.frame_history.len() > FRAME_HISTORY_CAPACITY {
|
||||
s.frame_history.pop_front();
|
||||
}
|
||||
|
||||
let sample_rate_hz = 1000.0 / 500.0_f64; // default tick; ESP32 frames arrive as fast as they come
|
||||
let (features, classification, breathing_rate_hz, sub_variances) =
|
||||
extract_features_from_frame(&frame, &s.frame_history, sample_rate_hz);
|
||||
|
||||
// Update RSSI history
|
||||
s.rssi_history.push_back(features.mean_rssi);
|
||||
if s.rssi_history.len() > 60 {
|
||||
@@ -1349,7 +1812,8 @@ async fn udp_receiver_task(state: SharedState, udp_port: u16) {
|
||||
features: features.clone(),
|
||||
classification,
|
||||
signal_field: generate_signal_field(
|
||||
features.mean_rssi, features.variance, motion_score, tick,
|
||||
features.mean_rssi, motion_score, breathing_rate_hz,
|
||||
features.variance.min(1.0), &sub_variances,
|
||||
),
|
||||
vital_signs: Some(vitals),
|
||||
enhanced_motion: None,
|
||||
@@ -1390,7 +1854,16 @@ async fn simulated_data_task(state: SharedState, tick_ms: u64) {
|
||||
let tick = s.tick;
|
||||
|
||||
let frame = generate_simulated_frame(tick);
|
||||
let (features, classification) = extract_features_from_frame(&frame);
|
||||
|
||||
// Append current amplitudes to history before feature extraction.
|
||||
s.frame_history.push_back(frame.amplitudes.clone());
|
||||
if s.frame_history.len() > FRAME_HISTORY_CAPACITY {
|
||||
s.frame_history.pop_front();
|
||||
}
|
||||
|
||||
let sample_rate_hz = 1000.0 / tick_ms as f64;
|
||||
let (features, classification, breathing_rate_hz, sub_variances) =
|
||||
extract_features_from_frame(&frame, &s.frame_history, sample_rate_hz);
|
||||
|
||||
s.rssi_history.push_back(features.mean_rssi);
|
||||
if s.rssi_history.len() > 60 {
|
||||
@@ -1407,6 +1880,9 @@ async fn simulated_data_task(state: SharedState, tick_ms: u64) {
|
||||
);
|
||||
s.latest_vitals = vitals.clone();
|
||||
|
||||
let frame_amplitudes = frame.amplitudes.clone();
|
||||
let frame_n_sub = frame.n_subcarriers;
|
||||
|
||||
let update = SensingUpdate {
|
||||
msg_type: "sensing_update".to_string(),
|
||||
timestamp: chrono::Utc::now().timestamp_millis() as f64 / 1000.0,
|
||||
@@ -1416,13 +1892,14 @@ async fn simulated_data_task(state: SharedState, tick_ms: u64) {
|
||||
node_id: 1,
|
||||
rssi_dbm: features.mean_rssi,
|
||||
position: [2.0, 0.0, 1.5],
|
||||
amplitude: frame.amplitudes,
|
||||
subcarrier_count: frame.n_subcarriers as usize,
|
||||
amplitude: frame_amplitudes,
|
||||
subcarrier_count: frame_n_sub as usize,
|
||||
}],
|
||||
features: features.clone(),
|
||||
classification,
|
||||
signal_field: generate_signal_field(
|
||||
features.mean_rssi, features.variance, motion_score, tick,
|
||||
features.mean_rssi, motion_score, breathing_rate_hz,
|
||||
features.variance.min(1.0), &sub_variances,
|
||||
),
|
||||
vital_signs: Some(vitals),
|
||||
enhanced_motion: None,
|
||||
@@ -2014,6 +2491,7 @@ async fn main() {
|
||||
let state: SharedState = Arc::new(RwLock::new(AppStateInner {
|
||||
latest_update: None,
|
||||
rssi_history: VecDeque::new(),
|
||||
frame_history: VecDeque::new(),
|
||||
tick: 0,
|
||||
source: source.into(),
|
||||
tx,
|
||||
|
||||
@@ -1,46 +1,73 @@
|
||||
# WiFi DensePose UI
|
||||
|
||||
A modular, modern web interface for the WiFi DensePose human tracking system. This UI provides real-time monitoring, configuration, and visualization of WiFi-based pose estimation.
|
||||
A modular, modern web interface for the WiFi DensePose human tracking system. Provides real-time monitoring, WiFi sensing visualization, and pose estimation from CSI (Channel State Information).
|
||||
|
||||
## 🏗️ Architecture
|
||||
## Architecture
|
||||
|
||||
The UI follows a modular architecture with clear separation of concerns:
|
||||
|
||||
```
|
||||
ui/
|
||||
├── app.js # Main application entry point
|
||||
├── index.html # Updated HTML with modular structure
|
||||
├── style.css # Complete CSS with additional styles
|
||||
├── config/ # Configuration modules
|
||||
│ └── api.config.js # API endpoints and configuration
|
||||
├── services/ # Service layer for API communication
|
||||
│ ├── api.service.js # HTTP API client
|
||||
│ ├── websocket.service.js # WebSocket client
|
||||
│ ├── pose.service.js # Pose estimation API wrapper
|
||||
│ ├── health.service.js # Health monitoring API wrapper
|
||||
│ └── stream.service.js # Streaming API wrapper
|
||||
├── components/ # UI components
|
||||
│ ├── TabManager.js # Tab navigation component
|
||||
│ ├── DashboardTab.js # Dashboard component with live data
|
||||
│ ├── HardwareTab.js # Hardware configuration component
|
||||
│ └── LiveDemoTab.js # Live demo with streaming
|
||||
├── utils/ # Utility functions and helpers
|
||||
│ └── mock-server.js # Mock server for testing
|
||||
└── tests/ # Comprehensive test suite
|
||||
├── test-runner.html # Test runner UI
|
||||
├── test-runner.js # Test framework and cases
|
||||
└── integration-test.html # Integration testing page
|
||||
├── app.js # Main application entry point
|
||||
├── index.html # HTML shell with tab structure
|
||||
├── style.css # Complete CSS design system
|
||||
├── config/
|
||||
│ └── api.config.js # API endpoints and configuration
|
||||
├── services/
|
||||
│ ├── api.service.js # HTTP API client
|
||||
│ ├── websocket.service.js # WebSocket connection manager
|
||||
│ ├── websocket-client.js # Low-level WebSocket client
|
||||
│ ├── pose.service.js # Pose estimation API wrapper
|
||||
│ ├── sensing.service.js # WiFi sensing data service (live + simulation fallback)
|
||||
│ ├── health.service.js # Health monitoring API wrapper
|
||||
│ ├── stream.service.js # Streaming API wrapper
|
||||
│ └── data-processor.js # Signal data processing utilities
|
||||
├── components/
|
||||
│ ├── TabManager.js # Tab navigation component
|
||||
│ ├── DashboardTab.js # Dashboard with live system metrics
|
||||
│ ├── SensingTab.js # WiFi sensing visualization (3D signal field, metrics)
|
||||
│ ├── LiveDemoTab.js # Live pose detection with setup guide
|
||||
│ ├── HardwareTab.js # Hardware configuration
|
||||
│ ├── SettingsPanel.js # Settings panel
|
||||
│ ├── PoseDetectionCanvas.js # Canvas-based pose skeleton renderer
|
||||
│ ├── gaussian-splats.js # 3D Gaussian splat signal field renderer (Three.js)
|
||||
│ ├── body-model.js # 3D body model
|
||||
│ ├── scene.js # Three.js scene management
|
||||
│ ├── signal-viz.js # Signal visualization utilities
|
||||
│ ├── environment.js # Environment/room visualization
|
||||
│ └── dashboard-hud.js # Dashboard heads-up display
|
||||
├── utils/
|
||||
│ ├── backend-detector.js # Auto-detect backend availability
|
||||
│ ├── mock-server.js # Mock server for testing
|
||||
│ └── pose-renderer.js # Pose rendering utilities
|
||||
└── tests/
|
||||
├── test-runner.html # Test runner UI
|
||||
├── test-runner.js # Test framework and cases
|
||||
└── integration-test.html # Integration testing page
|
||||
```
|
||||
|
||||
## 🚀 Features
|
||||
## Features
|
||||
|
||||
### Smart Backend Detection
|
||||
- **Automatic Detection**: Automatically detects if your FastAPI backend is running
|
||||
- **Real Backend Priority**: Always uses the real backend when available
|
||||
- **Mock Fallback**: Falls back to mock server only when backend is unavailable
|
||||
- **Testing Mode**: Can force mock mode for testing and development
|
||||
### WiFi Sensing Tab
|
||||
- 3D Gaussian-splat signal field visualization (Three.js)
|
||||
- Real-time RSSI, variance, motion band, breathing band metrics
|
||||
- Presence/motion classification with confidence scores
|
||||
- **Data source banner**: green "LIVE - ESP32", yellow "RECONNECTING...", or red "SIMULATED DATA"
|
||||
- Sparkline RSSI history graph
|
||||
- "About This Data" card explaining CSI capabilities per sensor count
|
||||
|
||||
### Real-time Dashboard
|
||||
### Live Demo Tab
|
||||
- WebSocket-based real-time pose skeleton rendering
|
||||
- **Estimation Mode badge**: green "Signal-Derived" or blue "Model Inference"
|
||||
- **Setup Guide panel** showing what each ESP32 count provides:
|
||||
- 1 ESP32: presence, breathing, gross motion
|
||||
- 2-3 ESP32s: body localization, motion direction
|
||||
- 4+ ESP32s + trained model: individual limb tracking, full pose
|
||||
- Debug mode with log export
|
||||
- Zone selection and force-reconnect controls
|
||||
- Performance metrics sidebar (frames, uptime, errors)
|
||||
|
||||
### Dashboard
|
||||
- Live system health monitoring
|
||||
- Real-time pose detection statistics
|
||||
- Zone occupancy tracking
|
||||
@@ -53,284 +80,118 @@ ui/
|
||||
- Configuration panels
|
||||
- Hardware status monitoring
|
||||
|
||||
### Live Demo
|
||||
- WebSocket-based real-time streaming
|
||||
- Signal visualization
|
||||
- Pose detection visualization
|
||||
- Interactive controls
|
||||
## Data Sources
|
||||
|
||||
### API Integration
|
||||
- Complete REST API coverage
|
||||
- WebSocket streaming support
|
||||
- Authentication handling
|
||||
- Error management
|
||||
- Request/response interceptors
|
||||
The sensing service (`sensing.service.js`) supports three connection states:
|
||||
|
||||
## 📋 API Coverage
|
||||
| State | Banner Color | Description |
|
||||
|-------|-------------|-------------|
|
||||
| **LIVE - ESP32** | Green | Connected to the Rust sensing server receiving real CSI data |
|
||||
| **RECONNECTING** | Yellow (pulsing) | WebSocket disconnected, retrying (up to 20 attempts) |
|
||||
| **SIMULATED DATA** | Red | Fallback to client-side simulation after 5+ failed reconnects |
|
||||
|
||||
The UI integrates with all WiFi DensePose API endpoints:
|
||||
Simulated frames include a `_simulated: true` marker so code can detect synthetic data.
|
||||
|
||||
### Health Endpoints
|
||||
- `GET /health/health` - System health check
|
||||
- `GET /health/ready` - Readiness check
|
||||
- `GET /health/live` - Liveness check
|
||||
- `GET /health/metrics` - System metrics
|
||||
- `GET /health/version` - Version information
|
||||
## Backends
|
||||
|
||||
### Pose Estimation
|
||||
- `GET /api/v1/pose/current` - Current pose data
|
||||
- `POST /api/v1/pose/analyze` - Trigger analysis
|
||||
- `GET /api/v1/pose/zones/{zone_id}/occupancy` - Zone occupancy
|
||||
- `GET /api/v1/pose/zones/summary` - All zones summary
|
||||
- `POST /api/v1/pose/historical` - Historical data
|
||||
- `GET /api/v1/pose/activities` - Recent activities
|
||||
- `POST /api/v1/pose/calibrate` - System calibration
|
||||
- `GET /api/v1/pose/stats` - Statistics
|
||||
### Rust Sensing Server (primary)
|
||||
The Rust-based `wifi-densepose-sensing-server` serves the UI and provides:
|
||||
- `GET /health` — server health
|
||||
- `GET /api/v1/sensing/latest` — latest sensing features
|
||||
- `GET /api/v1/vital-signs` — vital sign estimates (HR/RR)
|
||||
- `GET /api/v1/model/info` — RVF model container info
|
||||
- `WS /ws/sensing` — real-time sensing data stream
|
||||
- `WS /api/v1/stream/pose` — real-time pose keypoint stream
|
||||
|
||||
### Streaming
|
||||
- `WS /api/v1/stream/pose` - Real-time pose stream
|
||||
- `WS /api/v1/stream/events` - Event stream
|
||||
- `GET /api/v1/stream/status` - Stream status
|
||||
- `POST /api/v1/stream/start` - Start streaming
|
||||
- `POST /api/v1/stream/stop` - Stop streaming
|
||||
- `GET /api/v1/stream/clients` - Connected clients
|
||||
- `DELETE /api/v1/stream/clients/{client_id}` - Disconnect client
|
||||
### Python FastAPI (legacy)
|
||||
The original Python backend on port 8000 is still supported. The UI auto-detects which backend is available via `backend-detector.js`.
|
||||
|
||||
## 🧪 Testing
|
||||
|
||||
### Test Runner
|
||||
Open `tests/test-runner.html` to run the complete test suite:
|
||||
## Quick Start
|
||||
|
||||
### With Docker (recommended)
|
||||
```bash
|
||||
# Serve the UI directory on port 3000 (to avoid conflicts with FastAPI on 8000)
|
||||
cd /workspaces/wifi-densepose/ui
|
||||
cd docker/
|
||||
|
||||
# Default: auto-detects ESP32 on UDP 5005, falls back to simulation
|
||||
docker-compose up
|
||||
|
||||
# Force real ESP32 data
|
||||
CSI_SOURCE=esp32 docker-compose up
|
||||
|
||||
# Force simulation (no hardware needed)
|
||||
CSI_SOURCE=simulated docker-compose up
|
||||
```
|
||||
Open http://localhost:3000/ui/index.html
|
||||
|
||||
### With local Rust binary
|
||||
```bash
|
||||
cd rust-port/wifi-densepose-rs
|
||||
cargo build -p wifi-densepose-sensing-server --no-default-features
|
||||
|
||||
# Run with simulated data
|
||||
../../target/debug/sensing-server --source simulated --tick-ms 100 --ui-path ../../ui --http-port 3000
|
||||
|
||||
# Run with real ESP32
|
||||
../../target/debug/sensing-server --source esp32 --tick-ms 100 --ui-path ../../ui --http-port 3000
|
||||
```
|
||||
Open http://localhost:3000/ui/index.html
|
||||
|
||||
### With Python HTTP server (legacy)
|
||||
```bash
|
||||
# Start FastAPI backend on port 8000
|
||||
wifi-densepose start
|
||||
|
||||
# Serve the UI on port 3000
|
||||
cd ui/
|
||||
python -m http.server 3000
|
||||
# Open http://localhost:3000/tests/test-runner.html
|
||||
```
|
||||
Open http://localhost:3000
|
||||
|
||||
### Test Categories
|
||||
- **API Configuration Tests** - Configuration and URL building
|
||||
- **API Service Tests** - HTTP client functionality
|
||||
- **WebSocket Service Tests** - WebSocket connection management
|
||||
- **Pose Service Tests** - Pose estimation API wrapper
|
||||
- **Health Service Tests** - Health monitoring functionality
|
||||
- **UI Component Tests** - Component behavior and interaction
|
||||
- **Integration Tests** - End-to-end functionality
|
||||
## Pose Estimation Modes
|
||||
|
||||
### Integration Testing
|
||||
Use `tests/integration-test.html` for visual integration testing:
|
||||
| Mode | Badge | Requirements | Accuracy |
|
||||
|------|-------|-------------|----------|
|
||||
| **Signal-Derived** | Green | 1+ ESP32, no model needed | Presence, breathing, gross motion |
|
||||
| **Model Inference** | Blue | 4+ ESP32s + trained `.rvf` model | Full 17-keypoint COCO pose |
|
||||
|
||||
To use model inference, start the server with a trained model:
|
||||
```bash
|
||||
# Open http://localhost:3000/tests/integration-test.html
|
||||
sensing-server --source esp32 --model path/to/model.rvf --ui-path ./ui
|
||||
```
|
||||
|
||||
Features:
|
||||
- Mock server with realistic API responses
|
||||
- Visual testing of all components
|
||||
- Real-time data simulation
|
||||
- Error scenario testing
|
||||
- WebSocket stream testing
|
||||
|
||||
## 🛠️ Usage
|
||||
|
||||
### Basic Setup
|
||||
```html
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<link rel="stylesheet" href="style.css">
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<!-- Your content -->
|
||||
</div>
|
||||
<script type="module" src="app.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
### Using Services
|
||||
```javascript
|
||||
import { poseService } from './services/pose.service.js';
|
||||
import { healthService } from './services/health.service.js';
|
||||
|
||||
// Get current pose data
|
||||
const poseData = await poseService.getCurrentPose();
|
||||
|
||||
// Subscribe to health updates
|
||||
healthService.subscribeToHealth(health => {
|
||||
console.log('Health status:', health.status);
|
||||
});
|
||||
|
||||
// Start pose streaming
|
||||
poseService.startPoseStream({
|
||||
minConfidence: 0.7,
|
||||
maxFps: 30
|
||||
});
|
||||
|
||||
poseService.subscribeToPoseUpdates(update => {
|
||||
if (update.type === 'pose_update') {
|
||||
console.log('New pose data:', update.data);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### Using Components
|
||||
```javascript
|
||||
import { TabManager } from './components/TabManager.js';
|
||||
import { DashboardTab } from './components/DashboardTab.js';
|
||||
|
||||
// Initialize tab manager
|
||||
const container = document.querySelector('.container');
|
||||
const tabManager = new TabManager(container);
|
||||
tabManager.init();
|
||||
|
||||
// Initialize dashboard
|
||||
const dashboardContainer = document.getElementById('dashboard');
|
||||
const dashboard = new DashboardTab(dashboardContainer);
|
||||
await dashboard.init();
|
||||
```
|
||||
|
||||
## 🔧 Configuration
|
||||
## Configuration
|
||||
|
||||
### API Configuration
|
||||
Edit `config/api.config.js` to modify API settings:
|
||||
Edit `config/api.config.js`:
|
||||
|
||||
```javascript
|
||||
export const API_CONFIG = {
|
||||
BASE_URL: window.location.origin,
|
||||
API_VERSION: '/api/v1',
|
||||
|
||||
// Rate limiting
|
||||
RATE_LIMITS: {
|
||||
REQUESTS_PER_MINUTE: 60,
|
||||
BURST_LIMIT: 10
|
||||
},
|
||||
|
||||
// WebSocket configuration
|
||||
WS_CONFIG: {
|
||||
RECONNECT_DELAY: 5000,
|
||||
MAX_RECONNECT_ATTEMPTS: 5,
|
||||
MAX_RECONNECT_ATTEMPTS: 20,
|
||||
PING_INTERVAL: 30000
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### Authentication
|
||||
```javascript
|
||||
import { apiService } from './services/api.service.js';
|
||||
## Testing
|
||||
|
||||
// Set authentication token
|
||||
apiService.setAuthToken('your-jwt-token');
|
||||
Open `tests/test-runner.html` to run the test suite:
|
||||
|
||||
// Add request interceptor for auth
|
||||
apiService.addRequestInterceptor((url, options) => {
|
||||
// Modify request before sending
|
||||
return { url, options };
|
||||
});
|
||||
```
|
||||
|
||||
## 🎨 Styling
|
||||
|
||||
The UI uses a comprehensive CSS design system with:
|
||||
|
||||
- CSS Custom Properties for theming
|
||||
- Dark/light mode support
|
||||
- Responsive design
|
||||
- Component-based styling
|
||||
- Smooth animations and transitions
|
||||
|
||||
### Key CSS Variables
|
||||
```css
|
||||
:root {
|
||||
--color-primary: rgba(33, 128, 141, 1);
|
||||
--color-background: rgba(252, 252, 249, 1);
|
||||
--color-surface: rgba(255, 255, 253, 1);
|
||||
--color-text: rgba(19, 52, 59, 1);
|
||||
--space-16: 16px;
|
||||
--radius-lg: 12px;
|
||||
}
|
||||
```
|
||||
|
||||
## 🔍 Monitoring & Debugging
|
||||
|
||||
### Health Monitoring
|
||||
```javascript
|
||||
import { healthService } from './services/health.service.js';
|
||||
|
||||
// Start automatic health checks
|
||||
healthService.startHealthMonitoring(30000); // Every 30 seconds
|
||||
|
||||
// Check if system is healthy
|
||||
const isHealthy = healthService.isSystemHealthy();
|
||||
|
||||
// Get specific component status
|
||||
const apiStatus = healthService.getComponentStatus('api');
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
```javascript
|
||||
// Global error handling
|
||||
window.addEventListener('error', (event) => {
|
||||
console.error('Global error:', event.error);
|
||||
});
|
||||
|
||||
// API error handling
|
||||
apiService.addResponseInterceptor(async (response, url) => {
|
||||
if (!response.ok) {
|
||||
console.error(`API error: ${response.status} for ${url}`);
|
||||
}
|
||||
return response;
|
||||
});
|
||||
```
|
||||
|
||||
## 🚀 Deployment
|
||||
|
||||
### Development
|
||||
|
||||
**Option 1: Use the startup script**
|
||||
```bash
|
||||
cd /workspaces/wifi-densepose/ui
|
||||
./start-ui.sh
|
||||
```
|
||||
|
||||
**Option 2: Manual setup**
|
||||
```bash
|
||||
# First, start your FastAPI backend (runs on port 8000)
|
||||
wifi-densepose start
|
||||
# or from the main project directory:
|
||||
python -m wifi_densepose.main
|
||||
|
||||
# Then, start the UI server on a different port to avoid conflicts
|
||||
cd /workspaces/wifi-densepose/ui
|
||||
cd ui/
|
||||
python -m http.server 3000
|
||||
# or
|
||||
npx http-server . -p 3000
|
||||
|
||||
# Open the UI at http://localhost:3000
|
||||
# The UI will automatically detect and connect to your backend
|
||||
# Open http://localhost:3000/tests/test-runner.html
|
||||
```
|
||||
|
||||
### Backend Detection Behavior
|
||||
- **Real Backend Available**: UI connects to `http://localhost:8000` and shows ✅ "Connected to real backend"
|
||||
- **Backend Unavailable**: UI automatically uses mock server and shows ⚠️ "Mock server active - testing mode"
|
||||
- **Force Mock Mode**: Set `API_CONFIG.MOCK_SERVER.ENABLED = true` for testing
|
||||
Test categories: API configuration, API service, WebSocket, pose service, health service, UI components, integration.
|
||||
|
||||
### Production
|
||||
1. Configure `API_CONFIG.BASE_URL` for your backend
|
||||
2. Set up HTTPS for WebSocket connections
|
||||
3. Configure authentication if required
|
||||
4. Optimize assets (minify CSS/JS)
|
||||
5. Set up monitoring and logging
|
||||
## Styling
|
||||
|
||||
## 🤝 Contributing
|
||||
Uses a CSS design system with custom properties, dark/light mode, responsive layout, and component-based styling. Key variables in `:root` of `style.css`.
|
||||
|
||||
1. Follow the modular architecture
|
||||
2. Add tests for new functionality
|
||||
3. Update documentation
|
||||
4. Ensure TypeScript compatibility
|
||||
5. Test with mock server
|
||||
## License
|
||||
|
||||
## 📄 License
|
||||
|
||||
This project is part of the WiFi-DensePose system. See the main project LICENSE file for details.
|
||||
Part of the WiFi-DensePose system. See the main project LICENSE file.
|
||||
|
||||
@@ -14,7 +14,9 @@ export class LiveDemoTab {
|
||||
currentZone: 'zone_1',
|
||||
debugMode: false,
|
||||
autoReconnect: true,
|
||||
renderMode: 'skeleton'
|
||||
renderMode: 'skeleton',
|
||||
// 'unknown' | 'signal_derived' | 'model_inference'
|
||||
poseSource: 'unknown'
|
||||
};
|
||||
|
||||
this.components = {
|
||||
@@ -136,6 +138,48 @@ export class LiveDemoTab {
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="pose-source-panel">
|
||||
<h4>Estimation Mode</h4>
|
||||
<div class="pose-source-indicator" id="pose-source-indicator">
|
||||
<span class="pose-source-badge pose-source-unknown" id="pose-source-badge">Unknown</span>
|
||||
<p class="pose-source-description" id="pose-source-description">
|
||||
Waiting for first frame...
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="setup-guide-panel">
|
||||
<h4>Setup Guide</h4>
|
||||
<div class="setup-levels">
|
||||
<div class="setup-level">
|
||||
<span class="setup-level-icon">1x</span>
|
||||
<div class="setup-level-info">
|
||||
<strong>1 ESP32 + 1 AP</strong>
|
||||
<p>Presence, breathing, gross motion</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="setup-level">
|
||||
<span class="setup-level-icon">3x</span>
|
||||
<div class="setup-level-info">
|
||||
<strong>2-3 ESP32s</strong>
|
||||
<p>Body localization, motion direction</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="setup-level">
|
||||
<span class="setup-level-icon">4x+</span>
|
||||
<div class="setup-level-info">
|
||||
<strong>4+ ESP32s + trained model</strong>
|
||||
<p>Individual limb tracking, full pose</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p class="setup-note">
|
||||
Signal-Derived mode uses aggregate CSI features.
|
||||
For per-limb tracking, load a trained <code>.rvf</code> model
|
||||
with <code>--model path.rvf</code> and use 4+ sensors.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<div class="health-panel">
|
||||
<h4>System Health</h4>
|
||||
<div class="health-check">
|
||||
@@ -185,8 +229,8 @@ export class LiveDemoTab {
|
||||
flex-direction: column;
|
||||
height: 100%;
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
color: #333;
|
||||
background: #0a0f1a;
|
||||
color: #e0e0e0;
|
||||
}
|
||||
|
||||
.demo-header {
|
||||
@@ -194,10 +238,10 @@ export class LiveDemoTab {
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
padding: 20px 24px;
|
||||
background: rgba(255, 255, 255, 0.95);
|
||||
background: rgba(15, 20, 35, 0.95);
|
||||
backdrop-filter: blur(10px);
|
||||
border-bottom: 1px solid rgba(255, 255, 255, 0.2);
|
||||
box-shadow: 0 2px 20px rgba(0, 0, 0, 0.1);
|
||||
border-bottom: 1px solid rgba(255, 255, 255, 0.08);
|
||||
box-shadow: 0 2px 20px rgba(0, 0, 0, 0.3);
|
||||
position: relative;
|
||||
z-index: 10;
|
||||
}
|
||||
@@ -210,10 +254,10 @@ export class LiveDemoTab {
|
||||
|
||||
.demo-title h2 {
|
||||
margin: 0;
|
||||
color: #333;
|
||||
color: #e0e0e0;
|
||||
font-size: 22px;
|
||||
font-weight: 700;
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
background: linear-gradient(135deg, #667eea 0%, #a78bfa 100%);
|
||||
-webkit-background-clip: text;
|
||||
-webkit-text-fill-color: transparent;
|
||||
background-clip: text;
|
||||
@@ -224,9 +268,9 @@ export class LiveDemoTab {
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
padding: 8px 16px;
|
||||
background: rgba(248, 249, 250, 0.8);
|
||||
background: rgba(30, 40, 60, 0.8);
|
||||
border-radius: 20px;
|
||||
border: 1px solid rgba(222, 226, 230, 0.5);
|
||||
border: 1px solid rgba(255, 255, 255, 0.1);
|
||||
}
|
||||
|
||||
.status-indicator {
|
||||
@@ -260,7 +304,7 @@ export class LiveDemoTab {
|
||||
.status-text {
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
color: #495057;
|
||||
color: #b0b8c8;
|
||||
}
|
||||
|
||||
.demo-controls {
|
||||
@@ -294,19 +338,19 @@ export class LiveDemoTab {
|
||||
|
||||
.btn--primary:hover:not(:disabled) {
|
||||
transform: translateY(-2px);
|
||||
box-shadow: 0 4px 16px rgba(102, 126, 234, 0.3);
|
||||
box-shadow: 0 4px 16px rgba(102, 126, 234, 0.4);
|
||||
}
|
||||
|
||||
.btn--secondary {
|
||||
background: #f8f9fa;
|
||||
color: #495057;
|
||||
border-color: #dee2e6;
|
||||
background: rgba(30, 40, 60, 0.8);
|
||||
color: #b0b8c8;
|
||||
border-color: rgba(255, 255, 255, 0.1);
|
||||
}
|
||||
|
||||
.btn--secondary:hover:not(:disabled) {
|
||||
background: #e9ecef;
|
||||
background: rgba(40, 50, 75, 0.9);
|
||||
transform: translateY(-1px);
|
||||
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.15);
|
||||
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3);
|
||||
}
|
||||
|
||||
.btn:disabled {
|
||||
@@ -324,19 +368,20 @@ export class LiveDemoTab {
|
||||
|
||||
.zone-select {
|
||||
padding: 10px 14px;
|
||||
border: 1px solid #dee2e6;
|
||||
border: 1px solid rgba(255, 255, 255, 0.1);
|
||||
border-radius: 8px;
|
||||
background: white;
|
||||
background: rgba(30, 40, 60, 0.8);
|
||||
color: #b0b8c8;
|
||||
font-size: 14px;
|
||||
cursor: pointer;
|
||||
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1);
|
||||
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.2);
|
||||
transition: all 0.2s ease;
|
||||
}
|
||||
|
||||
.zone-select:focus {
|
||||
outline: none;
|
||||
border-color: #667eea;
|
||||
box-shadow: 0 0 0 3px rgba(102, 126, 234, 0.1);
|
||||
box-shadow: 0 0 0 3px rgba(102, 126, 234, 0.2);
|
||||
}
|
||||
|
||||
.demo-content {
|
||||
@@ -344,18 +389,17 @@ export class LiveDemoTab {
|
||||
flex: 1;
|
||||
gap: 24px;
|
||||
padding: 24px;
|
||||
background: rgba(255, 255, 255, 0.1);
|
||||
backdrop-filter: blur(10px);
|
||||
background: #0a0f1a;
|
||||
}
|
||||
|
||||
.demo-main {
|
||||
flex: 2;
|
||||
min-height: 500px;
|
||||
background: white;
|
||||
background: #111827;
|
||||
border-radius: 12px;
|
||||
overflow: hidden;
|
||||
box-shadow: 0 8px 32px rgba(0, 0, 0, 0.1);
|
||||
border: 1px solid rgba(255, 255, 255, 0.2);
|
||||
box-shadow: 0 8px 32px rgba(0, 0, 0, 0.3);
|
||||
border: 1px solid rgba(255, 255, 255, 0.06);
|
||||
}
|
||||
|
||||
.pose-detection-container {
|
||||
@@ -372,15 +416,15 @@ export class LiveDemoTab {
|
||||
}
|
||||
|
||||
.metrics-panel, .health-panel, .debug-panel {
|
||||
background: #fff;
|
||||
border: 1px solid #ddd;
|
||||
background: rgba(17, 24, 39, 0.9);
|
||||
border: 1px solid rgba(255, 255, 255, 0.08);
|
||||
border-radius: 8px;
|
||||
padding: 15px;
|
||||
}
|
||||
|
||||
.metrics-panel h4, .health-panel h4, .debug-panel h4 {
|
||||
margin: 0 0 15px 0;
|
||||
color: #333;
|
||||
color: #e0e0e0;
|
||||
font-size: 14px;
|
||||
font-weight: 600;
|
||||
}
|
||||
@@ -394,12 +438,12 @@ export class LiveDemoTab {
|
||||
}
|
||||
|
||||
.metric label, .health-check label {
|
||||
color: #666;
|
||||
color: #8899aa;
|
||||
}
|
||||
|
||||
.metric span, .health-check span {
|
||||
font-weight: 500;
|
||||
color: #333;
|
||||
color: #c8d0dc;
|
||||
}
|
||||
|
||||
.debug-actions {
|
||||
@@ -411,18 +455,20 @@ export class LiveDemoTab {
|
||||
|
||||
.debug-info textarea {
|
||||
width: 100%;
|
||||
border: 1px solid #ddd;
|
||||
border: 1px solid rgba(255, 255, 255, 0.1);
|
||||
border-radius: 4px;
|
||||
padding: 8px;
|
||||
font-family: monospace;
|
||||
font-size: 11px;
|
||||
resize: vertical;
|
||||
background: #0a0f1a;
|
||||
color: #c8d0dc;
|
||||
}
|
||||
|
||||
.error-display {
|
||||
background: #f8d7da;
|
||||
color: #721c24;
|
||||
border: 1px solid #f5c6cb;
|
||||
background: rgba(220, 53, 69, 0.15);
|
||||
color: #f5a0a8;
|
||||
border: 1px solid rgba(220, 53, 69, 0.3);
|
||||
border-radius: 4px;
|
||||
padding: 12px;
|
||||
margin: 10px 20px;
|
||||
@@ -432,6 +478,134 @@ export class LiveDemoTab {
|
||||
.health-good { color: #28a745; }
|
||||
.health-poor { color: #ffc107; }
|
||||
.health-bad { color: #dc3545; }
|
||||
|
||||
/* Pose estimation mode indicator */
|
||||
.pose-source-panel {
|
||||
background: rgba(17, 24, 39, 0.9);
|
||||
border: 1px solid rgba(255, 255, 255, 0.08);
|
||||
border-radius: 8px;
|
||||
padding: 15px;
|
||||
}
|
||||
|
||||
.pose-source-panel h4 {
|
||||
margin: 0 0 12px 0;
|
||||
color: #e0e0e0;
|
||||
font-size: 14px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.pose-source-indicator {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.pose-source-badge {
|
||||
display: inline-block;
|
||||
padding: 4px 12px;
|
||||
border-radius: 12px;
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
width: fit-content;
|
||||
}
|
||||
|
||||
.pose-source-unknown {
|
||||
background: rgba(108, 117, 125, 0.15);
|
||||
color: #8899aa;
|
||||
border: 1px solid rgba(108, 117, 125, 0.3);
|
||||
}
|
||||
|
||||
.pose-source-signal {
|
||||
background: rgba(0, 204, 136, 0.12);
|
||||
color: #00cc88;
|
||||
border: 1px solid rgba(0, 204, 136, 0.3);
|
||||
}
|
||||
|
||||
.pose-source-model {
|
||||
background: rgba(102, 126, 234, 0.12);
|
||||
color: #8ea4f0;
|
||||
border: 1px solid rgba(102, 126, 234, 0.3);
|
||||
}
|
||||
|
||||
.pose-source-description {
|
||||
margin: 0;
|
||||
font-size: 11px;
|
||||
color: #8899aa;
|
||||
line-height: 1.4;
|
||||
}
|
||||
|
||||
.setup-guide-panel {
|
||||
background: rgba(17, 24, 39, 0.9);
|
||||
border: 1px solid rgba(255, 255, 255, 0.08);
|
||||
border-radius: 8px;
|
||||
padding: 15px;
|
||||
}
|
||||
|
||||
.setup-guide-panel h4 {
|
||||
margin: 0 0 12px 0;
|
||||
color: #e0e0e0;
|
||||
font-size: 14px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.setup-levels {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.setup-level {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 10px;
|
||||
padding: 8px;
|
||||
border-radius: 6px;
|
||||
background: rgba(30, 40, 60, 0.6);
|
||||
border: 1px solid rgba(255, 255, 255, 0.06);
|
||||
}
|
||||
|
||||
.setup-level-icon {
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
color: white;
|
||||
font-size: 11px;
|
||||
font-weight: 700;
|
||||
width: 32px;
|
||||
height: 32px;
|
||||
border-radius: 50%;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.setup-level-info strong {
|
||||
font-size: 12px;
|
||||
color: #c8d0dc;
|
||||
display: block;
|
||||
}
|
||||
|
||||
.setup-level-info p {
|
||||
margin: 2px 0 0;
|
||||
font-size: 11px;
|
||||
color: #8899aa;
|
||||
}
|
||||
|
||||
.setup-note {
|
||||
margin: 10px 0 0;
|
||||
font-size: 11px;
|
||||
color: #6b7a8d;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.setup-note code {
|
||||
background: rgba(102, 126, 234, 0.12);
|
||||
color: #8ea4f0;
|
||||
padding: 1px 4px;
|
||||
border-radius: 3px;
|
||||
font-size: 10px;
|
||||
}
|
||||
`;
|
||||
|
||||
if (!document.querySelector('#live-demo-enhanced-styles')) {
|
||||
@@ -545,7 +719,11 @@ export class LiveDemoTab {
|
||||
handlePoseUpdate(data) {
|
||||
this.metrics.frameCount++;
|
||||
this.metrics.lastUpdate = Date.now();
|
||||
this.updateDebugOutput(`Pose update: ${data.persons?.length || 0} persons detected`);
|
||||
// Update pose source indicator if the backend supplies it
|
||||
if (data.pose_source && data.pose_source !== this.state.poseSource) {
|
||||
this.setState({ poseSource: data.pose_source });
|
||||
}
|
||||
this.updateDebugOutput(`Pose update: ${data.persons?.length || 0} persons detected (${data.pose_source || 'unknown'})`);
|
||||
}
|
||||
|
||||
handleCanvasError(error) {
|
||||
@@ -706,6 +884,7 @@ export class LiveDemoTab {
|
||||
this.updateStatusIndicator();
|
||||
this.updateControls();
|
||||
this.updateMetricsDisplay();
|
||||
this.updatePoseSourceIndicator();
|
||||
}
|
||||
|
||||
updateStatusIndicator() {
|
||||
@@ -789,6 +968,33 @@ export class LiveDemoTab {
|
||||
}
|
||||
}
|
||||
|
||||
updatePoseSourceIndicator() {
|
||||
const badge = this.container.querySelector('#pose-source-badge');
|
||||
const description = this.container.querySelector('#pose-source-description');
|
||||
|
||||
if (!badge || !description) return;
|
||||
|
||||
const source = this.state.poseSource;
|
||||
|
||||
if (source === 'model_inference') {
|
||||
badge.className = 'pose-source-badge pose-source-model';
|
||||
badge.textContent = 'Model Inference';
|
||||
description.textContent =
|
||||
'Pose is estimated by a trained neural network ' +
|
||||
'loaded from an RVF container.';
|
||||
} else if (source === 'signal_derived') {
|
||||
badge.className = 'pose-source-badge pose-source-signal';
|
||||
badge.textContent = 'Signal-Derived';
|
||||
description.textContent =
|
||||
'Keypoints are derived from live CSI signal features ' +
|
||||
'(motion power, breathing rate, variance).';
|
||||
} else {
|
||||
badge.className = 'pose-source-badge pose-source-unknown';
|
||||
badge.textContent = 'Unknown';
|
||||
description.textContent = 'Waiting for first frame...';
|
||||
}
|
||||
}
|
||||
|
||||
getHealthClass(status) {
|
||||
switch (status) {
|
||||
case 'connected': return 'good';
|
||||
|
||||
@@ -89,21 +89,17 @@ export class PoseDetectionCanvas {
|
||||
</div>
|
||||
</div>
|
||||
<div class="pose-canvas-controls" id="controls-${this.containerId}">
|
||||
<div class="control-group primary-controls">
|
||||
<button class="btn btn-start" id="start-btn-${this.containerId}">Start</button>
|
||||
<button class="btn btn-stop" id="stop-btn-${this.containerId}" disabled>Stop</button>
|
||||
<button class="btn btn-reconnect" id="reconnect-btn-${this.containerId}" disabled>Reconnect</button>
|
||||
<button class="btn btn-demo" id="demo-btn-${this.containerId}">Demo</button>
|
||||
</div>
|
||||
<div class="control-group secondary-controls">
|
||||
<select class="mode-select" id="mode-select-${this.containerId}">
|
||||
<option value="skeleton">Skeleton</option>
|
||||
<option value="keypoints">Keypoints</option>
|
||||
<option value="heatmap">Heatmap</option>
|
||||
<option value="dense">Dense</option>
|
||||
</select>
|
||||
<button class="btn btn-settings" id="settings-btn-${this.containerId}">⚙️ Settings</button>
|
||||
</div>
|
||||
<button class="btn btn-start" id="start-btn-${this.containerId}">▶ Start</button>
|
||||
<button class="btn btn-stop" id="stop-btn-${this.containerId}" disabled>■ Stop</button>
|
||||
<button class="btn btn-reconnect" id="reconnect-btn-${this.containerId}" disabled>↻ Reconnect</button>
|
||||
<button class="btn btn-demo" id="demo-btn-${this.containerId}">⚙ Demo</button>
|
||||
<select class="mode-select" id="mode-select-${this.containerId}">
|
||||
<option value="skeleton">Skeleton</option>
|
||||
<option value="keypoints">Keypoints</option>
|
||||
<option value="heatmap">Heatmap</option>
|
||||
<option value="dense">Dense</option>
|
||||
</select>
|
||||
<button class="btn btn-settings" id="settings-btn-${this.containerId}">⚙ Settings</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="pose-canvas-container">
|
||||
@@ -124,20 +120,20 @@ export class PoseDetectionCanvas {
|
||||
const style = document.createElement('style');
|
||||
style.textContent = `
|
||||
.pose-detection-canvas-wrapper {
|
||||
border: 1px solid #ddd;
|
||||
border: 1px solid rgba(255, 255, 255, 0.06);
|
||||
border-radius: 8px;
|
||||
overflow: hidden;
|
||||
background: #f9f9f9;
|
||||
font-family: Arial, sans-serif;
|
||||
background: #0d1117;
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
}
|
||||
|
||||
.pose-canvas-header {
|
||||
display: flex;
|
||||
justify-content: space-between;
|
||||
align-items: center;
|
||||
padding: 10px 15px;
|
||||
background: #f0f0f0;
|
||||
border-bottom: 1px solid #ddd;
|
||||
padding: 12px 16px;
|
||||
background: rgba(15, 20, 35, 0.95);
|
||||
border-bottom: 1px solid rgba(255, 255, 255, 0.06);
|
||||
}
|
||||
|
||||
.pose-canvas-title {
|
||||
@@ -148,156 +144,166 @@ export class PoseDetectionCanvas {
|
||||
|
||||
.pose-canvas-title h3 {
|
||||
margin: 0;
|
||||
color: #333;
|
||||
color: #e0e0e0;
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.connection-status {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 5px;
|
||||
gap: 6px;
|
||||
padding: 4px 10px;
|
||||
background: rgba(30, 40, 60, 0.6);
|
||||
border-radius: 12px;
|
||||
border: 1px solid rgba(255, 255, 255, 0.06);
|
||||
}
|
||||
|
||||
.status-indicator {
|
||||
width: 10px;
|
||||
height: 10px;
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
border-radius: 50%;
|
||||
background: #ccc;
|
||||
background: #4a5568;
|
||||
transition: background-color 0.3s;
|
||||
}
|
||||
|
||||
.status-indicator.connected { background: #28a745; }
|
||||
.status-indicator.connecting { background: #ffc107; }
|
||||
.status-indicator.error { background: #dc3545; }
|
||||
.status-indicator.disconnected { background: #6c757d; }
|
||||
.status-indicator.connected { background: #00cc88; box-shadow: 0 0 6px rgba(0, 204, 136, 0.5); }
|
||||
.status-indicator.connecting { background: #fbbf24; box-shadow: 0 0 6px rgba(251, 191, 36, 0.5); animation: pulse 1.5s ease-in-out infinite; }
|
||||
.status-indicator.error { background: #ef4444; box-shadow: 0 0 6px rgba(239, 68, 68, 0.5); }
|
||||
.status-indicator.disconnected { background: #4a5568; }
|
||||
|
||||
.status-text {
|
||||
font-size: 12px;
|
||||
color: #666;
|
||||
min-width: 80px;
|
||||
font-size: 11px;
|
||||
color: #8899aa;
|
||||
min-width: 70px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.pose-canvas-controls {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
gap: 15px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.control-group {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.primary-controls {
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.secondary-controls {
|
||||
flex-shrink: 0;
|
||||
flex-wrap: nowrap;
|
||||
}
|
||||
|
||||
.btn {
|
||||
padding: 8px 16px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 6px;
|
||||
background: #ffffff;
|
||||
color: #333333;
|
||||
border: 1px solid rgba(255, 255, 255, 0.1);
|
||||
border-radius: 8px;
|
||||
background: rgba(30, 40, 60, 0.8);
|
||||
color: #c8d0dc;
|
||||
cursor: pointer;
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
transition: all 0.2s ease;
|
||||
box-shadow: 0 1px 3px rgba(0,0,0,0.1);
|
||||
box-shadow: 0 2px 4px rgba(0, 0, 0, 0.2);
|
||||
text-decoration: none;
|
||||
display: inline-block;
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
min-width: 80px;
|
||||
text-align: center;
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
.btn:hover:not(:disabled) {
|
||||
background: #f8f9fa;
|
||||
border-color: #adb5bd;
|
||||
box-shadow: 0 2px 6px rgba(0,0,0,0.15);
|
||||
transform: translateY(-1px);
|
||||
box-shadow: 0 4px 8px rgba(0, 0, 0, 0.3);
|
||||
}
|
||||
|
||||
.btn:active:not(:disabled) {
|
||||
transform: translateY(0);
|
||||
box-shadow: 0 1px 3px rgba(0,0,0,0.1);
|
||||
box-shadow: 0 1px 3px rgba(0, 0, 0, 0.2);
|
||||
}
|
||||
|
||||
.btn:disabled {
|
||||
opacity: 0.6;
|
||||
opacity: 0.35;
|
||||
cursor: not-allowed;
|
||||
background: #e9ecef;
|
||||
color: #6c757d;
|
||||
background: rgba(20, 30, 50, 0.6);
|
||||
color: #4a5568;
|
||||
transform: none !important;
|
||||
box-shadow: none !important;
|
||||
}
|
||||
|
||||
.btn-start {
|
||||
background: #28a745;
|
||||
color: white;
|
||||
border-color: #28a745;
|
||||
.btn-start {
|
||||
background: rgba(0, 204, 136, 0.15);
|
||||
color: #00cc88;
|
||||
border-color: rgba(0, 204, 136, 0.3);
|
||||
}
|
||||
|
||||
.btn-start:hover:not(:disabled) {
|
||||
background: #218838;
|
||||
border-color: #1e7e34;
|
||||
.btn-start:hover:not(:disabled) {
|
||||
background: rgba(0, 204, 136, 0.25);
|
||||
border-color: rgba(0, 204, 136, 0.5);
|
||||
box-shadow: 0 4px 12px rgba(0, 204, 136, 0.2);
|
||||
}
|
||||
|
||||
.btn-stop {
|
||||
background: #dc3545;
|
||||
color: white;
|
||||
border-color: #dc3545;
|
||||
.btn-stop {
|
||||
background: rgba(239, 68, 68, 0.15);
|
||||
color: #ef4444;
|
||||
border-color: rgba(239, 68, 68, 0.3);
|
||||
}
|
||||
|
||||
.btn-stop:hover:not(:disabled) {
|
||||
background: #c82333;
|
||||
border-color: #bd2130;
|
||||
.btn-stop:hover:not(:disabled) {
|
||||
background: rgba(239, 68, 68, 0.25);
|
||||
border-color: rgba(239, 68, 68, 0.5);
|
||||
box-shadow: 0 4px 12px rgba(239, 68, 68, 0.2);
|
||||
}
|
||||
|
||||
.btn-reconnect {
|
||||
background: #17a2b8;
|
||||
color: white;
|
||||
border-color: #17a2b8;
|
||||
.btn-reconnect {
|
||||
background: rgba(59, 130, 246, 0.15);
|
||||
color: #60a5fa;
|
||||
border-color: rgba(59, 130, 246, 0.3);
|
||||
}
|
||||
|
||||
.btn-reconnect:hover:not(:disabled) {
|
||||
background: #138496;
|
||||
border-color: #117a8b;
|
||||
.btn-reconnect:hover:not(:disabled) {
|
||||
background: rgba(59, 130, 246, 0.25);
|
||||
border-color: rgba(59, 130, 246, 0.5);
|
||||
box-shadow: 0 4px 12px rgba(59, 130, 246, 0.2);
|
||||
}
|
||||
|
||||
.btn-demo {
|
||||
background: #6f42c1;
|
||||
color: white;
|
||||
border-color: #6f42c1;
|
||||
.btn-demo {
|
||||
background: rgba(139, 92, 246, 0.15);
|
||||
color: #a78bfa;
|
||||
border-color: rgba(139, 92, 246, 0.3);
|
||||
}
|
||||
|
||||
.btn-demo:hover:not(:disabled) {
|
||||
background: #5a32a3;
|
||||
border-color: #512a97;
|
||||
.btn-demo:hover:not(:disabled) {
|
||||
background: rgba(139, 92, 246, 0.25);
|
||||
border-color: rgba(139, 92, 246, 0.5);
|
||||
box-shadow: 0 4px 12px rgba(139, 92, 246, 0.2);
|
||||
}
|
||||
|
||||
.btn-settings {
|
||||
background: #6c757d;
|
||||
color: white;
|
||||
border-color: #6c757d;
|
||||
.btn-settings {
|
||||
background: rgba(100, 116, 139, 0.15);
|
||||
color: #94a3b8;
|
||||
border-color: rgba(100, 116, 139, 0.3);
|
||||
}
|
||||
|
||||
.btn-settings:hover:not(:disabled) {
|
||||
background: #5a6268;
|
||||
border-color: #545b62;
|
||||
.btn-settings:hover:not(:disabled) {
|
||||
background: rgba(100, 116, 139, 0.25);
|
||||
border-color: rgba(100, 116, 139, 0.5);
|
||||
}
|
||||
|
||||
.mode-select {
|
||||
padding: 5px 8px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 4px;
|
||||
background: #fff;
|
||||
font-size: 12px;
|
||||
padding: 8px 12px;
|
||||
border: 1px solid rgba(255, 255, 255, 0.1);
|
||||
border-radius: 8px;
|
||||
background: rgba(30, 40, 60, 0.8);
|
||||
color: #b0b8c8;
|
||||
font-size: 13px;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.mode-select:focus {
|
||||
outline: none;
|
||||
border-color: rgba(139, 92, 246, 0.5);
|
||||
box-shadow: 0 0 0 2px rgba(139, 92, 246, 0.15);
|
||||
}
|
||||
|
||||
.mode-select option {
|
||||
background: #1a2234;
|
||||
color: #c8d0dc;
|
||||
}
|
||||
|
||||
.pose-canvas-container {
|
||||
|
||||
@@ -33,6 +33,13 @@ export class SensingTab {
|
||||
_buildDOM() {
|
||||
this.container.innerHTML = `
|
||||
<h2>Live WiFi Sensing</h2>
|
||||
|
||||
<!-- Data-source status banner — updated by _onStateChange -->
|
||||
<div id="sensingSourceBanner" class="sensing-source-banner sensing-source-reconnecting"
|
||||
role="status" aria-live="polite">
|
||||
RECONNECTING...
|
||||
</div>
|
||||
|
||||
<div class="sensing-layout">
|
||||
<!-- 3D viewport -->
|
||||
<div class="sensing-viewport" id="sensingViewport">
|
||||
@@ -98,6 +105,17 @@ export class SensingTab {
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Setup info -->
|
||||
<div class="sensing-card">
|
||||
<div class="sensing-card-title">About This Data</div>
|
||||
<p class="sensing-about-text">
|
||||
Metrics are computed from WiFi Channel State Information (CSI).
|
||||
With <strong>1 ESP32</strong> you get presence detection, breathing
|
||||
estimation, and gross motion. Add <strong>3-4+ ESP32 nodes</strong>
|
||||
around the room for spatial resolution and limb-level tracking.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<!-- Extra info -->
|
||||
<div class="sensing-card">
|
||||
<div class="sensing-card-title">Details</div>
|
||||
@@ -178,19 +196,34 @@ export class SensingTab {
|
||||
}
|
||||
|
||||
_onStateChange(state) {
|
||||
const dot = this.container.querySelector('#sensingDot');
|
||||
const text = this.container.querySelector('#sensingState');
|
||||
if (!dot || !text) return;
|
||||
const dot = this.container.querySelector('#sensingDot');
|
||||
const text = this.container.querySelector('#sensingState');
|
||||
const banner = this.container.querySelector('#sensingSourceBanner');
|
||||
|
||||
const labels = {
|
||||
disconnected: 'Disconnected',
|
||||
connecting: 'Connecting...',
|
||||
connected: 'Connected',
|
||||
simulated: 'Simulated',
|
||||
};
|
||||
if (dot && text) {
|
||||
const stateLabels = {
|
||||
disconnected: 'Disconnected',
|
||||
connecting: 'Connecting...',
|
||||
connected: 'Connected',
|
||||
reconnecting: 'Reconnecting...',
|
||||
simulated: 'Simulated',
|
||||
};
|
||||
dot.className = 'sensing-dot ' + state;
|
||||
text.textContent = stateLabels[state] || state;
|
||||
}
|
||||
|
||||
dot.className = 'sensing-dot ' + state;
|
||||
text.textContent = labels[state] || state;
|
||||
if (banner) {
|
||||
// Map the service's dataSource to banner text and CSS modifier class.
|
||||
const dataSource = sensingService.dataSource;
|
||||
const bannerConfig = {
|
||||
live: { text: 'LIVE - ESP32', cls: 'sensing-source-live' },
|
||||
reconnecting: { text: 'RECONNECTING...', cls: 'sensing-source-reconnecting' },
|
||||
simulated: { text: 'SIMULATED DATA', cls: 'sensing-source-simulated' },
|
||||
};
|
||||
const cfg = bannerConfig[dataSource] || bannerConfig.reconnecting;
|
||||
banner.textContent = cfg.text;
|
||||
banner.className = 'sensing-source-banner ' + cfg.cls;
|
||||
}
|
||||
}
|
||||
|
||||
// ---- HUD update --------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
EXPO_PUBLIC_DEFAULT_SERVER_URL=http://192.168.1.100:8080
|
||||
@@ -0,0 +1,26 @@
|
||||
module.exports = {
|
||||
root: true,
|
||||
parser: '@typescript-eslint/parser',
|
||||
parserOptions: {
|
||||
ecmaVersion: 'latest',
|
||||
sourceType: 'module',
|
||||
ecmaFeatures: {
|
||||
jsx: true,
|
||||
},
|
||||
},
|
||||
plugins: ['@typescript-eslint', 'react', 'react-hooks'],
|
||||
extends: [
|
||||
'eslint:recommended',
|
||||
'plugin:react/recommended',
|
||||
'plugin:react-hooks/recommended',
|
||||
'plugin:@typescript-eslint/recommended',
|
||||
],
|
||||
settings: {
|
||||
react: {
|
||||
version: 'detect',
|
||||
},
|
||||
},
|
||||
rules: {
|
||||
'react/react-in-jsx-scope': 'off',
|
||||
},
|
||||
};
|
||||
@@ -0,0 +1,41 @@
|
||||
# Learn more https://docs.github.com/en/get-started/getting-started-with-git/ignoring-files
|
||||
|
||||
# dependencies
|
||||
node_modules/
|
||||
|
||||
# Expo
|
||||
.expo/
|
||||
dist/
|
||||
web-build/
|
||||
expo-env.d.ts
|
||||
|
||||
# Native
|
||||
.kotlin/
|
||||
*.orig.*
|
||||
*.jks
|
||||
*.p8
|
||||
*.p12
|
||||
*.key
|
||||
*.mobileprovision
|
||||
|
||||
# Metro
|
||||
.metro-health-check*
|
||||
|
||||
# debug
|
||||
npm-debug.*
|
||||
yarn-debug.*
|
||||
yarn-error.*
|
||||
|
||||
# macOS
|
||||
.DS_Store
|
||||
*.pem
|
||||
|
||||
# local env files
|
||||
.env*.local
|
||||
|
||||
# typescript
|
||||
*.tsbuildinfo
|
||||
|
||||
# generated native folders
|
||||
/ios
|
||||
/android
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"singleQuote": true,
|
||||
"trailingComma": "all"
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
import { useEffect } from 'react';
|
||||
import { NavigationContainer, DarkTheme } from '@react-navigation/native';
|
||||
import { GestureHandlerRootView } from 'react-native-gesture-handler';
|
||||
import { StatusBar } from 'expo-status-bar';
|
||||
import { SafeAreaProvider } from 'react-native-safe-area-context';
|
||||
import { apiService } from '@/services/api.service';
|
||||
import { rssiService } from '@/services/rssi.service';
|
||||
import { wsService } from '@/services/ws.service';
|
||||
import { ThemeProvider } from './src/theme/ThemeContext';
|
||||
import { usePoseStore } from './src/stores/poseStore';
|
||||
import { useSettingsStore } from './src/stores/settingsStore';
|
||||
import { RootNavigator } from './src/navigation/RootNavigator';
|
||||
|
||||
export default function App() {
|
||||
const serverUrl = useSettingsStore((state) => state.serverUrl);
|
||||
const rssiScanEnabled = useSettingsStore((state) => state.rssiScanEnabled);
|
||||
|
||||
useEffect(() => {
|
||||
apiService.setBaseUrl(serverUrl);
|
||||
const unsubscribe = wsService.subscribe(usePoseStore.getState().handleFrame);
|
||||
wsService.connect(serverUrl);
|
||||
|
||||
return () => {
|
||||
unsubscribe();
|
||||
wsService.disconnect();
|
||||
};
|
||||
}, [serverUrl]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!rssiScanEnabled) {
|
||||
rssiService.stopScanning();
|
||||
return;
|
||||
}
|
||||
|
||||
const unsubscribe = rssiService.subscribe(() => {
|
||||
// Consumers can subscribe elsewhere for RSSI events.
|
||||
});
|
||||
rssiService.startScanning(2000);
|
||||
|
||||
return () => {
|
||||
unsubscribe();
|
||||
rssiService.stopScanning();
|
||||
};
|
||||
}, [rssiScanEnabled]);
|
||||
|
||||
useEffect(() => {
|
||||
(globalThis as { __appStartTime?: number }).__appStartTime = Date.now();
|
||||
}, []);
|
||||
|
||||
const navigationTheme = {
|
||||
...DarkTheme,
|
||||
colors: {
|
||||
...DarkTheme.colors,
|
||||
background: '#0A0E1A',
|
||||
card: '#0D1117',
|
||||
text: '#E2E8F0',
|
||||
border: '#1E293B',
|
||||
primary: '#32B8C6',
|
||||
},
|
||||
};
|
||||
|
||||
return (
|
||||
<GestureHandlerRootView style={{ flex: 1 }}>
|
||||
<SafeAreaProvider>
|
||||
<ThemeProvider>
|
||||
<NavigationContainer theme={navigationTheme}>
|
||||
<RootNavigator />
|
||||
</NavigationContainer>
|
||||
</ThemeProvider>
|
||||
</SafeAreaProvider>
|
||||
<StatusBar style="light" />
|
||||
</GestureHandlerRootView>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,412 @@
|
||||
# WiFi-DensePose Mobile
|
||||
|
||||
**See through walls from your phone.** Real-time WiFi sensing, vital signs, and disaster response — in a cross-platform mobile app.
|
||||
|
||||
WiFi-DensePose Mobile is a React Native / Expo companion app for the [WiFi-DensePose](../../README.md) sensing platform. It connects to a WiFi sensing server over WebSocket, renders live 3D Gaussian splat visualizations of detected humans, displays breathing and heart rate in real time, and provides a full WiFi-MAT disaster triage dashboard — all from a single codebase that runs on iOS, Android, and Web.
|
||||
|
||||
> | Screen | What It Shows |
|
||||
> |--------|---------------|
|
||||
> | **Live** | 3D Gaussian splat body rendering with FPS counter, signal strength, confidence HUD |
|
||||
> | **Vitals** | Breathing rate (6-30 BPM) and heart rate (40-120 BPM) arc gauges with sparkline history |
|
||||
> | **Zones** | SVG floor plan with occupancy grid, zone legend, presence heatmap |
|
||||
> | **MAT** | Mass casualty assessment: survivor counter, triage alerts, zone management |
|
||||
> | **Settings** | Server URL, theme picker, RSSI-only toggle, alert sound control |
|
||||
|
||||
```bash
|
||||
# Quick start — web preview in 30 seconds
|
||||
cd ui/mobile
|
||||
npm install
|
||||
npx expo start --web
|
||||
```
|
||||
|
||||
<!-- Screenshot placeholder: replace with actual app screenshots -->
|
||||
<!--  -->
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
| | Feature | Details |
|
||||
|---|---------|---------|
|
||||
| **3D Live View** | Gaussian splat rendering | Three.js via WebView (native) or iframe (web), real-time pose overlay |
|
||||
| **Vital Signs** | Breathing + heart rate | Arc gauge components with sparkline 60-sample history, confidence indicators |
|
||||
| **Disaster Response** | WiFi-MAT dashboard | Survivor detection, START triage classification, priority alerts, zone scan tracking |
|
||||
| **Floor Plan** | SVG occupancy grid | Zone-level presence visualization, color-coded density, interactive legend |
|
||||
| **Cross-Platform** | iOS, Android, Web | Expo SDK 55, React Native 0.83, single codebase with platform-specific modules |
|
||||
| **Offline Capable** | Automatic simulation fallback | When the sensing server is unreachable, generates synthetic data so the UI stays functional |
|
||||
| **RSSI Mode** | No CSI hardware needed | Toggle RSSI-only scanning for coarse presence detection on consumer WiFi devices |
|
||||
| **Dark Theme** | Cyan accent (#32B8C6) | Dark-first design system with consistent color tokens, spacing scale, and monospace typography |
|
||||
| **Persistent State** | Zustand + AsyncStorage | Settings, connection preferences, and theme survive app restarts |
|
||||
| **Platform WiFi** | Native RSSI scanning | Android: `react-native-wifi-reborn`, iOS: stub (requires entitlement), Web: synthetic values |
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Requirement | Version | Notes |
|
||||
|-------------|---------|-------|
|
||||
| Node.js | 18+ | LTS recommended |
|
||||
| npm | 9+ | Ships with Node.js 18+ |
|
||||
| Expo CLI | Latest | Installed automatically via `npx` |
|
||||
| iOS Simulator | Xcode 15+ | macOS only; optional for iOS development |
|
||||
| Android Emulator | API 33+ | Android Studio; optional for Android development |
|
||||
| WiFi-DensePose Server | Any | Optional — app falls back to simulated data without a server |
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Web (fastest)
|
||||
|
||||
```bash
|
||||
cd ui/mobile
|
||||
npm install
|
||||
npx expo start --web
|
||||
```
|
||||
|
||||
Open `http://localhost:8081` in your browser. The app starts in simulation mode with synthetic pose and vital sign data.
|
||||
|
||||
### Android
|
||||
|
||||
```bash
|
||||
cd ui/mobile
|
||||
npm install
|
||||
npx expo start --android
|
||||
```
|
||||
|
||||
Requires Android Studio with an emulator running, or a physical device with Expo Go installed.
|
||||
|
||||
### iOS
|
||||
|
||||
```bash
|
||||
cd ui/mobile
|
||||
npm install
|
||||
npx expo start --ios
|
||||
```
|
||||
|
||||
Requires Xcode with a simulator, or a physical device with Expo Go. RSSI scanning on iOS requires the `com.apple.developer.networking.wifi-info` entitlement.
|
||||
|
||||
---
|
||||
|
||||
## Connecting to a Sensing Server
|
||||
|
||||
The app connects to the WiFi-DensePose sensing server over WebSocket for live data. Configure the server URL in the **Settings** tab.
|
||||
|
||||
| Server Location | URL | Notes |
|
||||
|----------------|-----|-------|
|
||||
| Local dev server | `http://localhost:3000` | Default; sensing WS auto-connects on port 3001 |
|
||||
| Docker container | `http://host.docker.internal:3000` | From emulator connecting to host Docker |
|
||||
| ESP32 mesh | `http://<esp32-ip>:3000` | Direct connection to ESP32 aggregator |
|
||||
| Remote server | `https://your-server.example.com` | TLS supported; WebSocket upgrades to `wss://` |
|
||||
|
||||
When the server is unreachable, the app automatically falls back to **simulation mode** after exhausting reconnect attempts (exponential backoff). A yellow `SIM` badge appears in the connection banner. Reconnection resumes automatically when the server becomes available.
|
||||
|
||||
---
|
||||
|
||||
<details>
|
||||
<summary><strong>Architecture</strong></summary>
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
ui/mobile/
|
||||
App.tsx Root component (providers, navigation, services)
|
||||
app.config.ts Expo configuration
|
||||
index.ts Entry point
|
||||
src/
|
||||
components/
|
||||
ConnectionBanner.tsx Server status banner (connected/simulated/disconnected)
|
||||
ErrorBoundary.tsx Crash boundary with fallback UI
|
||||
GaugeArc.tsx SVG arc gauge for vital sign display
|
||||
HudOverlay.tsx Heads-up display overlay
|
||||
LoadingSpinner.tsx Themed loading indicator
|
||||
ModeBadge.tsx LIVE / SIM / RSSI mode indicator
|
||||
OccupancyGrid.tsx Grid-based occupancy visualization
|
||||
SignalBar.tsx RSSI signal strength bars
|
||||
SparklineChart.tsx Mini sparkline for metric history
|
||||
StatusDot.tsx Connection status indicator dot
|
||||
ThemedText.tsx Text component with theme presets
|
||||
ThemedView.tsx View component with theme background
|
||||
constants/
|
||||
api.ts REST API path constants
|
||||
simulation.ts Simulation tick interval, defaults
|
||||
websocket.ts WS path, reconnect delays, max attempts
|
||||
hooks/
|
||||
usePoseStream.ts Subscribe to live or simulated sensing frames
|
||||
useRssiScanner.ts Platform RSSI scanning hook
|
||||
useServerReachability.ts HTTP health check polling
|
||||
useTheme.ts Dark/light/system theme resolution
|
||||
useWebViewBridge.ts WebView message bridge for Gaussian viewer
|
||||
navigation/
|
||||
MainTabs.tsx Bottom tab navigator (5 tabs with lazy loading)
|
||||
RootNavigator.tsx Root stack navigator
|
||||
types.ts Navigation param list types
|
||||
screens/
|
||||
LiveScreen/
|
||||
index.tsx 3D Gaussian splat view with HUD overlay
|
||||
GaussianSplatWebView.tsx Native WebView renderer (Three.js)
|
||||
GaussianSplatWebView.web.tsx Web iframe renderer
|
||||
LiveHUD.tsx FPS, RSSI, confidence, person count overlay
|
||||
useGaussianBridge.ts WebView message protocol
|
||||
VitalsScreen/
|
||||
index.tsx Breathing + heart rate dashboard
|
||||
BreathingGauge.tsx Arc gauge for breathing BPM
|
||||
HeartRateGauge.tsx Arc gauge for heart rate BPM
|
||||
MetricCard.tsx Vital sign metric card with sparkline
|
||||
ZonesScreen/
|
||||
index.tsx Floor plan occupancy view
|
||||
FloorPlanSvg.tsx SVG floor plan renderer
|
||||
useOccupancyGrid.ts Grid computation from sensing frames
|
||||
ZoneLegend.tsx Color-coded zone legend
|
||||
MATScreen/
|
||||
index.tsx Mass casualty assessment dashboard
|
||||
AlertCard.tsx Single triage alert card
|
||||
AlertList.tsx Scrollable alert list with priority sorting
|
||||
MatWebView.tsx MAT visualization WebView
|
||||
SurvivorCounter.tsx Survivor count by triage status
|
||||
useMatBridge.ts MAT WebView message protocol
|
||||
SettingsScreen/
|
||||
index.tsx App settings panel
|
||||
ServerUrlInput.tsx Server URL text input with validation
|
||||
RssiToggle.tsx RSSI-only mode switch
|
||||
ThemePicker.tsx Dark / light / system theme selector
|
||||
services/
|
||||
ws.service.ts WebSocket client with auto-reconnect + simulation fallback
|
||||
api.service.ts REST client (Axios) with retry logic
|
||||
rssi.service.ts Platform-agnostic RSSI scanner interface
|
||||
rssi.service.android.ts Android: react-native-wifi-reborn integration
|
||||
rssi.service.ios.ts iOS: stub (requires entitlement)
|
||||
rssi.service.web.ts Web: synthetic RSSI values
|
||||
simulation.service.ts Generates synthetic SensingFrame data
|
||||
stores/
|
||||
poseStore.ts Pose frames, connection status, frame history (Zustand)
|
||||
matStore.ts MAT survivors, zones, alerts, disaster events (Zustand)
|
||||
settingsStore.ts Server URL, theme, RSSI toggle (Zustand + persist)
|
||||
theme/
|
||||
colors.ts Color tokens (bg, surface, accent, danger, etc.)
|
||||
spacing.ts 4px-based spacing scale
|
||||
typography.ts Font families and size presets
|
||||
ThemeContext.tsx React context provider for theme
|
||||
index.ts Theme barrel export
|
||||
types/
|
||||
sensing.ts SensingFrame, SensingNode, VitalsData, Classification
|
||||
mat.ts Survivor, Alert, ScanZone, TriageStatus, DisasterType
|
||||
api.ts PoseStatus, ZoneConfig, HistoricalFrames, ApiError
|
||||
navigation.ts Navigation param lists
|
||||
utils/
|
||||
colorMap.ts Value-to-color mapping for heatmaps
|
||||
formatters.ts Number and date formatting utilities
|
||||
ringBuffer.ts Fixed-size circular buffer for frame history
|
||||
urlValidator.ts Server URL validation
|
||||
e2e/ Maestro end-to-end test specs
|
||||
assets/ App icons and images
|
||||
```
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
WiFi Sensing Server (Rust/Axum)
|
||||
|
|
||||
| WebSocket (ws://host:3001/ws/sensing)
|
||||
v
|
||||
ws.service.ts -----> [auto-reconnect with exponential backoff]
|
||||
| |
|
||||
| SensingFrame | (server unreachable)
|
||||
v v
|
||||
poseStore.ts simulation.service.ts
|
||||
| |
|
||||
| Zustand state | synthetic SensingFrame
|
||||
v v
|
||||
usePoseStream.ts <----------+
|
||||
|
|
||||
+---> LiveScreen (3D Gaussian splat + HUD)
|
||||
+---> VitalsScreen (breathing + heart rate gauges)
|
||||
+---> ZonesScreen (floor plan occupancy grid)
|
||||
|
||||
api.service.ts -----> REST API (GET /api/pose/status, /zones, /frames)
|
||||
|
|
||||
v
|
||||
matStore.ts -----> MATScreen (survivor counter, alerts, zones)
|
||||
|
||||
rssi.service.ts -----> Platform WiFi scan (Android / iOS / Web)
|
||||
|
|
||||
v
|
||||
useRssiScanner.ts -----> LiveScreen HUD (signal bars)
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
<details>
|
||||
<summary><strong>Screens</strong></summary>
|
||||
|
||||
### Live
|
||||
|
||||
The primary visualization screen. Renders a 3D Gaussian splat representation of detected humans using Three.js. On native platforms, the renderer runs inside a WebView; on web, it uses an iframe. A heads-up display overlays connection status, FPS, RSSI signal strength, detection confidence, and person count. Supports three modes: **LIVE** (connected to server), **SIM** (simulation fallback), and **RSSI** (RSSI-only scanning).
|
||||
|
||||
### Vitals
|
||||
|
||||
Displays real-time breathing rate and heart rate extracted from CSI signal processing. Each vital sign is shown as an animated arc gauge (`GaugeArc` component) with the current BPM value, a 60-sample sparkline history (`SparklineChart`), and a confidence percentage. Normal ranges: breathing 6-30 BPM, heart rate 40-120 BPM.
|
||||
|
||||
### Zones
|
||||
|
||||
A floor plan view that maps WiFi sensing coverage to physical space. Uses SVG rendering (`react-native-svg`) to draw zones with color-coded occupancy density. The `useOccupancyGrid` hook computes grid cell values from incoming sensing frames. A legend shows the color scale from empty to high-density zones.
|
||||
|
||||
### MAT
|
||||
|
||||
Mass Casualty Assessment Tool for disaster response. Displays a survivor counter grouped by START triage classification (Immediate / Delayed / Minor / Deceased), a scrollable alert list sorted by priority, and zone scan progress. Each alert card shows the survivor location, recommended action, and triage color. The MAT tab badge shows the active alert count.
|
||||
|
||||
### Settings
|
||||
|
||||
Configuration panel with four controls:
|
||||
- **Server URL** — text input with URL validation; changes trigger WebSocket reconnect
|
||||
- **Theme** — dark / light / system picker
|
||||
- **RSSI Scanning** — toggle for platform-native WiFi RSSI scanning
|
||||
- **Alert Sound** — toggle for MAT alert audio notifications
|
||||
|
||||
All settings persist across app restarts via Zustand with AsyncStorage.
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
<details>
|
||||
<summary><strong>API Integration</strong></summary>
|
||||
|
||||
### WebSocket Protocol
|
||||
|
||||
The app connects to the sensing server's WebSocket endpoint for real-time data streaming.
|
||||
|
||||
**Endpoint:** `ws://<host>:3001/ws/sensing`
|
||||
|
||||
**Frame format** (`SensingFrame`):
|
||||
|
||||
```typescript
|
||||
interface SensingFrame {
|
||||
type?: string;
|
||||
timestamp?: number;
|
||||
source?: string; // "live" | "simulated"
|
||||
tick?: number;
|
||||
nodes: SensingNode[]; // Per-node RSSI, position, amplitude
|
||||
features: FeatureSet; // mean_rssi, variance, motion_band_power, etc.
|
||||
classification: Classification; // motion_level, presence, confidence
|
||||
signal_field: SignalField; // 3D voxel grid values
|
||||
vital_signs?: VitalsData; // breathing_bpm, hr_proxy_bpm, confidence
|
||||
}
|
||||
```
|
||||
|
||||
The WebSocket service (`ws.service.ts`) handles:
|
||||
- Automatic reconnection with exponential backoff (1s, 2s, 4s, 8s, 16s)
|
||||
- Fallback to simulation after max reconnect attempts
|
||||
- Protocol upgrade (`http:` to `ws:`, `https:` to `wss:`)
|
||||
- Port mapping (HTTP 3000 maps to WS 3001)
|
||||
|
||||
### REST API
|
||||
|
||||
The REST client (`api.service.ts`) provides:
|
||||
|
||||
| Method | Path | Returns |
|
||||
|--------|------|---------|
|
||||
| `GET` | `/api/pose/status` | `PoseStatus` — server health and capabilities |
|
||||
| `GET` | `/api/pose/zones` | `ZoneConfig[]` — configured sensing zones |
|
||||
| `GET` | `/api/pose/frames?limit=N` | `HistoricalFrames` — recent frame history |
|
||||
|
||||
All requests use Axios with a 5-second timeout and automatic retry (2 attempts).
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
```bash
|
||||
cd ui/mobile
|
||||
npm test
|
||||
```
|
||||
|
||||
Runs the Jest test suite via `jest-expo`. Tests cover:
|
||||
|
||||
| Category | Files | What Is Tested |
|
||||
|----------|-------|----------------|
|
||||
| Components | 7 | `ConnectionBanner`, `GaugeArc`, `HudOverlay`, `OccupancyGrid`, `SignalBar`, `SparklineChart`, `StatusDot` |
|
||||
| Screens | 5 | `LiveScreen`, `VitalsScreen`, `ZonesScreen`, `MATScreen`, `SettingsScreen` |
|
||||
| Services | 4 | `ws.service`, `api.service`, `rssi.service`, `simulation.service` |
|
||||
| Stores | 3 | `poseStore`, `matStore`, `settingsStore` |
|
||||
| Hooks | 3 | `usePoseStream`, `useRssiScanner`, `useServerReachability` |
|
||||
| Utils | 3 | `colorMap`, `ringBuffer`, `urlValidator` |
|
||||
|
||||
### End-to-End Tests (Maestro)
|
||||
|
||||
```bash
|
||||
# Install Maestro CLI
|
||||
curl -Ls https://get.maestro.mobile.dev | bash
|
||||
|
||||
# Run all e2e specs
|
||||
maestro test e2e/
|
||||
```
|
||||
|
||||
Maestro YAML specs cover each screen:
|
||||
|
||||
| Spec | What It Verifies |
|
||||
|------|-----------------|
|
||||
| `live_screen.yaml` | 3D viewer loads, HUD elements visible, mode badge displays |
|
||||
| `vitals_screen.yaml` | Breathing and heart rate gauges render with values |
|
||||
| `zones_screen.yaml` | Floor plan SVG renders, zone legend visible |
|
||||
| `mat_screen.yaml` | Survivor counter displays, alert list populates |
|
||||
| `settings_screen.yaml` | URL input editable, theme picker works, toggles respond |
|
||||
| `offline_fallback.yaml` | App transitions to SIM mode when server unreachable |
|
||||
|
||||
---
|
||||
|
||||
## Tech Stack
|
||||
|
||||
| Layer | Technology | Version |
|
||||
|-------|-----------|---------|
|
||||
| Framework | Expo | 55 |
|
||||
| UI | React Native | 0.83 |
|
||||
| Language | TypeScript | 5.9 |
|
||||
| Navigation | React Navigation | 7.x |
|
||||
| State | Zustand | 5.x |
|
||||
| HTTP | Axios | 1.x |
|
||||
| SVG | react-native-svg | 15.x |
|
||||
| WebView | react-native-webview | 13.x |
|
||||
| WiFi | react-native-wifi-reborn | 4.x |
|
||||
| Charts | Victory Native | 41.x |
|
||||
| Animations | react-native-reanimated | 4.x |
|
||||
| Testing | Jest + jest-expo | 30.x |
|
||||
| E2E | Maestro | Latest |
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch from `main`
|
||||
3. Make changes in the `ui/mobile/` directory
|
||||
4. Run `npm test` and verify all tests pass
|
||||
5. Run `npx expo start --web` to verify the app renders correctly
|
||||
6. Submit a pull request
|
||||
|
||||
Follow the project's existing patterns:
|
||||
- Components go in `src/components/`
|
||||
- Screen-specific components go in `src/screens/<ScreenName>/`
|
||||
- Platform-specific files use the `.android.ts` / `.ios.ts` / `.web.ts` suffix convention
|
||||
- All state management uses Zustand stores in `src/stores/`
|
||||
- All types go in `src/types/`
|
||||
|
||||
---
|
||||
|
||||
## Credits
|
||||
|
||||
Mobile app by [@MaTriXy](https://github.com/MaTriXy) — original scaffold, screen architecture, and cross-platform service layer.
|
||||
|
||||
Built on the [WiFi-DensePose](../../README.md) sensing platform.
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
[MIT](../../LICENSE)
|
||||
@@ -0,0 +1,12 @@
|
||||
export default {
|
||||
name: 'WiFi-DensePose',
|
||||
slug: 'wifi-densepose',
|
||||
version: '1.0.0',
|
||||
ios: {
|
||||
bundleIdentifier: 'com.ruvnet.wifidensepose',
|
||||
},
|
||||
android: {
|
||||
package: 'com.ruvnet.wifidensepose',
|
||||
},
|
||||
// Use expo-env and app-level defaults from the project configuration when available.
|
||||
};
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"expo": {
|
||||
"name": "mobile",
|
||||
"slug": "mobile",
|
||||
"version": "1.0.0",
|
||||
"orientation": "portrait",
|
||||
"icon": "./assets/icon.png",
|
||||
"userInterfaceStyle": "light",
|
||||
"splash": {
|
||||
"image": "./assets/splash-icon.png",
|
||||
"resizeMode": "contain",
|
||||
"backgroundColor": "#ffffff"
|
||||
},
|
||||
"ios": {
|
||||
"supportsTablet": true
|
||||
},
|
||||
"android": {
|
||||
"adaptiveIcon": {
|
||||
"backgroundColor": "#E6F4FE",
|
||||
"foregroundImage": "./assets/android-icon-foreground.png",
|
||||
"backgroundImage": "./assets/android-icon-background.png",
|
||||
"monochromeImage": "./assets/android-icon-monochrome.png"
|
||||
},
|
||||
"predictiveBackGestureEnabled": false
|
||||
},
|
||||
"web": {
|
||||
"favicon": "./assets/favicon.png"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
After Width: | Height: | Size: 17 KiB |
|
After Width: | Height: | Size: 77 KiB |
|
After Width: | Height: | Size: 4.0 KiB |
|
After Width: | Height: | Size: 1.1 KiB |
|
After Width: | Height: | Size: 384 KiB |
|
After Width: | Height: | Size: 17 KiB |
@@ -0,0 +1,9 @@
|
||||
module.exports = function (api) {
|
||||
api.cache(true);
|
||||
return {
|
||||
presets: ['babel-preset-expo'],
|
||||
plugins: [
|
||||
'react-native-reanimated/plugin'
|
||||
]
|
||||
};
|
||||
};
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"cli": {
|
||||
"version": ">= 4.0.0"
|
||||
},
|
||||
"build": {
|
||||
"development": {
|
||||
"developmentClient": true,
|
||||
"distribution": "internal"
|
||||
},
|
||||
"preview": {
|
||||
"distribution": "internal"
|
||||
},
|
||||
"production": {
|
||||
"autoIncrement": true
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
import { registerRootComponent } from 'expo';
|
||||
import App from './App';
|
||||
|
||||
registerRootComponent(App);
|
||||
@@ -0,0 +1,8 @@
|
||||
module.exports = {
|
||||
preset: 'jest-expo',
|
||||
setupFilesAfterEnv: ['<rootDir>/jest.setup.ts'],
|
||||
testPathIgnorePatterns: ['/src/__tests__/'],
|
||||
transformIgnorePatterns: [
|
||||
'node_modules/(?!(expo|expo-.+|react-native|@react-native|react-native-webview|react-native-reanimated|react-native-svg|react-native-safe-area-context|react-native-screens|@react-navigation|@expo|@unimodules|expo-modules-core)/)',
|
||||
],
|
||||
};
|
||||
@@ -0,0 +1,24 @@
|
||||
jest.mock('@react-native-async-storage/async-storage', () =>
|
||||
require('@react-native-async-storage/async-storage/jest/async-storage-mock')
|
||||
);
|
||||
|
||||
jest.mock('react-native-wifi-reborn', () => ({
|
||||
loadWifiList: jest.fn(async () => []),
|
||||
}));
|
||||
|
||||
jest.mock('react-native-reanimated', () =>
|
||||
require('react-native-reanimated/mock')
|
||||
);
|
||||
|
||||
jest.mock('react-native-webview', () => {
|
||||
const React = require('react');
|
||||
const { View } = require('react-native');
|
||||
|
||||
const MockWebView = (props: unknown) => React.createElement(View, props);
|
||||
|
||||
return {
|
||||
__esModule: true,
|
||||
default: MockWebView,
|
||||
WebView: MockWebView,
|
||||
};
|
||||
});
|
||||
@@ -0,0 +1,11 @@
|
||||
const { getDefaultConfig } = require('expo/metro-config');
|
||||
|
||||
const config = getDefaultConfig(__dirname);
|
||||
|
||||
// Force CJS resolution for packages that use import.meta (not supported in Hermes script mode)
|
||||
config.resolver = {
|
||||
...config.resolver,
|
||||
unstable_enablePackageExports: false,
|
||||
};
|
||||
|
||||
module.exports = config;
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"name": "mobile",
|
||||
"version": "1.0.0",
|
||||
"main": "index.ts",
|
||||
"scripts": {
|
||||
"start": "expo start",
|
||||
"android": "expo start --android",
|
||||
"ios": "expo start --ios",
|
||||
"web": "expo start --web",
|
||||
"test": "jest",
|
||||
"lint": "eslint ."
|
||||
},
|
||||
"dependencies": {
|
||||
"@expo/vector-icons": "^15.0.2",
|
||||
"@react-native-async-storage/async-storage": "2.2.0",
|
||||
"@react-navigation/bottom-tabs": "^7.15.3",
|
||||
"@react-navigation/native": "^7.1.31",
|
||||
"@types/three": "^0.183.1",
|
||||
"axios": "^1.13.6",
|
||||
"expo": "~55.0.4",
|
||||
"expo-status-bar": "~55.0.4",
|
||||
"react": "19.2.0",
|
||||
"react-dom": "19.2.0",
|
||||
"react-native": "0.83.2",
|
||||
"react-native-gesture-handler": "~2.30.0",
|
||||
"react-native-reanimated": "4.2.1",
|
||||
"react-native-safe-area-context": "~5.6.2",
|
||||
"react-native-screens": "~4.23.0",
|
||||
"react-native-svg": "15.15.3",
|
||||
"react-native-web": "^0.21.0",
|
||||
"react-native-webview": "13.16.0",
|
||||
"react-native-wifi-reborn": "^4.13.6",
|
||||
"three": "^0.183.2",
|
||||
"victory-native": "^41.20.2",
|
||||
"zustand": "^5.0.11"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@testing-library/jest-native": "^5.4.3",
|
||||
"@testing-library/react-native": "^13.3.3",
|
||||
"@types/jest": "^30.0.0",
|
||||
"@types/react": "~19.2.2",
|
||||
"@typescript-eslint/eslint-plugin": "^8.56.1",
|
||||
"@typescript-eslint/parser": "^8.56.1",
|
||||
"babel-preset-expo": "^55.0.10",
|
||||
"eslint": "^10.0.2",
|
||||
"jest": "^30.2.0",
|
||||
"jest-expo": "^55.0.9",
|
||||
"prettier": "^3.8.1",
|
||||
"react-native-worklets": "^0.7.4",
|
||||
"typescript": "~5.9.2"
|
||||
},
|
||||
"private": true
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,36 @@
|
||||
import React, { PropsWithChildren } from 'react';
|
||||
import { render, type RenderOptions } from '@testing-library/react-native';
|
||||
import { NavigationContainer } from '@react-navigation/native';
|
||||
import { GestureHandlerRootView } from 'react-native-gesture-handler';
|
||||
import { SafeAreaProvider } from 'react-native-safe-area-context';
|
||||
import { ThemeProvider } from '@/theme/ThemeContext';
|
||||
|
||||
type TestProvidersProps = PropsWithChildren<object>;
|
||||
|
||||
const TestProviders = ({ children }: TestProvidersProps) => (
|
||||
<GestureHandlerRootView style={{ flex: 1 }}>
|
||||
<SafeAreaProvider>
|
||||
<ThemeProvider>{children}</ThemeProvider>
|
||||
</SafeAreaProvider>
|
||||
</GestureHandlerRootView>
|
||||
);
|
||||
|
||||
const TestProvidersWithNavigation = ({ children }: TestProvidersProps) => (
|
||||
<TestProviders>
|
||||
<NavigationContainer>{children}</NavigationContainer>
|
||||
</TestProviders>
|
||||
);
|
||||
|
||||
interface RenderWithProvidersOptions extends Omit<RenderOptions, 'wrapper'> {
|
||||
withNavigation?: boolean;
|
||||
}
|
||||
|
||||
export const renderWithProviders = (
|
||||
ui: React.ReactElement,
|
||||
{ withNavigation, ...options }: RenderWithProvidersOptions = {},
|
||||
) => {
|
||||
return render(ui, {
|
||||
...options,
|
||||
wrapper: withNavigation ? TestProvidersWithNavigation : TestProviders,
|
||||
});
|
||||
};
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,5 @@
|
||||
describe('placeholder', () => {
|
||||
it('passes', () => {
|
||||
expect(true).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,585 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta
|
||||
name="viewport"
|
||||
content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"
|
||||
/>
|
||||
<meta
|
||||
http-equiv="Content-Security-Policy"
|
||||
content="default-src 'self'; script-src 'self' https://cdnjs.cloudflare.com https://cdn.jsdelivr.net; style-src 'self' 'unsafe-inline'; img-src 'self' data:; connect-src 'self'"
|
||||
/>
|
||||
<title>WiFi DensePose Splat Viewer</title>
|
||||
<style>
|
||||
html,
|
||||
body,
|
||||
#gaussian-splat-root {
|
||||
margin: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
overflow: hidden;
|
||||
background: #0a0e1a;
|
||||
touch-action: none;
|
||||
}
|
||||
|
||||
#gaussian-splat-root {
|
||||
position: relative;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="gaussian-splat-root"></div>
|
||||
|
||||
<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r165/three.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.165.0/examples/js/controls/OrbitControls.js"></script>
|
||||
|
||||
<script>
|
||||
(function () {
|
||||
const postMessageToRN = (message) => {
|
||||
if (!window.ReactNativeWebView || typeof window.ReactNativeWebView.postMessage !== 'function') {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
window.ReactNativeWebView.postMessage(JSON.stringify(message));
|
||||
} catch (error) {
|
||||
console.error('Failed to post RN message', error);
|
||||
}
|
||||
};
|
||||
|
||||
const postError = (message) => {
|
||||
postMessageToRN({
|
||||
type: 'ERROR',
|
||||
payload: {
|
||||
message: typeof message === 'string' ? message : 'Unknown bridge error',
|
||||
},
|
||||
});
|
||||
};
|
||||
|
||||
// Use global THREE from CDN
|
||||
const getThree = () => window.THREE;
|
||||
|
||||
// ---- Custom Splat Shaders --------------------------------------------
|
||||
|
||||
const SPLAT_VERTEX = `
|
||||
attribute float splatSize;
|
||||
attribute vec3 splatColor;
|
||||
attribute float splatOpacity;
|
||||
|
||||
varying vec3 vColor;
|
||||
varying float vOpacity;
|
||||
|
||||
void main() {
|
||||
vColor = splatColor;
|
||||
vOpacity = splatOpacity;
|
||||
|
||||
vec4 mvPosition = modelViewMatrix * vec4(position, 1.0);
|
||||
gl_PointSize = splatSize * (300.0 / -mvPosition.z);
|
||||
gl_Position = projectionMatrix * mvPosition;
|
||||
}
|
||||
`;
|
||||
|
||||
const SPLAT_FRAGMENT = `
|
||||
varying vec3 vColor;
|
||||
varying float vOpacity;
|
||||
|
||||
void main() {
|
||||
// Circular soft-edge disc
|
||||
float dist = length(gl_PointCoord - vec2(0.5));
|
||||
if (dist > 0.5) discard;
|
||||
float alpha = smoothstep(0.5, 0.2, dist) * vOpacity;
|
||||
gl_FragColor = vec4(vColor, alpha);
|
||||
}
|
||||
`;
|
||||
|
||||
// ---- Color helpers ---------------------------------------------------
|
||||
|
||||
/** Map a scalar 0-1 to blue -> green -> red gradient */
|
||||
function valueToColor(v) {
|
||||
const clamped = Math.max(0, Math.min(1, v));
|
||||
// blue(0) -> cyan(0.25) -> green(0.5) -> yellow(0.75) -> red(1)
|
||||
let r;
|
||||
let g;
|
||||
let b;
|
||||
if (clamped < 0.5) {
|
||||
const t = clamped * 2;
|
||||
r = 0;
|
||||
g = t;
|
||||
b = 1 - t;
|
||||
} else {
|
||||
const t = (clamped - 0.5) * 2;
|
||||
r = t;
|
||||
g = 1 - t;
|
||||
b = 0;
|
||||
}
|
||||
return [r, g, b];
|
||||
}
|
||||
|
||||
// ---- GaussianSplatRenderer -------------------------------------------
|
||||
|
||||
class GaussianSplatRenderer {
|
||||
/** @param {HTMLElement} container - DOM element to attach the renderer to */
|
||||
constructor(container, opts = {}) {
|
||||
const THREE = getThree();
|
||||
if (!THREE) {
|
||||
throw new Error('Three.js not loaded');
|
||||
}
|
||||
|
||||
this.container = container;
|
||||
this.width = opts.width || container.clientWidth || 800;
|
||||
this.height = opts.height || 500;
|
||||
|
||||
// Scene
|
||||
this.scene = new THREE.Scene();
|
||||
this.scene.background = new THREE.Color(0x0a0e1a);
|
||||
|
||||
// Camera — perspective looking down at the room
|
||||
this.camera = new THREE.PerspectiveCamera(45, this.width / this.height, 0.1, 200);
|
||||
this.camera.position.set(0, 10, 12);
|
||||
this.camera.lookAt(0, 0, 0);
|
||||
|
||||
// Renderer
|
||||
this.renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
|
||||
this.renderer.setSize(this.width, this.height);
|
||||
this.renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2));
|
||||
container.appendChild(this.renderer.domElement);
|
||||
|
||||
// Lights
|
||||
const ambient = new THREE.AmbientLight(0x9ec7ff, 0.35);
|
||||
this.scene.add(ambient);
|
||||
|
||||
const directional = new THREE.DirectionalLight(0x9ec7ff, 0.65);
|
||||
directional.position.set(4, 10, 6);
|
||||
directional.castShadow = false;
|
||||
this.scene.add(directional);
|
||||
|
||||
// Grid & room
|
||||
this._createRoom(THREE);
|
||||
|
||||
// Signal field splats (20x20 = 400 points on the floor plane)
|
||||
this.gridSize = 20;
|
||||
this._createFieldSplats(THREE);
|
||||
|
||||
// Node markers (ESP32 / router positions)
|
||||
this._createNodeMarkers(THREE);
|
||||
|
||||
// Body disruption blob
|
||||
this._createBodyBlob(THREE);
|
||||
|
||||
// Orbit controls for drag + pinch zoom
|
||||
this.controls = new THREE.OrbitControls(this.camera, this.renderer.domElement);
|
||||
this.controls.target.set(0, 0, 0);
|
||||
this.controls.minDistance = 6;
|
||||
this.controls.maxDistance = 40;
|
||||
this.controls.enableDamping = true;
|
||||
this.controls.dampingFactor = 0.08;
|
||||
this.controls.update();
|
||||
|
||||
// Animation state
|
||||
this._animFrame = null;
|
||||
this._lastData = null;
|
||||
this._fpsFrames = [];
|
||||
this._lastFpsReport = 0;
|
||||
|
||||
// Start render loop
|
||||
this._animate();
|
||||
}
|
||||
|
||||
// ---- Scene setup ---------------------------------------------------
|
||||
|
||||
_createRoom(THREE) {
|
||||
// Floor grid (on y = 0), 20 units
|
||||
const grid = new THREE.GridHelper(20, 20, 0x1a3a4a, 0x0d1f28);
|
||||
grid.position.y = 0;
|
||||
this.scene.add(grid);
|
||||
|
||||
// Room boundary wireframe
|
||||
const boxGeo = new THREE.BoxGeometry(20, 6, 20);
|
||||
const edges = new THREE.EdgesGeometry(boxGeo);
|
||||
const line = new THREE.LineSegments(
|
||||
edges,
|
||||
new THREE.LineBasicMaterial({ color: 0x1a4a5a, opacity: 0.3, transparent: true }),
|
||||
);
|
||||
line.position.y = 3;
|
||||
this.scene.add(line);
|
||||
}
|
||||
|
||||
_createFieldSplats(THREE) {
|
||||
const count = this.gridSize * this.gridSize;
|
||||
|
||||
const positions = new Float32Array(count * 3);
|
||||
const sizes = new Float32Array(count);
|
||||
const colors = new Float32Array(count * 3);
|
||||
const opacities = new Float32Array(count);
|
||||
|
||||
// Lay splats on the floor plane (y = 0.05 to sit just above grid)
|
||||
for (let iz = 0; iz < this.gridSize; iz++) {
|
||||
for (let ix = 0; ix < this.gridSize; ix++) {
|
||||
const idx = iz * this.gridSize + ix;
|
||||
positions[idx * 3 + 0] = (ix - this.gridSize / 2) + 0.5; // x
|
||||
positions[idx * 3 + 1] = 0.05; // y
|
||||
positions[idx * 3 + 2] = (iz - this.gridSize / 2) + 0.5; // z
|
||||
|
||||
sizes[idx] = 1.5;
|
||||
colors[idx * 3] = 0.1;
|
||||
colors[idx * 3 + 1] = 0.2;
|
||||
colors[idx * 3 + 2] = 0.6;
|
||||
opacities[idx] = 0.15;
|
||||
}
|
||||
}
|
||||
|
||||
const geo = new THREE.BufferGeometry();
|
||||
geo.setAttribute('position', new THREE.BufferAttribute(positions, 3));
|
||||
geo.setAttribute('splatSize', new THREE.BufferAttribute(sizes, 1));
|
||||
geo.setAttribute('splatColor', new THREE.BufferAttribute(colors, 3));
|
||||
geo.setAttribute('splatOpacity', new THREE.BufferAttribute(opacities, 1));
|
||||
|
||||
const mat = new THREE.ShaderMaterial({
|
||||
vertexShader: SPLAT_VERTEX,
|
||||
fragmentShader: SPLAT_FRAGMENT,
|
||||
transparent: true,
|
||||
depthWrite: false,
|
||||
blending: THREE.AdditiveBlending,
|
||||
});
|
||||
|
||||
this.fieldPoints = new THREE.Points(geo, mat);
|
||||
this.scene.add(this.fieldPoints);
|
||||
}
|
||||
|
||||
_createNodeMarkers(THREE) {
|
||||
// Router at center — green sphere
|
||||
const routerGeo = new THREE.SphereGeometry(0.3, 16, 16);
|
||||
const routerMat = new THREE.MeshBasicMaterial({ color: 0x00ff88, transparent: true, opacity: 0.8 });
|
||||
this.routerMarker = new THREE.Mesh(routerGeo, routerMat);
|
||||
this.routerMarker.position.set(0, 0.5, 0);
|
||||
this.scene.add(this.routerMarker);
|
||||
|
||||
// ESP32 node — cyan sphere (default position, updated from data)
|
||||
const nodeGeo = new THREE.SphereGeometry(0.25, 16, 16);
|
||||
const nodeMat = new THREE.MeshBasicMaterial({ color: 0x00ccff, transparent: true, opacity: 0.8 });
|
||||
this.nodeMarker = new THREE.Mesh(nodeGeo, nodeMat);
|
||||
this.nodeMarker.position.set(2, 0.5, 1.5);
|
||||
this.scene.add(this.nodeMarker);
|
||||
}
|
||||
|
||||
_createBodyBlob(THREE) {
|
||||
// A cluster of splats representing body disruption
|
||||
const count = 64;
|
||||
const positions = new Float32Array(count * 3);
|
||||
const sizes = new Float32Array(count);
|
||||
const colors = new Float32Array(count * 3);
|
||||
const opacities = new Float32Array(count);
|
||||
|
||||
for (let i = 0; i < count; i++) {
|
||||
// Random sphere distribution
|
||||
const theta = Math.random() * Math.PI * 2;
|
||||
const phi = Math.acos(2 * Math.random() - 1);
|
||||
const r = Math.random() * 1.5;
|
||||
positions[i * 3] = r * Math.sin(phi) * Math.cos(theta);
|
||||
positions[i * 3 + 1] = r * Math.cos(phi) + 2;
|
||||
positions[i * 3 + 2] = r * Math.sin(phi) * Math.sin(theta);
|
||||
|
||||
sizes[i] = 2 + Math.random() * 3;
|
||||
colors[i * 3] = 0.2;
|
||||
colors[i * 3 + 1] = 0.8;
|
||||
colors[i * 3 + 2] = 0.3;
|
||||
opacities[i] = 0.0; // hidden until presence detected
|
||||
}
|
||||
|
||||
const geo = new THREE.BufferGeometry();
|
||||
geo.setAttribute('position', new THREE.BufferAttribute(positions, 3));
|
||||
geo.setAttribute('splatSize', new THREE.BufferAttribute(sizes, 1));
|
||||
geo.setAttribute('splatColor', new THREE.BufferAttribute(colors, 3));
|
||||
geo.setAttribute('splatOpacity', new THREE.BufferAttribute(opacities, 1));
|
||||
|
||||
const mat = new THREE.ShaderMaterial({
|
||||
vertexShader: SPLAT_VERTEX,
|
||||
fragmentShader: SPLAT_FRAGMENT,
|
||||
transparent: true,
|
||||
depthWrite: false,
|
||||
blending: THREE.AdditiveBlending,
|
||||
});
|
||||
|
||||
this.bodyBlob = new THREE.Points(geo, mat);
|
||||
this.scene.add(this.bodyBlob);
|
||||
}
|
||||
|
||||
// ---- Data update --------------------------------------------------
|
||||
|
||||
/**
|
||||
* Update the visualization with new sensing data.
|
||||
* @param {object} data - sensing_update JSON from ws_server
|
||||
*/
|
||||
update(data) {
|
||||
this._lastData = data;
|
||||
if (!data) return;
|
||||
|
||||
const features = data.features || {};
|
||||
const classification = data.classification || {};
|
||||
const signalField = data.signal_field || {};
|
||||
const nodes = data.nodes || [];
|
||||
|
||||
// -- Update signal field splats ------------------------------------
|
||||
if (signalField.values && this.fieldPoints) {
|
||||
const geo = this.fieldPoints.geometry;
|
||||
const clr = geo.attributes.splatColor.array;
|
||||
const sizes = geo.attributes.splatSize.array;
|
||||
const opac = geo.attributes.splatOpacity.array;
|
||||
const vals = signalField.values;
|
||||
const count = Math.min(vals.length, this.gridSize * this.gridSize);
|
||||
|
||||
for (let i = 0; i < count; i++) {
|
||||
const v = vals[i];
|
||||
const [r, g, b] = valueToColor(v);
|
||||
clr[i * 3] = r;
|
||||
clr[i * 3 + 1] = g;
|
||||
clr[i * 3 + 2] = b;
|
||||
sizes[i] = 1.0 + v * 4.0;
|
||||
opac[i] = 0.1 + v * 0.6;
|
||||
}
|
||||
|
||||
geo.attributes.splatColor.needsUpdate = true;
|
||||
geo.attributes.splatSize.needsUpdate = true;
|
||||
geo.attributes.splatOpacity.needsUpdate = true;
|
||||
}
|
||||
|
||||
// -- Update body blob ----------------------------------------------
|
||||
if (this.bodyBlob) {
|
||||
const bGeo = this.bodyBlob.geometry;
|
||||
const bOpac = bGeo.attributes.splatOpacity.array;
|
||||
const bClr = bGeo.attributes.splatColor.array;
|
||||
const bSize = bGeo.attributes.splatSize.array;
|
||||
|
||||
const presence = classification.presence || false;
|
||||
const motionLvl = classification.motion_level || 'absent';
|
||||
const confidence = classification.confidence || 0;
|
||||
const breathing = features.breathing_band_power || 0;
|
||||
|
||||
// Breathing pulsation
|
||||
const breathPulse = 1.0 + Math.sin(Date.now() * 0.004) * Math.min(breathing * 3, 0.4);
|
||||
|
||||
for (let i = 0; i < bOpac.length; i++) {
|
||||
if (presence) {
|
||||
bOpac[i] = confidence * 0.4;
|
||||
|
||||
// Color by motion level
|
||||
if (motionLvl === 'active') {
|
||||
bClr[i * 3] = 1.0;
|
||||
bClr[i * 3 + 1] = 0.2;
|
||||
bClr[i * 3 + 2] = 0.1;
|
||||
} else {
|
||||
bClr[i * 3] = 0.1;
|
||||
bClr[i * 3 + 1] = 0.8;
|
||||
bClr[i * 3 + 2] = 0.4;
|
||||
}
|
||||
|
||||
bSize[i] = (2 + Math.random() * 2) * breathPulse;
|
||||
} else {
|
||||
bOpac[i] = 0.0;
|
||||
}
|
||||
}
|
||||
|
||||
bGeo.attributes.splatOpacity.needsUpdate = true;
|
||||
bGeo.attributes.splatColor.needsUpdate = true;
|
||||
bGeo.attributes.splatSize.needsUpdate = true;
|
||||
}
|
||||
|
||||
// -- Update node positions -----------------------------------------
|
||||
if (nodes.length > 0 && nodes[0].position && this.nodeMarker) {
|
||||
const pos = nodes[0].position;
|
||||
this.nodeMarker.position.set(pos[0], 0.5, pos[2]);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Render loop -------------------------------------------------
|
||||
|
||||
_animate() {
|
||||
this._animFrame = requestAnimationFrame(() => this._animate());
|
||||
|
||||
const now = performance.now();
|
||||
|
||||
// Gentle router glow pulse
|
||||
if (this.routerMarker) {
|
||||
const pulse = 0.6 + 0.3 * Math.sin(now * 0.003);
|
||||
this.routerMarker.material.opacity = pulse;
|
||||
}
|
||||
|
||||
this.controls.update();
|
||||
this.renderer.render(this.scene, this.camera);
|
||||
|
||||
this._fpsFrames.push(now);
|
||||
while (this._fpsFrames.length > 0 && this._fpsFrames[0] < now - 1000) {
|
||||
this._fpsFrames.shift();
|
||||
}
|
||||
|
||||
if (now - this._lastFpsReport >= 1000) {
|
||||
const fps = this._fpsFrames.length;
|
||||
this._lastFpsReport = now;
|
||||
postMessageToRN({
|
||||
type: 'FPS_TICK',
|
||||
payload: { fps },
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// ---- Resize / cleanup --------------------------------------------
|
||||
|
||||
resize(width, height) {
|
||||
if (!width || !height) return;
|
||||
this.width = width;
|
||||
this.height = height;
|
||||
this.camera.aspect = width / height;
|
||||
this.camera.updateProjectionMatrix();
|
||||
this.renderer.setSize(width, height);
|
||||
}
|
||||
|
||||
dispose() {
|
||||
if (this._animFrame) {
|
||||
cancelAnimationFrame(this._animFrame);
|
||||
}
|
||||
|
||||
this.controls?.dispose();
|
||||
this.renderer.dispose();
|
||||
if (this.renderer.domElement.parentNode) {
|
||||
this.renderer.domElement.parentNode.removeChild(this.renderer.domElement);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Expose renderer constructor for debugging/interop
|
||||
window.GaussianSplatRenderer = GaussianSplatRenderer;
|
||||
|
||||
let renderer = null;
|
||||
let pendingFrame = null;
|
||||
let pendingResize = null;
|
||||
|
||||
const postSafeReady = () => {
|
||||
postMessageToRN({ type: 'READY' });
|
||||
};
|
||||
|
||||
const routeMessage = (event) => {
|
||||
let raw = event.data;
|
||||
if (typeof raw === 'object' && raw != null && 'data' in raw) {
|
||||
raw = raw.data;
|
||||
}
|
||||
|
||||
let message = raw;
|
||||
if (typeof raw === 'string') {
|
||||
try {
|
||||
message = JSON.parse(raw);
|
||||
} catch (err) {
|
||||
postError('Failed to parse RN message payload');
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
if (!message || typeof message !== 'object') {
|
||||
return;
|
||||
}
|
||||
|
||||
if (message.type === 'FRAME_UPDATE') {
|
||||
const payload = message.payload || null;
|
||||
if (!payload) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (!renderer) {
|
||||
pendingFrame = payload;
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
renderer.update(payload);
|
||||
} catch (error) {
|
||||
postError((error && error.message) || 'Failed to update frame');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (message.type === 'RESIZE') {
|
||||
const dims = message.payload || {};
|
||||
const w = Number(dims.width);
|
||||
const h = Number(dims.height);
|
||||
if (!Number.isFinite(w) || !Number.isFinite(h) || !w || !h) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (!renderer) {
|
||||
pendingResize = { width: w, height: h };
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
renderer.resize(w, h);
|
||||
} catch (error) {
|
||||
postError((error && error.message) || 'Failed to resize renderer');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (message.type === 'DISPOSE') {
|
||||
if (!renderer) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
renderer.dispose();
|
||||
} catch (error) {
|
||||
postError((error && error.message) || 'Failed to dispose renderer');
|
||||
}
|
||||
renderer = null;
|
||||
return;
|
||||
}
|
||||
};
|
||||
|
||||
const buildRenderer = () => {
|
||||
const container = document.getElementById('gaussian-splat-root');
|
||||
if (!container) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
renderer = new GaussianSplatRenderer(container, {
|
||||
width: container.clientWidth || window.innerWidth,
|
||||
height: container.clientHeight || window.innerHeight,
|
||||
});
|
||||
|
||||
if (pendingFrame) {
|
||||
renderer.update(pendingFrame);
|
||||
pendingFrame = null;
|
||||
}
|
||||
|
||||
if (pendingResize) {
|
||||
renderer.resize(pendingResize.width, pendingResize.height);
|
||||
pendingResize = null;
|
||||
}
|
||||
|
||||
postSafeReady();
|
||||
} catch (error) {
|
||||
renderer = null;
|
||||
postError((error && error.message) || 'Failed to initialize renderer');
|
||||
}
|
||||
};
|
||||
|
||||
if (document.readyState === 'loading') {
|
||||
document.addEventListener('DOMContentLoaded', buildRenderer);
|
||||
} else {
|
||||
buildRenderer();
|
||||
}
|
||||
|
||||
window.addEventListener('message', routeMessage);
|
||||
window.addEventListener('resize', () => {
|
||||
if (!renderer) {
|
||||
pendingResize = {
|
||||
width: window.innerWidth,
|
||||
height: window.innerHeight,
|
||||
};
|
||||
return;
|
||||
}
|
||||
renderer.resize(window.innerWidth, window.innerHeight);
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,505 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>MAT Dashboard</title>
|
||||
<style>
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
html,
|
||||
body {
|
||||
margin: 0;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
background: #0a0e1a;
|
||||
color: #e5e7eb;
|
||||
font-family: 'Courier New', 'Consolas', monospace;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
#app {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 8px;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
padding: 8px;
|
||||
}
|
||||
|
||||
#status {
|
||||
color: #6dd4df;
|
||||
font-size: 12px;
|
||||
letter-spacing: 0.5px;
|
||||
}
|
||||
|
||||
#mapCanvas {
|
||||
flex: 1;
|
||||
width: 100%;
|
||||
border: 1px solid #1e293b;
|
||||
border-radius: 8px;
|
||||
min-height: 180px;
|
||||
background: #0a0e1a;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div id="app">
|
||||
<div id="status">Initializing MAT dashboard...</div>
|
||||
<canvas id="mapCanvas"></canvas>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
(function () {
|
||||
const TRIAGE = {
|
||||
Immediate: 0,
|
||||
Delayed: 1,
|
||||
Minimal: 2,
|
||||
Expectant: 3,
|
||||
Unknown: 4,
|
||||
};
|
||||
|
||||
const TRIAGE_COLOR = ['#ff0000', '#ffcc00', '#00cc00', '#111111', '#888888'];
|
||||
const PRIORITY = { Critical: 0, High: 1, Medium: 2, Low: 3 };
|
||||
|
||||
const toRgba = (status) => TRIAGE_COLOR[status] || TRIAGE_COLOR[4];
|
||||
const safeId = () =>
|
||||
typeof crypto !== 'undefined' && crypto.randomUUID
|
||||
? crypto.randomUUID()
|
||||
: `id-${Date.now()}-${Math.floor(Math.random() * 1e6)}`;
|
||||
|
||||
const isNumber = (value) => typeof value === 'number' && Number.isFinite(value);
|
||||
|
||||
class MatDashboard {
|
||||
constructor() {
|
||||
this.event = null;
|
||||
this.zones = new Map();
|
||||
this.survivors = new Map();
|
||||
this.alerts = new Map();
|
||||
this.motionVector = { x: 0, y: 0 };
|
||||
}
|
||||
|
||||
createEvent(type, lat, lon, name) {
|
||||
const eventId = safeId();
|
||||
this.event = {
|
||||
event_id: eventId,
|
||||
disaster_type: type,
|
||||
latitude: lat,
|
||||
longitude: lon,
|
||||
description: name,
|
||||
createdAt: Date.now(),
|
||||
};
|
||||
this.zones.clear();
|
||||
this.survivors.clear();
|
||||
this.alerts.clear();
|
||||
return eventId;
|
||||
}
|
||||
|
||||
addRectangleZone(name, x, y, w, h) {
|
||||
const id = safeId();
|
||||
this.zones.set(id, {
|
||||
id,
|
||||
name,
|
||||
zone_type: 'rectangle',
|
||||
status: 0,
|
||||
scan_count: 0,
|
||||
detection_count: 0,
|
||||
x,
|
||||
y,
|
||||
width: w,
|
||||
height: h,
|
||||
});
|
||||
return id;
|
||||
}
|
||||
|
||||
addCircleZone(name, cx, cy, radius) {
|
||||
const id = safeId();
|
||||
this.zones.set(id, {
|
||||
id,
|
||||
name,
|
||||
zone_type: 'circle',
|
||||
status: 0,
|
||||
scan_count: 0,
|
||||
detection_count: 0,
|
||||
center_x: cx,
|
||||
center_y: cy,
|
||||
radius,
|
||||
});
|
||||
return id;
|
||||
}
|
||||
|
||||
addZoneFromPayload(payload) {
|
||||
if (!payload || typeof payload !== 'object') {
|
||||
return;
|
||||
}
|
||||
|
||||
const source = payload;
|
||||
const type = source.zone_type || source.type || 'rectangle';
|
||||
const name = source.name || `Zone-${safeId().slice(0, 4)}`;
|
||||
|
||||
if (type === 'circle' || source.center_x !== undefined) {
|
||||
const cx = isNumber(source.center_x) ? source.center_x : 120;
|
||||
const cy = isNumber(source.center_y) ? source.center_y : 120;
|
||||
const radius = isNumber(source.radius) ? source.radius : 50;
|
||||
return this.addCircleZone(name, cx, cy, radius);
|
||||
}
|
||||
|
||||
const x = isNumber(source.x) ? source.x : 40;
|
||||
const y = isNumber(source.y) ? source.y : 40;
|
||||
const width = isNumber(source.width) ? source.width : 100;
|
||||
const height = isNumber(source.height) ? source.height : 100;
|
||||
return this.addRectangleZone(name, x, y, width, height);
|
||||
}
|
||||
|
||||
inferTriage(vitalSigns, confidence) {
|
||||
const breathing = isNumber(vitalSigns?.breathing_rate) ? vitalSigns.breathing_rate : 14;
|
||||
const heart = isNumber(vitalSigns?.heart_rate)
|
||||
? vitalSigns.heart_rate
|
||||
: isNumber(vitalSigns?.hr)
|
||||
? vitalSigns.hr
|
||||
: 70;
|
||||
|
||||
if (!isNumber(confidence) || confidence > 0.82) {
|
||||
if (breathing < 10 || breathing > 35 || heart > 150) {
|
||||
return TRIAGE.Immediate;
|
||||
}
|
||||
if (breathing >= 8 && breathing <= 34) {
|
||||
return TRIAGE.Delayed;
|
||||
}
|
||||
}
|
||||
|
||||
if (breathing >= 6 && breathing <= 28 && heart > 45 && heart < 180) {
|
||||
return TRIAGE.Minimal;
|
||||
}
|
||||
|
||||
return TRIAGE.Expectant;
|
||||
}
|
||||
|
||||
locateZoneForPoint(x, y) {
|
||||
for (const [id, zone] of this.zones.entries()) {
|
||||
if (zone.zone_type === 'circle') {
|
||||
const dx = x - zone.center_x;
|
||||
const dy = y - zone.center_y;
|
||||
const inside = Math.sqrt(dx * dx + dy * dy) <= zone.radius;
|
||||
if (inside) {
|
||||
return id;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (x >= zone.x && x <= zone.x + zone.width && y >= zone.y && y <= zone.y + zone.height) {
|
||||
return id;
|
||||
}
|
||||
}
|
||||
return this.zones.size > 0 ? this.zones.keys().next().value : safeId();
|
||||
}
|
||||
|
||||
processSurvivorDetection(zone, confidence = 0.6, vital_signs = {}) {
|
||||
const zoneKey =
|
||||
typeof zone === 'string'
|
||||
? [...this.zones.values()].find((entry) => entry.id === zone || entry.name === zone)
|
||||
: null;
|
||||
|
||||
const selectedZone =
|
||||
zoneKey
|
||||
|| (this.zones.size > 0
|
||||
? [...this.zones.values()][Math.floor(Math.random() * Math.max(1, this.zones.size))]
|
||||
: null);
|
||||
|
||||
const bounds = this._pickPointInZone(selectedZone);
|
||||
const triageStatus = this.inferTriage(vital_signs, confidence);
|
||||
const breathingRate = isNumber(vital_signs?.breathing_rate)
|
||||
? vital_signs.breathing_rate
|
||||
: 10 + confidence * 28;
|
||||
const heartRate = isNumber(vital_signs?.heart_rate)
|
||||
? vital_signs.heart_rate
|
||||
: isNumber(vital_signs?.hr)
|
||||
? vital_signs.hr
|
||||
: 55 + confidence * 60;
|
||||
|
||||
const id = safeId();
|
||||
const zone_id = this.locateZoneForPoint(bounds.x, bounds.y);
|
||||
|
||||
const survivor = {
|
||||
id,
|
||||
zone_id,
|
||||
x: bounds.x,
|
||||
y: bounds.y,
|
||||
depth: -Math.abs(isNumber(vital_signs.depth) ? vital_signs.depth : Math.random() * 3),
|
||||
triage_status: triageStatus,
|
||||
triage_color: toRgba(triageStatus),
|
||||
confidence,
|
||||
breathing_rate: breathingRate,
|
||||
heart_rate: heartRate,
|
||||
first_detected: new Date().toISOString(),
|
||||
last_updated: new Date().toISOString(),
|
||||
is_deteriorating: false,
|
||||
};
|
||||
|
||||
this.survivors.set(id, survivor);
|
||||
if (selectedZone) {
|
||||
selectedZone.detection_count = (selectedZone.detection_count || 0) + 1;
|
||||
}
|
||||
|
||||
if (typeof this.postMessage === 'function') {
|
||||
this.postMessage({
|
||||
type: 'SURVIVOR_DETECTED',
|
||||
payload: survivor,
|
||||
});
|
||||
}
|
||||
|
||||
this.generateAlerts();
|
||||
return id;
|
||||
}
|
||||
|
||||
_pickPointInZone(zone) {
|
||||
if (!zone) {
|
||||
return {
|
||||
x: 220 + Math.random() * 80,
|
||||
y: 120 + Math.random() * 80,
|
||||
};
|
||||
}
|
||||
|
||||
if (zone.zone_type === 'circle') {
|
||||
const angle = Math.random() * Math.PI * 2;
|
||||
const radius = Math.random() * (zone.radius || 20);
|
||||
return {
|
||||
x: Math.max(10, Math.min(560, zone.center_x + Math.cos(angle) * radius)),
|
||||
y: Math.max(10, Math.min(280, zone.center_y + Math.sin(angle) * radius)),
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
x: Math.max(zone.x || 5, Math.min((zone.x || 5) + (zone.width || 40), (zone.x || 5) + Math.random() * (zone.width || 40))),
|
||||
y: Math.max(zone.y || 5, Math.min((zone.y || 5) + (zone.height || 40), (zone.y || 5) + Math.random() * (zone.height || 40))),
|
||||
};
|
||||
}
|
||||
|
||||
generateAlerts() {
|
||||
for (const survivor of this.survivors.values()) {
|
||||
if ((survivor.triage_status !== TRIAGE.Immediate && survivor.triage_status !== TRIAGE.Delayed)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const alertId = `alert-${survivor.id}`;
|
||||
if (this.alerts.has(alertId)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const priority =
|
||||
survivor.triage_status === TRIAGE.Immediate ? PRIORITY.Critical : PRIORITY.High;
|
||||
const message =
|
||||
survivor.triage_status === TRIAGE.Immediate
|
||||
? `Immediate rescue required at (${survivor.x.toFixed(0)}, ${survivor.y.toFixed(0)})`
|
||||
: `High-priority rescue needed at (${survivor.x.toFixed(0)}, ${survivor.y.toFixed(0)})`;
|
||||
const alert = {
|
||||
id: alertId,
|
||||
survivor_id: survivor.id,
|
||||
priority,
|
||||
title: survivor.triage_status === TRIAGE.Immediate ? 'URGENT' : 'HIGH',
|
||||
message,
|
||||
recommended_action: survivor.triage_status === TRIAGE.Immediate ? 'Dispatch now' : 'Coordinate rescue',
|
||||
triage_status: survivor.triage_status,
|
||||
location_x: survivor.x,
|
||||
location_y: survivor.y,
|
||||
created_at: new Date().toISOString(),
|
||||
priority_color: survivor.triage_status === TRIAGE.Immediate ? '#ff0000' : '#ff8c00',
|
||||
};
|
||||
|
||||
this.alerts.set(alertId, alert);
|
||||
if (typeof this.postMessage === 'function') {
|
||||
this.postMessage({
|
||||
type: 'ALERT_GENERATED',
|
||||
payload: alert,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
processFrame(frame) {
|
||||
const motion = Number(frame?.features?.motion_band_power || 0);
|
||||
const xDelta = isNumber(motion) ? (motion - 0.1) * 4 : 0;
|
||||
const yDelta = isNumber(frame?.features?.breathing_band_power || 0)
|
||||
? (frame.features.breathing_band_power - 0.1) * 3
|
||||
: 0;
|
||||
this.motionVector = { x: xDelta || 0, y: yDelta || 0 };
|
||||
|
||||
for (const survivor of this.survivors.values()) {
|
||||
const jitterX = (Math.random() - 0.5) * 2;
|
||||
const jitterY = (Math.random() - 0.5) * 2;
|
||||
survivor.x = Math.max(5, Math.min(560, survivor.x + this.motionVector.x + jitterX));
|
||||
survivor.y = Math.max(5, Math.min(280, survivor.y + this.motionVector.y + jitterY));
|
||||
survivor.last_updated = new Date().toISOString();
|
||||
}
|
||||
}
|
||||
|
||||
renderZones(ctx) {
|
||||
for (const zone of this.zones.values()) {
|
||||
const fill = 'rgba(0, 150, 255, 0.3)';
|
||||
ctx.strokeStyle = '#0096ff';
|
||||
ctx.fillStyle = fill;
|
||||
ctx.lineWidth = 2;
|
||||
|
||||
if (zone.zone_type === 'circle') {
|
||||
ctx.beginPath();
|
||||
ctx.arc(zone.center_x, zone.center_y, zone.radius, 0, Math.PI * 2);
|
||||
ctx.fill();
|
||||
ctx.stroke();
|
||||
ctx.fillStyle = '#ffffff';
|
||||
ctx.font = '12px monospace';
|
||||
ctx.fillText(zone.name, zone.center_x - 22, zone.center_y);
|
||||
} else {
|
||||
ctx.fillRect(zone.x, zone.y, zone.width, zone.height);
|
||||
ctx.strokeRect(zone.x, zone.y, zone.width, zone.height);
|
||||
ctx.fillStyle = '#ffffff';
|
||||
ctx.font = '12px monospace';
|
||||
ctx.fillText(zone.name, zone.x + 4, zone.y + 14);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
renderSurvivors(ctx) {
|
||||
for (const survivor of this.survivors.values()) {
|
||||
const radius = survivor.is_deteriorating ? 11 : 9;
|
||||
|
||||
if (survivor.triage_status === TRIAGE.Immediate) {
|
||||
ctx.fillStyle = 'rgba(255, 0, 0, 0.26)';
|
||||
ctx.beginPath();
|
||||
ctx.arc(survivor.x, survivor.y, radius + 6, 0, Math.PI * 2);
|
||||
ctx.fill();
|
||||
}
|
||||
|
||||
ctx.fillStyle = survivor.triage_color || toRgba(TRIAGE.Minimal);
|
||||
ctx.font = 'bold 18px monospace';
|
||||
ctx.textAlign = 'center';
|
||||
ctx.textBaseline = 'middle';
|
||||
ctx.fillText('✦', survivor.x, survivor.y);
|
||||
ctx.strokeStyle = '#ffffff';
|
||||
ctx.lineWidth = 1.5;
|
||||
ctx.beginPath();
|
||||
ctx.arc(survivor.x, survivor.y, radius, 0, Math.PI * 2);
|
||||
ctx.stroke();
|
||||
|
||||
if (survivor.depth < 0) {
|
||||
ctx.fillStyle = '#ffffff';
|
||||
ctx.font = '9px monospace';
|
||||
ctx.fillText(`${Math.abs(survivor.depth).toFixed(1)}m`, survivor.x + radius + 4, survivor.y + 4);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
render(ctx, width, height) {
|
||||
ctx.clearRect(0, 0, width, height);
|
||||
ctx.fillStyle = '#0a0e1a';
|
||||
ctx.fillRect(0, 0, width, height);
|
||||
|
||||
ctx.strokeStyle = '#1f2a3d';
|
||||
ctx.lineWidth = 1;
|
||||
const grid = 40;
|
||||
for (let x = 0; x <= width; x += grid) {
|
||||
ctx.beginPath();
|
||||
ctx.moveTo(x, 0);
|
||||
ctx.lineTo(x, height);
|
||||
ctx.stroke();
|
||||
}
|
||||
for (let y = 0; y <= height; y += grid) {
|
||||
ctx.beginPath();
|
||||
ctx.moveTo(0, y);
|
||||
ctx.lineTo(width, y);
|
||||
ctx.stroke();
|
||||
}
|
||||
|
||||
this.renderZones(ctx);
|
||||
this.renderSurvivors(ctx);
|
||||
|
||||
ctx.fillStyle = '#ffffff';
|
||||
ctx.font = '12px monospace';
|
||||
const stats = {
|
||||
survivors: this.survivors.size,
|
||||
alerts: this.alerts.size,
|
||||
};
|
||||
ctx.fillText(`Survivors: ${stats.survivors}`, 12, 20);
|
||||
ctx.fillText(`Alerts: ${stats.alerts}`, 12, 36);
|
||||
}
|
||||
|
||||
postMessage(message) {
|
||||
if (typeof window.ReactNativeWebView !== 'undefined' && window.ReactNativeWebView.postMessage) {
|
||||
window.ReactNativeWebView.postMessage(JSON.stringify(message));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const dashboard = new MatDashboard();
|
||||
const canvas = document.getElementById('mapCanvas');
|
||||
const ctx = canvas.getContext('2d');
|
||||
const status = document.getElementById('status');
|
||||
|
||||
const resize = () => {
|
||||
canvas.width = Math.max(200, Math.floor(canvas.parentElement.clientWidth - 2));
|
||||
canvas.height = Math.max(180, Math.floor(canvas.parentElement.clientHeight - 20));
|
||||
};
|
||||
|
||||
const startup = () => {
|
||||
dashboard.createEvent('earthquake', 37.7749, -122.4194, 'Training Scenario');
|
||||
dashboard.addRectangleZone('Zone A', 60, 45, 170, 120);
|
||||
dashboard.addCircleZone('Zone B', 300, 170, 70);
|
||||
dashboard.processSurvivorDetection('Zone A', 0.94, { breathing_rate: 11, hr: 128 });
|
||||
dashboard.processSurvivorDetection('Zone A', 0.88, { breathing_rate: 16, hr: 118 });
|
||||
dashboard.processSurvivorDetection('Zone B', 0.71, { breathing_rate: 9, hr: 142 });
|
||||
status.textContent = 'MAT dashboard ready';
|
||||
dashboard.postMessage({ type: 'READY' });
|
||||
};
|
||||
|
||||
const loop = () => {
|
||||
if (dashboard.zones.size > 0) {
|
||||
dashboard.render(ctx, canvas.width, canvas.height);
|
||||
}
|
||||
requestAnimationFrame(loop);
|
||||
};
|
||||
|
||||
window.addEventListener('resize', resize);
|
||||
|
||||
window.addEventListener('message', (evt) => {
|
||||
let incoming = evt.data;
|
||||
try {
|
||||
if (typeof incoming === 'string') {
|
||||
incoming = JSON.parse(incoming);
|
||||
}
|
||||
} catch {
|
||||
incoming = null;
|
||||
}
|
||||
|
||||
if (!incoming || typeof incoming !== 'object') {
|
||||
return;
|
||||
}
|
||||
|
||||
if (incoming.type === 'CREATE_EVENT') {
|
||||
const payload = incoming.payload || {};
|
||||
dashboard.createEvent(
|
||||
payload.type || payload.disaster_type || 'earthquake',
|
||||
payload.latitude || 0,
|
||||
payload.longitude || 0,
|
||||
payload.name || payload.description || 'Disaster Event',
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
if (incoming.type === 'ADD_ZONE') {
|
||||
dashboard.addZoneFromPayload(incoming.payload || {});
|
||||
return;
|
||||
}
|
||||
|
||||
if (incoming.type === 'FRAME_UPDATE') {
|
||||
dashboard.processFrame(incoming.payload || {});
|
||||
}
|
||||
});
|
||||
|
||||
resize();
|
||||
startup();
|
||||
loop();
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,70 @@
|
||||
import { StyleSheet, View } from 'react-native';
|
||||
import { ThemedText } from './ThemedText';
|
||||
|
||||
type ConnectionState = 'connected' | 'simulated' | 'disconnected';
|
||||
|
||||
type ConnectionBannerProps = {
|
||||
status: ConnectionState;
|
||||
};
|
||||
|
||||
const resolveState = (status: ConnectionState) => {
|
||||
if (status === 'connected') {
|
||||
return {
|
||||
label: 'LIVE STREAM',
|
||||
backgroundColor: '#0F6B2A',
|
||||
textColor: '#E2FFEA',
|
||||
};
|
||||
}
|
||||
|
||||
if (status === 'disconnected') {
|
||||
return {
|
||||
label: 'DISCONNECTED',
|
||||
backgroundColor: '#8A1E2A',
|
||||
textColor: '#FFE3E7',
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
label: 'SIMULATED DATA',
|
||||
backgroundColor: '#9A5F0C',
|
||||
textColor: '#FFF3E1',
|
||||
};
|
||||
};
|
||||
|
||||
export const ConnectionBanner = ({ status }: ConnectionBannerProps) => {
|
||||
const state = resolveState(status);
|
||||
|
||||
return (
|
||||
<View
|
||||
style={[
|
||||
styles.banner,
|
||||
{
|
||||
backgroundColor: state.backgroundColor,
|
||||
borderBottomColor: state.textColor,
|
||||
},
|
||||
]}
|
||||
>
|
||||
<ThemedText preset="labelMd" style={[styles.text, { color: state.textColor }]}>
|
||||
{state.label}
|
||||
</ThemedText>
|
||||
</View>
|
||||
);
|
||||
};
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
banner: {
|
||||
position: 'absolute',
|
||||
left: 0,
|
||||
right: 0,
|
||||
top: 0,
|
||||
zIndex: 100,
|
||||
paddingVertical: 6,
|
||||
borderBottomWidth: 2,
|
||||
alignItems: 'center',
|
||||
justifyContent: 'center',
|
||||
},
|
||||
text: {
|
||||
letterSpacing: 2,
|
||||
fontWeight: '700',
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,66 @@
|
||||
import { Component, ErrorInfo, ReactNode } from 'react';
|
||||
import { Button, StyleSheet, View } from 'react-native';
|
||||
import { ThemedText } from './ThemedText';
|
||||
import { ThemedView } from './ThemedView';
|
||||
|
||||
type ErrorBoundaryProps = {
|
||||
children: ReactNode;
|
||||
};
|
||||
|
||||
type ErrorBoundaryState = {
|
||||
hasError: boolean;
|
||||
error?: Error;
|
||||
};
|
||||
|
||||
export class ErrorBoundary extends Component<ErrorBoundaryProps, ErrorBoundaryState> {
|
||||
constructor(props: ErrorBoundaryProps) {
|
||||
super(props);
|
||||
this.state = { hasError: false };
|
||||
}
|
||||
|
||||
static getDerivedStateFromError(error: Error): ErrorBoundaryState {
|
||||
return { hasError: true, error };
|
||||
}
|
||||
|
||||
componentDidCatch(error: Error, errorInfo: ErrorInfo) {
|
||||
console.error('ErrorBoundary caught an error', error, errorInfo);
|
||||
}
|
||||
|
||||
handleRetry = () => {
|
||||
this.setState({ hasError: false, error: undefined });
|
||||
};
|
||||
|
||||
render() {
|
||||
if (this.state.hasError) {
|
||||
return (
|
||||
<ThemedView style={styles.container}>
|
||||
<ThemedText preset="displayMd">Something went wrong</ThemedText>
|
||||
<ThemedText preset="bodySm" style={styles.message}>
|
||||
{this.state.error?.message ?? 'An unexpected error occurred.'}
|
||||
</ThemedText>
|
||||
<View style={styles.buttonWrap}>
|
||||
<Button title="Retry" onPress={this.handleRetry} />
|
||||
</View>
|
||||
</ThemedView>
|
||||
);
|
||||
}
|
||||
|
||||
return this.props.children;
|
||||
}
|
||||
}
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
container: {
|
||||
flex: 1,
|
||||
justifyContent: 'center',
|
||||
alignItems: 'center',
|
||||
padding: 20,
|
||||
gap: 12,
|
||||
},
|
||||
message: {
|
||||
textAlign: 'center',
|
||||
},
|
||||
buttonWrap: {
|
||||
marginTop: 8,
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,117 @@
|
||||
import { useEffect, useMemo } from 'react';
|
||||
import { StyleSheet, View } from 'react-native';
|
||||
import Animated, { interpolateColor, useAnimatedProps, useSharedValue, withSpring } from 'react-native-reanimated';
|
||||
import Svg, { Circle, G, Text as SvgText } from 'react-native-svg';
|
||||
|
||||
type GaugeArcProps = {
|
||||
value: number;
|
||||
min?: number;
|
||||
max: number;
|
||||
label: string;
|
||||
unit: string;
|
||||
color: string;
|
||||
colorTo?: string;
|
||||
size?: number;
|
||||
};
|
||||
|
||||
const AnimatedCircle = Animated.createAnimatedComponent(Circle);
|
||||
|
||||
const clamp = (value: number, min: number, max: number) => Math.max(min, Math.min(max, value));
|
||||
|
||||
export const GaugeArc = ({ value, min = 0, max, label, unit, color, colorTo, size = 140 }: GaugeArcProps) => {
|
||||
const radius = (size - 20) / 2;
|
||||
const circumference = 2 * Math.PI * radius;
|
||||
const arcLength = circumference * 0.75;
|
||||
const strokeWidth = 12;
|
||||
const progress = useSharedValue(0);
|
||||
|
||||
const normalized = useMemo(() => {
|
||||
const span = max - min;
|
||||
const safeSpan = span > 0 ? span : 1;
|
||||
return clamp((value - min) / safeSpan, 0, 1);
|
||||
}, [value, min, max]);
|
||||
|
||||
const displayValue = useMemo(() => {
|
||||
if (!Number.isFinite(value)) {
|
||||
return '--';
|
||||
}
|
||||
return `${Math.max(min, Math.min(max, value)).toFixed(1)} ${unit}`;
|
||||
}, [max, min, unit, value]);
|
||||
|
||||
useEffect(() => {
|
||||
progress.value = withSpring(normalized, {
|
||||
damping: 16,
|
||||
stiffness: 140,
|
||||
mass: 1,
|
||||
});
|
||||
}, [normalized, progress]);
|
||||
|
||||
const animatedStroke = useAnimatedProps(() => {
|
||||
const dashOffset = arcLength - arcLength * progress.value;
|
||||
const strokeColor = colorTo ? interpolateColor(progress.value, [0, 1], [color, colorTo]) : color;
|
||||
|
||||
return {
|
||||
strokeDashoffset: dashOffset,
|
||||
stroke: strokeColor,
|
||||
};
|
||||
});
|
||||
|
||||
return (
|
||||
<View style={styles.wrapper}>
|
||||
<Svg width={size} height={size} viewBox={`0 0 ${size} ${size}`}>
|
||||
<G transform={`rotate(-135 ${size / 2} ${size / 2})`}>
|
||||
<Circle
|
||||
cx={size / 2}
|
||||
cy={size / 2}
|
||||
r={radius}
|
||||
strokeWidth={strokeWidth}
|
||||
stroke="#1E293B"
|
||||
fill="none"
|
||||
strokeDasharray={`${arcLength} ${circumference}`}
|
||||
strokeLinecap="round"
|
||||
/>
|
||||
<AnimatedCircle
|
||||
cx={size / 2}
|
||||
cy={size / 2}
|
||||
r={radius}
|
||||
strokeWidth={strokeWidth}
|
||||
stroke={color}
|
||||
fill="none"
|
||||
strokeDasharray={`${arcLength} ${circumference}`}
|
||||
strokeLinecap="round"
|
||||
animatedProps={animatedStroke}
|
||||
/>
|
||||
</G>
|
||||
<SvgText
|
||||
x={size / 2}
|
||||
y={size / 2 - 8}
|
||||
fill="#E2E8F0"
|
||||
fontSize={Math.round(size * 0.16)}
|
||||
fontFamily="Courier New"
|
||||
fontWeight="700"
|
||||
textAnchor="middle"
|
||||
>
|
||||
{displayValue}
|
||||
</SvgText>
|
||||
<SvgText
|
||||
x={size / 2}
|
||||
y={size / 2 + 18}
|
||||
fill="#94A3B8"
|
||||
fontSize={Math.round(size * 0.085)}
|
||||
fontFamily="Courier New"
|
||||
textAnchor="middle"
|
||||
letterSpacing="0.6"
|
||||
>
|
||||
{label}
|
||||
</SvgText>
|
||||
</Svg>
|
||||
</View>
|
||||
);
|
||||
};
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
wrapper: {
|
||||
alignItems: 'center',
|
||||
justifyContent: 'center',
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,60 @@
|
||||
import { useEffect } from 'react';
|
||||
import { StyleSheet, ViewStyle } from 'react-native';
|
||||
import Animated, { Easing, useAnimatedStyle, useSharedValue, withRepeat, withTiming } from 'react-native-reanimated';
|
||||
import Svg, { Circle } from 'react-native-svg';
|
||||
import { colors } from '../theme/colors';
|
||||
|
||||
type LoadingSpinnerProps = {
|
||||
size?: number;
|
||||
color?: string;
|
||||
style?: ViewStyle;
|
||||
};
|
||||
|
||||
export const LoadingSpinner = ({ size = 36, color = colors.accent, style }: LoadingSpinnerProps) => {
|
||||
const rotation = useSharedValue(0);
|
||||
const strokeWidth = Math.max(4, size * 0.14);
|
||||
const center = size / 2;
|
||||
const radius = center - strokeWidth;
|
||||
const circumference = 2 * Math.PI * radius;
|
||||
|
||||
useEffect(() => {
|
||||
rotation.value = withRepeat(withTiming(360, { duration: 900, easing: Easing.linear }), -1);
|
||||
}, [rotation]);
|
||||
|
||||
const animatedStyle = useAnimatedStyle(() => ({
|
||||
transform: [{ rotateZ: `${rotation.value}deg` }],
|
||||
}));
|
||||
|
||||
return (
|
||||
<Animated.View style={[styles.container, { width: size, height: size }, style, animatedStyle]} pointerEvents="none">
|
||||
<Svg width={size} height={size} viewBox={`0 0 ${size} ${size}`}>
|
||||
<Circle
|
||||
cx={center}
|
||||
cy={center}
|
||||
r={radius}
|
||||
stroke="rgba(255,255,255,0.2)"
|
||||
strokeWidth={strokeWidth}
|
||||
fill="none"
|
||||
/>
|
||||
<Circle
|
||||
cx={center}
|
||||
cy={center}
|
||||
r={radius}
|
||||
stroke={color}
|
||||
strokeWidth={strokeWidth}
|
||||
fill="none"
|
||||
strokeLinecap="round"
|
||||
strokeDasharray={`${circumference * 0.3} ${circumference * 0.7}`}
|
||||
strokeDashoffset={circumference * 0.2}
|
||||
/>
|
||||
</Svg>
|
||||
</Animated.View>
|
||||
);
|
||||
};
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
container: {
|
||||
alignItems: 'center',
|
||||
justifyContent: 'center',
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,71 @@
|
||||
import { StyleSheet } from 'react-native';
|
||||
import { ThemedText } from './ThemedText';
|
||||
import { colors } from '../theme/colors';
|
||||
|
||||
type Mode = 'CSI' | 'RSSI' | 'SIM' | 'LIVE';
|
||||
|
||||
const modeStyle: Record<
|
||||
Mode,
|
||||
{
|
||||
background: string;
|
||||
border: string;
|
||||
color: string;
|
||||
}
|
||||
> = {
|
||||
CSI: {
|
||||
background: 'rgba(50, 184, 198, 0.25)',
|
||||
border: colors.accent,
|
||||
color: colors.accent,
|
||||
},
|
||||
RSSI: {
|
||||
background: 'rgba(255, 165, 2, 0.2)',
|
||||
border: colors.warn,
|
||||
color: colors.warn,
|
||||
},
|
||||
SIM: {
|
||||
background: 'rgba(255, 71, 87, 0.18)',
|
||||
border: colors.simulated,
|
||||
color: colors.simulated,
|
||||
},
|
||||
LIVE: {
|
||||
background: 'rgba(46, 213, 115, 0.18)',
|
||||
border: colors.connected,
|
||||
color: colors.connected,
|
||||
},
|
||||
};
|
||||
|
||||
type ModeBadgeProps = {
|
||||
mode: Mode;
|
||||
};
|
||||
|
||||
export const ModeBadge = ({ mode }: ModeBadgeProps) => {
|
||||
const style = modeStyle[mode];
|
||||
|
||||
return (
|
||||
<ThemedText
|
||||
preset="labelMd"
|
||||
style={[
|
||||
styles.badge,
|
||||
{
|
||||
backgroundColor: style.background,
|
||||
borderColor: style.border,
|
||||
color: style.color,
|
||||
},
|
||||
]}
|
||||
>
|
||||
{mode}
|
||||
</ThemedText>
|
||||
);
|
||||
};
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
badge: {
|
||||
paddingHorizontal: 10,
|
||||
paddingVertical: 4,
|
||||
borderRadius: 999,
|
||||
borderWidth: 1,
|
||||
overflow: 'hidden',
|
||||
letterSpacing: 1,
|
||||
textAlign: 'center',
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,147 @@
|
||||
import { useEffect, useMemo, useRef } from 'react';
|
||||
import { StyleProp, ViewStyle } from 'react-native';
|
||||
import Animated, { interpolateColor, useAnimatedProps, useSharedValue, withTiming, type SharedValue } from 'react-native-reanimated';
|
||||
import Svg, { Circle, G, Rect } from 'react-native-svg';
|
||||
import { colors } from '../theme/colors';
|
||||
|
||||
type Point = {
|
||||
x: number;
|
||||
y: number;
|
||||
};
|
||||
|
||||
type OccupancyGridProps = {
|
||||
values: number[];
|
||||
personPositions?: Point[];
|
||||
size?: number;
|
||||
style?: StyleProp<ViewStyle>;
|
||||
};
|
||||
|
||||
const GRID_DIMENSION = 20;
|
||||
const CELLS = GRID_DIMENSION * GRID_DIMENSION;
|
||||
|
||||
const toColor = (value: number): string => {
|
||||
const clamped = Math.max(0, Math.min(1, value));
|
||||
let r: number;
|
||||
let g: number;
|
||||
let b: number;
|
||||
|
||||
if (clamped < 0.5) {
|
||||
const t = clamped * 2;
|
||||
r = Math.round(255 * 0);
|
||||
g = Math.round(255 * t);
|
||||
b = Math.round(255 * (1 - t));
|
||||
} else {
|
||||
const t = (clamped - 0.5) * 2;
|
||||
r = Math.round(255 * t);
|
||||
g = Math.round(255 * (1 - t));
|
||||
b = 0;
|
||||
}
|
||||
|
||||
return `rgb(${r}, ${g}, ${b})`;
|
||||
};
|
||||
|
||||
const AnimatedRect = Animated.createAnimatedComponent(Rect);
|
||||
|
||||
const normalizeValues = (values: number[]) => {
|
||||
const normalized = new Array(CELLS).fill(0);
|
||||
for (let i = 0; i < CELLS; i += 1) {
|
||||
const value = values?.[i] ?? 0;
|
||||
normalized[i] = Number.isFinite(value) ? Math.max(0, Math.min(1, value)) : 0;
|
||||
}
|
||||
return normalized;
|
||||
};
|
||||
|
||||
type CellProps = {
|
||||
index: number;
|
||||
size: number;
|
||||
progress: SharedValue<number>;
|
||||
previousColors: string[];
|
||||
nextColors: string[];
|
||||
};
|
||||
|
||||
const Cell = ({ index, size, progress, previousColors, nextColors }: CellProps) => {
|
||||
const col = index % GRID_DIMENSION;
|
||||
const row = Math.floor(index / GRID_DIMENSION);
|
||||
const cellSize = size / GRID_DIMENSION;
|
||||
const x = col * cellSize;
|
||||
const y = row * cellSize;
|
||||
|
||||
const animatedProps = useAnimatedProps(() => ({
|
||||
fill: interpolateColor(
|
||||
progress.value,
|
||||
[0, 1],
|
||||
[previousColors[index] ?? colors.surfaceAlt, nextColors[index] ?? colors.surfaceAlt],
|
||||
),
|
||||
}));
|
||||
|
||||
return (
|
||||
<AnimatedRect
|
||||
x={x}
|
||||
y={y}
|
||||
width={cellSize}
|
||||
height={cellSize}
|
||||
rx={1}
|
||||
animatedProps={animatedProps}
|
||||
/>
|
||||
);
|
||||
};
|
||||
|
||||
export const OccupancyGrid = ({
|
||||
values,
|
||||
personPositions = [],
|
||||
size = 320,
|
||||
style,
|
||||
}: OccupancyGridProps) => {
|
||||
const normalizedValues = useMemo(() => normalizeValues(values), [values]);
|
||||
const previousColors = useRef<string[]>(normalizedValues.map(toColor));
|
||||
const nextColors = useRef<string[]>(normalizedValues.map(toColor));
|
||||
const progress = useSharedValue(1);
|
||||
|
||||
useEffect(() => {
|
||||
const next = normalizeValues(values);
|
||||
previousColors.current = normalizedValues.map(toColor);
|
||||
nextColors.current = next.map(toColor);
|
||||
progress.value = 0;
|
||||
progress.value = withTiming(1, { duration: 500 });
|
||||
}, [values, normalizedValues, progress]);
|
||||
|
||||
const markers = useMemo(() => {
|
||||
const cellSize = size / GRID_DIMENSION;
|
||||
return personPositions.map(({ x, y }, idx) => {
|
||||
const clampedX = Math.max(0, Math.min(GRID_DIMENSION - 1, Math.round(x)));
|
||||
const clampedY = Math.max(0, Math.min(GRID_DIMENSION - 1, Math.round(y)));
|
||||
const cx = (clampedX + 0.5) * cellSize;
|
||||
const cy = (clampedY + 0.5) * cellSize;
|
||||
const markerRadius = Math.max(3, cellSize * 0.25);
|
||||
return (
|
||||
<Circle
|
||||
key={`person-${idx}`}
|
||||
cx={cx}
|
||||
cy={cy}
|
||||
r={markerRadius}
|
||||
fill={colors.accent}
|
||||
stroke={colors.textPrimary}
|
||||
strokeWidth={1}
|
||||
/>
|
||||
);
|
||||
});
|
||||
}, [personPositions, size]);
|
||||
|
||||
return (
|
||||
<Svg width={size} height={size} style={style} viewBox={`0 0 ${size} ${size}`}>
|
||||
<G>
|
||||
{Array.from({ length: CELLS }).map((_, index) => (
|
||||
<Cell
|
||||
key={index}
|
||||
index={index}
|
||||
size={size}
|
||||
progress={progress}
|
||||
previousColors={previousColors.current}
|
||||
nextColors={nextColors.current}
|
||||
/>
|
||||
))}
|
||||
</G>
|
||||
{markers}
|
||||
</Svg>
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,62 @@
|
||||
import { useEffect } from 'react';
|
||||
import { StyleSheet, View } from 'react-native';
|
||||
import Animated, { useAnimatedStyle, useSharedValue, withTiming } from 'react-native-reanimated';
|
||||
import { ThemedText } from './ThemedText';
|
||||
import { colors } from '../theme/colors';
|
||||
|
||||
type SignalBarProps = {
|
||||
value: number;
|
||||
label: string;
|
||||
color?: string;
|
||||
};
|
||||
|
||||
const clamp01 = (value: number) => Math.max(0, Math.min(1, value));
|
||||
|
||||
export const SignalBar = ({ value, label, color = colors.accent }: SignalBarProps) => {
|
||||
const progress = useSharedValue(clamp01(value));
|
||||
|
||||
useEffect(() => {
|
||||
progress.value = withTiming(clamp01(value), { duration: 250 });
|
||||
}, [value, progress]);
|
||||
|
||||
const animatedFill = useAnimatedStyle(() => ({
|
||||
width: `${progress.value * 100}%`,
|
||||
}));
|
||||
|
||||
return (
|
||||
<View style={styles.container}>
|
||||
<ThemedText preset="bodySm" style={styles.label}>
|
||||
{label}
|
||||
</ThemedText>
|
||||
<View style={styles.track}>
|
||||
<Animated.View style={[styles.fill, { backgroundColor: color }, animatedFill]} />
|
||||
</View>
|
||||
<ThemedText preset="bodySm" style={styles.percent}>
|
||||
{Math.round(clamp01(value) * 100)}%
|
||||
</ThemedText>
|
||||
</View>
|
||||
);
|
||||
};
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
container: {
|
||||
gap: 6,
|
||||
},
|
||||
label: {
|
||||
marginBottom: 4,
|
||||
},
|
||||
track: {
|
||||
height: 8,
|
||||
borderRadius: 4,
|
||||
backgroundColor: colors.surfaceAlt,
|
||||
overflow: 'hidden',
|
||||
},
|
||||
fill: {
|
||||
height: '100%',
|
||||
borderRadius: 4,
|
||||
},
|
||||
percent: {
|
||||
textAlign: 'right',
|
||||
color: colors.textSecondary,
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,64 @@
|
||||
import { useMemo } from 'react';
|
||||
import { View, ViewStyle } from 'react-native';
|
||||
import { colors } from '../theme/colors';
|
||||
|
||||
type SparklineChartProps = {
|
||||
data: number[];
|
||||
color?: string;
|
||||
height?: number;
|
||||
style?: ViewStyle;
|
||||
};
|
||||
|
||||
const defaultHeight = 72;
|
||||
|
||||
export const SparklineChart = ({
|
||||
data,
|
||||
color = colors.accent,
|
||||
height = defaultHeight,
|
||||
style,
|
||||
}: SparklineChartProps) => {
|
||||
const normalizedData = data.length > 0 ? data : [0];
|
||||
|
||||
const chartData = useMemo(
|
||||
() =>
|
||||
normalizedData.map((value, index) => ({
|
||||
x: index,
|
||||
y: value,
|
||||
})),
|
||||
[normalizedData],
|
||||
);
|
||||
|
||||
const yValues = normalizedData.map((value) => Number(value) || 0);
|
||||
const yMin = Math.min(...yValues);
|
||||
const yMax = Math.max(...yValues);
|
||||
const yPadding = yMax - yMin === 0 ? 1 : (yMax - yMin) * 0.2;
|
||||
|
||||
return (
|
||||
<View style={style}>
|
||||
<View
|
||||
accessibilityRole="image"
|
||||
style={{
|
||||
height,
|
||||
width: '100%',
|
||||
borderRadius: 4,
|
||||
borderWidth: 1,
|
||||
borderColor: color,
|
||||
opacity: 0.2,
|
||||
backgroundColor: 'transparent',
|
||||
}}
|
||||
>
|
||||
<View
|
||||
style={{
|
||||
flex: 1,
|
||||
justifyContent: 'center',
|
||||
alignItems: 'center',
|
||||
}}
|
||||
>
|
||||
{chartData.map((point) => (
|
||||
<View key={point.x} style={{ position: 'absolute', left: `${(point.x / Math.max(normalizedData.length - 1, 1)) * 100}%` }} />
|
||||
))}
|
||||
</View>
|
||||
</View>
|
||||
</View>
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,83 @@
|
||||
import { useEffect } from 'react';
|
||||
import { StyleSheet, ViewStyle } from 'react-native';
|
||||
import Animated, {
|
||||
cancelAnimation,
|
||||
Easing,
|
||||
useAnimatedStyle,
|
||||
useSharedValue,
|
||||
withRepeat,
|
||||
withSequence,
|
||||
withTiming,
|
||||
} from 'react-native-reanimated';
|
||||
import { colors } from '../theme/colors';
|
||||
|
||||
type StatusType = 'connected' | 'simulated' | 'disconnected' | 'connecting';
|
||||
|
||||
type StatusDotProps = {
|
||||
status: StatusType;
|
||||
size?: number;
|
||||
style?: ViewStyle;
|
||||
};
|
||||
|
||||
const resolveColor = (status: StatusType): string => {
|
||||
if (status === 'connecting') return colors.warn;
|
||||
return colors[status];
|
||||
};
|
||||
|
||||
export const StatusDot = ({ status, size = 10, style }: StatusDotProps) => {
|
||||
const scale = useSharedValue(1);
|
||||
const opacity = useSharedValue(1);
|
||||
const isConnecting = status === 'connecting';
|
||||
|
||||
useEffect(() => {
|
||||
if (isConnecting) {
|
||||
scale.value = withRepeat(
|
||||
withSequence(
|
||||
withTiming(1.35, { duration: 800, easing: Easing.out(Easing.cubic) }),
|
||||
withTiming(1, { duration: 800, easing: Easing.in(Easing.cubic) }),
|
||||
),
|
||||
-1,
|
||||
);
|
||||
opacity.value = withRepeat(
|
||||
withSequence(
|
||||
withTiming(0.4, { duration: 800, easing: Easing.out(Easing.quad) }),
|
||||
withTiming(1, { duration: 800, easing: Easing.in(Easing.quad) }),
|
||||
),
|
||||
-1,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
cancelAnimation(scale);
|
||||
cancelAnimation(opacity);
|
||||
scale.value = 1;
|
||||
opacity.value = 1;
|
||||
}, [isConnecting, opacity, scale]);
|
||||
|
||||
const animatedStyle = useAnimatedStyle(() => ({
|
||||
transform: [{ scale: scale.value }],
|
||||
opacity: opacity.value,
|
||||
}));
|
||||
|
||||
return (
|
||||
<Animated.View
|
||||
style={[
|
||||
styles.dot,
|
||||
{
|
||||
width: size,
|
||||
height: size,
|
||||
backgroundColor: resolveColor(status),
|
||||
borderRadius: size / 2,
|
||||
},
|
||||
animatedStyle,
|
||||
style,
|
||||
]}
|
||||
/>
|
||||
);
|
||||
};
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
dot: {
|
||||
borderRadius: 999,
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,28 @@
|
||||
import { ComponentPropsWithoutRef } from 'react';
|
||||
import { StyleProp, Text, TextStyle } from 'react-native';
|
||||
import { useTheme } from '../hooks/useTheme';
|
||||
import { colors } from '../theme/colors';
|
||||
import { typography } from '../theme/typography';
|
||||
|
||||
type TextPreset = keyof typeof typography;
|
||||
type ColorKey = keyof typeof colors;
|
||||
|
||||
type ThemedTextProps = Omit<ComponentPropsWithoutRef<typeof Text>, 'style'> & {
|
||||
preset?: TextPreset;
|
||||
color?: ColorKey;
|
||||
style?: StyleProp<TextStyle>;
|
||||
};
|
||||
|
||||
export const ThemedText = ({
|
||||
preset = 'bodyMd',
|
||||
color = 'textPrimary',
|
||||
style,
|
||||
...props
|
||||
}: ThemedTextProps) => {
|
||||
const { colors, typography } = useTheme();
|
||||
|
||||
const presetStyle = (typography as Record<TextPreset, TextStyle>)[preset];
|
||||
const colorStyle = { color: colors[color] };
|
||||
|
||||
return <Text {...props} style={[presetStyle, colorStyle, style]} />;
|
||||
};
|
||||
@@ -0,0 +1,24 @@
|
||||
import { PropsWithChildren, forwardRef } from 'react';
|
||||
import { View, ViewProps } from 'react-native';
|
||||
import { useTheme } from '../hooks/useTheme';
|
||||
|
||||
type ThemedViewProps = PropsWithChildren<ViewProps>;
|
||||
|
||||
export const ThemedView = forwardRef<View, ThemedViewProps>(({ children, style, ...props }, ref) => {
|
||||
const { colors } = useTheme();
|
||||
|
||||
return (
|
||||
<View
|
||||
ref={ref}
|
||||
{...props}
|
||||
style={[
|
||||
{
|
||||
backgroundColor: colors.bg,
|
||||
},
|
||||
style,
|
||||
]}
|
||||
>
|
||||
{children}
|
||||
</View>
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,14 @@
|
||||
export const API_ROOT = '/api/v1';
|
||||
|
||||
export const API_POSE_STATUS_PATH = '/api/v1/pose/status';
|
||||
export const API_POSE_FRAMES_PATH = '/api/v1/pose/frames';
|
||||
export const API_POSE_ZONES_PATH = '/api/v1/pose/zones';
|
||||
export const API_POSE_CURRENT_PATH = '/api/v1/pose/current';
|
||||
export const API_STREAM_STATUS_PATH = '/api/v1/stream/status';
|
||||
export const API_STREAM_POSE_PATH = '/api/v1/stream/pose';
|
||||
export const API_MAT_EVENTS_PATH = '/api/v1/mat/events';
|
||||
|
||||
export const API_HEALTH_PATH = '/health';
|
||||
export const API_HEALTH_SYSTEM_PATH = '/health/health';
|
||||
export const API_HEALTH_READY_PATH = '/health/ready';
|
||||
export const API_HEALTH_LIVE_PATH = '/health/live';
|
||||
@@ -0,0 +1,20 @@
|
||||
export const SIMULATION_TICK_INTERVAL_MS = 500;
|
||||
export const SIMULATION_GRID_SIZE = 20;
|
||||
|
||||
export const RSSI_BASE_DBM = -45;
|
||||
export const RSSI_AMPLITUDE_DBM = 3;
|
||||
|
||||
export const VARIANCE_BASE = 1.5;
|
||||
export const VARIANCE_AMPLITUDE = 1.0;
|
||||
|
||||
export const MOTION_BAND_MIN = 0.05;
|
||||
export const MOTION_BAND_AMPLITUDE = 0.15;
|
||||
export const BREATHING_BAND_MIN = 0.03;
|
||||
export const BREATHING_BAND_AMPLITUDE = 0.08;
|
||||
|
||||
export const SIGNAL_FIELD_PRESENCE_LEVEL = 0.8;
|
||||
|
||||
export const BREATHING_BPM_MIN = 12;
|
||||
export const BREATHING_BPM_MAX = 24;
|
||||
export const HEART_BPM_MIN = 58;
|
||||
export const HEART_BPM_MAX = 96;
|
||||
@@ -0,0 +1,3 @@
|
||||
export const WS_PATH = '/api/v1/stream/pose';
|
||||
export const RECONNECT_DELAYS = [1000, 2000, 4000, 8000, 16000];
|
||||
export const MAX_RECONNECT_ATTEMPTS = 10;
|
||||
@@ -0,0 +1,32 @@
|
||||
import { useEffect } from 'react';
|
||||
import { wsService } from '@/services/ws.service';
|
||||
import { usePoseStore } from '@/stores/poseStore';
|
||||
import { useSettingsStore } from '@/stores/settingsStore';
|
||||
|
||||
export interface UsePoseStreamResult {
|
||||
connectionStatus: ReturnType<typeof usePoseStore.getState>['connectionStatus'];
|
||||
lastFrame: ReturnType<typeof usePoseStore.getState>['lastFrame'];
|
||||
isSimulated: boolean;
|
||||
}
|
||||
|
||||
export function usePoseStream(): UsePoseStreamResult {
|
||||
const connectionStatus = usePoseStore((state) => state.connectionStatus);
|
||||
const lastFrame = usePoseStore((state) => state.lastFrame);
|
||||
const isSimulated = usePoseStore((state) => state.isSimulated);
|
||||
const serverUrl = useSettingsStore((state) => state.serverUrl);
|
||||
|
||||
useEffect(() => {
|
||||
const unsubscribe = wsService.subscribe((frame) => {
|
||||
usePoseStore.getState().handleFrame(frame);
|
||||
});
|
||||
|
||||
// Auto-connect to sensing server on mount
|
||||
wsService.connect(serverUrl);
|
||||
|
||||
return () => {
|
||||
unsubscribe();
|
||||
};
|
||||
}, [serverUrl]);
|
||||
|
||||
return { connectionStatus, lastFrame, isSimulated };
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
import { useEffect, useState } from 'react';
|
||||
import { rssiService, type WifiNetwork } from '@/services/rssi.service';
|
||||
import { useSettingsStore } from '@/stores/settingsStore';
|
||||
|
||||
export function useRssiScanner(): { networks: WifiNetwork[]; isScanning: boolean } {
|
||||
const enabled = useSettingsStore((state) => state.rssiScanEnabled);
|
||||
const [networks, setNetworks] = useState<WifiNetwork[]>([]);
|
||||
const [isScanning, setIsScanning] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
if (!enabled) {
|
||||
rssiService.stopScanning();
|
||||
setIsScanning(false);
|
||||
return;
|
||||
}
|
||||
|
||||
const unsubscribe = rssiService.subscribe((result) => {
|
||||
setNetworks(result);
|
||||
});
|
||||
rssiService.startScanning(2000);
|
||||
setIsScanning(true);
|
||||
|
||||
return () => {
|
||||
unsubscribe();
|
||||
rssiService.stopScanning();
|
||||
setIsScanning(false);
|
||||
};
|
||||
}, [enabled]);
|
||||
|
||||
return { networks, isScanning };
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
import { useEffect, useState } from 'react';
|
||||
import { apiService } from '@/services/api.service';
|
||||
|
||||
interface ServerReachability {
|
||||
reachable: boolean;
|
||||
latencyMs: number | null;
|
||||
}
|
||||
|
||||
const POLL_MS = 10000;
|
||||
|
||||
export function useServerReachability(): ServerReachability {
|
||||
const [state, setState] = useState<ServerReachability>({
|
||||
reachable: false,
|
||||
latencyMs: null,
|
||||
});
|
||||
|
||||
useEffect(() => {
|
||||
let active = true;
|
||||
|
||||
const check = async () => {
|
||||
const started = Date.now();
|
||||
try {
|
||||
await apiService.getStatus();
|
||||
if (!active) {
|
||||
return;
|
||||
}
|
||||
setState({
|
||||
reachable: true,
|
||||
latencyMs: Date.now() - started,
|
||||
});
|
||||
} catch {
|
||||
if (!active) {
|
||||
return;
|
||||
}
|
||||
setState({
|
||||
reachable: false,
|
||||
latencyMs: null,
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
void check();
|
||||
const timer = setInterval(check, POLL_MS);
|
||||
|
||||
return () => {
|
||||
active = false;
|
||||
clearInterval(timer);
|
||||
};
|
||||
}, []);
|
||||
|
||||
return state;
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
import { useContext } from 'react';
|
||||
import { ThemeContext, ThemeContextValue } from '../theme/ThemeContext';
|
||||
|
||||
export const useTheme = (): ThemeContextValue => useContext(ThemeContext);
|
||||
@@ -0,0 +1,132 @@
|
||||
import React, { Suspense } from 'react';
|
||||
import { ActivityIndicator } from 'react-native';
|
||||
import { createBottomTabNavigator } from '@react-navigation/bottom-tabs';
|
||||
import { Ionicons } from '@expo/vector-icons';
|
||||
import { ThemedText } from '../components/ThemedText';
|
||||
import { ThemedView } from '../components/ThemedView';
|
||||
import { colors } from '../theme/colors';
|
||||
import { useMatStore } from '../stores/matStore';
|
||||
import { MainTabsParamList } from './types';
|
||||
|
||||
const createPlaceholder = (label: string) => {
|
||||
const Placeholder = () => (
|
||||
<ThemedView style={{ flex: 1, alignItems: 'center', justifyContent: 'center' }}>
|
||||
<ThemedText preset="bodyLg">{label} screen not implemented yet</ThemedText>
|
||||
<ThemedText preset="bodySm" color="textSecondary">
|
||||
Placeholder shell
|
||||
</ThemedText>
|
||||
</ThemedView>
|
||||
);
|
||||
const LazyPlaceholder = React.lazy(async () => ({ default: Placeholder }));
|
||||
|
||||
const Wrapped = () => (
|
||||
<Suspense
|
||||
fallback={
|
||||
<ThemedView style={{ flex: 1, alignItems: 'center', justifyContent: 'center' }}>
|
||||
<ActivityIndicator color={colors.accent} />
|
||||
<ThemedText preset="bodySm" color="textSecondary" style={{ marginTop: 8 }}>
|
||||
Loading {label}
|
||||
</ThemedText>
|
||||
</ThemedView>
|
||||
}
|
||||
>
|
||||
<LazyPlaceholder />
|
||||
</Suspense>
|
||||
);
|
||||
|
||||
return Wrapped;
|
||||
};
|
||||
|
||||
const wrapLazy = (
|
||||
loader: () => Promise<{ default: React.ComponentType }>,
|
||||
label: string,
|
||||
) => {
|
||||
const fallback = createPlaceholder(label);
|
||||
return React.lazy(async () => {
|
||||
try {
|
||||
const module = await loader();
|
||||
if (module?.default) {
|
||||
return module;
|
||||
}
|
||||
} catch {
|
||||
// keep fallback for shell-only screens
|
||||
}
|
||||
return { default: fallback } as { default: React.ComponentType };
|
||||
});
|
||||
};
|
||||
|
||||
const LiveScreen = wrapLazy(() => import('../screens/LiveScreen'), 'Live');
|
||||
const VitalsScreen = wrapLazy(() => import('../screens/VitalsScreen'), 'Vitals');
|
||||
const ZonesScreen = wrapLazy(() => import('../screens/ZonesScreen'), 'Zones');
|
||||
const MATScreen = wrapLazy(() => import('../screens/MATScreen'), 'MAT');
|
||||
const SettingsScreen = wrapLazy(() => import('../screens/SettingsScreen'), 'Settings');
|
||||
|
||||
const toIconName = (routeName: keyof MainTabsParamList) => {
|
||||
switch (routeName) {
|
||||
case 'Live':
|
||||
return 'wifi';
|
||||
case 'Vitals':
|
||||
return 'heart';
|
||||
case 'Zones':
|
||||
return 'grid';
|
||||
case 'MAT':
|
||||
return 'shield-checkmark';
|
||||
case 'Settings':
|
||||
return 'settings';
|
||||
default:
|
||||
return 'ellipse';
|
||||
}
|
||||
};
|
||||
|
||||
const screens: ReadonlyArray<{ name: keyof MainTabsParamList; component: React.ComponentType }> = [
|
||||
{ name: 'Live', component: LiveScreen },
|
||||
{ name: 'Vitals', component: VitalsScreen },
|
||||
{ name: 'Zones', component: ZonesScreen },
|
||||
{ name: 'MAT', component: MATScreen },
|
||||
{ name: 'Settings', component: SettingsScreen },
|
||||
];
|
||||
|
||||
const Tab = createBottomTabNavigator<MainTabsParamList>();
|
||||
|
||||
const Suspended = ({ component: Component }: { component: React.ComponentType }) => (
|
||||
<Suspense fallback={<ActivityIndicator color={colors.accent} />}>
|
||||
<Component />
|
||||
</Suspense>
|
||||
);
|
||||
|
||||
export const MainTabs = () => {
|
||||
const matAlertCount = useMatStore((state) => state.alerts.length);
|
||||
|
||||
return (
|
||||
<Tab.Navigator
|
||||
screenOptions={({ route }) => ({
|
||||
headerShown: false,
|
||||
tabBarActiveTintColor: colors.accent,
|
||||
tabBarInactiveTintColor: colors.textSecondary,
|
||||
tabBarStyle: {
|
||||
backgroundColor: '#0D1117',
|
||||
borderTopColor: colors.border,
|
||||
borderTopWidth: 1,
|
||||
},
|
||||
tabBarIcon: ({ color, size }) => <Ionicons name={toIconName(route.name)} size={size} color={color} />,
|
||||
tabBarLabelStyle: {
|
||||
fontFamily: 'Courier New',
|
||||
textTransform: 'uppercase',
|
||||
fontSize: 10,
|
||||
},
|
||||
tabBarLabel: ({ children, color }) => <ThemedText style={{ color }}>{children}</ThemedText>,
|
||||
})}
|
||||
>
|
||||
{screens.map(({ name, component }) => (
|
||||
<Tab.Screen
|
||||
key={name}
|
||||
name={name}
|
||||
options={{
|
||||
tabBarBadge: name === 'MAT' ? (matAlertCount > 0 ? matAlertCount : undefined) : undefined,
|
||||
}}
|
||||
component={() => <Suspended component={component} />}
|
||||
/>
|
||||
))}
|
||||
</Tab.Navigator>
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,5 @@
|
||||
import { MainTabs } from './MainTabs';
|
||||
|
||||
export const RootNavigator = () => {
|
||||
return <MainTabs />;
|
||||
};
|
||||
@@ -0,0 +1,11 @@
|
||||
export type RootStackParamList = {
|
||||
MainTabs: undefined;
|
||||
};
|
||||
|
||||
export type MainTabsParamList = {
|
||||
Live: undefined;
|
||||
Vitals: undefined;
|
||||
Zones: undefined;
|
||||
MAT: undefined;
|
||||
Settings: undefined;
|
||||
};
|
||||
@@ -0,0 +1,41 @@
|
||||
import { LayoutChangeEvent, StyleSheet } from 'react-native';
|
||||
import type { RefObject } from 'react';
|
||||
import { WebView, type WebViewMessageEvent } from 'react-native-webview';
|
||||
import GAUSSIAN_SPLATS_HTML from '@/assets/webview/gaussian-splats.html';
|
||||
|
||||
type GaussianSplatWebViewProps = {
|
||||
onMessage: (event: WebViewMessageEvent) => void;
|
||||
onError: () => void;
|
||||
webViewRef: RefObject<WebView | null>;
|
||||
onLayout?: (event: LayoutChangeEvent) => void;
|
||||
};
|
||||
|
||||
export const GaussianSplatWebView = ({
|
||||
onMessage,
|
||||
onError,
|
||||
webViewRef,
|
||||
onLayout,
|
||||
}: GaussianSplatWebViewProps) => {
|
||||
const html = typeof GAUSSIAN_SPLATS_HTML === 'string' ? GAUSSIAN_SPLATS_HTML : '';
|
||||
|
||||
return (
|
||||
<WebView
|
||||
ref={webViewRef}
|
||||
source={{ html }}
|
||||
originWhitelist={['*']}
|
||||
allowFileAccess={false}
|
||||
javaScriptEnabled
|
||||
onMessage={onMessage}
|
||||
onError={onError}
|
||||
onLayout={onLayout}
|
||||
style={styles.webView}
|
||||
/>
|
||||
);
|
||||
};
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
webView: {
|
||||
flex: 1,
|
||||
backgroundColor: '#0A0E1A',
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,700 @@
|
||||
import { useCallback, useEffect, useRef } from 'react';
|
||||
import { StyleSheet, View } from 'react-native';
|
||||
import * as THREE from 'three';
|
||||
import type { SensingFrame } from '@/types/sensing';
|
||||
|
||||
type Props = {
|
||||
onReady: () => void;
|
||||
onFps: (fps: number) => void;
|
||||
onError: (msg: string) => void;
|
||||
frame: SensingFrame | null;
|
||||
};
|
||||
|
||||
// COCO skeleton bones
|
||||
const BONES: [number, number][] = [
|
||||
[0,1],[0,2],[1,3],[2,4],[5,6],[5,7],[7,9],[6,8],[8,10],
|
||||
[5,11],[6,12],[11,12],[11,13],[13,15],[12,14],[14,16],
|
||||
];
|
||||
|
||||
// Standing pose (meters, Y-up)
|
||||
const BASE_POSE: [number, number, number][] = [
|
||||
[ 0.00, 1.72, 0.04], // 0 nose
|
||||
[-0.03, 1.76, 0.05], // 1 left eye
|
||||
[ 0.03, 1.76, 0.05], // 2 right eye
|
||||
[-0.08, 1.74,-0.01], // 3 left ear
|
||||
[ 0.08, 1.74,-0.01], // 4 right ear
|
||||
[-0.20, 1.45, 0.00], // 5 left shoulder
|
||||
[ 0.20, 1.45, 0.00], // 6 right shoulder
|
||||
[-0.26, 1.12, 0.04], // 7 left elbow
|
||||
[ 0.26, 1.12, 0.04], // 8 right elbow
|
||||
[-0.28, 0.82, 0.02], // 9 left wrist
|
||||
[ 0.28, 0.82, 0.02], // 10 right wrist
|
||||
[-0.11, 0.95, 0.00], // 11 left hip
|
||||
[ 0.11, 0.95, 0.00], // 12 right hip
|
||||
[-0.12, 0.50, 0.02], // 13 left knee
|
||||
[ 0.12, 0.50, 0.02], // 14 right knee
|
||||
[-0.12, 0.04, 0.00], // 15 left ankle
|
||||
[ 0.12, 0.04, 0.00], // 16 right ankle
|
||||
];
|
||||
|
||||
// DensePose-style body part colors (24 parts → simplified per-segment)
|
||||
const DENSEPOSE_COLORS: Record<string, number> = {
|
||||
head: 0xf4a582, // warm skin
|
||||
neck: 0xd6604d, // darker warm
|
||||
torsoFront: 0x92c5de, // blue-gray
|
||||
torsoSide: 0x4393c3, // steel blue
|
||||
pelvis: 0x2166ac, // deep blue
|
||||
lUpperArm: 0xd73027, // red
|
||||
rUpperArm: 0xf46d43, // orange-red
|
||||
lForearm: 0xfdae61, // orange
|
||||
rForearm: 0xfee090, // light orange
|
||||
lHand: 0xffffbf, // pale yellow
|
||||
rHand: 0xffffbf,
|
||||
lThigh: 0xa6d96a, // green
|
||||
rThigh: 0x66bd63, // darker green
|
||||
lShin: 0x1a9850, // deep green
|
||||
rShin: 0x006837, // forest
|
||||
lFoot: 0x762a83, // purple
|
||||
rFoot: 0x9970ab, // light purple
|
||||
};
|
||||
|
||||
// Body segments: [jointA, jointB, topRadius, botRadius, colorKey]
|
||||
const BODY_SEGS: [number, number, number, number, string][] = [
|
||||
[5, 6, 0.10, 0.10, 'torsoFront'], // collar
|
||||
[5, 11, 0.09, 0.07, 'torsoSide'], // L torso
|
||||
[6, 12, 0.09, 0.07, 'torsoSide'], // R torso
|
||||
[11, 12, 0.08, 0.08, 'pelvis'], // pelvis
|
||||
[5, 7, 0.045,0.040,'lUpperArm'], // L upper arm
|
||||
[7, 9, 0.038,0.032,'lForearm'], // L forearm
|
||||
[6, 8, 0.045,0.040,'rUpperArm'], // R upper arm
|
||||
[8, 10, 0.038,0.032,'rForearm'], // R forearm
|
||||
[11, 13, 0.065,0.050,'lThigh'], // L thigh
|
||||
[13, 15, 0.048,0.038,'lShin'], // L shin
|
||||
[12, 14, 0.065,0.050,'rThigh'], // R thigh
|
||||
[14, 16, 0.048,0.038,'rShin'], // R shin
|
||||
];
|
||||
|
||||
function makePart(scene: THREE.Scene, rTop: number, rBot: number, color: number, glow: boolean = false): THREE.Mesh {
|
||||
const geo = new THREE.CapsuleGeometry((rTop + rBot) / 2, 1, 6, 12);
|
||||
const mat = new THREE.MeshPhysicalMaterial({
|
||||
color, emissive: color,
|
||||
emissiveIntensity: glow ? 0.4 : 0.08,
|
||||
transparent: true, opacity: glow ? 0.12 : 0.85,
|
||||
roughness: 0.35, metalness: 0.1,
|
||||
clearcoat: glow ? 0 : 0.3, clearcoatRoughness: 0.4,
|
||||
side: glow ? THREE.BackSide : THREE.FrontSide,
|
||||
});
|
||||
const m = new THREE.Mesh(geo, mat);
|
||||
m.visible = false;
|
||||
m.castShadow = !glow;
|
||||
scene.add(m);
|
||||
return m;
|
||||
}
|
||||
|
||||
function positionLimb(mesh: THREE.Mesh, a: THREE.Vector3, b: THREE.Vector3, rTop: number, rBot: number) {
|
||||
const mid = new THREE.Vector3().addVectors(a, b).multiplyScalar(0.5);
|
||||
mesh.position.copy(mid);
|
||||
const len = a.distanceTo(b);
|
||||
// CapsuleGeometry height param = 1, so scale Y to actual length
|
||||
mesh.scale.set((rTop + rBot) * 10, len, (rTop + rBot) * 10);
|
||||
const dir = new THREE.Vector3().subVectors(b, a).normalize();
|
||||
const up = new THREE.Vector3(0, 1, 0);
|
||||
const quat = new THREE.Quaternion().setFromUnitVectors(up, dir);
|
||||
mesh.quaternion.copy(quat);
|
||||
}
|
||||
|
||||
function lerp3(out: THREE.Vector3, target: THREE.Vector3, alpha: number) {
|
||||
out.x += (target.x - out.x) * alpha;
|
||||
out.y += (target.y - out.y) * alpha;
|
||||
out.z += (target.z - out.z) * alpha;
|
||||
}
|
||||
|
||||
export const GaussianSplatWebViewWeb = ({ onReady, onFps, onError, frame }: Props) => {
|
||||
const containerRef = useRef<HTMLDivElement>(null);
|
||||
const frameRef = useRef<SensingFrame | null>(null);
|
||||
const sceneRef = useRef<any>(null);
|
||||
frameRef.current = frame;
|
||||
|
||||
const cleanup = useCallback(() => {
|
||||
const s = sceneRef.current;
|
||||
if (!s) return;
|
||||
cancelAnimationFrame(s.animId);
|
||||
s.renderer.dispose();
|
||||
s.scene.traverse((obj: any) => {
|
||||
if (obj.geometry) obj.geometry.dispose();
|
||||
if (obj.material) {
|
||||
const mats = Array.isArray(obj.material) ? obj.material : [obj.material];
|
||||
mats.forEach((m: any) => m.dispose());
|
||||
}
|
||||
});
|
||||
sceneRef.current = null;
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
const container = containerRef.current;
|
||||
if (!container) return;
|
||||
try {
|
||||
const W = () => container.clientWidth || window.innerWidth;
|
||||
const H = () => container.clientHeight || window.innerHeight;
|
||||
|
||||
// --- Renderer ---
|
||||
const renderer = new THREE.WebGLRenderer({ antialias: true, powerPreference: 'high-performance' });
|
||||
renderer.setSize(W(), H());
|
||||
renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2));
|
||||
renderer.setClearColor(0x080c16);
|
||||
renderer.shadowMap.enabled = true;
|
||||
renderer.shadowMap.type = THREE.PCFSoftShadowMap;
|
||||
renderer.toneMapping = THREE.ACESFilmicToneMapping;
|
||||
renderer.toneMappingExposure = 1.1;
|
||||
container.appendChild(renderer.domElement);
|
||||
|
||||
const scene = new THREE.Scene();
|
||||
scene.background = new THREE.Color(0x080c16);
|
||||
scene.fog = new THREE.FogExp2(0x080c16, 0.018);
|
||||
|
||||
const camera = new THREE.PerspectiveCamera(45, W() / H(), 0.1, 200);
|
||||
camera.position.set(0, 1.4, 3.5);
|
||||
camera.lookAt(0, 0.9, 0);
|
||||
|
||||
// --- Lighting (3-point + rim) ---
|
||||
scene.add(new THREE.AmbientLight(0x223344, 0.5));
|
||||
|
||||
const key = new THREE.DirectionalLight(0xddeeff, 1.0);
|
||||
key.position.set(2, 5, 3);
|
||||
key.castShadow = true;
|
||||
key.shadow.mapSize.set(1024, 1024);
|
||||
key.shadow.camera.near = 0.5;
|
||||
key.shadow.camera.far = 15;
|
||||
key.shadow.camera.left = -3;
|
||||
key.shadow.camera.right = 3;
|
||||
key.shadow.camera.top = 3;
|
||||
key.shadow.camera.bottom = -1;
|
||||
scene.add(key);
|
||||
|
||||
const rim = new THREE.PointLight(0x32b8c6, 1.5, 12);
|
||||
rim.position.set(-1.5, 2.5, -2);
|
||||
scene.add(rim);
|
||||
|
||||
const fill = new THREE.PointLight(0x554488, 0.5, 8);
|
||||
fill.position.set(1.5, 0.8, 2.5);
|
||||
scene.add(fill);
|
||||
|
||||
const under = new THREE.PointLight(0x225566, 0.4, 5);
|
||||
under.position.set(0, 0.1, 1);
|
||||
scene.add(under);
|
||||
|
||||
// --- Ground ---
|
||||
const groundGeo = new THREE.PlaneGeometry(20, 20);
|
||||
const groundMat = new THREE.MeshStandardMaterial({
|
||||
color: 0x0a0e1a, roughness: 0.9, metalness: 0.1,
|
||||
});
|
||||
const ground = new THREE.Mesh(groundGeo, groundMat);
|
||||
ground.rotation.x = -Math.PI / 2;
|
||||
ground.receiveShadow = true;
|
||||
scene.add(ground);
|
||||
|
||||
const gridH = new THREE.GridHelper(20, 40, 0x1a3050, 0x0e1826);
|
||||
gridH.position.y = 0.002;
|
||||
scene.add(gridH);
|
||||
|
||||
// --- Signal field (20x20) ---
|
||||
const GS = 20;
|
||||
const cellGeo = new THREE.PlaneGeometry(0.38, 0.38);
|
||||
const cellMat = new THREE.MeshBasicMaterial({ color: 0x32b8c6, transparent: true, opacity: 0.25, side: THREE.DoubleSide });
|
||||
const sigGrid = new THREE.InstancedMesh(cellGeo, cellMat, GS * GS);
|
||||
sigGrid.rotation.x = -Math.PI / 2; sigGrid.position.y = 0.005;
|
||||
const dum = new THREE.Object3D();
|
||||
for (let z = 0; z < GS; z++) for (let x = 0; x < GS; x++) {
|
||||
dum.position.set((x - GS / 2) * 0.4, (z - GS / 2) * 0.4, 0);
|
||||
dum.updateMatrix();
|
||||
sigGrid.setMatrixAt(z * GS + x, dum.matrix);
|
||||
sigGrid.setColorAt(z * GS + x, new THREE.Color(0x080c16));
|
||||
}
|
||||
sigGrid.instanceMatrix.needsUpdate = true;
|
||||
if (sigGrid.instanceColor) sigGrid.instanceColor.needsUpdate = true;
|
||||
scene.add(sigGrid);
|
||||
|
||||
// --- ESP32 nodes ---
|
||||
const nodeGeo = new THREE.OctahedronGeometry(0.08, 1);
|
||||
const nodeMs: THREE.Mesh[] = [];
|
||||
for (let i = 0; i < 8; i++) {
|
||||
const mat = new THREE.MeshStandardMaterial({ color: 0x00ff88, emissive: 0x00ff88, emissiveIntensity: 0.7, wireframe: true });
|
||||
const m = new THREE.Mesh(nodeGeo, mat); m.visible = false; scene.add(m); nodeMs.push(m);
|
||||
}
|
||||
|
||||
// --- Human body: DensePose-colored capsule mesh ---
|
||||
// Head: slightly oblate sphere
|
||||
const headGeo = new THREE.SphereGeometry(0.105, 20, 16);
|
||||
headGeo.scale(1, 1.08, 1);
|
||||
const headMat = new THREE.MeshPhysicalMaterial({
|
||||
color: DENSEPOSE_COLORS.head, emissive: DENSEPOSE_COLORS.head,
|
||||
emissiveIntensity: 0.08, roughness: 0.3, metalness: 0.05,
|
||||
clearcoat: 0.4, clearcoatRoughness: 0.3, transparent: true, opacity: 0.9,
|
||||
});
|
||||
const headM = new THREE.Mesh(headGeo, headMat);
|
||||
headM.castShadow = true; headM.visible = false; scene.add(headM);
|
||||
|
||||
// Head glow
|
||||
const headGlowGeo = new THREE.SphereGeometry(0.14, 12, 10);
|
||||
const headGlowMat = new THREE.MeshBasicMaterial({
|
||||
color: DENSEPOSE_COLORS.head, transparent: true, opacity: 0.08, side: THREE.BackSide,
|
||||
});
|
||||
const headGlowM = new THREE.Mesh(headGlowGeo, headGlowMat);
|
||||
headGlowM.visible = false; scene.add(headGlowM);
|
||||
|
||||
// Eyes
|
||||
const eyeGeo = new THREE.SphereGeometry(0.015, 8, 6);
|
||||
const eyeMat = new THREE.MeshBasicMaterial({ color: 0xeeffff });
|
||||
const eyeL = new THREE.Mesh(eyeGeo, eyeMat);
|
||||
const eyeR = new THREE.Mesh(eyeGeo, eyeMat.clone());
|
||||
eyeL.visible = eyeR.visible = false;
|
||||
scene.add(eyeL); scene.add(eyeR);
|
||||
|
||||
// Pupils
|
||||
const pupilGeo = new THREE.SphereGeometry(0.008, 6, 4);
|
||||
const pupilMat = new THREE.MeshBasicMaterial({ color: 0x112233 });
|
||||
const pupilL = new THREE.Mesh(pupilGeo, pupilMat);
|
||||
const pupilR = new THREE.Mesh(pupilGeo, pupilMat.clone());
|
||||
pupilL.visible = pupilR.visible = false;
|
||||
scene.add(pupilL); scene.add(pupilR);
|
||||
|
||||
// Neck
|
||||
const neckGeo = new THREE.CapsuleGeometry(0.04, 0.08, 4, 8);
|
||||
const neckMat = new THREE.MeshPhysicalMaterial({
|
||||
color: DENSEPOSE_COLORS.neck, emissive: DENSEPOSE_COLORS.neck,
|
||||
emissiveIntensity: 0.05, roughness: 0.4, transparent: true, opacity: 0.85,
|
||||
});
|
||||
const neckM = new THREE.Mesh(neckGeo, neckMat);
|
||||
neckM.castShadow = true; neckM.visible = false; scene.add(neckM);
|
||||
|
||||
// Torso: front plate
|
||||
const torsoGeo = new THREE.BoxGeometry(0.34, 0.50, 0.18, 2, 3, 2);
|
||||
// Round the torso vertices slightly
|
||||
const torsoPos = torsoGeo.attributes.position;
|
||||
for (let i = 0; i < torsoPos.count; i++) {
|
||||
const x = torsoPos.getX(i), y = torsoPos.getY(i), z = torsoPos.getZ(i);
|
||||
const r = Math.sqrt(x * x + z * z);
|
||||
if (r > 0.01) {
|
||||
const bulge = 1 + 0.15 * Math.cos(y * 3.5); // chest & hip curvature
|
||||
torsoPos.setX(i, x * bulge);
|
||||
torsoPos.setZ(i, z * bulge);
|
||||
}
|
||||
}
|
||||
torsoGeo.computeVertexNormals();
|
||||
const torsoMat = new THREE.MeshPhysicalMaterial({
|
||||
color: DENSEPOSE_COLORS.torsoFront, emissive: DENSEPOSE_COLORS.torsoFront,
|
||||
emissiveIntensity: 0.06, roughness: 0.35, metalness: 0.05,
|
||||
clearcoat: 0.2, transparent: true, opacity: 0.88,
|
||||
});
|
||||
const torsoM = new THREE.Mesh(torsoGeo, torsoMat);
|
||||
torsoM.castShadow = true; torsoM.visible = false; scene.add(torsoM);
|
||||
|
||||
// Torso glow
|
||||
const torsoGlowGeo = new THREE.BoxGeometry(0.40, 0.55, 0.24);
|
||||
const torsoGlowMat = new THREE.MeshBasicMaterial({
|
||||
color: DENSEPOSE_COLORS.torsoFront, transparent: true, opacity: 0.06, side: THREE.BackSide,
|
||||
});
|
||||
const torsoGlowM = new THREE.Mesh(torsoGlowGeo, torsoGlowMat);
|
||||
torsoGlowM.visible = false; scene.add(torsoGlowM);
|
||||
|
||||
// Hands (small boxes)
|
||||
const handGeo = new THREE.BoxGeometry(0.05, 0.08, 0.025, 1, 1, 1);
|
||||
const handLMat = new THREE.MeshPhysicalMaterial({ color: DENSEPOSE_COLORS.lHand, emissive: DENSEPOSE_COLORS.lHand, emissiveIntensity: 0.1, roughness: 0.3, transparent: true, opacity: 0.85 });
|
||||
const handRMat = new THREE.MeshPhysicalMaterial({ color: DENSEPOSE_COLORS.rHand, emissive: DENSEPOSE_COLORS.rHand, emissiveIntensity: 0.1, roughness: 0.3, transparent: true, opacity: 0.85 });
|
||||
const handL = new THREE.Mesh(handGeo, handLMat); handL.visible = false; scene.add(handL);
|
||||
const handR = new THREE.Mesh(handGeo, handRMat); handR.visible = false; scene.add(handR);
|
||||
|
||||
// Feet (wedge-like boxes)
|
||||
const footGeo = new THREE.BoxGeometry(0.06, 0.04, 0.14, 1, 1, 1);
|
||||
const footLMat = new THREE.MeshPhysicalMaterial({ color: DENSEPOSE_COLORS.lFoot, emissive: DENSEPOSE_COLORS.lFoot, emissiveIntensity: 0.1, roughness: 0.4, transparent: true, opacity: 0.85 });
|
||||
const footRMat = new THREE.MeshPhysicalMaterial({ color: DENSEPOSE_COLORS.rFoot, emissive: DENSEPOSE_COLORS.rFoot, emissiveIntensity: 0.1, roughness: 0.4, transparent: true, opacity: 0.85 });
|
||||
const footL = new THREE.Mesh(footGeo, footLMat); footL.visible = false; scene.add(footL);
|
||||
const footR = new THREE.Mesh(footGeo, footRMat); footR.visible = false; scene.add(footR);
|
||||
|
||||
// Limb capsules + glow capsules
|
||||
const limbMs = BODY_SEGS.map(([,, rT, rB, ck]) => makePart(scene, rT, rB, DENSEPOSE_COLORS[ck]));
|
||||
const limbGlowMs = BODY_SEGS.map(([,, rT, rB, ck]) => makePart(scene, rT * 1.6, rB * 1.6, DENSEPOSE_COLORS[ck], true));
|
||||
|
||||
// Joint dots
|
||||
const jDotGeo = new THREE.SphereGeometry(0.018, 6, 4);
|
||||
const jDots = Array.from({ length: 17 }, () => {
|
||||
const mat = new THREE.MeshBasicMaterial({ color: 0x88ddee, transparent: true, opacity: 0.7 });
|
||||
const m = new THREE.Mesh(jDotGeo, mat); m.visible = false; scene.add(m); return m;
|
||||
});
|
||||
|
||||
// Skeleton lines (thin wireframe overlay)
|
||||
const skelMat = new THREE.LineBasicMaterial({ color: 0x55ccdd, transparent: true, opacity: 0.25 });
|
||||
const skelLines = BONES.map(([a, b]) => {
|
||||
const g = new THREE.BufferGeometry().setFromPoints([new THREE.Vector3(), new THREE.Vector3()]);
|
||||
const l = new THREE.Line(g, skelMat); l.visible = false; scene.add(l); return { line: l, a, b };
|
||||
});
|
||||
|
||||
// Heart ring
|
||||
const hrGeo = new THREE.TorusGeometry(0.18, 0.006, 8, 32);
|
||||
const hrMat = new THREE.MeshBasicMaterial({ color: 0xff3355, transparent: true, opacity: 0 });
|
||||
const hrRing = new THREE.Mesh(hrGeo, hrMat); hrRing.visible = false; scene.add(hrRing);
|
||||
|
||||
// Breathing indicator rings (concentric around chest)
|
||||
const brRings = [0.22, 0.28, 0.34].map((r) => {
|
||||
const geo = new THREE.TorusGeometry(r, 0.003, 6, 32);
|
||||
const mat = new THREE.MeshBasicMaterial({ color: 0x44ddaa, transparent: true, opacity: 0 });
|
||||
const m = new THREE.Mesh(geo, mat); m.visible = false; scene.add(m); return m;
|
||||
});
|
||||
|
||||
// WiFi pulse rings
|
||||
const wifiRings = [1.0, 1.8, 2.6].map((r) => {
|
||||
const geo = new THREE.TorusGeometry(r, 0.01, 6, 48);
|
||||
const mat = new THREE.MeshBasicMaterial({ color: 0x32b8c6, transparent: true, opacity: 0.15 });
|
||||
const m = new THREE.Mesh(geo, mat); m.rotation.x = Math.PI / 2; m.position.y = 0.01; scene.add(m); return m;
|
||||
});
|
||||
|
||||
// Particles
|
||||
const NP = 400;
|
||||
const pGeo = new THREE.BufferGeometry();
|
||||
const pA = new Float32Array(NP * 3);
|
||||
for (let i = 0; i < NP; i++) {
|
||||
pA[i * 3] = (Math.random() - 0.5) * 12;
|
||||
pA[i * 3 + 1] = Math.random() * 3.5;
|
||||
pA[i * 3 + 2] = (Math.random() - 0.5) * 12;
|
||||
}
|
||||
pGeo.setAttribute('position', new THREE.BufferAttribute(pA, 3));
|
||||
scene.add(new THREE.Points(pGeo, new THREE.PointsMaterial({
|
||||
color: 0x3399bb, size: 0.018, transparent: true, opacity: 0.25,
|
||||
})));
|
||||
|
||||
// --- HUD ---
|
||||
const hudC = document.createElement('canvas'); hudC.width = 640; hudC.height = 128;
|
||||
const hudT = new THREE.CanvasTexture(hudC);
|
||||
const hudS = new THREE.Sprite(new THREE.SpriteMaterial({ map: hudT, transparent: true }));
|
||||
hudS.scale.set(3.2, 0.64, 1); hudS.position.set(0, 3.2, 0); scene.add(hudS);
|
||||
|
||||
// --- Smooth keypoints ---
|
||||
const smoothKps: THREE.Vector3[] = BASE_POSE.map(([x, y, z]) => new THREE.Vector3(x, y, z));
|
||||
const targetKps: THREE.Vector3[] = BASE_POSE.map(([x, y, z]) => new THREE.Vector3(x, y, z));
|
||||
const tmpA = new THREE.Vector3();
|
||||
const tmpB = new THREE.Vector3();
|
||||
const hc = new THREE.Color();
|
||||
|
||||
// State
|
||||
const state: any = {
|
||||
renderer, scene, camera, animId: 0,
|
||||
camAngle: 0, camR: 3.5, camY: 1.4,
|
||||
drag: false, fCount: 0, fpsT: performance.now(),
|
||||
prevPresence: false, fadeIn: 0,
|
||||
};
|
||||
sceneRef.current = state;
|
||||
|
||||
// Input
|
||||
const cvs = renderer.domElement;
|
||||
cvs.addEventListener('mousedown', () => { state.drag = true; });
|
||||
cvs.addEventListener('mouseup', () => { state.drag = false; });
|
||||
cvs.addEventListener('mouseleave', () => { state.drag = false; });
|
||||
cvs.addEventListener('mousemove', (e: MouseEvent) => {
|
||||
if (state.drag) { state.camAngle += e.movementX * 0.006; state.camY = Math.max(0.2, Math.min(4, state.camY - e.movementY * 0.006)); }
|
||||
});
|
||||
cvs.addEventListener('wheel', (e: WheelEvent) => {
|
||||
state.camR = Math.max(1.5, Math.min(10, state.camR + e.deltaY * 0.003));
|
||||
}, { passive: true });
|
||||
const onR = () => { camera.aspect = W() / H(); camera.updateProjectionMatrix(); renderer.setSize(W(), H()); };
|
||||
window.addEventListener('resize', onR);
|
||||
|
||||
// --- Animate ---
|
||||
const animate = () => {
|
||||
state.animId = requestAnimationFrame(animate);
|
||||
const t = performance.now() * 0.001;
|
||||
const fr = frameRef.current;
|
||||
|
||||
// Camera
|
||||
if (!state.drag) state.camAngle += 0.001;
|
||||
camera.position.set(Math.sin(state.camAngle) * state.camR, state.camY, Math.cos(state.camAngle) * state.camR);
|
||||
camera.lookAt(0, 0.95, 0);
|
||||
|
||||
const pres = fr?.classification?.presence ?? false;
|
||||
const mot = fr?.classification?.motion_level ?? 'absent';
|
||||
const conf = fr?.classification?.confidence ?? 0;
|
||||
const mPow = fr?.features?.motion_band_power ?? 0;
|
||||
const bPow = fr?.features?.breathing_band_power ?? 0;
|
||||
const rssi = fr?.features?.mean_rssi ?? -80;
|
||||
|
||||
// Fade body in/out (gradual transitions)
|
||||
if (pres && conf > 0.2) state.fadeIn = Math.min(1, state.fadeIn + 0.015);
|
||||
else state.fadeIn = Math.max(0, state.fadeIn - 0.008);
|
||||
const show = state.fadeIn > 0.01;
|
||||
const alpha = state.fadeIn;
|
||||
|
||||
// --- Compute target keypoints ---
|
||||
for (let i = 0; i < 17; i++) {
|
||||
const [bx, by, bz] = BASE_POSE[i];
|
||||
let ax = bx, ay = by, az = bz;
|
||||
|
||||
if (pres) {
|
||||
// Breathing: gentle chest rise/fall
|
||||
const bFreq = 0.25 + bPow * 0.5; // ~15 bpm base
|
||||
const bAmp = 0.004 + bPow * 0.008;
|
||||
const bPhase = Math.sin(t * bFreq * Math.PI * 2);
|
||||
if (i >= 5 && i <= 10) { ay += bPhase * bAmp; }
|
||||
if (i <= 4) ay += bPhase * bAmp * 0.3;
|
||||
|
||||
// Very subtle sway
|
||||
ax += Math.sin(t * 0.35) * 0.004;
|
||||
az += Math.cos(t * 0.25) * 0.002;
|
||||
|
||||
if (mot === 'active') {
|
||||
const ws = 1.8 + mPow * 2;
|
||||
const wa = 0.03 + mPow * 0.06;
|
||||
const ph = t * ws;
|
||||
|
||||
// Legs
|
||||
if (i === 13) { az += Math.sin(ph) * wa * 0.7; ay -= Math.abs(Math.sin(ph)) * 0.015; }
|
||||
if (i === 14) { az += Math.sin(ph + Math.PI) * wa * 0.7; ay -= Math.abs(Math.sin(ph + Math.PI)) * 0.015; }
|
||||
if (i === 15) { az += Math.sin(ph - 0.2) * wa * 0.8; }
|
||||
if (i === 16) { az += Math.sin(ph + Math.PI - 0.2) * wa * 0.8; }
|
||||
|
||||
// Arms counter-swing (subtle)
|
||||
if (i === 7) az += Math.sin(ph + Math.PI) * wa * 0.35;
|
||||
if (i === 8) az += Math.sin(ph) * wa * 0.35;
|
||||
if (i === 9) az += Math.sin(ph + Math.PI) * wa * 0.45;
|
||||
if (i === 10) az += Math.sin(ph) * wa * 0.45;
|
||||
|
||||
// Tiny vertical bob
|
||||
ay += Math.abs(Math.sin(ph)) * 0.006;
|
||||
|
||||
} else if (mot === 'present_still') {
|
||||
const it = t * 0.25;
|
||||
// Very subtle weight shift
|
||||
if (i >= 11) ax += Math.sin(it * 0.4) * 0.004;
|
||||
// Barely perceptible hand drift
|
||||
if (i === 9) { ax += Math.sin(it * 0.8) * 0.005; }
|
||||
if (i === 10) { ax += Math.sin(it * 0.6 + 0.5) * 0.005; }
|
||||
}
|
||||
}
|
||||
targetKps[i].set(ax, ay, az);
|
||||
}
|
||||
|
||||
// Smooth interpolation (lower = smoother, less jumpy)
|
||||
const lerpA = 0.04;
|
||||
for (let i = 0; i < 17; i++) lerp3(smoothKps[i], targetKps[i], lerpA);
|
||||
|
||||
// --- Head ---
|
||||
headM.visible = headGlowM.visible = show;
|
||||
if (show) {
|
||||
tmpA.copy(smoothKps[0]).add(new THREE.Vector3(0, 0.06, 0));
|
||||
headM.position.copy(tmpA);
|
||||
headGlowM.position.copy(tmpA);
|
||||
(headM.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.9;
|
||||
headGlowMat.opacity = alpha * 0.08;
|
||||
}
|
||||
|
||||
// Eyes + pupils
|
||||
eyeL.visible = eyeR.visible = pupilL.visible = pupilR.visible = show;
|
||||
if (show) {
|
||||
const headPos = headM.position;
|
||||
eyeL.position.set(headPos.x - 0.032, headPos.y + 0.01, headPos.z + 0.09);
|
||||
eyeR.position.set(headPos.x + 0.032, headPos.y + 0.01, headPos.z + 0.09);
|
||||
pupilL.position.set(eyeL.position.x, eyeL.position.y, eyeL.position.z + 0.012);
|
||||
pupilR.position.set(eyeR.position.x, eyeR.position.y, eyeR.position.z + 0.012);
|
||||
}
|
||||
|
||||
// Neck
|
||||
neckM.visible = show;
|
||||
if (show) {
|
||||
const neckTop = new THREE.Vector3().copy(smoothKps[0]).add(new THREE.Vector3(0, -0.04, 0));
|
||||
const neckBot = tmpA.addVectors(smoothKps[5], smoothKps[6]).multiplyScalar(0.5).add(new THREE.Vector3(0, 0.04, 0));
|
||||
neckM.position.addVectors(neckTop, neckBot).multiplyScalar(0.5);
|
||||
neckM.scale.y = neckTop.distanceTo(neckBot) * 4;
|
||||
(neckM.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.85;
|
||||
}
|
||||
|
||||
// Torso
|
||||
torsoM.visible = torsoGlowM.visible = show;
|
||||
if (show) {
|
||||
const mSh = tmpA.addVectors(smoothKps[5], smoothKps[6]).multiplyScalar(0.5);
|
||||
const mHp = tmpB.addVectors(smoothKps[11], smoothKps[12]).multiplyScalar(0.5);
|
||||
const tPos = new THREE.Vector3().addVectors(mSh, mHp).multiplyScalar(0.5);
|
||||
torsoM.position.copy(tPos);
|
||||
torsoGlowM.position.copy(tPos);
|
||||
const bScale = 1 + Math.sin(t * (0.9 + bPow * 4) * Math.PI * 2) * 0.02 * (1 + bPow * 3);
|
||||
torsoM.scale.set(1, 1, bScale);
|
||||
(torsoM.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.88;
|
||||
torsoGlowMat.opacity = alpha * 0.06;
|
||||
}
|
||||
|
||||
// Hands
|
||||
handL.visible = handR.visible = show;
|
||||
if (show) {
|
||||
handL.position.copy(smoothKps[9]).add(new THREE.Vector3(0, -0.04, 0));
|
||||
handR.position.copy(smoothKps[10]).add(new THREE.Vector3(0, -0.04, 0));
|
||||
(handL.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.85;
|
||||
(handR.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.85;
|
||||
}
|
||||
|
||||
// Feet
|
||||
footL.visible = footR.visible = show;
|
||||
if (show) {
|
||||
footL.position.copy(smoothKps[15]).add(new THREE.Vector3(0, 0.02, 0.04));
|
||||
footR.position.copy(smoothKps[16]).add(new THREE.Vector3(0, 0.02, 0.04));
|
||||
(footL.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.85;
|
||||
(footR.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.85;
|
||||
}
|
||||
|
||||
// Limb capsules — emissive reacts to motion intensity
|
||||
BODY_SEGS.forEach(([ai, bi, rT, rB], idx) => {
|
||||
limbMs[idx].visible = limbGlowMs[idx].visible = show;
|
||||
if (show) {
|
||||
positionLimb(limbMs[idx], smoothKps[ai], smoothKps[bi], rT, rB);
|
||||
positionLimb(limbGlowMs[idx], smoothKps[ai], smoothKps[bi], rT * 1.6, rB * 1.6);
|
||||
const limbMat = limbMs[idx].material as THREE.MeshPhysicalMaterial;
|
||||
limbMat.opacity = alpha * 0.82;
|
||||
// Glow brighter with more motion (direct sensor feedback)
|
||||
limbMat.emissiveIntensity = 0.06 + mPow * 0.4;
|
||||
const glowMat = limbGlowMs[idx].material as THREE.MeshPhysicalMaterial;
|
||||
glowMat.opacity = alpha * (0.06 + mPow * 0.15);
|
||||
}
|
||||
});
|
||||
|
||||
// Joint dots & skeleton lines
|
||||
jDots.forEach((d, i) => { d.visible = show; if (show) d.position.copy(smoothKps[i]); });
|
||||
skelLines.forEach(({ line, a, b }) => {
|
||||
line.visible = show;
|
||||
if (show) {
|
||||
const p = line.geometry.attributes.position as THREE.BufferAttribute;
|
||||
p.setXYZ(0, smoothKps[a].x, smoothKps[a].y, smoothKps[a].z);
|
||||
p.setXYZ(1, smoothKps[b].x, smoothKps[b].y, smoothKps[b].z);
|
||||
p.needsUpdate = true;
|
||||
}
|
||||
});
|
||||
|
||||
// Heart ring
|
||||
const vs = fr?.vital_signs as Record<string, unknown> | undefined;
|
||||
const hrBpm = Number(vs?.hr_proxy_bpm ?? vs?.heart_rate_bpm ?? 0);
|
||||
hrRing.visible = show && hrBpm > 0;
|
||||
if (hrRing.visible) {
|
||||
const chst = tmpA.addVectors(smoothKps[5], smoothKps[6]).multiplyScalar(0.5);
|
||||
chst.y -= 0.08;
|
||||
hrRing.position.copy(chst);
|
||||
hrRing.lookAt(camera.position);
|
||||
const bp = (t * (hrBpm / 60) * Math.PI * 2) % (Math.PI * 2);
|
||||
const beat = Math.pow(Math.max(0, Math.sin(bp)), 10);
|
||||
hrMat.opacity = beat * 0.5 * alpha;
|
||||
hrRing.scale.setScalar(1 + beat * 0.12);
|
||||
}
|
||||
|
||||
// Breathing rings
|
||||
brRings.forEach((ring, ri) => {
|
||||
ring.visible = show && bPow > 0.01;
|
||||
if (ring.visible) {
|
||||
const chst = tmpA.addVectors(smoothKps[5], smoothKps[6]).multiplyScalar(0.5);
|
||||
chst.y -= 0.05;
|
||||
ring.position.copy(chst);
|
||||
ring.lookAt(camera.position);
|
||||
const bph = Math.sin(t * (0.9 + bPow * 4) * Math.PI * 2 - ri * 0.5);
|
||||
(ring.material as THREE.MeshBasicMaterial).opacity = Math.max(0, bph * 0.2 * alpha);
|
||||
ring.scale.setScalar(1 + bph * 0.08);
|
||||
}
|
||||
});
|
||||
|
||||
// WiFi pulse rings
|
||||
wifiRings.forEach((wr, wi) => {
|
||||
const phase = (t * 0.5 + wi * 0.4) % 1;
|
||||
wr.scale.setScalar(0.8 + phase * 1.5 + mPow);
|
||||
(wr.material as THREE.MeshBasicMaterial).opacity = (1 - phase) * 0.12 * (pres ? 1 : 0.3);
|
||||
});
|
||||
|
||||
// ESP32 nodes
|
||||
(fr?.nodes || []).forEach((n, i) => {
|
||||
if (i < nodeMs.length) {
|
||||
const [px, py, pz] = n.position;
|
||||
nodeMs[i].position.set(px * 2, py + 0.12, pz * 2);
|
||||
nodeMs[i].visible = true; nodeMs[i].rotation.y = t * 0.4 + i;
|
||||
(nodeMs[i].material as THREE.MeshStandardMaterial).emissiveIntensity = 0.5 + Math.sin(t * 3 + i) * 0.3;
|
||||
}
|
||||
});
|
||||
for (let i = (fr?.nodes || []).length; i < nodeMs.length; i++) nodeMs[i].visible = false;
|
||||
|
||||
// Signal field
|
||||
const sf = fr?.signal_field;
|
||||
if (sf?.values?.length) {
|
||||
const gx = sf.grid_size[0], gz = sf.grid_size[2];
|
||||
for (let zi = 0; zi < Math.min(gz, GS); zi++) for (let xi = 0; xi < Math.min(gx, GS); xi++) {
|
||||
const v = sf.values[zi * gx + xi] || 0;
|
||||
if (v < 0.25) hc.setRGB(0.03, 0.05 + v * 1.8, 0.08 + v * 1.8);
|
||||
else if (v < 0.5) hc.setRGB(0.03, 0.2 + (v - 0.25) * 2.4, 0.5 - (v - 0.25) * 1.2);
|
||||
else if (v < 0.75) hc.setRGB((v - 0.5) * 4, 0.7 + (v - 0.5) * 0.6, 0.1);
|
||||
else hc.setRGB(1, 1 - (v - 0.75) * 3, 0.05);
|
||||
sigGrid.setColorAt(zi * GS + xi, hc);
|
||||
}
|
||||
if (sigGrid.instanceColor) sigGrid.instanceColor.needsUpdate = true;
|
||||
}
|
||||
|
||||
// Lighting follows data
|
||||
rim.intensity = 0.8 + Math.abs(rssi + 50) * 0.015;
|
||||
|
||||
// Particles
|
||||
const pp = pGeo.attributes.position as THREE.BufferAttribute;
|
||||
for (let i = 0; i < NP; i++) {
|
||||
(pp.array as Float32Array)[i * 3 + 1] += Math.sin(t * 0.8 + i * 0.5) * 0.0006 + mPow * 0.001;
|
||||
if ((pp.array as Float32Array)[i * 3 + 1] > 3.5) (pp.array as Float32Array)[i * 3 + 1] = 0;
|
||||
}
|
||||
pp.needsUpdate = true;
|
||||
|
||||
// HUD
|
||||
const ctx = hudC.getContext('2d');
|
||||
if (ctx && fr) {
|
||||
ctx.clearRect(0, 0, 640, 128);
|
||||
ctx.font = 'bold 14px "SF Mono", Menlo, monospace';
|
||||
ctx.fillStyle = '#32b8c6';
|
||||
ctx.fillText(`WIFI-DENSEPOSE [${(fr.source || '--').toUpperCase()}]`, 12, 20);
|
||||
ctx.font = '12px "SF Mono", Menlo, monospace';
|
||||
ctx.fillStyle = '#7799aa';
|
||||
ctx.fillText(`Nodes: ${(fr.nodes || []).length} RSSI: ${rssi.toFixed(1)} dBm Motion: ${mot} Conf: ${(conf * 100).toFixed(0)}%`, 12, 42);
|
||||
if (vs) {
|
||||
const br = Number(vs.breathing_bpm ?? vs.breathing_rate_bpm ?? 0);
|
||||
if (br > 0 || hrBpm > 0) {
|
||||
ctx.fillStyle = '#44ddaa';
|
||||
ctx.fillText(`Breathing: ${br.toFixed(1)} bpm Heart: ${hrBpm.toFixed(1)} bpm`, 12, 62);
|
||||
}
|
||||
}
|
||||
if (show) {
|
||||
ctx.fillStyle = pres ? (mot === 'active' ? '#ff8844' : '#44bbcc') : '#556677';
|
||||
const mBar = Math.min(20, Math.round(mPow * 40));
|
||||
const mBarStr = '\u2588'.repeat(mBar) + '\u2591'.repeat(20 - mBar);
|
||||
ctx.fillText(`Motion: [${mBarStr}] ${(mPow * 100).toFixed(0)}%`, 12, 82);
|
||||
ctx.fillStyle = '#556677';
|
||||
ctx.font = '10px "SF Mono", Menlo, monospace';
|
||||
ctx.fillText('Pose: procedural (load NN model for limb tracking)', 12, 100);
|
||||
}
|
||||
hudT.needsUpdate = true;
|
||||
}
|
||||
|
||||
renderer.render(scene, camera);
|
||||
|
||||
state.fCount++;
|
||||
if (performance.now() - state.fpsT >= 1000) {
|
||||
onFps(state.fCount); state.fCount = 0; state.fpsT = performance.now();
|
||||
}
|
||||
};
|
||||
|
||||
animate();
|
||||
onReady();
|
||||
|
||||
return () => {
|
||||
cvs.removeEventListener('mousedown', () => {});
|
||||
window.removeEventListener('resize', onR);
|
||||
cleanup();
|
||||
if (container.contains(renderer.domElement)) container.removeChild(renderer.domElement);
|
||||
};
|
||||
} catch (err) {
|
||||
onError(err instanceof Error ? err.message : 'Failed to initialize 3D renderer');
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, []);
|
||||
|
||||
return (
|
||||
<View style={styles.container}>
|
||||
<div ref={containerRef} style={{ width: '100%', height: '100%', backgroundColor: '#080c16' }} />
|
||||
</View>
|
||||
);
|
||||
};
|
||||
|
||||
const styles = StyleSheet.create({ container: { flex: 1, backgroundColor: '#080c16' } });
|
||||
export default GaussianSplatWebViewWeb;
|
||||
@@ -0,0 +1,164 @@
|
||||
import { Pressable, StyleSheet, View } from 'react-native';
|
||||
import { memo, useCallback, useState } from 'react';
|
||||
import Animated, { useAnimatedStyle, useSharedValue, withTiming } from 'react-native-reanimated';
|
||||
import { StatusDot } from '@/components/StatusDot';
|
||||
import { ModeBadge } from '@/components/ModeBadge';
|
||||
import { ThemedText } from '@/components/ThemedText';
|
||||
import { formatConfidence, formatRssi } from '@/utils/formatters';
|
||||
import { colors, spacing } from '@/theme';
|
||||
import type { ConnectionStatus } from '@/types/sensing';
|
||||
|
||||
type LiveMode = 'LIVE' | 'SIM' | 'RSSI';
|
||||
|
||||
type LiveHUDProps = {
|
||||
rssi?: number;
|
||||
connectionStatus: ConnectionStatus;
|
||||
fps: number;
|
||||
confidence: number;
|
||||
personCount: number;
|
||||
mode: LiveMode;
|
||||
};
|
||||
|
||||
const statusTextMap: Record<ConnectionStatus, string> = {
|
||||
connected: 'Connected',
|
||||
simulated: 'Simulated',
|
||||
connecting: 'Connecting',
|
||||
disconnected: 'Disconnected',
|
||||
};
|
||||
|
||||
const statusDotStatusMap: Record<ConnectionStatus, 'connected' | 'simulated' | 'disconnected' | 'connecting'> = {
|
||||
connected: 'connected',
|
||||
simulated: 'simulated',
|
||||
connecting: 'connecting',
|
||||
disconnected: 'disconnected',
|
||||
};
|
||||
|
||||
export const LiveHUD = memo(
|
||||
({ rssi, connectionStatus, fps, confidence, personCount, mode }: LiveHUDProps) => {
|
||||
const [panelVisible, setPanelVisible] = useState(true);
|
||||
const panelAlpha = useSharedValue(1);
|
||||
|
||||
const togglePanel = useCallback(() => {
|
||||
const next = !panelVisible;
|
||||
setPanelVisible(next);
|
||||
panelAlpha.value = withTiming(next ? 1 : 0, { duration: 220 });
|
||||
}, [panelAlpha, panelVisible]);
|
||||
|
||||
const animatedPanelStyle = useAnimatedStyle(() => ({
|
||||
opacity: panelAlpha.value,
|
||||
}));
|
||||
|
||||
const statusText = statusTextMap[connectionStatus];
|
||||
|
||||
return (
|
||||
<Pressable style={StyleSheet.absoluteFill} onPress={togglePanel}>
|
||||
<Animated.View pointerEvents="none" style={[StyleSheet.absoluteFill, animatedPanelStyle]}>
|
||||
{/* App title */}
|
||||
<View style={styles.topLeft}>
|
||||
<ThemedText preset="labelLg" style={styles.appTitle}>
|
||||
WiFi-DensePose
|
||||
</ThemedText>
|
||||
</View>
|
||||
|
||||
{/* Status + FPS */}
|
||||
<View style={styles.topRight}>
|
||||
<View style={styles.row}>
|
||||
<StatusDot status={statusDotStatusMap[connectionStatus]} size={10} />
|
||||
<ThemedText preset="labelMd" style={styles.statusText}>
|
||||
{statusText}
|
||||
</ThemedText>
|
||||
</View>
|
||||
{fps > 0 && (
|
||||
<View style={styles.row}>
|
||||
<ThemedText preset="labelMd">{fps} FPS</ThemedText>
|
||||
</View>
|
||||
)}
|
||||
</View>
|
||||
|
||||
{/* Bottom panel */}
|
||||
<View style={styles.bottomPanel}>
|
||||
<View style={styles.bottomCell}>
|
||||
<ThemedText preset="bodySm">RSSI</ThemedText>
|
||||
<ThemedText preset="displayMd" style={styles.bigValue}>
|
||||
{formatRssi(rssi)}
|
||||
</ThemedText>
|
||||
</View>
|
||||
|
||||
<View style={styles.bottomCell}>
|
||||
<ModeBadge mode={mode} />
|
||||
</View>
|
||||
|
||||
<View style={styles.bottomCellRight}>
|
||||
<ThemedText preset="bodySm">Confidence</ThemedText>
|
||||
<ThemedText preset="bodyMd" style={styles.metaText}>
|
||||
{formatConfidence(confidence)}
|
||||
</ThemedText>
|
||||
<ThemedText preset="bodySm">People: {personCount}</ThemedText>
|
||||
</View>
|
||||
</View>
|
||||
</Animated.View>
|
||||
</Pressable>
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
topLeft: {
|
||||
position: 'absolute',
|
||||
top: spacing.md,
|
||||
left: spacing.md,
|
||||
},
|
||||
appTitle: {
|
||||
color: colors.textPrimary,
|
||||
},
|
||||
topRight: {
|
||||
position: 'absolute',
|
||||
top: spacing.md,
|
||||
right: spacing.md,
|
||||
alignItems: 'flex-end',
|
||||
gap: 4,
|
||||
},
|
||||
row: {
|
||||
flexDirection: 'row',
|
||||
alignItems: 'center',
|
||||
gap: spacing.sm,
|
||||
},
|
||||
statusText: {
|
||||
color: colors.textPrimary,
|
||||
},
|
||||
bottomPanel: {
|
||||
position: 'absolute',
|
||||
left: spacing.sm,
|
||||
right: spacing.sm,
|
||||
bottom: spacing.sm,
|
||||
minHeight: 72,
|
||||
borderRadius: 12,
|
||||
backgroundColor: 'rgba(10,14,26,0.72)',
|
||||
borderWidth: 1,
|
||||
borderColor: 'rgba(50,184,198,0.35)',
|
||||
paddingHorizontal: spacing.md,
|
||||
paddingVertical: spacing.sm,
|
||||
flexDirection: 'row',
|
||||
justifyContent: 'space-between',
|
||||
alignItems: 'center',
|
||||
},
|
||||
bottomCell: {
|
||||
flex: 1,
|
||||
alignItems: 'center',
|
||||
},
|
||||
bottomCellRight: {
|
||||
flex: 1,
|
||||
alignItems: 'flex-end',
|
||||
},
|
||||
bigValue: {
|
||||
color: colors.accent,
|
||||
marginTop: 2,
|
||||
marginBottom: 2,
|
||||
},
|
||||
metaText: {
|
||||
color: colors.textPrimary,
|
||||
marginBottom: 4,
|
||||
},
|
||||
});
|
||||
|
||||
LiveHUD.displayName = 'LiveHUD';
|
||||
@@ -0,0 +1,142 @@
|
||||
import { useCallback, useEffect, useRef, useState } from 'react';
|
||||
import { Button, Platform, StyleSheet, View } from 'react-native';
|
||||
import { ErrorBoundary } from '@/components/ErrorBoundary';
|
||||
import { LoadingSpinner } from '@/components/LoadingSpinner';
|
||||
import { ThemedText } from '@/components/ThemedText';
|
||||
import { ThemedView } from '@/components/ThemedView';
|
||||
import { usePoseStream } from '@/hooks/usePoseStream';
|
||||
import { colors, spacing } from '@/theme';
|
||||
import type { ConnectionStatus, SensingFrame } from '@/types/sensing';
|
||||
import { LiveHUD } from './LiveHUD';
|
||||
|
||||
type LiveMode = 'LIVE' | 'SIM' | 'RSSI';
|
||||
|
||||
const getMode = (
|
||||
status: ConnectionStatus,
|
||||
isSimulated: boolean,
|
||||
frame: SensingFrame | null,
|
||||
): LiveMode => {
|
||||
if (isSimulated || frame?.source === 'simulated') return 'SIM';
|
||||
if (status === 'connected') return 'LIVE';
|
||||
return 'RSSI';
|
||||
};
|
||||
|
||||
const isWeb = Platform.OS === 'web';
|
||||
|
||||
type ViewerProps = {
|
||||
frame: SensingFrame | null;
|
||||
onReady: () => void;
|
||||
onFps: (fps: number) => void;
|
||||
onError: (msg: string) => void;
|
||||
};
|
||||
|
||||
const WebLiveViewer = ({ frame, onReady, onFps, onError }: ViewerProps) => {
|
||||
const [Viewer, setViewer] = useState<React.ComponentType<any> | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
import('./GaussianSplatWebView.web').then((mod) => {
|
||||
setViewer(() => mod.GaussianSplatWebViewWeb);
|
||||
}).catch(() => onError('Failed to load web viewer'));
|
||||
}, [onError]);
|
||||
|
||||
if (!Viewer) return null;
|
||||
return <Viewer frame={frame} onReady={onReady} onFps={onFps} onError={onError} />;
|
||||
};
|
||||
|
||||
const NativeLiveViewer = ({ frame, onReady, onFps, onError }: ViewerProps) => {
|
||||
const webViewRef = useRef(null);
|
||||
const [WVComponent, setWVComponent] = useState<React.ComponentType<any> | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
try {
|
||||
const { GaussianSplatWebView } = require('./GaussianSplatWebView');
|
||||
setWVComponent(() => GaussianSplatWebView);
|
||||
} catch {
|
||||
onError('WebView not available on this platform');
|
||||
}
|
||||
}, [onError]);
|
||||
|
||||
if (!WVComponent) return null;
|
||||
|
||||
return (
|
||||
<WVComponent
|
||||
webViewRef={webViewRef}
|
||||
onMessage={(event: any) => {
|
||||
try {
|
||||
const data = typeof event.nativeEvent.data === 'string'
|
||||
? JSON.parse(event.nativeEvent.data)
|
||||
: event.nativeEvent.data;
|
||||
if (data.type === 'READY') onReady();
|
||||
else if (data.type === 'FPS_TICK') onFps(data.payload?.fps ?? 0);
|
||||
else if (data.type === 'ERROR') onError(data.payload?.message ?? 'Unknown error');
|
||||
} catch { /* ignore */ }
|
||||
}}
|
||||
onError={() => onError('WebView renderer failed')}
|
||||
/>
|
||||
);
|
||||
};
|
||||
|
||||
export const LiveScreen = () => {
|
||||
const { lastFrame, connectionStatus, isSimulated } = usePoseStream();
|
||||
const [ready, setReady] = useState(false);
|
||||
const [fps, setFps] = useState(0);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
const [viewerKey, setViewerKey] = useState(0);
|
||||
|
||||
const handleReady = useCallback(() => { setReady(true); setError(null); }, []);
|
||||
const handleFps = useCallback((f: number) => setFps(Math.max(0, Math.floor(f))), []);
|
||||
const handleError = useCallback((msg: string) => { setError(msg); setReady(false); }, []);
|
||||
const handleRetry = useCallback(() => { setError(null); setReady(false); setFps(0); setViewerKey((v) => v + 1); }, []);
|
||||
|
||||
const rssi = lastFrame?.features?.mean_rssi;
|
||||
const personCount = lastFrame?.classification?.presence ? 1 : 0;
|
||||
const mode = getMode(connectionStatus, isSimulated, lastFrame);
|
||||
|
||||
if (error) {
|
||||
return (
|
||||
<ThemedView style={styles.fallbackWrap}>
|
||||
<ThemedText preset="bodyLg">Live visualization failed</ThemedText>
|
||||
<ThemedText preset="bodySm" color="textSecondary" style={styles.errorText}>{error}</ThemedText>
|
||||
<Button title="Retry" onPress={handleRetry} />
|
||||
</ThemedView>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<ErrorBoundary>
|
||||
<View style={styles.container}>
|
||||
{isWeb ? (
|
||||
<WebLiveViewer key={viewerKey} frame={lastFrame} onReady={handleReady} onFps={handleFps} onError={handleError} />
|
||||
) : (
|
||||
<NativeLiveViewer key={viewerKey} frame={lastFrame} onReady={handleReady} onFps={handleFps} onError={handleError} />
|
||||
)}
|
||||
|
||||
<LiveHUD
|
||||
connectionStatus={connectionStatus}
|
||||
fps={fps}
|
||||
rssi={rssi}
|
||||
confidence={lastFrame?.classification?.confidence ?? 0}
|
||||
personCount={personCount}
|
||||
mode={mode}
|
||||
/>
|
||||
|
||||
{!ready && (
|
||||
<View style={styles.loadingWrap}>
|
||||
<LoadingSpinner />
|
||||
<ThemedText preset="bodyMd" style={styles.loadingText}>Loading live renderer</ThemedText>
|
||||
</View>
|
||||
)}
|
||||
</View>
|
||||
</ErrorBoundary>
|
||||
);
|
||||
};
|
||||
|
||||
export default LiveScreen;
|
||||
|
||||
const styles = StyleSheet.create({
|
||||
container: { flex: 1, backgroundColor: colors.bg },
|
||||
loadingWrap: { ...StyleSheet.absoluteFillObject, backgroundColor: colors.bg, alignItems: 'center', justifyContent: 'center', gap: spacing.md },
|
||||
loadingText: { color: colors.textSecondary },
|
||||
fallbackWrap: { flex: 1, alignItems: 'center', justifyContent: 'center', gap: spacing.md, padding: spacing.lg },
|
||||
errorText: { textAlign: 'center' },
|
||||
});
|
||||
@@ -0,0 +1,97 @@
|
||||
import { useCallback, useState } from 'react';
|
||||
import type { RefObject } from 'react';
|
||||
import type { WebViewMessageEvent } from 'react-native-webview';
|
||||
import { WebView } from 'react-native-webview';
|
||||
import type { SensingFrame } from '@/types/sensing';
|
||||
|
||||
export type GaussianBridgeMessageType = 'READY' | 'FPS_TICK' | 'ERROR';
|
||||
|
||||
type BridgeMessage = {
|
||||
type: GaussianBridgeMessageType;
|
||||
payload?: {
|
||||
fps?: number;
|
||||
message?: string;
|
||||
};
|
||||
};
|
||||
|
||||
const toJsonScript = (message: unknown): string => {
|
||||
const serialized = JSON.stringify(message);
|
||||
return `window.dispatchEvent(new MessageEvent('message', { data: ${JSON.stringify(serialized)} })); true;`;
|
||||
};
|
||||
|
||||
export const useGaussianBridge = (webViewRef: RefObject<WebView | null>) => {
|
||||
const [isReady, setIsReady] = useState(false);
|
||||
const [fps, setFps] = useState(0);
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
const send = useCallback((message: unknown) => {
|
||||
const webView = webViewRef.current;
|
||||
if (!webView) {
|
||||
return;
|
||||
}
|
||||
|
||||
webView.injectJavaScript(toJsonScript(message));
|
||||
}, [webViewRef]);
|
||||
|
||||
const sendFrame = useCallback(
|
||||
(frame: SensingFrame) => {
|
||||
send({
|
||||
type: 'FRAME_UPDATE',
|
||||
payload: frame,
|
||||
});
|
||||
},
|
||||
[send],
|
||||
);
|
||||
|
||||
const onMessage = useCallback((event: WebViewMessageEvent) => {
|
||||
let parsed: BridgeMessage | null = null;
|
||||
const raw = event.nativeEvent.data;
|
||||
|
||||
if (typeof raw === 'string') {
|
||||
try {
|
||||
parsed = JSON.parse(raw) as BridgeMessage;
|
||||
} catch {
|
||||
setError('Invalid bridge message format');
|
||||
return;
|
||||
}
|
||||
} else if (typeof raw === 'object' && raw !== null) {
|
||||
parsed = raw as BridgeMessage;
|
||||
}
|
||||
|
||||
if (!parsed) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (parsed.type === 'READY') {
|
||||
setIsReady(true);
|
||||
setError(null);
|
||||
return;
|
||||
}
|
||||
|
||||
if (parsed.type === 'FPS_TICK') {
|
||||
const fpsValue = parsed.payload?.fps;
|
||||
if (typeof fpsValue === 'number' && Number.isFinite(fpsValue)) {
|
||||
setFps(Math.max(0, Math.floor(fpsValue)));
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (parsed.type === 'ERROR') {
|
||||
setError(parsed.payload?.message ?? 'Unknown bridge error');
|
||||
setIsReady(false);
|
||||
}
|
||||
}, []);
|
||||
|
||||
return {
|
||||
sendFrame,
|
||||
onMessage,
|
||||
isReady,
|
||||
fps,
|
||||
error,
|
||||
reset: () => {
|
||||
setIsReady(false);
|
||||
setFps(0);
|
||||
setError(null);
|
||||
},
|
||||
};
|
||||
};
|
||||
@@ -0,0 +1,84 @@
|
||||
import { View } from 'react-native';
|
||||
import { ThemedText } from '@/components/ThemedText';
|
||||
import { colors } from '@/theme/colors';
|
||||
import { spacing } from '@/theme/spacing';
|
||||
import { AlertPriority, type Alert } from '@/types/mat';
|
||||
|
||||
type SeverityLevel = 'URGENT' | 'HIGH' | 'NORMAL';
|
||||
|
||||
type AlertCardProps = {
|
||||
alert: Alert;
|
||||
};
|
||||
|
||||
type SeverityMeta = {
|
||||
label: SeverityLevel;
|
||||
icon: string;
|
||||
color: string;
|
||||
};
|
||||
|
||||
const resolveSeverity = (alert: Alert): SeverityMeta => {
|
||||
if (alert.priority === AlertPriority.Critical) {
|
||||
return {
|
||||
label: 'URGENT',
|
||||
icon: '‼',
|
||||
color: colors.danger,
|
||||
};
|
||||
}
|
||||
|
||||
if (alert.priority === AlertPriority.High) {
|
||||
return {
|
||||
label: 'HIGH',
|
||||
icon: '⚠',
|
||||
color: colors.warn,
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
label: 'NORMAL',
|
||||
icon: '•',
|
||||
color: colors.accent,
|
||||
};
|
||||
};
|
||||
|
||||
const formatTime = (value?: string): string => {
|
||||
if (!value) {
|
||||
return 'Unknown';
|
||||
}
|
||||
|
||||
try {
|
||||
return new Date(value).toLocaleTimeString();
|
||||
} catch {
|
||||
return 'Unknown';
|
||||
}
|
||||
};
|
||||
|
||||
export const AlertCard = ({ alert }: AlertCardProps) => {
|
||||
const severity = resolveSeverity(alert);
|
||||
|
||||
return (
|
||||
<View
|
||||
style={{
|
||||
backgroundColor: '#111827',
|
||||
borderWidth: 1,
|
||||
borderColor: `${severity.color}55`,
|
||||
padding: spacing.md,
|
||||
borderRadius: 10,
|
||||
marginBottom: spacing.sm,
|
||||
}}
|
||||
>
|
||||
<View style={{ flexDirection: 'row', alignItems: 'center', gap: 8 }}>
|
||||
<ThemedText preset="labelMd" style={{ color: severity.color }}>
|
||||
{severity.icon} {severity.label}
|
||||
</ThemedText>
|
||||
<View style={{ flex: 1 }}>
|
||||
<ThemedText preset="bodySm" style={{ color: colors.textSecondary }}>
|
||||
{formatTime(alert.created_at)}
|
||||
</ThemedText>
|
||||
</View>
|
||||
</View>
|
||||
<ThemedText preset="bodyMd" style={{ color: colors.textPrimary, marginTop: 6 }}>
|
||||
{alert.message}
|
||||
</ThemedText>
|
||||
</View>
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,41 @@
|
||||
import { FlatList, View } from 'react-native';
|
||||
import { ThemedText } from '@/components/ThemedText';
|
||||
import { colors } from '@/theme/colors';
|
||||
import { spacing } from '@/theme/spacing';
|
||||
import type { Alert } from '@/types/mat';
|
||||
import { AlertCard } from './AlertCard';
|
||||
|
||||
type AlertListProps = {
|
||||
alerts: Alert[];
|
||||
};
|
||||
|
||||
export const AlertList = ({ alerts }: AlertListProps) => {
|
||||
if (alerts.length === 0) {
|
||||
return (
|
||||
<View
|
||||
style={{
|
||||
alignItems: 'center',
|
||||
justifyContent: 'center',
|
||||
padding: spacing.md,
|
||||
borderWidth: 1,
|
||||
borderColor: colors.border,
|
||||
borderRadius: 12,
|
||||
backgroundColor: '#111827',
|
||||
}}
|
||||
>
|
||||
<ThemedText preset="bodyMd">No alerts — system nominal</ThemedText>
|
||||
</View>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<FlatList
|
||||
data={alerts}
|
||||
keyExtractor={(item) => item.id}
|
||||
renderItem={({ item }) => <AlertCard alert={item} />}
|
||||
contentContainerStyle={{ paddingBottom: spacing.md }}
|
||||
showsVerticalScrollIndicator={false}
|
||||
removeClippedSubviews={false}
|
||||
/>
|
||||
);
|
||||
};
|
||||