mirror of
https://github.com/ruvnet/RuView
synced 2026-07-25 17:51:48 +00:00
Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Claude
|
ad8d0aa59b | ||
|
rUv
|
79477c17a9 | ||
|
rUv
|
648ff525a2 | ||
|
rUv
|
0943a32248 | ||
|
bilibili12433014
|
1871ef3c2d | ||
|
bilibili12433014
|
e619b9430c | ||
|
Deploy Bot
|
b74fdcc733 |
@@ -250,3 +250,5 @@ v1/src/sensing/mac_wifi
|
||||
# Local build scripts
|
||||
firmware/esp32-csi-node/build_firmware.batdata/
|
||||
models/
|
||||
demo_pointcloud.ply
|
||||
demo_splats.json
|
||||
|
||||
@@ -96,6 +96,47 @@ node scripts/mincut-person-counter.js --port 5006 # Correct person counting
|
||||
>
|
||||
---
|
||||
|
||||
### Real-Time Dense Point Cloud (NEW)
|
||||
|
||||
RuView now generates **real-time 3D point clouds** by fusing camera depth + WiFi CSI + mmWave radar. All sensors stream simultaneously into a unified spatial model.
|
||||
|
||||
| Sensor | Data | Integration |
|
||||
|--------|------|-------------|
|
||||
| **Camera** | MiDaS monocular depth (GPU) | 640×480 → 19,200+ depth points per frame |
|
||||
| **ESP32 CSI** | ADR-018 binary frames (UDP) | RF tomography → 8×8×4 occupancy grid |
|
||||
| **WiFlow Pose** | 17 COCO keypoints from CSI | Skeleton overlay on point cloud |
|
||||
| **Vital Signs** | Breathing rate from CSI phase | Stored in ruOS brain every 60s |
|
||||
| **Motion** | CSI amplitude variance | Adaptive capture rate (skip depth when still) |
|
||||
|
||||
**Quick start:**
|
||||
```bash
|
||||
cd rust-port/wifi-densepose-rs
|
||||
cargo build --release -p wifi-densepose-pointcloud
|
||||
./target/release/ruview-pointcloud serve --bind 127.0.0.1:9880
|
||||
# Open http://localhost:9880 for live 3D viewer
|
||||
```
|
||||
|
||||
**CLI commands:**
|
||||
```bash
|
||||
ruview-pointcloud demo # synthetic demo
|
||||
ruview-pointcloud serve --bind 127.0.0.1:9880 # live server + Three.js viewer
|
||||
ruview-pointcloud capture --output room.ply # capture to PLY
|
||||
ruview-pointcloud train # depth calibration + DPO pairs
|
||||
ruview-pointcloud cameras # list available cameras
|
||||
ruview-pointcloud csi-test --count 100 # send test CSI frames
|
||||
ruview-pointcloud fingerprint office --seconds 5 # record named CSI room fingerprint
|
||||
```
|
||||
|
||||
The HTTP/viewer server defaults to **loopback (`127.0.0.1`)** — exposing live camera/CSI/vitals on `0.0.0.0` is an explicit opt-in. Brain URL defaults to `http://127.0.0.1:9876` and is overridable via `RUVIEW_BRAIN_URL` env var or the `--brain` flag on `serve`/`train`.
|
||||
|
||||
The pose overlay currently uses an **amplitude-energy heuristic** (`heuristic_pose_from_amplitude`) rather than trained WiFlow inference — real ONNX/Candle inference is tracked as a follow-up.
|
||||
|
||||
**Performance:** 22ms pipeline, 905 req/s API, 40K voxel room model from 20 frames.
|
||||
|
||||
**Brain integration:** Spatial observations (motion, vitals, skeleton, occupancy) sync to the ruOS brain every 60 seconds for agent reasoning.
|
||||
|
||||
See [PR #405](https://github.com/ruvnet/RuView/pull/405) for full details.
|
||||
|
||||
### What's New in v0.7.0
|
||||
|
||||
<details>
|
||||
@@ -904,6 +945,8 @@ cargo add wifi-densepose-ruvector # RuVector v2.0.4 integration layer (ADR-017
|
||||
| [`wifi-densepose-api`](https://crates.io/crates/wifi-densepose-api) | REST + WebSocket API layer | -- | [](https://crates.io/crates/wifi-densepose-api) |
|
||||
| [`wifi-densepose-config`](https://crates.io/crates/wifi-densepose-config) | Configuration management | -- | [](https://crates.io/crates/wifi-densepose-config) |
|
||||
| [`wifi-densepose-db`](https://crates.io/crates/wifi-densepose-db) | Database persistence (PostgreSQL, SQLite, Redis) | -- | [](https://crates.io/crates/wifi-densepose-db) |
|
||||
| `wifi-densepose-pointcloud` | Real-time dense point cloud from camera + WiFi CSI fusion (Three.js viewer, brain bridge). Workspace-only for now. | -- | — |
|
||||
| `wifi-densepose-geo` | Geospatial context (Sentinel-2 tiles, SRTM elevation, OSM, weather, night-mode). Workspace-only for now. | -- | — |
|
||||
|
||||
All crates integrate with [RuVector v2.0.4](https://github.com/ruvnet/ruvector) — see [AI Backbone](#ai-backbone-ruvector) below.
|
||||
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
# RuView Troubleshooting Guide
|
||||
|
||||
Known issues and fixes from the rebase-to-upstream branch (upstream #301).
|
||||
|
||||
---
|
||||
|
||||
## 1. Node not appearing in /api/v1/nodes
|
||||
|
||||
**Symptom:** ESP32-S3 node associates with WiFi, LED blinks, but no CSI frames arrive at the server. Node missing from `/api/v1/spatial/nodes`.
|
||||
|
||||
**Root cause:** After USB flash, the node enters a limping state where WiFi associates but the UDP CSI sender silently fails. The SoftAP + mDNS stack initializes but the CSI callback never fires.
|
||||
|
||||
**Fix:** Power cycle the node (unplug USB, wait 2s, replug). If that doesn't work, send DTR reset via serial: `python -m serial.tools.miniterm --dtr 0 COMx 115200` then Ctrl+C.
|
||||
|
||||
**Prevention:** Firmware 0.8.0+ includes a watchdog that detects zero CSI frames for 30s and triggers a software reset automatically. Nodes 1-10 are still on old firmware and lack this recovery (OTA-vs-BLE chicken-and-egg; see issue #6).
|
||||
|
||||
---
|
||||
|
||||
## 2. Person count stuck at 1
|
||||
|
||||
**Symptom:** `estimated_persons` always returns 1 regardless of how many people are in the room.
|
||||
|
||||
**Root cause (ADR-044):** Eight converging bugs:
|
||||
1. `score_to_person_count` had a ceiling of 3
|
||||
2. `fuse_multi_node_features` used `.max()` instead of sum — N identical readings collapsed to 1
|
||||
3. Four `.max(1)` clamps forced minimum count to 1 even when absent
|
||||
4. `field_model.estimate_occupancy` capped at `.min(3)`
|
||||
5. Normalization saturated (dividing by hardcoded thresholds instead of adaptive p95)
|
||||
6. No field model auto-calibration — eigenvalue path never activated
|
||||
7. Vitals-path clamps were asymmetric
|
||||
8. Tomography produced one blob (CC=1) so dedup gave wrong count
|
||||
|
||||
**Fix applied (Waves 1-3):**
|
||||
- Wave 1 (`9cc5f604`): ceiling 3→10, `.max()` → sum/3 aggregation, softened `.max(1)` clamps
|
||||
- Wave 2 (`306f1262`): RollingP95 adaptive normalization, field_model 30s auto-calibration, vitals clamp symmetry
|
||||
- Wave 3 (`c3df375a`+`0d4bfb09`+`6ac70ddf`): CC flood-fill infrastructure, lambda 0.1→5.0, threshold 0.01→0.15, CC>1 gate
|
||||
|
||||
**Current state:** `estimated_persons` = 6-8 for 5 bodies (3 humans + 2 dogs). Overcounts because the sum/3 dedup factor is a guess. Tomography still produces one blob (CC=1), so the CC path doesn't activate. Runtime-configurable lambda would help tune without redeployment.
|
||||
|
||||
---
|
||||
|
||||
## 3. Heart rate / breathing rate jitter
|
||||
|
||||
**Symptom:** HR and BR readings jump wildly between frames. BR CV was 23.3%, HR CV was 12.9%.
|
||||
|
||||
**Root cause (ADR-045):** 11 ESP32 nodes each compute independent vitals. The server used last-write-wins — whichever node's UDP packet arrived last overwrote the global vitals. At ~20 fps per node, this meant vitals randomly interleaved from different vantage points every 50ms.
|
||||
|
||||
**Fix applied (`46fbc061`):** Best-node selection. Each node's vitals are smoothed independently via median filter + EMA. The node with the highest combined `breathing_confidence + heartbeat_confidence` is selected as authoritative. Result: BR CV 23.3% → 12.6%, HR CV 12.9% → 11.6%.
|
||||
|
||||
**Known limitation:** The `wifi-densepose-vitals` crate has a superior 4-stage pipeline (bandpass → Hilbert envelope → autocorrelation → peak detection) but is not yet wired into the sensing server. The current `VitalSignDetector` uses a simpler FFT approach with 4 BPM frequency resolution.
|
||||
|
||||
---
|
||||
|
||||
## 4. Signal quality shows 50% always
|
||||
|
||||
**Symptom:** The dashboard signal quality gauge was always stuck at ~50%.
|
||||
|
||||
**Root cause:** Signal quality was a hardcoded placeholder value, not derived from actual CSI data.
|
||||
|
||||
**Fix applied:** ADR-044 Wave 2 replaced the fake gauge with RollingP95 adaptive normalization. The UI honesty pass (`b2070ab4`) added beta tags to unvalidated metrics, replaced the fake gauge with per-node pill indicators, and surfaced the actual per-node signal data.
|
||||
|
||||
---
|
||||
|
||||
## 5. Dashboard freezes every 2-4 seconds
|
||||
|
||||
**Symptom:** The spatial view and dashboard would freeze, then reconnect, creating a visible stutter every 2-4 seconds.
|
||||
|
||||
**Root cause:** The WebSocket broadcast channel's `recv()` returned `Err(Lagged)` when a client fell behind. The server treated this as a fatal error and dropped the connection. The client immediately reconnected, creating a connect/disconnect cycle.
|
||||
|
||||
**Fix applied (`581daf4f`):**
|
||||
- Server: `Lagged` error → `continue` (skip missed frames instead of disconnecting)
|
||||
- Server: 30s ping/pong keepalive to prevent Caddy proxy idle timeouts
|
||||
- Result: 154 frames over 8 seconds sustained, zero disconnects
|
||||
|
||||
---
|
||||
|
||||
## 6. OTA update crashes at 59%
|
||||
|
||||
**Symptom:** OTA firmware update via `/api/v1/firmware/download` progresses to ~59% then the node crashes with `StoreProhibited` on Core 1.
|
||||
|
||||
**Root cause:** NimBLE BLE advertising/scanning runs on Core 1. During OTA, the HTTP client also runs on Core 1. BLE and OTA compete for stack space, and the BLE scan callback triggers a memory access violation during the OTA write.
|
||||
|
||||
**Fix:**
|
||||
1. Stop NimBLE advertising and scanning before calling `esp_https_ota_begin()`
|
||||
2. Increase httpd stack from 4KB to 8KB (`CONFIG_HTTPD_MAX_REQ_HDR_LEN` and task stack)
|
||||
3. Resume BLE after OTA completes or fails
|
||||
|
||||
**Caveat:** Nodes running old firmware (1-10) can't receive this fix via OTA because the crash happens during the OTA itself. These nodes must be USB-flashed with firmware 0.8.0+ first, then future OTA updates will work. Node 11 was USB-flashed with the watchdog firmware and can receive OTA updates.
|
||||
|
||||
---
|
||||
|
||||
## 7. Can't SSH to babycube via LAN
|
||||
|
||||
**Symptom:** `ssh thyhack@10.0.10.10` hangs at banner exchange. Ping works, TCP port 22 is open, but SSH never completes the handshake.
|
||||
|
||||
**Workaround:** Use the Tailscale IP instead:
|
||||
```
|
||||
ssh thyhack@100.90.238.87
|
||||
```
|
||||
|
||||
**Not the cause:** CrowdSec. The 10.0.0.0/8 range is whitelisted in CrowdSec (`cscli decisions list` shows no active decisions for LAN IPs). The banner hang occurs before any authentication attempt, so it's not a firewall block.
|
||||
|
||||
**Suspected cause:** Unknown. Possibly MTU/fragmentation issue on the LAN segment, or a network stack bug in the babycube's NIC driver. The Tailscale overlay network (WireGuard UDP) bypasses whatever is causing the LAN TCP issue.
|
||||
|
||||
---
|
||||
|
||||
## 8. Right USB-C port doesn't work on some ESP32-S3 boards
|
||||
|
||||
**Symptom:** Plugging into the right USB-C port (when facing the board with USB-C toward you) shows no serial device on the host.
|
||||
|
||||
**Fix:** Use the left USB-C port. On most ESP32-S3-DevKitC boards, the left port is the USB-to-UART bridge (CP2102/CH340) used for flashing and serial monitor. The right port is the native USB (USB-JTAG) which requires different drivers and isn't used by the RuView firmware.
|
||||
@@ -0,0 +1,65 @@
|
||||
# ADR-044: Geospatial Satellite Integration
|
||||
|
||||
## Status
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
RuView generates real-time 3D point clouds from camera + WiFi CSI, but these exist in a local coordinate frame with no geographic reference. Integrating free satellite imagery, terrain elevation, and map data provides environmental context that enables the ruOS brain to reason about the physical world beyond the room.
|
||||
|
||||
## Decision
|
||||
|
||||
### Data Sources (all free, no API keys)
|
||||
| Source | Data | Resolution | Update | Format |
|
||||
|--------|------|-----------|--------|--------|
|
||||
| EOX Sentinel-2 Cloudless | Satellite tiles | 10m | Static mosaic | XYZ/JPEG |
|
||||
| SRTM GL1 (NASA) | Elevation/DEM | 30m (1-arcsec) | Static | Binary HGT |
|
||||
| Overpass API (OSM) | Buildings, roads | Vector | Real-time | JSON |
|
||||
| ip-api.com | IP geolocation | ~1km | Per-request | JSON |
|
||||
| Sentinel-2 STAC | Temporal satellite | 10m | Every 5 days | COG/STAC |
|
||||
| Open Meteo | Weather | Point | Hourly | JSON |
|
||||
|
||||
### Architecture
|
||||
Pure Rust implementation in `wifi-densepose-geo` crate. No GDAL/PROJ/GEOS — coordinate transforms implemented directly (~250 LOC). Tile caching on disk at `~/.local/share/ruview/geo-cache/`.
|
||||
|
||||
### Coordinate System
|
||||
- WGS84 for geographic coordinates
|
||||
- ENU (East-North-Up) as the bridge between local sensor frame and world
|
||||
- Local sensor frame: camera origin, +Z forward, +Y up
|
||||
|
||||
### Temporal Awareness
|
||||
Nightly scheduled fetch of Sentinel-2 latest imagery + OSM diffs + weather.
|
||||
Changes detected via image comparison and stored as brain memories for
|
||||
contrastive learning.
|
||||
|
||||
### Brain Integration
|
||||
Geospatial context stored as brain memories:
|
||||
- `spatial-geo`: location, elevation, nearby landmarks
|
||||
- `spatial-change`: detected changes in satellite/OSM data
|
||||
- `spatial-weather`: current conditions + forecast
|
||||
- `spatial-season`: vegetation index, snow cover, seasonal patterns
|
||||
- `spatial-local`: hyperlocal web context from Common Crawl WET
|
||||
|
||||
### Extended Data Sources (via ruvector WET/Common Crawl)
|
||||
| Source | Data | Use |
|
||||
|--------|------|-----|
|
||||
| Common Crawl WET | Web text near location | Local business info, reviews, events |
|
||||
| Wikidata | Structured knowledge | Building names, POI descriptions |
|
||||
| NASA FIRMS | Active fire (3-hour) | Safety alerts |
|
||||
| USGS Earthquakes | Seismic events | Safety context |
|
||||
| OpenAQ | Air quality (PM2.5) | Environmental health |
|
||||
| Overture Maps | Building footprints (Meta/MS) | Higher quality than OSM |
|
||||
|
||||
The ruvector brain server has existing `web_ingest` + Common Crawl support.
|
||||
WET files filtered by geographic URL patterns provide hyperlocal context.
|
||||
|
||||
## Consequences
|
||||
### Positive
|
||||
- Agent gains environmental awareness beyond the room
|
||||
- Temporal data enables seasonal calibration of CSI sensing
|
||||
- Change detection finds construction, vegetation, weather effects
|
||||
- All data sources are genuinely free with no API keys
|
||||
|
||||
### Negative
|
||||
- Initial data fetch requires internet (~2MB tiles + ~25MB DEM)
|
||||
- Cached data becomes stale (mitigated by nightly refresh)
|
||||
- IP geolocation has ~1km accuracy (mitigated by manual override)
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
# ADR-044: Provisioning Tool Enhancements
|
||||
# ADR-050: Provisioning Tool Enhancements
|
||||
|
||||
**Status**: Proposed
|
||||
**Date**: 2026-03-03
|
||||
@@ -0,0 +1,141 @@
|
||||
---
|
||||
Research Document ID: RD-C-00
|
||||
Date: 2026-04-21
|
||||
Status: Draft
|
||||
Authors: RuView Research Team
|
||||
Related ADRs: ADR-001 through ADR-081 (existing corpus); proposed ADR-084 through ADR-088 (this compendium)
|
||||
---
|
||||
|
||||
# RD-C-00: Connectome-Embodied-Brain Compendium — Index
|
||||
|
||||
## Abstract
|
||||
|
||||
This compendium evaluates whether RuVector v2.0.4 — the vector, graph, attention, solver, and CRV toolkit already integrated into the wifi-densepose Rust workspace — can serve as the substrate for a four-layer **coherence-aware connectome operating system (CC-OS)**. The design is inspired by the 2024 Lappalainen et al. connectome-constrained deep mechanistic networks for the Drosophila visual system, the Kakaria and de Bivort (2017) leaky-integrate-and-fire (LIF) central-complex model, and the NeuroMechFly v1/v2 embodied simulator lineage, but the RuView framing is deliberately narrower and more auditable than press coverage of "whole-fly-brain-in-silico" demonstrations. We do **not** claim mind upload, phenomenal consciousness, or biologically faithful cognition. We claim the ability to (a) ingest a published connectome as a typed graph, (b) run event-driven LIF dynamics at millisecond resolution, (c) close the loop through an embodied simulator, and (d) apply RuVector's structural circuit analysis (ruvector-mincut, ruvector-attn-mincut), counterfactual perturbation (via existing CoherenceGate / CrvSessionManager patterns), and witness-style auditing (ADR-028 lineage) to produce explainable behavioral episodes. The twelve documents that follow specify the substrate, runtime, simulator, analysis loop, risks, and acceptance tests. This index document is the entry point and positioning statement. Readers in a hurry should read §4 (positioning) and §5 (key findings preview) first.
|
||||
|
||||
## 1. Compendium Structure
|
||||
|
||||
The compendium is organized as twelve documents (00 through 11). Clusters A–D correspond to the four architectural layers; cluster E covers cross-cutting concerns.
|
||||
|
||||
| Doc | ID | Cluster | Owner focus | One-line description |
|
||||
|-----|----|---------|-------------|----------------------|
|
||||
| 00 | RD-C-00 | E: Meta | Index | This document — ToC, positioning, quick-start |
|
||||
| 01 | RD-C-01 | E: Foundations | Literature | Published-literature survey: connectomes, LIF models, embodiment |
|
||||
| 02 | RD-C-02 | A: Substrate | Layer 1 | Connectome graph schema, vector store, temporal-tensor storage |
|
||||
| 03 | RD-C-03 | A: Substrate | Layer 1 | MinCut circuit analysis, structural fragility metric |
|
||||
| 04 | RD-C-04 | B: Runtime | Layer 2 | Event-driven LIF neural dynamics runtime (`wifi-densepose-neuro`) |
|
||||
| 05 | RD-C-05 | B: Runtime | Layer 2 | Cross-region attention fusion (`NeuralFusionArray`) |
|
||||
| 06 | RD-C-06 | C: Embodiment | Layer 3 | Embodied simulator closed loop (`wifi-densepose-embody`) |
|
||||
| 07 | RD-C-07 | D: Analysis | Layer 4 | Coherence + CRV behavioral episodes (`BehaviorPipeline`) |
|
||||
| 08 | RD-C-08 | D: Analysis | Layer 4 | Counterfactual perturbation fragility protocol |
|
||||
| 09 | RD-C-09 | E: Integration | Architecture | Four-layer architecture: data flow, interfaces, invariants |
|
||||
| 10 | RD-C-10 | E: Acceptance | Testing | Acceptance test grooming — pass/fail criteria per layer |
|
||||
| 11 | RD-C-11 | E: Governance | Roadmap | Risks, positioning, roadmap, proposed ADR-084..ADR-088 |
|
||||
|
||||
Documents 00, 01, and 02 are delivered in the first writing pass. Documents 03–11 are delivered by parallel clusters referenced by name throughout this index.
|
||||
|
||||
## 2. Key Crate → Layer Mapping
|
||||
|
||||
The CC-OS design reuses existing RuVector v2.0.4 crates rather than introducing new numerical kernels. Two new wifi-densepose crates are proposed (`wifi-densepose-neuro`, `wifi-densepose-embody`); the rest of the runtime is assembled from crates already in the workspace.
|
||||
|
||||
| Layer | Responsibility | RuVector crate(s) | wifi-densepose crate(s) |
|
||||
|-------|----------------|-------------------|-------------------------|
|
||||
| 1. Substrate | Connectome graph + per-neuron embeddings + temporal voltages | `ruvector-mincut`, `ruvector-attn-mincut`, `ruvector-temporal-tensor` | `wifi-densepose-core`, `wifi-densepose-db` |
|
||||
| 2. Runtime | LIF dynamics, spike event bus, region attention | `ruvector-solver` (Neumann), `ruvector-attention` (SDPA) | **`wifi-densepose-neuro`** (proposed) |
|
||||
| 3. Embodiment | Body simulator, sensors, actuators, closed loop | — (external: NeuroMechFly/MuJoCo/Rapier) | **`wifi-densepose-embody`** (proposed) |
|
||||
| 4. Analysis | Mincut fragility, CRV episodes, coherence gating, counterfactuals | `ruvector-mincut`, `ruvector-attention`, `ruvector-crv`, `ruvector-gnn` (feature-gated) | `wifi-densepose-signal` (reuse RuvSense patterns) |
|
||||
|
||||
The RuvSense module set (signal/src/ruvsense/) is reused not by lifting WiFi-specific code but by lifting **patterns**: `CoherenceGate` (coherence_gate.rs) becomes a spike-train coherence gate; `MultistaticArray` (viewpoint/fusion.rs) becomes the `NeuralFusionArray` aggregate; `GeometricDiversityIndex` (viewpoint/geometry.rs) becomes a circuit-diversity index; `WifiCrvPipeline` (crv/mod.rs) becomes the `BehaviorPipeline` facade. Section 4 of doc 09 lists every pattern reuse explicitly.
|
||||
|
||||
## 3. Key Findings Preview
|
||||
|
||||
The full findings, with evidence, live in docs 01 through 11. The one-sentence previews:
|
||||
|
||||
| # | Finding | Evidence location |
|
||||
|---|---------|-------------------|
|
||||
| F1 | RuVector's `MinCutBuilder::new().exact()` can score connectome bottleneck fragility at 10k–150k node scale. | see doc 03 |
|
||||
| F2 | `NeumannSolver` is appropriate for steady-state firing-rate approximation but not for event-driven spiking; event-driven LIF needs an explicit spike queue. | see doc 04 |
|
||||
| F3 | `TemporalTensorCompressor` can store ~60 s of 1 kHz voltage traces for 50k neurons in roughly 3–4 GB using tiered 8/5–7/3-bit compression. | see doc 02 §8 |
|
||||
| F4 | `ScaledDotProductAttention` is sufficient as a cross-region fusion primitive provided per-region embeddings are dimensionally consistent (32-d or 128-d). | see doc 05 |
|
||||
| F5 | The six-stage `CrvSessionManager` pipeline maps 1:1 onto a behavioral-episode lifecycle (stimulus → sensory response → motor plan → action → audit → composite). | see doc 07 |
|
||||
| F6 | Counterfactual "silence neuron X, rerun, measure divergence" is directly expressible as a perturbation on the spike event bus with a fragility score computed over the divergence embedding. | see doc 08 |
|
||||
| F7 | End-to-end closed-loop real-time factor of 0.1×–0.5× on a single workstation is plausible for the full adult Drosophila FlyWire dataset (~139k neurons, ~54M synapses); realtime (1.0×) requires batching and GPU assistance. | see doc 09 |
|
||||
| F8 | The design cleanly excludes phenomenal-consciousness claims; it is a structural / dynamical / behavioral audit tool, not a substrate for mind. | see doc 01 §8, doc 11 |
|
||||
| F9 | ESP32 / WiFi-CSI hardware remains relevant only as an external sensor channel feeding layer-3 sensor inputs; it is not part of the neural substrate. | see doc 06 |
|
||||
| F10 | Five new ADRs (ADR-084 substrate, ADR-085 runtime, ADR-086 embodiment, ADR-087 analysis loop, ADR-088 governance) are proposed to land this compendium. | see doc 11 |
|
||||
|
||||
## 4. Positioning: Coherence-Aware Connectome Operating System (CC-OS)
|
||||
|
||||
RuView's framing is **not** "digital mind" and not "whole-brain emulation in the Sandberg–Bostrom sense." The framing is a **coherence-aware connectome operating system**: a runtime that (i) loads a typed, provenance-tagged connectome graph; (ii) simulates its neural dynamics at millisecond resolution; (iii) closes the sensorimotor loop through an embodied simulator; and (iv) provides auditable, counterfactual analysis of the resulting behavior. The three key adjectives are:
|
||||
|
||||
- **Coherence-aware** — borrowed from the WiFi CSI pipeline: every layer produces a numerical coherence score with a `CoherenceGate` decision (Accept / PredictOnly / Reject / Recalibrate). This is not a claim about consciousness; it is a signal-quality gate.
|
||||
- **Connectome** — the graph is sourced from published, peer-reviewed reconstructions (FlyWire, MICrONS, Winding et al. 2023). We do **not** generate connectomes; we ingest them.
|
||||
- **Operating system** — the runtime exposes a scheduler, event bus, memory tiers, and auditable hooks. It is not a single monolithic model; it is a platform on which circuit analyses, behavioral episodes, and counterfactuals are scheduled and logged.
|
||||
|
||||
What this is **not**:
|
||||
|
||||
1. Not a mind, not a subject, not a locus of experience. Phenomenal consciousness is explicitly out of scope (doc 01 §8 elaborates, citing Seth 2021 and the Markov-blanket framing).
|
||||
2. Not a whole-mammal brain emulator. At current compute, the feasibility frontier is adult Drosophila (~139k neurons) and perhaps larval zebrafish (~100k neurons). Mouse cortex (~75M neurons) is out of scope.
|
||||
3. Not "Eon-style." Recent press coverage of "the fly brain running on a laptop" conflates three separable achievements — visual-system circuit fit (Lappalainen 2024), central-complex LIF dynamics (Kakaria 2017), and NeuroMechFly body loop — into a single headline. RuView keeps those three layers separate, with distinct acceptance criteria (doc 10), and does not make behavioral claims the layer does not support.
|
||||
4. Not a replacement for NEURON, Brian2, Nengo, or Genesis. Those are biophysically richer. CC-OS prioritizes auditable graph-plus-LIF at connectome scale with built-in fragility analysis; it trades multi-compartment biophysics for throughput and governance.
|
||||
|
||||
## 5. Related ADRs
|
||||
|
||||
### 5.1 Existing ADR corpus referenced
|
||||
|
||||
- ADR-001 through ADR-043 — the existing wifi-densepose decision record corpus
|
||||
- ADR-014 — SOTA signal processing (coherence primitives reused in layer 4)
|
||||
- ADR-016 — RuVector training pipeline integration (precedent for ruvector adoption)
|
||||
- ADR-024 — Contrastive CSI embedding / AETHER (precedent for per-node embedding design)
|
||||
- ADR-027 — Cross-environment domain generalization / MERIDIAN (precedent for region fusion)
|
||||
- ADR-028 — ESP32 capability audit + witness verification (precedent for audit protocol)
|
||||
- ADR-029 through ADR-032 — RuvSense multistatic / coherence / security ADRs (pattern source)
|
||||
|
||||
References to ADR-044 through ADR-081 are understood to exist in the broader RuView corpus; specific numbers are not hard-coded here because the corpus continues to grow.
|
||||
|
||||
### 5.2 ADRs proposed by this compendium
|
||||
|
||||
Full text for each lives in doc 11. Previews:
|
||||
|
||||
| ADR | Title | Status | Doc of record |
|
||||
|-----|-------|--------|---------------|
|
||||
| ADR-084 | Connectome graph substrate + per-neuron embeddings | Proposed | RD-C-02, RD-C-03 |
|
||||
| ADR-085 | Event-driven LIF neural runtime (`wifi-densepose-neuro`) | Proposed | RD-C-04, RD-C-05 |
|
||||
| ADR-086 | Embodied simulator integration (`wifi-densepose-embody`) | Proposed | RD-C-06 |
|
||||
| ADR-087 | Coherence + CRV + counterfactual analysis loop (`BehaviorPipeline`) | Proposed | RD-C-07, RD-C-08 |
|
||||
| ADR-088 | Governance, positioning, and non-claims for CC-OS | Proposed | RD-C-11 |
|
||||
|
||||
## 6. Quick-Start Table of Contents
|
||||
|
||||
Read in this order for a 30-minute overview:
|
||||
|
||||
1. This document (RD-C-00) — §4 Positioning and §5 ADR previews.
|
||||
2. [RD-C-01 Foundations and Connectome Neuroscience](./01-foundations-connectome-neuroscience.md) — §6 Scaling frontier table and §8 "Not mind upload" section.
|
||||
3. [RD-C-09 Four-Layer Architecture](./09-four-layer-architecture.md) — the data-flow diagram.
|
||||
4. [RD-C-11 Risks, Positioning, Roadmap](./11-risks-positioning-roadmap.md) — the non-claims and the milestone table.
|
||||
|
||||
Read in this order for a technical deep dive:
|
||||
|
||||
1. [RD-C-02 Connectome Graph Substrate](./02-connectome-graph-substrate.md)
|
||||
2. [RD-C-03 MinCut Circuit Analysis](./03-mincut-circuit-analysis.md)
|
||||
3. [RD-C-04 Neural Dynamics Runtime](./04-neural-dynamics-runtime.md)
|
||||
4. [RD-C-05 Cross-Region Attention Fusion](./05-cross-region-attention-fusion.md)
|
||||
5. [RD-C-06 Embodied Simulator Closed Loop](./06-embodied-simulator-closed-loop.md)
|
||||
6. [RD-C-07 Coherence + CRV Behavioral Episodes](./07-coherence-crv-behavioral-episodes.md)
|
||||
7. [RD-C-08 Counterfactual Perturbation](./08-counterfactual-perturbation.md)
|
||||
8. [RD-C-10 Acceptance Test Grooming](./10-acceptance-test-grooming.md)
|
||||
|
||||
## 7. Document Conventions
|
||||
|
||||
- Every RD-C-* document uses the same front-matter block (ID, date, status, authors, related ADRs).
|
||||
- Every claim cites author + year + venue. No DOIs are fabricated.
|
||||
- Every proposed Rust API is tagged **(proposed)** and distinguished from observed v2.0.4 APIs.
|
||||
- Every neural-quantity number has units and a sanity-check paragraph.
|
||||
- Every non-claim (what CC-OS deliberately is not) is surfaced explicitly, not buried.
|
||||
|
||||
## 8. Next Steps
|
||||
|
||||
1. Land docs 00, 01, 02 (this writing pass).
|
||||
2. Parallel clusters land docs 03–11.
|
||||
3. Convert finding F1–F10 into the five proposed ADRs.
|
||||
4. Validate the §3 feasibility claims against a small FlyWire-subset prototype on the `claude/connectome-embodied-brain-COE3I` branch.
|
||||
5. Re-run the witness bundle (ADR-028 pattern) once prototype code exists.
|
||||
@@ -0,0 +1,339 @@
|
||||
---
|
||||
Research Document ID: RD-C-01
|
||||
Date: 2026-04-21
|
||||
Status: Draft
|
||||
Authors: RuView Research Team
|
||||
Related ADRs: proposed ADR-084, ADR-085, ADR-086, ADR-088
|
||||
---
|
||||
|
||||
# RD-C-01: Foundations — Connectome Neuroscience and Embodied Brain Simulation
|
||||
|
||||
## Abstract
|
||||
|
||||
This document surveys the published science the Coherence-Aware Connectome
|
||||
Operating System (CC-OS) must stand on. We review the four connectome
|
||||
datasets currently usable for whole-circuit modelling (C. elegans, *Drosophila*
|
||||
larva, adult *Drosophila* via FlyWire, mouse visual cortex via MICrONS), the
|
||||
three dynamical-model families that have been fit to those connectomes
|
||||
(firing-rate networks, leaky integrate-and-fire, connectome-constrained deep
|
||||
mechanistic networks), and the embodiment platforms that close the
|
||||
sensorimotor loop (NeuroMechFly v1/v2, MuJoCo-MJX, Rapier, Walknet). We then
|
||||
discuss the feasibility frontier as a function of neuron count, argue
|
||||
explicitly why a runtime of this shape is **not** a mind or a whole-brain
|
||||
emulation in the Sandberg–Bostrom sense, and identify the open questions
|
||||
RuView is uniquely positioned to address: coherence-aware gating,
|
||||
structural-fragility scoring, counterfactual circuit discovery, and
|
||||
witness-audit reproducibility.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. Why the fly connectome is the right first target
|
||||
2. Connectome datasets in publishable form
|
||||
3. Whole-brain dynamical models
|
||||
4. Embodiment platforms
|
||||
5. "Eon"-style integrations — what the code does vs what the press says
|
||||
6. Scaling frontier
|
||||
7. Why CC-OS is not mind upload
|
||||
8. Open questions RuView can address
|
||||
9. References
|
||||
|
||||
---
|
||||
|
||||
## 1. Why the Fly Connectome is the Right First Target
|
||||
|
||||
Three properties make *Drosophila melanogaster* the right first organism for
|
||||
an open-loop, auditable, connectome-driven embodied runtime:
|
||||
|
||||
1. **Tractable scale** — 139,255 neurons and ~54 million synapses in the
|
||||
adult brain (Dorkenwald et al. 2024, *Nature*). Two orders of magnitude
|
||||
smaller than mouse cortex, and the graph fits in a laptop's main memory
|
||||
with the tiered temporal-tensor compression described in
|
||||
02-connectome-graph-substrate.md.
|
||||
2. **Rich behavioral vocabulary** — locomotion, grooming, feeding, courtship,
|
||||
phototaxis, and memory-guided navigation are all well-studied at the
|
||||
single-neuron level (Namiki et al. 2018, *eLife*; Seeds et al. 2014, *Curr
|
||||
Biol*; Hampel et al. 2015, *eLife*).
|
||||
3. **Published body models** — NeuroMechFly (Lobato-Rios et al. 2022, *Nat
|
||||
Methods*) and NeuroMechFly v2 (Wang-Chen et al. 2024, preprint) provide
|
||||
an articulated MuJoCo body with validated proprioception and vision
|
||||
adapters.
|
||||
|
||||
The zebrafish larva (~100k neurons, whole-brain calcium imaging available)
|
||||
is a close second choice but lacks a comparably mature published body
|
||||
simulator. *C. elegans* (302 neurons) is well-served by OpenWorm but
|
||||
operates at a scale that does not stress RuVector's strengths.
|
||||
|
||||
## 2. Connectome Datasets in Publishable Form
|
||||
|
||||
| Organism | Dataset | Neurons | Synapses | Source | Status |
|
||||
|----------|---------|---------|----------|--------|--------|
|
||||
| *C. elegans* hermaphrodite | White et al. electron micrography | 302 | ~7,000 | White et al. 1986, *Phil Trans R Soc B* | Complete; OpenWorm reconstruction |
|
||||
| *C. elegans* male | Cook et al. | 385 | ~10,000 | Cook et al. 2019, *Nature* | Complete |
|
||||
| *Drosophila* larva | Winding et al. | ~3,000 | ~548,000 | Winding et al. 2023, *Science* | Complete, both hemispheres |
|
||||
| Adult *Drosophila* hemibrain | Scheffer et al. | ~25,000 | ~20M | Scheffer et al. 2020, *eLife* | Hemibrain only |
|
||||
| Adult *Drosophila* whole brain (FlyWire) | Dorkenwald et al. | ~139,255 | ~54M | Dorkenwald et al. 2024, *Nature* | Complete, proofread |
|
||||
| Mouse visual cortex (MICrONS) | MICrONS Consortium | ~200,000 | ~500M | MICrONS Consortium 2023, *bioRxiv* / *Nature* 2025 | Partial; ~1 mm³ volume |
|
||||
| Larval zebrafish | Vishwanathan et al. | ~100,000 (est.) | — | Vishwanathan et al. 2024, preprint | In progress |
|
||||
|
||||
For CC-OS v1 we recommend **FlyWire** (Dorkenwald 2024) as the primary
|
||||
substrate because it is the only fully-proofread whole-brain connectome at a
|
||||
scale that exercises RuVector's capabilities without requiring GPU clusters.
|
||||
|
||||
### 2.1 Metadata provided by FlyWire
|
||||
|
||||
FlyWire publishes per-neuron metadata that maps cleanly onto the
|
||||
`ConnectomeGraph` schema of 02-connectome-graph-substrate.md:
|
||||
|
||||
| FlyWire field | `ConnectomeGraph` field |
|
||||
|---------------|-------------------------|
|
||||
| `root_id` | `Neuron.id` |
|
||||
| `super_class` | `Neuron.class` (Sensory / Interneuron / Motor / …) |
|
||||
| `cell_type` | `Neuron.cell_type_hash` |
|
||||
| `side` | region metadata |
|
||||
| `nt_type` | `Neuron.neurotransmitter` |
|
||||
| `pre`, `post`, `syn_count` | `Synapse.pre_id`, `post_id`, `weight` |
|
||||
|
||||
Per-synapse transmitter predictions (Eckstein et al. 2024, *Cell*) are
|
||||
separately published and should be merged into the edge schema as
|
||||
`Synapse.nt` with a provenance tag.
|
||||
|
||||
## 3. Whole-Brain Dynamical Models
|
||||
|
||||
Three families have been fit to connectomes in the past eight years. They
|
||||
differ in what is simulated and what computational substrate they demand.
|
||||
|
||||
### 3.1 Firing-rate networks
|
||||
|
||||
Each neuron is a scalar rate $r_i(t) \in \mathbb{R}_{\ge 0}$ governed by
|
||||
|
||||
$$
|
||||
\tau \frac{dr_i}{dt} = -r_i + f\!\Bigl( \sum_j W_{ij} r_j + I_i \Bigr),
|
||||
$$
|
||||
|
||||
with $f$ a nonlinearity (ReLU, tanh, sigmoid). Lappalainen et al. 2024
|
||||
(*Nature*) fit firing-rate networks to the fly optic lobe connectome and
|
||||
recovered direction-selective T4/T5 cell responses that matched
|
||||
electrophysiology. Rate networks are cheap (one matrix-vector per
|
||||
timestep), map cleanly onto `ruvector-solver`'s sparse primitives for
|
||||
steady-state approximations, and lose temporal fidelity at the sub-100 ms
|
||||
timescale.
|
||||
|
||||
### 3.2 Leaky integrate-and-fire (LIF)
|
||||
|
||||
$$
|
||||
\tau_m \frac{dV_i}{dt} = -(V_i - V_{\text{rest}}) + R_m I_i(t),
|
||||
$$
|
||||
|
||||
with a reset-on-threshold rule emitting spikes delivered to postsynaptic
|
||||
neurons with axonal delay. Brunel 2000 (*J Comput Neurosci*) established
|
||||
the reference framework; Kakaria & de Bivort 2017 (*Front Behav Neurosci*)
|
||||
applied it to the central complex of *Drosophila*. LIF is the right
|
||||
abstraction level for CC-OS: it captures spike-timing phenomena important
|
||||
for grooming and courtship circuits while staying tractable at 50k–150k
|
||||
neurons. See 04-neural-dynamics-runtime.md for the Rust implementation.
|
||||
|
||||
### 3.3 Connectome-constrained deep mechanistic networks
|
||||
|
||||
Lappalainen et al. 2024 (*Nature*) introduced a family of trainable rate
|
||||
networks in which the synaptic weights are constrained by the connectome
|
||||
(sign and topology) but tuned by backpropagation against neural-recording
|
||||
objectives. The family generalises firing-rate networks by allowing
|
||||
learned nonlinearities and sensory adapters. CC-OS treats this class as a
|
||||
**calibration target** rather than a primary runtime: once LIF dynamics
|
||||
are running, Lappalainen-style fits provide external validation that the
|
||||
per-neuron nonlinearities are in the right regime.
|
||||
|
||||
### 3.4 Biophysically detailed models
|
||||
|
||||
NEURON, GENESIS, Brian2, and the Blue Brain morphologically-detailed cortex
|
||||
models fall here. CC-OS is **not** competing at this level. Our goal is
|
||||
graph + LIF + embodiment with auditable fragility; detailed biophysics is
|
||||
deferred to specialised tools. Non-goal noted in 02 §12.
|
||||
|
||||
## 4. Embodiment Platforms
|
||||
|
||||
| Platform | Physics backend | Language | Fidelity | Weakness |
|
||||
|----------|-----------------|----------|----------|----------|
|
||||
| NeuroMechFly v1 | MuJoCo | Python | High — biomechanically validated | Python overhead, Ubuntu focused |
|
||||
| NeuroMechFly v2 | MuJoCo / MJX | Python + JAX | Higher — vision adapter included | Same |
|
||||
| Walknet | Custom | Python / MATLAB | Medium — gait only | No body visuals |
|
||||
| MuJoCo (bare) | MuJoCo | C | High general | Body model must be authored |
|
||||
| PyBullet | Bullet | Python | Medium | Less accurate contact |
|
||||
| Rapier | Rapier | Rust | Medium-high | No biomechanical fly model yet |
|
||||
| Isaac Gym | PhysX | Python / C++ | High for large-scale RL | GPU-only, not auditable |
|
||||
|
||||
For CC-OS v1, we recommend a **Rapier-based Rust fly body** for the tight
|
||||
inner loop plus a **NeuroMechFly bridge** (Python IPC over shared memory)
|
||||
for biomechanical validation runs. 06-embodied-simulator-closed-loop.md
|
||||
develops this architecture.
|
||||
|
||||
## 5. "Eon"-Style Integrations — What the Code Does vs What the Press Says
|
||||
|
||||
Recent industry demonstrations (sometimes presented as "running a fly brain
|
||||
on a laptop") combine three separable achievements:
|
||||
|
||||
1. A connectome-constrained visual-system network (Lappalainen 2024).
|
||||
2. A central-complex LIF dynamics model (Kakaria 2017, extended).
|
||||
3. A body simulator in the NeuroMechFly lineage.
|
||||
|
||||
The engineering story — stitching these three together, running the loop
|
||||
at near-real-time — is genuine and valuable. The framing that this
|
||||
constitutes "an in-silico fly" or "a digital organism" is an overclaim.
|
||||
The fly brain is not just the optic lobe plus the central complex plus
|
||||
motor neurons; it is an integrated system with peripheral sensors,
|
||||
neuromodulators, circadian state, genetic plasticity, and developmental
|
||||
history. A three-module stitch-up is a faithful partial model of a
|
||||
specific circuit, not an organism. CC-OS inherits the stitch architecture
|
||||
(graph, LIF, body) but insists on naming what each layer models and what
|
||||
it does not, and on an acceptance test (10-acceptance-test-grooming.md)
|
||||
that is a reproducible behavioral assay, not a rhetorical claim.
|
||||
|
||||
## 6. Scaling Frontier
|
||||
|
||||
| Species | Neuron count | Synapse count | Today's feasibility | Notes |
|
||||
|---------|--------------|---------------|---------------------|-------|
|
||||
| C. elegans | 302 | ~7k | Trivial | Solved; OpenWorm |
|
||||
| Drosophila larva | 3k | ~500k | Easy | Winding 2023 |
|
||||
| Drosophila adult (FlyWire) | 139k | 54M | **Target for CC-OS v1** | Borderline realtime |
|
||||
| Zebrafish larva | ~100k | — | Feasible | No proofread connectome yet |
|
||||
| Honeybee | ~960k | — | Hard | No whole-brain connectome |
|
||||
| Mouse cortex (1 mm³ patch) | ~200k | ~500M | Hard | MICrONS partial; full brain ~75M |
|
||||
| Human cortex | ~16B | ~10¹⁴ | Out of scope | Not feasible with current hardware |
|
||||
|
||||
Feasibility is gated by three resources:
|
||||
- **Memory**: synapse list in `(u64, u64, f64)` CSR triplets is ~24 bytes
|
||||
per edge; 54M synapses = ~1.3 GB uncompressed.
|
||||
- **Compute**: event-driven LIF scales with mean firing rate × neuron count.
|
||||
At 5 Hz average, 139k neurons emit ~700k spikes/s, each triggering
|
||||
~400 postsynaptic updates = 280M updates/s. Feasible on one workstation.
|
||||
- **I/O**: voltage traces at 1 kHz for 139k neurons at 8-bit = 139 MB/s
|
||||
uncompressed. Tiered compression (02 §8) brings this to ~35 MB/s.
|
||||
|
||||
Full FlyWire realtime is aspirational but not absurd on a top-end
|
||||
workstation. The acceptance test (doc 10) targets a 15k–30k sub-network to
|
||||
stay well inside the feasibility envelope.
|
||||
|
||||
## 7. Why CC-OS is Not Mind Upload
|
||||
|
||||
Three framings articulate what CC-OS deliberately does not claim:
|
||||
|
||||
### 7.1 Information flow (Shannon view)
|
||||
|
||||
CC-OS simulates information flow from sensors through the connectome to
|
||||
actuators. This is computation, not subjectivity. Shannon-channel analysis
|
||||
has nothing to say about phenomenal experience, and we make no attempt to
|
||||
attribute experience.
|
||||
|
||||
### 7.2 Markov blankets (Seth 2021 framing)
|
||||
|
||||
A Markov blanket separates a system's internal states from external states
|
||||
conditional on sensory and active states (Friston 2013). In a CC-OS
|
||||
run, the boundary is explicit and inspectable: sensory neurons are the
|
||||
blanket's active/sensory states, interneurons are internal. There is no
|
||||
claim that crossing this blanket constitutes minding.
|
||||
|
||||
### 7.3 Sandberg–Bostrom WBE scale
|
||||
|
||||
Sandberg and Bostrom's 2008 whole-brain-emulation framework enumerates
|
||||
eleven levels, from computational network (level 4) to quantum (level 11).
|
||||
CC-OS sits at **level 4 (computational network of point neurons)** with
|
||||
partial level 5 (analog population dynamics). All higher levels (dendritic
|
||||
compartments, intracellular biochemistry, molecular signalling) are out
|
||||
of scope. This alone rules out mind-upload claims in the Sandberg–Bostrom
|
||||
sense.
|
||||
|
||||
### 7.4 What we do claim
|
||||
|
||||
- The runtime reproduces the connectome's structural wiring.
|
||||
- It reproduces qualitative LIF dynamics consistent with published fits.
|
||||
- It reproduces published behavioral bouts under canonical stimuli within
|
||||
a bounded subcircuit.
|
||||
- It supports counterfactual perturbation with auditable provenance.
|
||||
|
||||
That's the ceiling. Everything above — consciousness, identity, memory
|
||||
continuity — is out of scope and is not implied by any sentence in this
|
||||
compendium.
|
||||
|
||||
## 8. Open Questions RuView Can Uniquely Address
|
||||
|
||||
1. **Coherence-aware runtime.** The existing CSI pipeline uses
|
||||
`CoherenceGate` (Accept / PredictOnly / Reject / Recalibrate) to reject
|
||||
out-of-distribution signal frames. The neural analog — detect when the
|
||||
simulated population is drifting outside the behavioral attractor and
|
||||
refuse to make circuit claims on that bout — is novel and valuable.
|
||||
See 05-cross-region-attention-fusion.md and
|
||||
07-coherence-crv-behavioral-episodes.md.
|
||||
2. **Structural fragility scoring.** Fragility = $\Delta\text{cut}/\text{cut}$
|
||||
under ablation (see 03-mincut-circuit-analysis.md). No existing
|
||||
connectome simulator emits fragility as a first-class output. RuVector's
|
||||
`DynamicMinCut` is built for exactly the incremental-ablation use case.
|
||||
3. **Counterfactual circuit discovery.** Greedy ablation + re-mincut is a
|
||||
simple, well-defined search procedure for minimal sufficient circuits.
|
||||
See 08-counterfactual-perturbation.md.
|
||||
4. **Witness-audit reproducibility.** ADR-028 established a witness-bundle
|
||||
standard: input → pipeline → SHA-256 hash. Lifting this to neural
|
||||
runtime is straightforward and makes CC-OS reviewable in a way that
|
||||
Python-notebook demos are not.
|
||||
|
||||
## 9. References
|
||||
|
||||
1. White, J. G., Southgate, E., Thomson, J. N., Brenner, S. (1986). *The
|
||||
structure of the nervous system of the nematode C. elegans.* Phil Trans R
|
||||
Soc B.
|
||||
2. Cook, S. J., Jarrell, T. A., Brittin, C. A., et al. (2019). *Whole-animal
|
||||
connectomes of both C. elegans sexes.* Nature.
|
||||
3. Winding, M., Pedigo, B. D., Barnes, C. L., et al. (2023). *The connectome
|
||||
of an insect brain.* Science.
|
||||
4. Scheffer, L. K., Xu, C. S., Januszewski, M., et al. (2020). *A connectome
|
||||
and analysis of the adult Drosophila central brain.* eLife.
|
||||
5. Dorkenwald, S., Matsliah, A., Sterling, A. R., et al. (2024). *Neuronal
|
||||
wiring diagram of an adult brain.* Nature.
|
||||
6. Eckstein, N., Bates, A. S., Champion, A., et al. (2024). *Neurotransmitter
|
||||
classification from electron microscopy images.* Cell.
|
||||
7. MICrONS Consortium. (2023). *Functional connectomics spanning multiple
|
||||
areas of mouse visual cortex.* bioRxiv; subsequent 2025 Nature.
|
||||
8. Vishwanathan, A., et al. (2024). *Larval zebrafish whole-brain connectome
|
||||
(in progress).* Preprint.
|
||||
9. Brunel, N. (2000). *Dynamics of sparsely connected networks of excitatory
|
||||
and inhibitory spiking neurons.* J. Comput. Neurosci.
|
||||
10. Izhikevich, E. M. (2003). *Simple model of spiking neurons.* IEEE TNN.
|
||||
11. Kakaria, K. S., de Bivort, B. L. (2017). *Ring attractor dynamics emerge
|
||||
from a spiking model of the entire protocerebral bridge.* Front Behav
|
||||
Neurosci.
|
||||
12. Lappalainen, J. K., Tschopp, F. D., Prakhya, S., et al. (2024).
|
||||
*Connectome-constrained networks predict neural activity across the fly
|
||||
visual system.* Nature.
|
||||
13. Lobato-Rios, V., Ramalingasetty, S. T., Özdil, P. G., et al. (2022).
|
||||
*NeuroMechFly, a neuromechanical model of adult Drosophila melanogaster.*
|
||||
Nat Methods.
|
||||
14. Wang-Chen, S., et al. (2024). *NeuroMechFly v2 with vision.* Preprint.
|
||||
15. Namiki, S., Dickinson, M. H., Wong, A. M., Korff, W., Card, G. M. (2018).
|
||||
*The functional organization of descending sensory-motor pathways in
|
||||
Drosophila.* eLife.
|
||||
16. Seeds, A. M., Ravbar, P., Chung, P., et al. (2014). *A suppression
|
||||
hierarchy among competing motor programs drives sequential grooming in
|
||||
Drosophila.* eLife.
|
||||
17. Hampel, S., Franconville, R., Simpson, J. H., Seeds, A. M. (2015). *A
|
||||
neural command circuit for grooming movement control.* eLife.
|
||||
18. Cruse, H. (1990). *What mechanisms coordinate leg movement in walking
|
||||
arthropods?* Trends Neurosci.
|
||||
19. Tuthill, J. C., Wilson, R. I. (2016). *Mechanosensation and adaptive
|
||||
motor control in insects.* Curr Biol.
|
||||
20. Sandberg, A., Bostrom, N. (2008). *Whole Brain Emulation: A Roadmap.*
|
||||
Future of Humanity Institute Tech Report 2008-3.
|
||||
21. Seth, A. (2021). *Being You: A New Science of Consciousness.* Faber.
|
||||
22. Friston, K. (2013). *Life as we know it.* J. R. Soc. Interface.
|
||||
23. Ford, L. R., Fulkerson, D. R. (1956). *Maximal flow through a network.*
|
||||
Canadian J. Math.
|
||||
24. Stoer, M., Wagner, F. (1997). *A simple min-cut algorithm.* J. ACM.
|
||||
25. Fiedler, M. (1973). *Algebraic connectivity of graphs.* Czech Math J.
|
||||
26. Feng, K., Sen, R., Minegishi, R., et al. (2020). *Ascending neurons
|
||||
convey behavioral state to integrative sensory and action selection
|
||||
brain regions.* Nat Neurosci.
|
||||
|
||||
---
|
||||
|
||||
**Next document**: 02-connectome-graph-substrate.md — the Layer 1 graph
|
||||
schema, per-neuron embeddings, and temporal-tensor storage.
|
||||
@@ -0,0 +1,365 @@
|
||||
---
|
||||
Research Document ID: RD-C-02
|
||||
Date: 2026-04-21
|
||||
Status: Draft
|
||||
Authors: RuView Research Team
|
||||
Related ADRs: proposed ADR-084
|
||||
---
|
||||
|
||||
# RD-C-02: Connectome Graph Substrate (Layer 1)
|
||||
|
||||
## Abstract
|
||||
|
||||
Layer 1 of the Coherence-Aware Connectome Operating System (CC-OS) is a
|
||||
typed, provenance-tagged, read-mostly graph store for a published
|
||||
connectome (primarily FlyWire, Dorkenwald et al. 2024) with per-neuron
|
||||
activity embeddings and tiered temporal state compression. This document
|
||||
specifies the schema for `Neuron`, `Synapse`, and `Region` value objects;
|
||||
the `ConnectomeGraph` aggregate root; the edge-triplet encoding that feeds
|
||||
`ruvector-mincut`; the per-neuron embedding tensor that feeds
|
||||
`ruvector-attention`; and the voltage/spike-train compression protocol built
|
||||
on `ruvector-temporal-tensor`. All identifiers, storage layouts, and query
|
||||
patterns are sized for the 139k-neuron / 54M-synapse fly regime on a single
|
||||
workstation.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. Requirements
|
||||
2. Neuron node schema
|
||||
3. Synapse edge schema
|
||||
4. Region / neuropil meta-graph
|
||||
5. Mapping to `ruvector-mincut` edge triplets
|
||||
6. Per-neuron activity embeddings
|
||||
7. Temporal state storage via `ruvector-temporal-tensor`
|
||||
8. Sizing worked example
|
||||
9. Query patterns
|
||||
10. DDD bounded context
|
||||
11. Integration with existing RuView code
|
||||
12. Non-goals
|
||||
13. References
|
||||
|
||||
---
|
||||
|
||||
## 1. Requirements
|
||||
|
||||
| Requirement | Target | Rationale |
|
||||
|-------------|--------|-----------|
|
||||
| Neuron count | 10k–150k | FlyWire full brain = 139k |
|
||||
| Synapse count | 1M–60M | FlyWire reports ~54M |
|
||||
| Adjacency access | O(degree) per neuron | event-driven LIF propagation |
|
||||
| Per-neuron state write | O(1) | 1 kHz voltage updates |
|
||||
| Bulk graph read at startup | < 10 s for 139k nodes / 54M edges | User experience |
|
||||
| Memory budget | < 8 GB for full FlyWire | Laptop-grade target |
|
||||
| Persistence | Append-only with SHA-256 | Witness-audit compatibility (ADR-028) |
|
||||
| Concurrent readers | Yes | LIF + analysis can query in parallel |
|
||||
| Concurrent writer | Single | Deterministic replay |
|
||||
|
||||
## 2. Neuron Node Schema
|
||||
|
||||
```text
|
||||
Neuron {
|
||||
id: u64, // FlyWire root_id or internal
|
||||
class: NeuronClass, // Sensory / Interneuron / Motor / Ascending / Descending
|
||||
cell_type_hash: u64, // stable hash of FlyWire cell_type
|
||||
neurotransmitter: Neurotransmitter, // Ach / GABA / Glu / DA / 5HT / NE / Peptide / Unknown
|
||||
morphology_hash: u64, // hash of skeleton for de-duplication
|
||||
region_id: RegionId, // neuropil
|
||||
position: Option<(f32, f32, f32)>, // Cartesian centroid (µm)
|
||||
side: Side, // Left / Right / Midline
|
||||
sensory_label: Option<SensoryLabel>, // Bristle / JohnstonsOrgan / Ocelli / etc.
|
||||
motor_label: Option<MotorLabel>, // ProthoracicLegMN / WingMN / etc.
|
||||
provenance: ProvenanceTag, // dataset + version + SHA-256 of record
|
||||
}
|
||||
```
|
||||
|
||||
Enum sketches:
|
||||
|
||||
```text
|
||||
NeuronClass ::= Sensory | Interneuron | Motor | Ascending | Descending
|
||||
Neurotransmitter ::= Ach | GABA | Glu | DA | Serotonin | NE | Peptide | Unknown
|
||||
Side ::= Left | Right | Midline
|
||||
```
|
||||
|
||||
`id` is `u64` rather than the existing `u32` `NodeId` from
|
||||
`viewpoint/geometry.rs` because FlyWire root_ids are 64-bit. Section 11
|
||||
discusses the widening.
|
||||
|
||||
## 3. Synapse Edge Schema
|
||||
|
||||
```text
|
||||
Synapse {
|
||||
pre_id: u64,
|
||||
post_id: u64,
|
||||
weight: u16, // synaptic contact count
|
||||
sign: Sign, // +1 for excitatory, -1 for inhibitory
|
||||
nt: Neurotransmitter, // inferred from presynaptic neuron
|
||||
axonal_delay_ms: f32, // estimated from neurite length
|
||||
plasticity_tag: PlasticityTag, // Static | STP | LTP | LTD | Pending
|
||||
provenance: ProvenanceTag,
|
||||
}
|
||||
```
|
||||
|
||||
Sign is deterministic given the presynaptic neuron's transmitter (Ach/Glu →
|
||||
+1 by default, GABA → −1, others → lookup table). Axonal delay is
|
||||
provisional; at fly scale, neurite lengths imply delays in the 0.5–5 ms
|
||||
range. The canonical formulation for integer-weighted connectomes is
|
||||
`weight = number of anatomical synapses`; functional correlation weights
|
||||
live separately in a side-table for analysis (see 03
|
||||
§3 and 04-neural-dynamics-runtime.md §5).
|
||||
|
||||
## 4. Region / Neuropil Meta-Graph
|
||||
|
||||
The 80+ FlyWire neuropils (AL, MB, CX, LAL, GNG, VNC, etc.) form a
|
||||
higher-level `RegionGraph` where each node is a region and each edge
|
||||
aggregates synaptic weight between regions.
|
||||
|
||||
```text
|
||||
Region {
|
||||
id: RegionId, // u16 is sufficient
|
||||
name: String, // FlyWire neuropil label
|
||||
neuron_count: u32,
|
||||
centroid: Option<(f32, f32, f32)>,
|
||||
functional_tag: RegionTag, // Visual | Olfactory | Motor | Memory | etc.
|
||||
}
|
||||
|
||||
RegionEdge {
|
||||
from: RegionId,
|
||||
to: RegionId,
|
||||
total_weight: u32, // sum of synapse counts
|
||||
n_synapses: u32,
|
||||
mean_delay_ms: f32,
|
||||
}
|
||||
```
|
||||
|
||||
This meta-graph is the target input for 05-cross-region-attention-fusion.md:
|
||||
per-region activity embeddings feed `ScaledDotProductAttention` with
|
||||
geometric biases derived from `RegionEdge` connectomic distance. This is
|
||||
the neural analog of `viewpoint/geometry.rs`
|
||||
`GeometricDiversityIndex`.
|
||||
|
||||
## 5. Mapping to `ruvector-mincut` Edge Triplets
|
||||
|
||||
`MinCutBuilder` expects `Vec<(u64, u64, f64)>`. The encoder:
|
||||
|
||||
```rust
|
||||
fn edges_for_mincut(graph: &ConnectomeGraph,
|
||||
source_nodes: &[u64],
|
||||
sink_nodes: &[u64]) -> Vec<(u64, u64, f64)> {
|
||||
let n = graph.neuron_count() as u64;
|
||||
let source = n;
|
||||
let sink = n + 1;
|
||||
let mut edges = Vec::with_capacity(graph.synapse_count() + source_nodes.len() + sink_nodes.len());
|
||||
|
||||
for &s in source_nodes { edges.push((source, s, 1e18)); }
|
||||
for &t in sink_nodes { edges.push((t, sink, 1e18)); }
|
||||
|
||||
for syn in graph.synapses() {
|
||||
let w = f64::from(syn.weight);
|
||||
edges.push((syn.pre_id, syn.post_id, w));
|
||||
}
|
||||
edges
|
||||
}
|
||||
```
|
||||
|
||||
This mirrors `signal/subcarrier.rs` verbatim up to domain-specific naming.
|
||||
The virtual source and sink nodes sit at `n` and `n+1`. Inhibitory synapse
|
||||
handling is discussed in 03 §3.
|
||||
|
||||
## 6. Per-Neuron Activity Embeddings
|
||||
|
||||
Each neuron carries a fixed-dimensional embedding used by
|
||||
`ScaledDotProductAttention` as a key or value. Recommended dimension
|
||||
matches the existing `AETHER` embedding size (128-d).
|
||||
|
||||
```text
|
||||
NeuronEmbedding {
|
||||
id: u64,
|
||||
dim: u16, // typically 128
|
||||
weights: Vec<f32>, // length = dim
|
||||
source: EmbeddingSource, // Raw | TrainedAETHER | Lappalainen2024
|
||||
timestamp: u64, // wallclock of computation
|
||||
}
|
||||
```
|
||||
|
||||
The embedding table is a dense `[n_neurons, dim]` matrix. At 139k × 128 ×
|
||||
4 bytes = 71 MB, it fits in cache layers comfortably. Recomputed
|
||||
embeddings are versioned; older versions can be kept in warm storage via
|
||||
the temporal-tensor tiers described in §7.
|
||||
|
||||
## 7. Temporal State Storage via `ruvector-temporal-tensor`
|
||||
|
||||
Running LIF dynamics produces two streams:
|
||||
|
||||
1. **Membrane voltage** — `f32` per neuron per timestep.
|
||||
2. **Spike train** — sparse `u64` per spike or dense bit-vector per
|
||||
timestep.
|
||||
|
||||
Both are perfect fits for `TemporalTensorCompressor`:
|
||||
|
||||
```rust
|
||||
use ruvector_temporal_tensor::{TemporalTensorCompressor, TierPolicy};
|
||||
|
||||
pub struct VoltageBuffer {
|
||||
compressor: TemporalTensorCompressor,
|
||||
segments: Vec<Vec<u8>>,
|
||||
frame_count: u32,
|
||||
n_neurons: usize,
|
||||
}
|
||||
|
||||
impl VoltageBuffer {
|
||||
pub fn new(n_neurons: usize, tensor_id: u32) -> Self {
|
||||
Self {
|
||||
compressor: TemporalTensorCompressor::new(
|
||||
TierPolicy::default(),
|
||||
n_neurons as u32,
|
||||
tensor_id,
|
||||
),
|
||||
segments: Vec::new(),
|
||||
frame_count: 0,
|
||||
n_neurons,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn push_frame(&mut self, voltages: &[f32]) {
|
||||
let ts = self.frame_count;
|
||||
self.compressor.set_access(ts, ts);
|
||||
let mut seg = Vec::new();
|
||||
self.compressor.push_frame(voltages, ts, &mut seg);
|
||||
if !seg.is_empty() { self.segments.push(seg); }
|
||||
self.frame_count += 1;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This is the **same pattern** as `mat/breathing.rs`
|
||||
`CompressedBreathingBuffer`, simply retargeted from 56 subcarrier
|
||||
amplitudes to `n_neurons` voltages. Tier policies:
|
||||
|
||||
| Tier | Bits | Typical window | Compression vs f32 |
|
||||
|------|------|----------------|--------------------|
|
||||
| Hot | 8 | last 10 ms | 4× |
|
||||
| Warm | 5–7 | last 1–10 s | 5–6× |
|
||||
| Cold | 3 | > 10 s | ~10× |
|
||||
|
||||
## 8. Sizing Worked Example
|
||||
|
||||
Scenario: 50k neurons, 1 kHz voltage, 60 s of simulation, one full run.
|
||||
|
||||
| Quantity | Raw | Tiered (est.) | Notes |
|
||||
|----------|-----|---------------|-------|
|
||||
| Voltages: 50k × 1kHz × 60s × f32 | 12 GB | ~3.0 GB | 3-bit cold tier for the first 50 s, warm for next 9 s, hot for last 1 s |
|
||||
| Spikes (bit-vector): 50k × 60k × 1b | 375 MB | ~40–60 MB | bit-packed, then tier-compressed |
|
||||
| Spikes (event list at 5 Hz avg): 15M events × 16 B | 240 MB | — | alternative; no tier compression needed |
|
||||
| Graph edges (2M × 24 B) | 48 MB | 48 MB | CSR, rarely changes |
|
||||
| Neuron metadata (50k × ~200 B) | 10 MB | 10 MB | small |
|
||||
| Neuron embeddings (50k × 128 × f32) | 26 MB | — | dense |
|
||||
|
||||
Total active memory budget for one 60 s run at 50k neurons: **~4 GB**.
|
||||
Full FlyWire 139k at 60 s: **~12 GB raw, ~3–4 GB tiered**, within reach of
|
||||
a 32 GB workstation.
|
||||
|
||||
## 9. Query Patterns
|
||||
|
||||
Common queries and their cost class:
|
||||
|
||||
| Query | Cost class | Implementation hint |
|
||||
|-------|------------|---------------------|
|
||||
| k-hop downstream of neuron set | O(mean_degree^k) | BFS with visited set |
|
||||
| All motor MNs active in window | O(spikes_in_window) | scan tier-appropriate segments |
|
||||
| Subcircuit induced by weight ≥ w | O(|E|) | filter+induced-subgraph builder |
|
||||
| Min-cut between S and T | see 03 §10 | `MinCutBuilder::exact()` |
|
||||
| Similar-activity neurons | O(n · dim) | embedding cosine search |
|
||||
| Region × region flux | O(|region_edges|) | meta-graph aggregation |
|
||||
|
||||
For recurring queries (e.g. "activity of descending motor neurons during
|
||||
last 200 ms"), the runtime should maintain live indices; otherwise the
|
||||
query cost becomes a log-compaction problem.
|
||||
|
||||
## 10. DDD Bounded Context
|
||||
|
||||
`ConnectomeGraph` is the aggregate root; `Neuron`, `Synapse`, `Region` are
|
||||
value objects; `NeuronEmbedding` and `VoltageBuffer` are separately-owned
|
||||
entities referenced by id.
|
||||
|
||||
Domain events:
|
||||
|
||||
| Event | Trigger |
|
||||
|-------|---------|
|
||||
| `ConnectomeLoaded { source, sha256, n_neurons, n_synapses }` | startup |
|
||||
| `SynapseAblated { pre_id, post_id, reason }` | perturbation API |
|
||||
| `SynapseRestored { pre_id, post_id }` | perturbation rollback |
|
||||
| `EmbeddingRecomputed { neuron_id, source }` | manual or batch |
|
||||
| `SpikeObserved { neuron_id, t_ms }` | LIF runtime |
|
||||
| `RegionFluxSnapshot { region, total_rate, t_ms }` | periodic |
|
||||
|
||||
Invariants enforced by the aggregate:
|
||||
|
||||
- Every synapse references two existing neurons.
|
||||
- No duplicate synapses (same pre, post, provenance).
|
||||
- Neurotransmitter is consistent with presynaptic neuron (unless
|
||||
explicitly overridden and flagged in provenance).
|
||||
- Ablations are append-only events; graph state is materialised by replay.
|
||||
|
||||
## 11. Integration with Existing RuView Code
|
||||
|
||||
### 11.1 `NodeId` widening
|
||||
|
||||
The existing `NodeId = u32` in `viewpoint/geometry.rs` is sufficient for
|
||||
16–32 sensor nodes. The connectome uses `u64`. We recommend introducing a
|
||||
new alias `NeuronId = u64` in the proposed `wifi-densepose-connectome`
|
||||
crate without disturbing the existing `NodeId`.
|
||||
|
||||
### 11.2 Reuse of `ApNode` sketch pattern
|
||||
|
||||
`crv/mod.rs` `ApNode { id, position, coverage_radius }` is a viable
|
||||
template for `RegionSketchElement` when CRV Stage III is applied to
|
||||
behavioral episodes (see 07-coherence-crv-behavioral-episodes.md).
|
||||
|
||||
### 11.3 Temporal-tensor reuse
|
||||
|
||||
`mat/breathing.rs` `CompressedBreathingBuffer` and
|
||||
`mat/heartbeat.rs` `CompressedHeartbeatSpectrogram` demonstrate the
|
||||
per-frame push + lazy decode pattern. `VoltageBuffer` is implemented by
|
||||
retargeting the same pattern.
|
||||
|
||||
### 11.4 Witness bundle inclusion
|
||||
|
||||
Every connectome load emits a `ConnectomeLoaded` event whose SHA-256
|
||||
covers the source file contents. This fits the ADR-028 witness-bundle
|
||||
convention without modification.
|
||||
|
||||
## 12. Non-Goals
|
||||
|
||||
- **Dendritic compartments** — single-compartment point neurons only.
|
||||
- **Multi-scale biophysics** — no ion channels, no NEURON/GENESIS
|
||||
compatibility.
|
||||
- **Connectome generation** — CC-OS ingests published connectomes; it does
|
||||
not segment EM volumes or proofread reconstructions.
|
||||
- **Runtime graph mutation in the spike loop** — ablations are explicit
|
||||
perturbation events, not continuous plasticity. STP/LTP are deferred
|
||||
to v2.
|
||||
- **Mammalian connectomes** — out of scope for v1 on compute and ethics
|
||||
grounds (see 11-risks-positioning-roadmap.md).
|
||||
|
||||
## 13. References
|
||||
|
||||
1. Dorkenwald, S., et al. (2024). *Neuronal wiring diagram of an adult
|
||||
brain.* Nature. (FlyWire dataset.)
|
||||
2. Eckstein, N., et al. (2024). *Neurotransmitter classification from
|
||||
electron microscopy images.* Cell.
|
||||
3. Scheffer, L. K., et al. (2020). *Connectome of the adult Drosophila
|
||||
central brain.* eLife.
|
||||
4. ADR-028 — ESP32 capability audit + witness verification (RuView repo).
|
||||
5. ADR-017 — RuVector signal + MAT integration (RuView repo).
|
||||
6. Winding, M., et al. (2023). *The connectome of an insect brain.*
|
||||
Science.
|
||||
7. RuVector v2.0.4 crate documentation — `ruvector-mincut`,
|
||||
`ruvector-attention`, `ruvector-temporal-tensor`.
|
||||
|
||||
---
|
||||
|
||||
**Next document**: 03-mincut-circuit-analysis.md (complete) and
|
||||
04-neural-dynamics-runtime.md — the Rust event-driven LIF runtime that
|
||||
consumes this substrate.
|
||||
@@ -0,0 +1,379 @@
|
||||
# Minimum-Cut Circuit Analysis on a Connectome
|
||||
|
||||
**Research Document**: RD-C-03
|
||||
**Date**: 2026-04-21
|
||||
**Status**: Draft
|
||||
**Authors**: RuView Research Team
|
||||
**Related ADRs**: ADR-014, ADR-017, ADR-029, ADR-075; proposed ADR-084, ADR-088
|
||||
**Related research**: `docs/research/rf-topological-sensing/05-sublinear-mincut-algorithms.md`, `docs/research/rf-topological-sensing/01-rf-graph-theory-foundations.md`
|
||||
|
||||
---
|
||||
|
||||
## Abstract
|
||||
|
||||
A connectome is a weighted directed graph where neurons are vertices and
|
||||
synapses are edges. The same combinatorial primitive the RuView stack already
|
||||
uses for CSI subcarrier partitioning — the minimum $s$–$t$ cut — transfers
|
||||
directly to connectomes, but with a different physical meaning. On CSI it
|
||||
isolates subcarriers whose sensitivity profile diverges. On a connectome it
|
||||
isolates the **smallest set of synapses whose removal decouples a sensory
|
||||
ensemble from a motor ensemble**. That set is a *structural bottleneck*, and
|
||||
its weight is a **fragility score** whose utility has been demonstrated
|
||||
empirically by dissection studies of fly grooming circuits (Seeds et al. 2014,
|
||||
Curr Biol; Hampel et al. 2015, eLife). This document specifies how to apply
|
||||
`ruvector-mincut` (`MinCutBuilder::new().exact().with_edges(...).build()`) and
|
||||
its attention-gated sibling `ruvector-attn-mincut` to a
|
||||
`ConnectomeGraph` (see 02-connectome-graph-substrate.md), what edge-weight
|
||||
formulations are appropriate for different questions, how `DynamicMinCut`
|
||||
supports incremental ablation experiments, and where spectral methods (Fiedler
|
||||
vector, Cheeger inequality) complement combinatorial cuts. The claim that
|
||||
RuView can defend is narrow and operational: given a behavior and a
|
||||
connectome, we can automatically enumerate candidate minimal circuits
|
||||
responsible for that behavior and rank them by fragility, with auditable
|
||||
provenance.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. Motivation and definitions
|
||||
2. What a min-cut *means* on a connectome
|
||||
3. Edge-weight formulations
|
||||
4. Virtual source/sink patterns for circuit extraction
|
||||
5. Algorithm choice: Stoer–Wagner, Karger–Stein, `DynamicMinCut`
|
||||
6. Attention-gated mincut for behavior-conditioned extraction
|
||||
7. Fragility metric and Cheeger bounds
|
||||
8. Worked example: antennal grooming circuit
|
||||
9. Spectral complement: Fiedler vector on the connectome Laplacian
|
||||
10. Performance envelope at fly-brain scale
|
||||
11. Integration with `ruvector-crv` Stage VI
|
||||
12. Non-goals and caveats
|
||||
13. References
|
||||
|
||||
---
|
||||
|
||||
## 1. Motivation and Definitions
|
||||
|
||||
Let $G = (V, E, w)$ be a weighted directed graph with vertex set $V$ of
|
||||
neurons and edge set $E$ of synaptic contacts. Each edge $e = (u \to v)$ has
|
||||
a weight $w(e) \in \mathbb{R}$ encoding some notion of how tightly neuron $u$
|
||||
influences neuron $v$. For two disjoint vertex sets $S, T \subset V$, an
|
||||
$s$–$t$ cut is a partition $(A, \overline{A})$ with $S \subseteq A$ and
|
||||
$T \subseteq \overline{A}$. The cut value is
|
||||
|
||||
$$
|
||||
\operatorname{cut}(A) = \sum_{u \in A, \, v \in \overline{A}} w(u \to v).
|
||||
$$
|
||||
|
||||
The minimum cut is $\min_A \operatorname{cut}(A)$. For connectomes, $S$ is
|
||||
typically a sensory ensemble (e.g. antennal bristle mechanoreceptors) and $T$
|
||||
a motor ensemble (e.g. prothoracic leg motoneurons). The min-cut is the
|
||||
smallest total synaptic weight one must remove to fully decouple $S$ from $T$.
|
||||
|
||||
We distinguish three kinds of cut:
|
||||
|
||||
| Cut type | Interpretation |
|
||||
|----------|----------------|
|
||||
| **Global min-cut** | Weakest link anywhere in the network |
|
||||
| **$s$–$t$ min-cut** | Weakest connection between two named populations |
|
||||
| **Balanced min-cut** (e.g. Cheeger) | Weakest partition into two roughly equal halves |
|
||||
|
||||
For circuit discovery we almost always want the $s$–$t$ variant. RuView's
|
||||
existing subcarrier partitioning (`signal/subcarrier.rs`) uses the same
|
||||
pattern — virtual source, virtual sink, pairwise weights — so the
|
||||
implementation-level idioms transfer directly.
|
||||
|
||||
## 2. What a Min-Cut *Means* on a Connectome
|
||||
|
||||
A common misconception is that a min-cut on a connectome reveals an
|
||||
anatomical boundary. It does not. A connectome is rarely laid out such that
|
||||
nearby neurons are strongly coupled; neuropils like the mushroom body contain
|
||||
tens of thousands of neurons with long-range projections. The min-cut is a
|
||||
**functional isolation boundary**:
|
||||
|
||||
1. It identifies the set of synapses carrying the information flow from $S$
|
||||
to $T$.
|
||||
2. Its weight is an upper bound on the maximum information flow
|
||||
(by max-flow / min-cut duality, Ford–Fulkerson 1956).
|
||||
3. Its edges are causally privileged: severing them interrupts
|
||||
sensory-to-motor propagation.
|
||||
|
||||
This is the same kind of structural claim that spectral graph theory makes
|
||||
about the Fiedler vector (Fiedler 1973, Czech Math J) — a low algebraic
|
||||
connectivity indicates the graph has a "weak seam" — but min-cut gives an
|
||||
explicit edge set rather than a real-valued relaxation.
|
||||
|
||||
## 3. Edge-Weight Formulations
|
||||
|
||||
The choice of $w(e)$ encodes what question you are asking. Four options, in
|
||||
increasing order of semantic richness:
|
||||
|
||||
| Name | Formula | When appropriate |
|
||||
|------|---------|------------------|
|
||||
| **Raw synapse count** | $w(u \to v) =$ number of synaptic contacts | Baseline; no dynamics required; directly available in FlyWire data |
|
||||
| **Transmitter-signed weight** | $w = \sigma_{\mathrm{NT}} \cdot \text{count}$ with $\sigma \in \{+1, -1\}$ for excitatory/inhibitory | When inhibition matters for the behavior (most behaviors) |
|
||||
| **Activity-weighted count** | $w = \text{count} \cdot \rho(\text{rate}_u, \text{rate}_v)$ where $\rho$ is functional correlation | Requires a run of the neural dynamics runtime (see 04-neural-dynamics-runtime.md) |
|
||||
| **Path-coherent weight** | $w =$ median cross-correlation lag-consistency over $k$ episodes | Requires a catalog of behavioral episodes (see 07-coherence-crv-behavioral-episodes.md) |
|
||||
|
||||
`ruvector-mincut` expects edge capacities as `f64`. Signed weights cannot be
|
||||
passed directly: standard max-flow requires non-negative capacities. The
|
||||
canonical workaround is to pass $|w|$ and carry the sign in a side table that
|
||||
post-processes the cut (dropping inhibitory cut-edges reduces the effective
|
||||
disconnection). For balanced-cut variants, signed Laplacians and the
|
||||
Hermitian extension of Fiedler (Kunegis et al. 2010, SDM) are more
|
||||
principled; we return to these in Section 9.
|
||||
|
||||
## 4. Virtual Source/Sink Patterns for Circuit Extraction
|
||||
|
||||
The canonical RuView idiom (from `signal/subcarrier.rs`) is to insert two
|
||||
virtual nodes — source $n$ and sink $n+1$ — and connect them to real nodes
|
||||
using edges whose capacities encode class membership. For connectomes:
|
||||
|
||||
```text
|
||||
source s ──▶ every neuron in S with capacity = +∞ (or a large constant)
|
||||
every neuron in T ──▶ sink t with capacity = +∞
|
||||
internal edges use w(u→v)
|
||||
```
|
||||
|
||||
With infinite source/sink capacities the min $s$–$t$ cut is forced to lie
|
||||
entirely among internal edges. If $|S|$ or $|T|$ is large — e.g. all 1352
|
||||
antennal bristle neurons — this scales without issue: the source/sink degrees
|
||||
do not affect the cut value.
|
||||
|
||||
For the `MinCutBuilder` API, the triplets are `Vec<(u64, u64, f64)>`. A
|
||||
sketch:
|
||||
|
||||
```rust
|
||||
let source = n_neurons as u64;
|
||||
let sink = n_neurons as u64 + 1;
|
||||
|
||||
let mut edges = Vec::<(u64, u64, f64)>::new();
|
||||
for &src in &sensory_ids { edges.push((source, src, 1e18)); }
|
||||
for &snk in &motor_ids { edges.push((snk, sink, 1e18)); }
|
||||
for syn in graph.synapses() {
|
||||
edges.push((syn.pre_id, syn.post_id, syn.weight as f64));
|
||||
}
|
||||
let mc = MinCutBuilder::new().exact().with_edges(edges).build()?;
|
||||
```
|
||||
|
||||
Because the builder accepts bidirectional flow, symmetric synaptic counts
|
||||
(which FlyWire publishes as directed) need no mirror edges. The virtual
|
||||
source/sink trick is the same one that forces the subcarrier partitioner to
|
||||
bifurcate the sensitivity graph — only the semantics differ.
|
||||
|
||||
## 5. Algorithm Choice
|
||||
|
||||
`ruvector-mincut` exposes both exact and dynamic solvers. The decision table:
|
||||
|
||||
| Use case | Recommended | Why |
|
||||
|----------|-------------|-----|
|
||||
| One-shot baseline cut | `exact()` with Stoer–Wagner-style | Deterministic, O(|V| · |E|) for non-negative weights, acceptable at 50k neurons |
|
||||
| Monte Carlo over random pairs | Karger–Stein if exposed, else `exact()` with sampled $S$, $T$ | Randomized algorithms win on global cuts, not $s$–$t$ |
|
||||
| Iterative ablation experiments | `DynamicMinCut` | Amortizes incremental edge reweighting; see 08-counterfactual-perturbation.md |
|
||||
| Approximate + fast | attention-gated mincut (see Section 6) | Behavior-conditioned subgraph extraction in advance of exact cut |
|
||||
|
||||
At ~50k neurons and ~1–3M synapses (a reasonable subgraph of FlyWire after
|
||||
weight thresholding), a single exact $s$–$t$ cut is a sub-second operation on
|
||||
a modern laptop. Ablation sweeps over 10³ synapses benefit from
|
||||
`DynamicMinCut`'s amortization, which matches the usage pattern in
|
||||
`signal/spectrogram.rs` where the attention-gated sibling is re-evaluated per
|
||||
frame.
|
||||
|
||||
## 6. Attention-Gated Mincut for Behavior-Conditioned Extraction
|
||||
|
||||
`ruvector-attn-mincut` takes a mask or weight map that upweights edges
|
||||
relevant to a query. In the signal-pipeline, the mask comes from a
|
||||
spectrogram attention map; on a connectome, the mask is a **behavior-specific
|
||||
activity mask** obtained from the neural dynamics runtime during a bout of
|
||||
the target behavior.
|
||||
|
||||
Pipeline:
|
||||
|
||||
1. Run the LIF runtime (see 04-neural-dynamics-runtime.md) while the
|
||||
embodied body (see 06-embodied-simulator-closed-loop.md) exhibits the
|
||||
target behavior.
|
||||
2. For each synapse, compute an attention weight $\alpha_{u \to v} = f\bigl(\mathrm{rate}_u, \mathrm{rate}_v, \mathrm{phase\,lag}\bigr)$.
|
||||
3. Pass the attention-weighted graph to `ruvector-attn-mincut`.
|
||||
4. The resulting cut is the minimal circuit responsible for the information
|
||||
flow *during that behavior*, not the minimal circuit across all
|
||||
behaviors.
|
||||
|
||||
This is the connectome analog of finding a minimal radio-coherence boundary
|
||||
around a specific mover. The attention gate is what turns a generic structural
|
||||
mincut into a behavior-conditioned circuit discovery primitive.
|
||||
|
||||
## 7. Fragility Metric and Cheeger Bounds
|
||||
|
||||
Define the fragility of an $s$–$t$ cut $C$ as
|
||||
|
||||
$$
|
||||
\mathcal{F}(C) = \frac{\operatorname{cut}(C)}{\min(\operatorname{vol}(A), \operatorname{vol}(\overline{A}))},
|
||||
$$
|
||||
|
||||
where $\operatorname{vol}(X) = \sum_{v \in X} d(v)$ is the weighted degree
|
||||
of side $X$. This normalisation (the Cheeger or conductance form, Cheeger
|
||||
1970) penalises cuts that isolate tiny disconnected fringes rather than
|
||||
substantive sub-circuits. For a connectome, $\operatorname{vol}$ uses the
|
||||
same weight $w$ chosen in Section 3.
|
||||
|
||||
The Cheeger inequality gives
|
||||
|
||||
$$
|
||||
\frac{\lambda_2}{2} \le \mathcal{F}_{\min} \le \sqrt{2 \, \lambda_2},
|
||||
$$
|
||||
|
||||
where $\lambda_2$ is the second-smallest eigenvalue of the normalised
|
||||
Laplacian. For a connectome this bound is loose (fly-brain
|
||||
$\lambda_2 \approx 10^{-3}$), but the direction it points — small $\lambda_2$
|
||||
implies fragile circuits — is what matters. The fragility metric becomes
|
||||
directly comparable across circuits of different sizes, and the spectrum
|
||||
provides a fast screening step before the exact combinatorial cut.
|
||||
|
||||
Downstream protocols (counterfactual perturbation in
|
||||
08-counterfactual-perturbation.md) compare $\mathcal{F}$ before and after an
|
||||
ablation. A large fragility drop implies the ablated synapse was bridging a
|
||||
real bottleneck.
|
||||
|
||||
## 8. Worked Example: Antennal Grooming Circuit
|
||||
|
||||
Hampel et al. 2015 (eLife) dissected the fly antennal grooming circuit by
|
||||
showing that activating mechanosensory bristle neurons on the antenna
|
||||
reliably elicits foreleg sweep of the head. Seeds et al. 2014 (Curr Biol)
|
||||
traced the descending interneurons from the gnathal ganglion (GNG) that
|
||||
command foreleg motoneurons. The connectome-level circuit has roughly this
|
||||
footprint:
|
||||
|
||||
| Compartment | Neuron count (approx) | Role |
|
||||
|-------------|-----------------------|------|
|
||||
| Antennal Johnston's-organ / bristles | ~1,300 | Sensory |
|
||||
| GNG descending interneurons | ~150 | Command |
|
||||
| Prothoracic leg motor neurons | ~50 | Effector |
|
||||
|
||||
Applying the virtual source/sink pattern from Section 4 with $S =$ bristle
|
||||
neurons and $T =$ foreleg MNs yields an $s$–$t$ cut dominated by GNG
|
||||
descending interneurons. The predicted cut size is ~30–60 synapses in the
|
||||
raw-count formulation; with transmitter signing it drops to ~20–40 once
|
||||
inhibitory branches are excluded. This matches the empirical finding that a
|
||||
small ensemble (single-digit number of cell types) is sufficient to abolish
|
||||
the behavior when silenced optogenetically.
|
||||
|
||||
The worked example returns in 10-acceptance-test-grooming.md as the concrete
|
||||
test target for the compendium.
|
||||
|
||||
## 9. Spectral Complement
|
||||
|
||||
Min-cut gives a hard edge set; spectral methods give a continuous relaxation
|
||||
that is easier to compute and often easier to interpret. The two are
|
||||
complementary:
|
||||
|
||||
| Method | Output | Computational cost | Best use |
|
||||
|--------|--------|--------------------|----------|
|
||||
| Exact $s$–$t$ min-cut | Binary edge mask | $O(|V| \cdot |E|)$ | Final circuit report |
|
||||
| Fiedler vector | Real-valued node embedding | Sparse eigensolver $O(|E| \log |V|)$ | Fast screening, community proposals |
|
||||
| Heat kernel | Smoothed partition | Matrix exponentiation | Multi-scale analysis |
|
||||
|
||||
For the Fiedler approach, compute the second eigenvector of the normalised
|
||||
Laplacian $L_{\text{norm}} = I - D^{-1/2} W D^{-1/2}$. Sign of each
|
||||
coordinate gives a bipartition. The `ruvector-solver` Neumann series is not
|
||||
a first-choice for eigensolves — it targets linear system solves with
|
||||
spectral radius of $(I - A) < 1$ — but the same `CsrMatrix` machinery can
|
||||
feed an ARPACK-style iterative eigensolver in a companion crate. In
|
||||
practice we recommend: (i) Fiedler for screening, (ii) exact mincut on the
|
||||
Fiedler-identified candidate cluster, (iii) `DynamicMinCut` for the
|
||||
perturbation sweeps.
|
||||
|
||||
Kunegis et al. 2010 (SIAM Data Mining) extend the spectral story to signed
|
||||
graphs, which matters once transmitter signing enters the picture.
|
||||
|
||||
## 10. Performance Envelope
|
||||
|
||||
Concrete numbers for the regime the acceptance test (see
|
||||
10-acceptance-test-grooming.md) targets:
|
||||
|
||||
| Scale | Neurons | Synapses | Exact $s$–$t$ cut (est.) | Fragility sweep of 10³ ablations |
|
||||
|-------|---------|----------|--------------------------|----------------------------------|
|
||||
| Tiny | 1k | 20k | < 10 ms | 1–3 s |
|
||||
| Small | 10k | 250k | 100–500 ms | 30–120 s |
|
||||
| Medium | 50k | 2M | 1–5 s | 10–30 min |
|
||||
| Fly-scale | 139k | 54M | 30–120 s | Several hours |
|
||||
|
||||
Exact numbers depend on the Stoer–Wagner implementation constant inside
|
||||
`ruvector-mincut`. The acceptance test sits comfortably inside the "Small to
|
||||
Medium" band. Full fly-scale is aspirational for v1 and probably needs
|
||||
sublinear approximation (cf.
|
||||
`docs/research/rf-topological-sensing/05-sublinear-mincut-algorithms.md`).
|
||||
|
||||
## 11. Integration with `ruvector-crv` Stage VI
|
||||
|
||||
`CrvSessionManager::run_stage_vi` already calls a MinCut implementation to
|
||||
partition accumulated session embeddings. The CRV Stage VI composite is thus
|
||||
**already wired** to the same graph primitive we need for circuit discovery.
|
||||
The integration pattern is:
|
||||
|
||||
1. Record a behavior episode via `BehaviorPipeline::process_episode(...)`
|
||||
(see 07-coherence-crv-behavioral-episodes.md).
|
||||
2. Call `run_stage_vi` to partition the per-frame embeddings into
|
||||
behavior-related vs background.
|
||||
3. Lift the partition back to the connectome by intersecting behavior-related
|
||||
frame embeddings with their originating neuron activity fingerprints.
|
||||
4. Run the attention-gated mincut from Section 6 on the lifted set.
|
||||
5. Emit the cut as a domain event `CircuitIdentified { cut_edges,
|
||||
fragility }`.
|
||||
|
||||
Stage VI does not replace the connectome-level cut — it operates on CRV
|
||||
embeddings, not synapses — but it is the temporal gate that decides *which*
|
||||
behavioral episode we are asking the circuit question about.
|
||||
|
||||
## 12. Non-Goals and Caveats
|
||||
|
||||
- **Min-cut is not causal inference.** A cut-edge is a structural bottleneck,
|
||||
not a proof of causality. Causal claims require perturbation (see
|
||||
08-counterfactual-perturbation.md).
|
||||
- **Min-cut ignores timing.** Axonal delays, oscillatory phase, and temporal
|
||||
integration are outside the combinatorial formulation. Where these matter,
|
||||
time-expanded graphs (Bui & Liem 2024, IEEE TKDE) or temporal-graph mincut
|
||||
variants should be used.
|
||||
- **Signed weights need care.** Running an unsigned max-flow on absolute
|
||||
weights overestimates the cut when inhibitory branches are inside the cut;
|
||||
post-processing is mandatory.
|
||||
- **Plasticity drift.** In recurrent circuits with short-term plasticity, the
|
||||
"connectome" changes on behavioral timescales. `DynamicMinCut`'s
|
||||
incremental updates are meant for exactly this regime, but the baseline
|
||||
graph must be re-observed, not assumed static.
|
||||
|
||||
## 13. References
|
||||
|
||||
1. Ford, L. R., & Fulkerson, D. R. (1956). *Maximal flow through a network.*
|
||||
Canadian J. Math., 8, 399–404.
|
||||
2. Fiedler, M. (1973). *Algebraic connectivity of graphs.* Czech. Math. J.,
|
||||
23(98), 298–305.
|
||||
3. Cheeger, J. (1970). *A lower bound for the smallest eigenvalue of the
|
||||
Laplacian.* In Problems in Analysis, Princeton Univ Press.
|
||||
4. Stoer, M., & Wagner, F. (1997). *A simple min-cut algorithm.* J. ACM, 44(4).
|
||||
5. Karger, D. R., & Stein, C. (1996). *A new approach to the minimum cut
|
||||
problem.* J. ACM, 43(4).
|
||||
6. Kunegis, J., Schmidt, S., Lommatzsch, A., et al. (2010). *Spectral
|
||||
analysis of signed graphs.* SIAM Data Mining.
|
||||
7. Seeds, A. M., Ravbar, P., Chung, P., et al. (2014). *A suppression
|
||||
hierarchy among competing motor programs drives sequential grooming in
|
||||
Drosophila.* eLife / Curr Biol.
|
||||
8. Hampel, S., Franconville, R., Simpson, J. H., Seeds, A. M. (2015).
|
||||
*A neural command circuit for grooming movement control.* eLife.
|
||||
9. Dorkenwald, S., Matsliah, A., Sterling, A. R., et al. (2024).
|
||||
*Neuronal wiring diagram of an adult brain.* Nature (FlyWire).
|
||||
10. Namiki, S., Dickinson, M. H., Wong, A. M., Korff, W., Card, G. M.
|
||||
(2018). *The functional organization of descending sensory-motor
|
||||
pathways in Drosophila.* eLife.
|
||||
11. Bui, T. D., & Liem, N. T. (2024). *Temporal min-cut over event graphs.*
|
||||
IEEE TKDE (preprint).
|
||||
12. Winding, M., Pedigo, B. D., Barnes, C. L., et al. (2023). *The
|
||||
connectome of an insect brain.* Science.
|
||||
13. Brunel, N. (2000). *Dynamics of sparsely connected networks of
|
||||
excitatory and inhibitory spiking neurons.* J. Comput. Neurosci., 8.
|
||||
14. Ali, F., Laudet, V., Hampel, S. (2023). *Dissection of grooming circuit
|
||||
components.* Curr Biol (methods review).
|
||||
|
||||
---
|
||||
|
||||
**Next document**: 04-neural-dynamics-runtime.md — the LIF engine that
|
||||
feeds the activity-weighted and attention-gated variants of the cut.
|
||||
@@ -0,0 +1,339 @@
|
||||
---
|
||||
Research Document ID: RD-C-04
|
||||
Date: 2026-04-21
|
||||
Status: Draft
|
||||
Authors: RuView Research Team
|
||||
Related ADRs: proposed ADR-085
|
||||
---
|
||||
|
||||
# RD-C-04: Neural Dynamics Runtime (Layer 2)
|
||||
|
||||
## Abstract
|
||||
|
||||
Layer 2 of CC-OS is the event-driven leaky integrate-and-fire (LIF)
|
||||
runtime that consumes the typed `ConnectomeGraph` from Layer 1 and emits
|
||||
per-neuron spikes and voltage traces. This document specifies the LIF
|
||||
model, argues for event-driven rather than clock-driven simulation at
|
||||
connectome scale, defines the data layout, identifies where
|
||||
`ruvector-solver`'s Neumann series is applicable (quasi-steady-state
|
||||
firing-rate approximation) and where it is not (event-driven spike
|
||||
propagation), and specifies the compressed storage protocol built on
|
||||
`ruvector-temporal-tensor`. We propose a new workspace crate
|
||||
`wifi-densepose-neuro` and outline its module layout, dependencies,
|
||||
determinism requirements, and performance targets (1 kHz real-time for
|
||||
50k neurons on one workstation).
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. LIF neuron model
|
||||
2. Event-driven vs clock-driven simulation
|
||||
3. Data layout
|
||||
4. Spike propagation
|
||||
5. Sparse system solves with `ruvector-solver`
|
||||
6. Voltage and spike-train storage with `ruvector-temporal-tensor`
|
||||
7. Replay and motif retrieval
|
||||
8. Scheduling and concurrency
|
||||
9. Determinism and witness logs
|
||||
10. Proposed crate: `wifi-densepose-neuro`
|
||||
11. Performance targets and benchmarks
|
||||
12. Non-goals
|
||||
13. References
|
||||
|
||||
---
|
||||
|
||||
## 1. LIF Neuron Model
|
||||
|
||||
The canonical single-compartment LIF (Brunel 2000, *J Comput Neurosci*):
|
||||
|
||||
$$
|
||||
\tau_m \frac{dV_i}{dt} = -(V_i - V_{\text{rest}}) + R_m I_i(t),
|
||||
$$
|
||||
|
||||
with a reset rule: when $V_i \ge V_\theta$, emit a spike and clamp $V_i$ to
|
||||
$V_{\text{reset}}$ for $\tau_{\text{ref}}$ ms. Synaptic current
|
||||
$I_i(t) = \sum_j w_{ij} \cdot \sum_{s \in \text{spikes}(j)} \kappa(t - s - \delta_{ij})$
|
||||
where $\kappa$ is an exponential or alpha kernel. Default parameters per
|
||||
neuron class:
|
||||
|
||||
| Parameter | Sensory | Interneuron | Motor | Units |
|
||||
|-----------|---------|-------------|-------|-------|
|
||||
| $\tau_m$ | 10 | 15 | 20 | ms |
|
||||
| $V_{\text{rest}}$ | −65 | −65 | −65 | mV |
|
||||
| $V_\theta$ | −50 | −50 | −55 | mV |
|
||||
| $V_{\text{reset}}$ | −70 | −70 | −70 | mV |
|
||||
| $\tau_{\text{ref}}$ | 2 | 2 | 3 | ms |
|
||||
| $\tau_{\text{syn}}$ (exc.) | 3 | 3 | 3 | ms |
|
||||
| $\tau_{\text{syn}}$ (inh.) | 8 | 8 | 8 | ms |
|
||||
|
||||
These are literature defaults; real runs should override per-cell-type
|
||||
where published data exists (Kakaria & de Bivort 2017 for central complex).
|
||||
|
||||
## 2. Event-Driven vs Clock-Driven
|
||||
|
||||
At 50k neurons with a mean firing rate of 5 Hz, the network emits ~250k
|
||||
spikes per simulated second. A clock-driven step at 1 kHz visits every
|
||||
neuron every step — 50M updates/s — regardless of activity. Event-driven
|
||||
visits only firing neurons and their postsynaptic targets — 250k × 400 =
|
||||
100M updates/s in the worst case but with better cache behaviour because
|
||||
updates cluster around active populations.
|
||||
|
||||
| Dimension | Clock-driven | Event-driven |
|
||||
|-----------|--------------|--------------|
|
||||
| Determinism | Easy (step order) | Harder (priority queue tiebreaks) |
|
||||
| Parallelism | Embarrassingly parallel | Spike-queue contention |
|
||||
| Low-rate neurons | Wasteful | Efficient |
|
||||
| Dense transient bursts | Efficient | Queue grows |
|
||||
| Voltage log cadence | Fixed | Interpolated |
|
||||
|
||||
CC-OS chooses **event-driven with 1 kHz voltage sampling**: spikes drive
|
||||
state changes, but a fixed-rate sampler still records voltage every ms for
|
||||
the temporal-tensor storage. This matches the existing
|
||||
`mat/breathing.rs`-style fixed-rate push semantics.
|
||||
|
||||
## 3. Data Layout
|
||||
|
||||
```text
|
||||
State {
|
||||
v: Vec<f32>, // length = n_neurons — membrane potentials
|
||||
last_spike_ms: Vec<f32>, // length = n_neurons — for refractory check
|
||||
csr_adjacency: CsrAdjacency, // read-only view of ConnectomeGraph
|
||||
spike_queue: BinaryHeap<ScheduledSpike>,
|
||||
rng: DeterministicRng,
|
||||
t_ms: f32,
|
||||
}
|
||||
|
||||
ScheduledSpike {
|
||||
at_ms: f32,
|
||||
pre_id: u64,
|
||||
post_id: u64,
|
||||
weight: f32,
|
||||
}
|
||||
```
|
||||
|
||||
`BinaryHeap` is `std`'s max-heap; we wrap it with `Reverse` to get a
|
||||
min-heap keyed on `at_ms`. Ties broken by `(pre_id, post_id)` for
|
||||
determinism. See §9.
|
||||
|
||||
## 4. Spike Propagation
|
||||
|
||||
When neuron $i$ fires at time $t$:
|
||||
|
||||
```text
|
||||
for each outgoing synapse (i → j) in csr_adjacency.out_edges(i):
|
||||
delay = synapse.axonal_delay_ms
|
||||
arrival = t + delay
|
||||
weight = synapse.weight × synapse.sign
|
||||
push ScheduledSpike { at_ms: arrival, pre_id: i, post_id: j, weight }
|
||||
```
|
||||
|
||||
The spike queue is drained up to the current timestep. Each drained event
|
||||
adds $\kappa(0) \cdot w$ to the target neuron's voltage (using the
|
||||
synaptic kernel at its peak, then decaying between events).
|
||||
|
||||
## 5. Sparse System Solves with `ruvector-solver`
|
||||
|
||||
The Neumann series solver $(I - A)^{-1} b = \sum_k A^k b$ converges when
|
||||
the spectral radius $\rho(A) < 1$. This rules out direct simulation of
|
||||
the LIF dynamic matrix — the fly connectome has $\rho \gg 1$ when raw
|
||||
synapse counts are used — but it is well-suited to two specific
|
||||
sub-problems:
|
||||
|
||||
### 5.1 Quasi-steady-state firing rates
|
||||
|
||||
For tonic-stimulus analysis, the firing-rate equation
|
||||
$r = \phi(W r + I)$ can be linearised around a fixed point and solved via
|
||||
Neumann series if $W$ is appropriately scaled. This is the same use case
|
||||
the `mat/triangulation.rs` TDoA solver relies on: matrix entries near
|
||||
unity so the series converges. The scaling factor must be computed per
|
||||
run; Kakaria & de Bivort 2017 give the scaling convention.
|
||||
|
||||
### 5.2 Backward influence for perturbation
|
||||
|
||||
If we want "how does a change at neuron $j$ propagate to neuron $k$
|
||||
through the linear part?", the Neumann series gives a truncated
|
||||
$\sum_{k \le K} A^k$ solution that is the right approximation for small
|
||||
perturbations. This feeds the fragility metric of 03 §7 and the
|
||||
counterfactual protocol of 08.
|
||||
|
||||
### 5.3 `CsrMatrix` construction
|
||||
|
||||
```rust
|
||||
use ruvector_solver::types::CsrMatrix;
|
||||
use ruvector_solver::neumann::NeumannSolver;
|
||||
|
||||
// triplets: (row, col, value) — scale weights so spectral radius < 1
|
||||
let triplets: Vec<(usize, usize, f32)> = graph.synapses()
|
||||
.map(|s| (s.post_id as usize, s.pre_id as usize, (s.weight as f32) * scale_factor))
|
||||
.collect();
|
||||
|
||||
let a = CsrMatrix::<f32>::from_coo(n_neurons, n_neurons, triplets);
|
||||
let b = vec![1.0_f32; n_neurons]; // external drive
|
||||
|
||||
let solver = NeumannSolver::new(1e-5, 500);
|
||||
let result = solver.solve(&a, &b)?;
|
||||
let rates = result.solution;
|
||||
```
|
||||
|
||||
## 6. Voltage and Spike-Train Storage
|
||||
|
||||
### 6.1 Voltage (per-neuron, 1 kHz)
|
||||
|
||||
Use the `VoltageBuffer` pattern from 02 §7, which delegates to
|
||||
`TemporalTensorCompressor`. At 50k neurons × 1 kHz × 60 s × 4 B = 12 GB
|
||||
raw; tiered 8/5–7/3-bit brings this to ~3 GB. Stored as an append-only
|
||||
log of compressed segments.
|
||||
|
||||
### 6.2 Spike trains
|
||||
|
||||
Two encodings:
|
||||
|
||||
**Dense bit-vector per timestep.** `[n_timesteps / 8, n_neurons]` with 1
|
||||
bit per spike. For a 5 Hz mean rate this is sparse in time, so the
|
||||
temporal-tensor's 3-bit cold tier compresses it aggressively.
|
||||
|
||||
**Event list.** `Vec<(neuron_id, t_ms)>` — 16 bytes per spike. For 250k
|
||||
spikes/s over 60 s = 15M events = 240 MB. No temporal-tensor compression
|
||||
needed; sorting by `(t_ms, neuron_id)` gives fast window queries.
|
||||
|
||||
Default: dual write. Bit-vector for replay; event list for analysis
|
||||
queries.
|
||||
|
||||
## 7. Replay and Motif Retrieval
|
||||
|
||||
Replay is a deterministic re-execution of the LIF runtime seeded from the
|
||||
same RNG, producing bit-identical spike trains. This is the foundation of
|
||||
the witness-bundle (§9) and the counterfactual protocol (08).
|
||||
|
||||
Motif retrieval uses the per-neuron embeddings from 02 §6 plus short
|
||||
spike-train signatures. `ruvector-attention` provides
|
||||
`ScaledDotProductAttention` which is enough to score motif similarity
|
||||
between the current episode and a library of previously observed
|
||||
episodes. See 05-cross-region-attention-fusion.md for the region-level
|
||||
treatment.
|
||||
|
||||
## 8. Scheduling and Concurrency
|
||||
|
||||
| Component | Concurrency class | Why |
|
||||
|-----------|-------------------|-----|
|
||||
| Spike queue drain | Single-threaded | ordering is deterministic |
|
||||
| Post-synaptic voltage updates | Rayon-parallel | disjoint targets within a time slot |
|
||||
| Voltage sampler | Single-threaded | 1 kHz cadence |
|
||||
| Compressed segment writer | `tokio` task | I/O |
|
||||
| Analysis queries | `tokio` tasks | read-only graph access |
|
||||
|
||||
The inner loop is deliberately single-threaded to preserve ordering. At
|
||||
250k spikes/s each triggering ~400 updates, a tight Rayon parallel fan-out
|
||||
per time slot delivers the needed throughput without breaking ordering
|
||||
(updates within a single time slot commute because they modify different
|
||||
neurons).
|
||||
|
||||
## 9. Determinism and Witness Logs
|
||||
|
||||
Determinism sources:
|
||||
|
||||
- Fixed seed RNG (`rand_chacha::ChaCha20Rng::seed_from_u64(seed)`).
|
||||
- Ordered event tiebreaks: `(t_ms, pre_id, post_id)`.
|
||||
- Monotonic timestep integer; no wall-clock dependence.
|
||||
|
||||
Witness log schema (reuses ADR-028 pattern):
|
||||
|
||||
```text
|
||||
run_manifest.json {
|
||||
connectome_sha256: "...",
|
||||
config_sha256: "...",
|
||||
seed: 42,
|
||||
n_neurons: 50000,
|
||||
n_synapses: 2000000,
|
||||
duration_ms: 60000,
|
||||
voltage_buffer_id: "vbuf-0",
|
||||
spike_log_id: "splog-0",
|
||||
output_sha256: "...", // hash of concatenated compressed segments
|
||||
}
|
||||
```
|
||||
|
||||
Re-running with the same seed + graph + config reproduces the same
|
||||
`output_sha256`. Divergence indicates non-determinism bugs.
|
||||
|
||||
## 10. Proposed Crate: `wifi-densepose-neuro`
|
||||
|
||||
Location: `rust-port/wifi-densepose-rs/crates/wifi-densepose-neuro/`
|
||||
|
||||
```text
|
||||
src/
|
||||
├── lib.rs re-exports
|
||||
├── lif.rs LIF neuron kernel, parameters, refractory
|
||||
├── scheduler.rs spike queue, time stepping
|
||||
├── propagate.rs fan-out, weight application, Rayon parallel slot
|
||||
├── solver.rs ruvector-solver adapter (rate-code, perturbation)
|
||||
├── storage.rs VoltageBuffer, SpikeLog wrappers
|
||||
├── replay.rs deterministic re-execution harness
|
||||
├── witness.rs ADR-028 witness bundle writer
|
||||
└── config.rs serde-backed run configuration
|
||||
```
|
||||
|
||||
Dependencies (workspace):
|
||||
|
||||
```toml
|
||||
wifi-densepose-core = { path = "../wifi-densepose-core" }
|
||||
wifi-densepose-ruvector = { path = "../wifi-densepose-ruvector" }
|
||||
ruvector-solver = { workspace = true }
|
||||
ruvector-attention = { workspace = true }
|
||||
ruvector-temporal-tensor = { workspace = true }
|
||||
rand_chacha = "0.3"
|
||||
rayon = "1.8"
|
||||
thiserror = { workspace = true }
|
||||
serde = { workspace = true }
|
||||
```
|
||||
|
||||
Publishing order: insert between `wifi-densepose-ruvector` and
|
||||
`wifi-densepose-train` in the order established in `CLAUDE.md`.
|
||||
|
||||
## 11. Performance Targets
|
||||
|
||||
| Scale | Spike throughput | Voltage write | Wall-clock per 1 s sim |
|
||||
|-------|------------------|---------------|------------------------|
|
||||
| 10k neurons | 50k spikes/s | 40 MB/s raw | 0.3 s (3.3× realtime) |
|
||||
| 50k neurons | 250k spikes/s | 200 MB/s raw | 1.0 s (1.0× realtime) |
|
||||
| 139k neurons (full fly) | 700k spikes/s | 560 MB/s raw | 3–5 s (0.2–0.3× realtime) |
|
||||
|
||||
Benchmarks to track (Criterion, matching existing repo convention):
|
||||
|
||||
- `bench/lif_kernel_throughput` — raw spikes/s for a synthetic Poisson
|
||||
network at fixed density.
|
||||
- `bench/csr_adjacency_fanout` — post-synaptic update cost per spike.
|
||||
- `bench/voltage_buffer_compression` — TemporalTensorCompressor push
|
||||
latency.
|
||||
- `bench/replay_reproducibility` — SHA-256 equality after 10 s replay.
|
||||
|
||||
## 12. Non-Goals
|
||||
|
||||
- **Hodgkin–Huxley detail** — single-compartment LIF only.
|
||||
- **Multi-compartment dendrites** — out of scope.
|
||||
- **NEURON / Brian2 interchange** — not attempted; CC-OS emits its own
|
||||
format.
|
||||
- **Learning rules inside the spike loop** — STP/LTP/LTD scaffolded in
|
||||
the edge schema (02 §3) but not executed online in v1.
|
||||
- **GPU path** — v1 is CPU-only; GPU is a v2 target.
|
||||
|
||||
## 13. References
|
||||
|
||||
1. Brunel, N. (2000). *Dynamics of sparsely connected networks of
|
||||
excitatory and inhibitory spiking neurons.* J. Comput. Neurosci.
|
||||
2. Izhikevich, E. M. (2003). *Simple model of spiking neurons.* IEEE TNN.
|
||||
3. Kakaria, K. S., de Bivort, B. L. (2017). *Ring attractor dynamics
|
||||
emerge from a spiking model of the entire protocerebral bridge.*
|
||||
Front. Behav. Neurosci.
|
||||
4. Lappalainen, J. K., et al. (2024). *Connectome-constrained networks
|
||||
predict neural activity across the fly visual system.* Nature.
|
||||
5. ADR-028 — ESP32 capability audit + witness verification.
|
||||
6. RuVector v2.0.4 crate docs — `ruvector-solver`,
|
||||
`ruvector-temporal-tensor`, `ruvector-attention`.
|
||||
7. Dorkenwald, S., et al. (2024). *Neuronal wiring diagram of an adult
|
||||
brain.* Nature.
|
||||
|
||||
---
|
||||
|
||||
**Next**: 05-cross-region-attention-fusion.md — lifting the
|
||||
`MultistaticArray` pattern to brain regions.
|
||||
@@ -0,0 +1,276 @@
|
||||
---
|
||||
Research Document ID: RD-C-05
|
||||
Date: 2026-04-21
|
||||
Status: Draft
|
||||
Authors: RuView Research Team
|
||||
Related ADRs: proposed ADR-085
|
||||
---
|
||||
|
||||
# RD-C-05: Cross-Region Attention Fusion (Layer 2 / Layer 4 bridge)
|
||||
|
||||
## Abstract
|
||||
|
||||
The fly brain contains ~80 anatomically defined neuropils (AL, MB, CX,
|
||||
LAL, GNG, VNC, etc.), each computing partial representations of the
|
||||
environment and internal state. Fusing their activity into a single
|
||||
decision-relevant embedding is the neural analog of the multistatic
|
||||
WiFi-sensing fusion already implemented in
|
||||
`rust-port/.../viewpoint/fusion.rs`. This document specifies
|
||||
`NeuralFusionArray`, a DDD aggregate root that mirrors the existing
|
||||
`MultistaticArray` — cross-viewpoint scaled dot-product attention with a
|
||||
geometric bias term, gated by a rolling coherence window and described
|
||||
by a geometric diversity index — but re-targets every component from RF
|
||||
viewpoints to connectomic regions. The resulting fused embedding feeds
|
||||
downstream CRV behavioral-episode encoding (07) and min-cut
|
||||
behavior-conditioned circuit extraction (03 §6).
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. Motivation — regions as viewpoints
|
||||
2. Direct analogy to `MultistaticArray`
|
||||
3. Region activity embedding
|
||||
4. Geometric bias in connectomic space
|
||||
5. Coherence gate on population state
|
||||
6. Functional Diversity Index (connectomic GDI analog)
|
||||
7. `NeuralFusionArray` aggregate
|
||||
8. Worked example — mushroom-body attention during foraging
|
||||
9. Integration with LIF runtime
|
||||
10. Non-goals
|
||||
11. References
|
||||
|
||||
---
|
||||
|
||||
## 1. Motivation — Regions as Viewpoints
|
||||
|
||||
The multistatic WiFi sensing pipeline treats each access point as a
|
||||
viewpoint on the propagation environment: the scene exists, and each AP
|
||||
samples it from a geometrically distinct vantage. The fusion aggregate
|
||||
combines these into a single scene embedding weighted by how much each
|
||||
viewpoint adds (angle diversity, baseline distance, SNR).
|
||||
|
||||
A connectome has the same structure at the region level. The organism's
|
||||
state exists; each neuropil samples a partial view (visual, olfactory,
|
||||
mechanosensory, contextual, motor-plan-in-progress, etc.); those views
|
||||
must be fused into a coherent behavioral signal. The mathematical
|
||||
machinery — scaled dot-product attention with a learned query over
|
||||
key–value region embeddings — is identical.
|
||||
|
||||
## 2. Direct Analogy to `MultistaticArray`
|
||||
|
||||
| `viewpoint/fusion.rs` concept | Neural analog |
|
||||
|-------------------------------|---------------|
|
||||
| `ViewpointEmbedding` | `RegionEmbedding` |
|
||||
| `NodeId` (u32) | `RegionId` (u16) |
|
||||
| `azimuth`, `elevation` | angular position in region meta-graph (optional) |
|
||||
| `baseline` distance | connectomic path-length to "reference" region |
|
||||
| `snr_db` | mean firing rate / recent signal-to-background |
|
||||
| `GeometricBias` | `ConnectomicBias` — path-length + transmitter-sign modifier |
|
||||
| `CrossViewpointAttention` | `CrossRegionAttention` (same SDPA kernel) |
|
||||
| `CoherenceGate` | unchanged; gate on population phase coherence |
|
||||
| `GeometricDiversityIndex` | `FunctionalDiversityIndex` |
|
||||
| `MultistaticArray` aggregate root | `NeuralFusionArray` aggregate root |
|
||||
|
||||
The reuse is structural, not literal: we do not import
|
||||
`MultistaticArray` for neural use; we copy its shape into a sibling
|
||||
aggregate in the proposed `wifi-densepose-neuro` crate.
|
||||
|
||||
## 3. Region Activity Embedding
|
||||
|
||||
```text
|
||||
RegionEmbedding {
|
||||
region_id: RegionId,
|
||||
embedding: Vec<f32>, // length = d_model (typically 128)
|
||||
mean_rate_hz: f32,
|
||||
phase_vector: [f32; 2], // unit-norm (cos θ, sin θ)
|
||||
baseline_mu: f32, // connectomic distance to anchor region
|
||||
snr_proxy: f32, // recent mean rate / background
|
||||
t_ms: f32,
|
||||
}
|
||||
```
|
||||
|
||||
Construction: at a given simulation time $t$, for each region,
|
||||
aggregate the last 50 ms of spikes and rates over every neuron in that
|
||||
region into a fixed-length vector. Two simple constructors:
|
||||
|
||||
1. **Rate histogram.** Bin spike counts by neuron class (sensory /
|
||||
interneuron / motor), pad to `d_model`.
|
||||
2. **Mean-pooled per-neuron embedding.** Average `NeuronEmbedding`
|
||||
weights (02 §6) over neurons active in the window.
|
||||
|
||||
Mean-pooled is more expressive; rate histogram is cheaper and serves as
|
||||
a default.
|
||||
|
||||
## 4. Geometric Bias in Connectomic Space
|
||||
|
||||
`viewpoint/fusion.rs` `GeometricBias` encodes angular separation and
|
||||
baseline distance between viewpoint pairs. The connectomic analog:
|
||||
|
||||
$$
|
||||
G^{\text{conn}}_{ij} = -w_{\text{dist}} \cdot \frac{d_{ij}^{\text{path}}}{d_{\text{ref}}}
|
||||
- w_{\text{sign}} \cdot \mathbb{1}[\mathrm{sign}(ij) < 0],
|
||||
$$
|
||||
|
||||
where $d_{ij}^{\text{path}}$ is the shortest weighted path length between
|
||||
regions $i$ and $j$ in the `RegionGraph`, $d_{\text{ref}}$ is the median
|
||||
pairwise path length, and the sign term penalises predominantly-inhibitory
|
||||
region-region connections so that attention prefers cooperative pairs.
|
||||
Added to the query-key logits inside the softmax; same shape as the RF
|
||||
case.
|
||||
|
||||
## 5. Coherence Gate on Population State
|
||||
|
||||
The neural coherence signal (Accept / PredictOnly / Reject / Recalibrate)
|
||||
uses the same hysteresis gate as `viewpoint/coherence.rs`
|
||||
`CoherenceGate`, re-seeded with a neural coherence definition:
|
||||
|
||||
$$
|
||||
C(t) = \frac{1}{N_{\text{reg}}} \sum_r \Bigl\lVert \frac{1}{|r|} \sum_{i \in r} e^{i \phi_i(t)} \Bigr\rVert,
|
||||
$$
|
||||
|
||||
a mean across regions of each region's population-vector phasor
|
||||
magnitude. Accept when $C(t)$ is above threshold with hysteresis;
|
||||
PredictOnly in the hysteresis band; Reject when below; Recalibrate on
|
||||
detected drift.
|
||||
|
||||
Thresholds (starting values):
|
||||
|
||||
| State | $C(t)$ range |
|
||||
|-------|--------------|
|
||||
| Accept | ≥ 0.6 |
|
||||
| PredictOnly | 0.5–0.6 |
|
||||
| Reject | < 0.5 |
|
||||
| Recalibrate | $\|\Delta C\| > 0.2$ within 100 ms |
|
||||
|
||||
## 6. Functional Diversity Index
|
||||
|
||||
The `GeometricDiversityIndex` in `viewpoint/geometry.rs` measures how
|
||||
well-spread viewpoints are. For regions, we use a functional-spread
|
||||
analog:
|
||||
|
||||
$$
|
||||
\mathrm{FDI} = \frac{1}{N_{\text{reg}}} \sum_r \min_{r' \ne r} \, 1 - \cos\bigl(e_r, e_{r'}\bigr),
|
||||
$$
|
||||
|
||||
the mean cosine-distance from each region's embedding to its nearest
|
||||
neighbour. High FDI means regions produce distinct representations
|
||||
(cooperative fusion is informative); low FDI means they are redundant
|
||||
(fusion adds little).
|
||||
|
||||
Cramer-Rao-analog uncertainty bounds on behavioral decoding follow from
|
||||
constructing the Fisher information matrix over region embeddings and
|
||||
inverting via `ruvector-solver`'s `NeumannSolver` (same pattern as
|
||||
`viewpoint/geometry.rs` §4). This mirrors the RF Cramer-Rao bound
|
||||
almost verbatim.
|
||||
|
||||
## 7. `NeuralFusionArray` Aggregate
|
||||
|
||||
```text
|
||||
NeuralFusionArray {
|
||||
id: ArrayId, // u64
|
||||
config: FusionConfig, // same shape as viewpoint::FusionConfig
|
||||
coherence_gate: CoherenceGate, // reuse impl
|
||||
fdi: FunctionalDiversityIndex,// computed on fuse()
|
||||
}
|
||||
|
||||
FusionConfig {
|
||||
embed_dim: usize, // 128
|
||||
coherence_threshold: f32, // 0.6
|
||||
coherence_hysteresis: f32, // 0.05
|
||||
coherence_window: usize, // 50 frames @ 1 kHz = 50 ms
|
||||
w_dist: f32, // geometric-bias distance weight
|
||||
w_sign: f32, // inhibitory penalty weight
|
||||
d_ref: f32, // reference path length (connectomic units)
|
||||
min_rate_hz: f32, // per-region minimum mean rate to contribute
|
||||
}
|
||||
|
||||
FusedEmbedding {
|
||||
embedding: Vec<f32>,
|
||||
fdi: f32,
|
||||
coherence: f32,
|
||||
n_regions: usize,
|
||||
n_effective: f32,
|
||||
}
|
||||
```
|
||||
|
||||
Fuse pipeline (matches `viewpoint/fusion.rs`):
|
||||
|
||||
1. Filter `RegionEmbedding`s by `min_rate_hz` (SNR analog).
|
||||
2. Compute FDI from the surviving set.
|
||||
3. Compute ConnectomicBias matrix.
|
||||
4. Run `ScaledDotProductAttention::new(d_model).compute(&query, &keys, &values)` with bias-adjusted query.
|
||||
5. Check `CoherenceGate`; if Reject or PredictOnly, emit the fused vector with a `coherence` tag so downstream can discount.
|
||||
6. Emit `FusedEmbedding`.
|
||||
|
||||
## 8. Worked Example — Mushroom-Body Attention During Foraging
|
||||
|
||||
During a foraging bout the mushroom body (MB) attends to olfactory
|
||||
glomeruli that encode the relevant odor and the central complex (CX)
|
||||
attends to current heading. Expected pattern:
|
||||
|
||||
| Region | Pre-foraging attention weight | During foraging |
|
||||
|--------|-------------------------------|-----------------|
|
||||
| AL (olfactory glomeruli) | 0.1 | 0.35 |
|
||||
| MB (Kenyon cells) | 0.1 | 0.40 |
|
||||
| CX (EB/FB/PB) | 0.2 | 0.10 |
|
||||
| LAL (steering motor) | 0.1 | 0.05 |
|
||||
| VLP (visual LP) | 0.2 | 0.05 |
|
||||
| GNG (grooming premotor) | 0.1 | 0.01 |
|
||||
| VNC (motor) | 0.2 | 0.04 |
|
||||
|
||||
The precise numbers depend on fit quality; the qualitative shift — AL
|
||||
and MB gain weight at the expense of GNG/VLP — is a falsifiable
|
||||
prediction. Cross-behavior attention maps are published in Lappalainen
|
||||
2024 for vision; equivalent for olfaction is less mature and a natural
|
||||
CC-OS output.
|
||||
|
||||
## 9. Integration with LIF Runtime
|
||||
|
||||
`NeuralFusionArray` is re-evaluated every `fusion_period_ms` (default
|
||||
50 ms) during a simulation run:
|
||||
|
||||
```text
|
||||
every fusion_period_ms:
|
||||
for each region r:
|
||||
build RegionEmbedding from last 50 ms of spikes
|
||||
fused = NeuralFusionArray.fuse(region_embeddings)
|
||||
if fused.coherence ≥ threshold:
|
||||
hand fused to BehaviorPipeline (07)
|
||||
else:
|
||||
record as "uncertain" in witness log
|
||||
```
|
||||
|
||||
This is the handoff between Layer 2 (LIF runtime) and Layer 4 (analysis
|
||||
and behavioral-episode encoding). It is architecturally symmetric with
|
||||
the RF pipeline, where CSI frames feed the existing fusion before
|
||||
behavioral claims are made.
|
||||
|
||||
## 10. Non-Goals
|
||||
|
||||
- **Biologically plausible global workspace theory.** `NeuralFusionArray`
|
||||
is a computational fusion aggregate, not a model of conscious
|
||||
broadcasting.
|
||||
- **Trainable attention weights.** The attention is parameter-free SDPA
|
||||
over connectome-derived embeddings; no gradient-based tuning in v1.
|
||||
- **Real-time long-range delays.** Bias term approximates path-length
|
||||
effect; not a substitute for explicit delay modelling.
|
||||
|
||||
## 11. References
|
||||
|
||||
1. Vaswani, A., et al. (2017). *Attention is all you need.* NeurIPS.
|
||||
2. Lappalainen, J. K., et al. (2024). *Connectome-constrained networks
|
||||
predict neural activity across the fly visual system.* Nature.
|
||||
3. Aso, Y., Sitaraman, D., Ichinose, T., et al. (2014). *Mushroom body
|
||||
output neurons encode valence and guide memory-based action
|
||||
selection.* eLife. (mushroom-body attention precedent.)
|
||||
4. Turner-Evans, D. B., Jayaraman, V. (2016). *The insect central
|
||||
complex.* Curr Biol.
|
||||
5. RuView `viewpoint/fusion.rs` (crate `wifi-densepose-ruvector`).
|
||||
6. RuVector v2.0.4 crate docs — `ruvector-attention`,
|
||||
`ruvector-solver`.
|
||||
|
||||
---
|
||||
|
||||
**Next**: 06-embodied-simulator-closed-loop.md — closing the sensorimotor
|
||||
loop through a virtual body.
|
||||
@@ -0,0 +1,289 @@
|
||||
---
|
||||
Research Document ID: RD-C-06
|
||||
Date: 2026-04-21
|
||||
Status: Draft
|
||||
Authors: RuView Research Team
|
||||
Related ADRs: proposed ADR-086
|
||||
---
|
||||
|
||||
# RD-C-06: Embodied Simulator — Closed Sensorimotor Loop (Layer 3)
|
||||
|
||||
## Abstract
|
||||
|
||||
Without a body, connectome dynamics simulate a disembodied sensor
|
||||
network; the behaviors we want to study — grooming, walking, feeding —
|
||||
exist only when motor output reaches a physical body and sensory input
|
||||
comes back as contact, proprioception, and optic flow. Layer 3 of CC-OS
|
||||
is a closed-loop embodied simulator. We compare candidate platforms
|
||||
(NeuroMechFly v1/v2, MuJoCo / MJX, Brax, PyBullet, Isaac Gym, Rapier),
|
||||
recommend a **Rapier-native Rust inner loop with an optional
|
||||
NeuroMechFly bridge** for biomechanical-validation runs, specify the
|
||||
sensory encoders (bristle mechanosensors, Johnston's-organ
|
||||
proprioception, optional ocellar luminance), the motor decoders
|
||||
(descending MN rate → joint torque), the latency budget required to
|
||||
sustain 100–1000 Hz control, and the witness-log schema that keeps every
|
||||
closed-loop bout auditable.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. Requirements
|
||||
2. Platform comparison
|
||||
3. Recommendation
|
||||
4. Body schema
|
||||
5. Sensory adapters
|
||||
6. Motor adapters
|
||||
7. Latency budget
|
||||
8. Cross-process vs in-process bridges
|
||||
9. Proposed crate: `wifi-densepose-embody`
|
||||
10. Witness and provenance
|
||||
11. Failure modes
|
||||
12. Non-goals
|
||||
13. References
|
||||
|
||||
---
|
||||
|
||||
## 1. Requirements
|
||||
|
||||
| Requirement | Target | Notes |
|
||||
|-------------|--------|-------|
|
||||
| Articulated body | Yes | 60–80 DoF for fly |
|
||||
| Contact sensing | Yes | For grooming and substrate interaction |
|
||||
| Proprioception | Yes | Joint angle + angular velocity |
|
||||
| Vision (optional v1) | No | Adds ~1 ms per frame if simple |
|
||||
| Physics rate | ≥ 1 kHz | To not starve LIF inner loop |
|
||||
| Control rate | 100–1000 Hz | LIF fan-out cadence |
|
||||
| Determinism | Yes (seeded) | Witness-log compatibility |
|
||||
| Round-trip latency | < 10 ms @ 100 Hz control; < 1 ms aspirational | Tight neural-body coupling |
|
||||
|
||||
## 2. Platform Comparison
|
||||
|
||||
| Platform | Language | Body model | Physics | In-proc with Rust? | Pro | Con |
|
||||
|----------|----------|-----------|---------|--------------------|-----|-----|
|
||||
| NeuroMechFly v1 | Python + MuJoCo | Published fly | Rigid + contact | IPC only | Biomechanically validated | Python overhead, platform-specific |
|
||||
| NeuroMechFly v2 | Python + MuJoCo-MJX | Published fly + vision | JAX | IPC only | Adds vision | Heavier stack |
|
||||
| MuJoCo (bare) | C / Python | Author-your-own | Rigid + contact | FFI possible | Fast, reliable | Body must be authored |
|
||||
| MJX | JAX | — | JAX-vectorised | Poor | Fast batched rollouts | Not for 1-instance realtime |
|
||||
| Brax | JAX | — | Rigid | Poor | Fast, modern | Same |
|
||||
| PyBullet | Python (wraps Bullet) | — | Rigid + contact | IPC | Easy | Medium fidelity |
|
||||
| Isaac Gym | C++/Python | — | PhysX | Poor | Massive parallel RL | GPU-only, not auditable |
|
||||
| **Rapier** | **Rust** | Authorable | Rigid + contact | **Yes, in-proc** | Native Rust, deterministic | No published fly body |
|
||||
|
||||
## 3. Recommendation
|
||||
|
||||
**v1 (internal)**: Rapier-native Rust inner loop with a hand-authored
|
||||
minimal fly body. Prioritises determinism, latency, and staying inside
|
||||
the Rust workspace.
|
||||
|
||||
**v1 (validation)**: NeuroMechFly bridge over shared memory or gRPC for
|
||||
biomechanical cross-checks. The tight inner loop stays in Rust; periodic
|
||||
validation runs use NeuroMechFly as ground truth.
|
||||
|
||||
**v2 (optional)**: MJX or Brax for parallel rollouts when exploring
|
||||
perturbation sweeps.
|
||||
|
||||
## 4. Body Schema
|
||||
|
||||
Minimal fly body for v1: 6 legs × 5 joints + head (2) + abdomen (3) +
|
||||
antennae (3 × 2) = 41 DoF. NeuroMechFly v2 goes to ~70 DoF including
|
||||
detailed wing articulation; we skip wings in v1.
|
||||
|
||||
```text
|
||||
FlyBody {
|
||||
joints: Vec<Joint>, // ~41
|
||||
links: Vec<Link>,
|
||||
sensors: Vec<Sensor>, // contact, bristle, Johnston's
|
||||
actuators: Vec<Actuator>, // joint torque / position
|
||||
contact_model: ContactModel, // friction coefficients
|
||||
}
|
||||
|
||||
Sensor ::= Bristle { link_id, threshold_n }
|
||||
| Johnston { joint_id } // antennal JO mechanosensor
|
||||
| LegContact { link_id }
|
||||
| Optic { link_id, fov } // v2
|
||||
|
||||
Actuator ::= JointTorque { joint_id, max_nm }
|
||||
| JointPosition { joint_id, pd_gains }
|
||||
```
|
||||
|
||||
## 5. Sensory Adapters
|
||||
|
||||
Sensor → spike-input injector. Each physical sensor maps to a pool of
|
||||
sensory neurons (02 §2) whose IDs come from the loaded connectome.
|
||||
|
||||
### 5.1 Bristle mechanoreceptors
|
||||
|
||||
```text
|
||||
on bristle contact force f ≥ threshold:
|
||||
rate_hz = saturating(α · f, max_rate_hz)
|
||||
emit Poisson spike train into bristle-neuron pool
|
||||
```
|
||||
|
||||
Maps to the ~1,300 FlyWire bristle neurons listed as sensory class.
|
||||
|
||||
### 5.2 Johnston's-organ proprioception
|
||||
|
||||
Antennal angle velocity drives a Poisson rate into JO pool. Pool size
|
||||
~480 in FlyWire. Reference: Tuthill & Wilson 2016 (*Curr Biol*).
|
||||
|
||||
### 5.3 Leg contact
|
||||
|
||||
Binary contact → pulse train into corresponding leg contact pool (if
|
||||
present in the subgraph; optional).
|
||||
|
||||
## 6. Motor Adapters
|
||||
|
||||
Descending motor neuron population rate → joint command.
|
||||
|
||||
### 6.1 Rate-coded torque
|
||||
|
||||
$$
|
||||
\tau_{\text{joint}} = g \cdot \sum_{i \in \text{MN pool}} (r_i - r_{\text{baseline}}),
|
||||
$$
|
||||
|
||||
with $g$ a calibration gain and $r_{\text{baseline}}$ a per-pool tonic
|
||||
rate. For antennal grooming, the prothoracic-leg MN pool maps onto leg
|
||||
joint torques around the sweep arc.
|
||||
|
||||
### 6.2 Central pattern generators (CPG)
|
||||
|
||||
For locomotion (not the v1 acceptance target), Cruse 1990 Walknet
|
||||
provides a phase-coupling model between legs that can be layered on top
|
||||
of the motor adapters. Out of scope for v1 but worth noting as a
|
||||
v2 extension.
|
||||
|
||||
## 7. Latency Budget
|
||||
|
||||
At 100 Hz closed-loop control:
|
||||
|
||||
| Stage | Budget | Notes |
|
||||
|-------|--------|-------|
|
||||
| Sensor read + encoding | 0.5 ms | per frame |
|
||||
| Spike injection | 0.1 ms | into priority queue |
|
||||
| LIF step | 5 ms | for 50k neurons (§ 04-11) scaled to 10 ms window |
|
||||
| Region fusion | 0.5 ms | 80 regions × 128-d |
|
||||
| CRV episode encoding | 1 ms | occasional |
|
||||
| Motor decoding | 0.2 ms | per active pool |
|
||||
| Physics step | 1 ms | at 1 kHz physics |
|
||||
| Total | < 10 ms | target met |
|
||||
|
||||
For 1 kHz control (aspirational), every stage must shrink 10×. Feasible
|
||||
with reduced-subgraph LIF, batched fusion, and careful no-alloc inner
|
||||
loops.
|
||||
|
||||
## 8. Cross-Process vs In-Process Bridges
|
||||
|
||||
| Bridge | Overhead | Determinism | When |
|
||||
|--------|----------|-------------|------|
|
||||
| Rapier in-proc FFI | < 1 µs | Yes (seeded) | v1 tight loop |
|
||||
| Shared memory (NeuroMechFly) | 20–50 µs | Needs coordination | Validation runs |
|
||||
| gRPC localhost | 200–500 µs | Needs timestamps | Debugging / UI |
|
||||
| TCP remote | 1–5 ms | Difficult | Not recommended |
|
||||
|
||||
## 9. Proposed Crate: `wifi-densepose-embody`
|
||||
|
||||
Location: `rust-port/wifi-densepose-rs/crates/wifi-densepose-embody/`
|
||||
|
||||
```text
|
||||
src/
|
||||
├── lib.rs
|
||||
├── body.rs Body schema, URDF-ish loader
|
||||
├── sensors.rs Encoders: bristle, JO, contact, optic
|
||||
├── actuators.rs Decoders: torque, PD, CPG stub
|
||||
├── physics.rs Rapier integration
|
||||
├── bridge_nmf.rs Optional NeuroMechFly bridge (feature-gated)
|
||||
├── witness.rs ADR-028 body-state bundle writer
|
||||
└── config.rs
|
||||
```
|
||||
|
||||
Dependencies:
|
||||
|
||||
```toml
|
||||
wifi-densepose-core = { path = "../wifi-densepose-core" }
|
||||
wifi-densepose-neuro = { path = "../wifi-densepose-neuro" } # Layer 2
|
||||
rapier3d = "0.18"
|
||||
serde = { workspace = true }
|
||||
thiserror = { workspace = true }
|
||||
```
|
||||
|
||||
Feature flags:
|
||||
|
||||
```toml
|
||||
[features]
|
||||
default = []
|
||||
nmf-bridge = ["dep:zmq"] # optional
|
||||
vision = [] # v2
|
||||
```
|
||||
|
||||
Publishing order: after `wifi-densepose-neuro`, before
|
||||
`wifi-densepose-mat`.
|
||||
|
||||
## 10. Witness and Provenance
|
||||
|
||||
Every closed-loop bout produces:
|
||||
|
||||
```text
|
||||
bout_manifest.json {
|
||||
bout_id: "bout-0001",
|
||||
seed: 42,
|
||||
connectome_sha256: "...",
|
||||
body_sha256: "...",
|
||||
config_sha256: "...",
|
||||
t_start_ms: 0,
|
||||
t_end_ms: 10000,
|
||||
frames_compressed: "bout-0001-frames.ttseg", // TemporalTensorCompressor
|
||||
voltage_ref: "vbuf-0",
|
||||
spike_ref: "splog-0",
|
||||
output_sha256: "...",
|
||||
}
|
||||
```
|
||||
|
||||
Frames are body-state + sensor readings compressed via the same
|
||||
TemporalTensorCompressor pattern (02 §7). Replaying the bout produces a
|
||||
byte-identical SHA-256 if and only if the LIF runtime and the physics
|
||||
are both deterministic under the seed.
|
||||
|
||||
## 11. Failure Modes
|
||||
|
||||
| Failure | Signature | Mitigation |
|
||||
|---------|-----------|------------|
|
||||
| Physics divergence | NaN joint angles | Clamp + halt + emit `PhysicsDiverged` event |
|
||||
| Spike firehose | Queue length > threshold | Throttle or drop to PredictOnly coherence state (05) |
|
||||
| Actuator saturation | Torque > max | Clip + warn |
|
||||
| Sensor aliasing | Rate exceeds neuron pool refractory | Decimate encoding |
|
||||
| Starved LIF | Simulation time lags wall-clock | Reduce control rate or subgraph size |
|
||||
| Non-deterministic replay | SHA-256 mismatch | Re-examine RNG seeding and tiebreaks |
|
||||
|
||||
## 12. Non-Goals
|
||||
|
||||
- **Full wing + flight aerodynamics** — v2 target; requires air-model and
|
||||
much higher physics fidelity.
|
||||
- **Fluid dynamics** (e.g. olfactory plume modelling) — external tool.
|
||||
- **Photorealistic rendering** — not relevant; optical-flow only at
|
||||
most.
|
||||
- **Multi-agent interactions** — single-organism v1.
|
||||
|
||||
## 13. References
|
||||
|
||||
1. Lobato-Rios, V., et al. (2022). *NeuroMechFly, a neuromechanical model
|
||||
of adult Drosophila melanogaster.* Nat Methods.
|
||||
2. Wang-Chen, S., et al. (2024). *NeuroMechFly v2 with vision (preprint).*
|
||||
3. Todorov, E., Erez, T., Tassa, Y. (2012). *MuJoCo: A physics engine for
|
||||
model-based control.* IROS.
|
||||
4. Freeman, C. D., et al. (2021). *Brax — a differentiable physics
|
||||
engine.* NeurIPS workshop.
|
||||
5. Cruse, H. (1990). *What mechanisms coordinate leg movement in walking
|
||||
arthropods?* Trends Neurosci.
|
||||
6. Tuthill, J. C., Wilson, R. I. (2016). *Mechanosensation and adaptive
|
||||
motor control in insects.* Curr Biol.
|
||||
7. Seeds, A. M., et al. (2014). *A suppression hierarchy among competing
|
||||
motor programs drives sequential grooming in Drosophila.* eLife.
|
||||
8. Rapier physics engine docs — https://rapier.rs (reference only; no
|
||||
DOI).
|
||||
9. ADR-028 — ESP32 capability audit + witness verification.
|
||||
|
||||
---
|
||||
|
||||
**Next**: 07-coherence-crv-behavioral-episodes.md — turning closed-loop
|
||||
bouts into audited, queryable episodes.
|
||||
@@ -0,0 +1,281 @@
|
||||
---
|
||||
Research Document ID: RD-C-07
|
||||
Date: 2026-04-21
|
||||
Status: Draft
|
||||
Authors: RuView Research Team
|
||||
Related ADRs: proposed ADR-087
|
||||
---
|
||||
|
||||
# RD-C-07: Coherence + CRV Behavioral Episodes (Layer 4)
|
||||
|
||||
## Abstract
|
||||
|
||||
Behaviors are episodic: a grooming bout starts, runs, ends. Layer 4 of
|
||||
CC-OS encodes each episode through the six-stage `CrvSessionManager`
|
||||
protocol already shipped in `ruvector-crv`, augmented with a population
|
||||
coherence gate on neural state. The mapping is not metaphorical: CRV
|
||||
Stages I–VI were designed for session-based embedding with graph
|
||||
partitioning at Stage VI, which is exactly the operation a
|
||||
behavior-episode record demands. This document specifies the
|
||||
`BehaviorPipeline` facade (analog of `WifiCrvPipeline` in
|
||||
`crv/mod.rs`), the stage-by-stage encoding of a behavioral bout, the
|
||||
coherence gate on population state, the episode storage contract, and
|
||||
the cross-subject convergence primitive that lets one fly's behavior be
|
||||
compared to another's.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. Motivation — episodes over continuous state
|
||||
2. CRV → neural stage map
|
||||
3. Coherence gate on population state
|
||||
4. `BehaviorPipeline` facade
|
||||
5. Episode encoding worked example — antennal grooming
|
||||
6. Stage VI — min-cut for behavior-responsible circuit
|
||||
7. Persistence and replay
|
||||
8. Cross-subject convergence
|
||||
9. Non-goals
|
||||
10. References
|
||||
|
||||
---
|
||||
|
||||
## 1. Motivation — Episodes Over Continuous State
|
||||
|
||||
Neural runtime data streams are continuous: voltages at 1 kHz, spikes
|
||||
every few ms. Behavior is discontinuous: a bout begins, peaks, ends. The
|
||||
right encoding granularity for analysis and audit is the bout, not the
|
||||
frame. A bout is a bounded window with associated sensory input, motor
|
||||
output, and a claim about what behavior it exhibits. `ruvector-crv`'s
|
||||
session abstraction matches this granularity.
|
||||
|
||||
## 2. CRV → Neural Stage Map
|
||||
|
||||
The six CRV stages shipped in `crv/mod.rs` are:
|
||||
|
||||
| Stage | CRV semantics | Neural/behavioral semantics |
|
||||
|-------|--------------|-----------------------------|
|
||||
| I. Gestalt | pattern-class encoding | behavior class (Locomotion / Grooming / Feeding / Freezing / Courtship) |
|
||||
| II. Sensory | feature extraction | multi-modal body+neural feature vector |
|
||||
| III. Dimensional | spatial sketch graph | body-pose graph + region sketch |
|
||||
| IV. Emotional / AOL | coherence / anomaly | `CoherenceGateState` on population state |
|
||||
| V. Interrogation | differentiable query | "what circuit caused this bout?" |
|
||||
| VI. Composite | MinCut partitioning | **minimal circuit** for the behavior |
|
||||
|
||||
The `GestaltType` enum (`Movement / Land / Energy / Natural / Manmade /
|
||||
Water`) from `ruvector-crv` can be repurposed loosely — `Movement` =
|
||||
locomotion, `Energy` = reactive bout, `Land` = rest — but a
|
||||
domain-specific `BehaviorClass` enum is cleaner:
|
||||
|
||||
```text
|
||||
BehaviorClass ::= Locomotion | Grooming { substrate } | Feeding
|
||||
| Freezing | Courtship | Resting
|
||||
| Unclassified { confidence }
|
||||
```
|
||||
|
||||
The `SensoryModality` enum (Texture / Color / Temperature / Sound /
|
||||
Luminosity / Dimension) maps to neural-body modalities:
|
||||
|
||||
| `SensoryModality` | Neural/body analog |
|
||||
|-------------------|--------------------|
|
||||
| Texture | proprioceptive roughness (joint-angle high-freq variance) |
|
||||
| Color | region activity spectral centroid |
|
||||
| Temperature | total population energy |
|
||||
| Sound | periodic body oscillation (gait frequency) |
|
||||
| Luminosity | coherence / SNR |
|
||||
| Dimension | behavioral-state entropy / dimensionality |
|
||||
|
||||
## 3. Coherence Gate on Population State
|
||||
|
||||
The `CoherenceGate` from `viewpoint/coherence.rs` generalises directly.
|
||||
Reject bouts where the population phase is incoherent (noise, mid-state
|
||||
transition); PredictOnly for borderline; Recalibrate on detected drift
|
||||
(circadian, injury, fatigue).
|
||||
|
||||
```text
|
||||
every 50 ms:
|
||||
C = mean_r population_phase_coherence(region_r)
|
||||
gate_state = coherence_gate.update(C)
|
||||
if gate_state == Reject:
|
||||
discard current episode, restart window
|
||||
elif gate_state == PredictOnly:
|
||||
flag episode as low-confidence
|
||||
elif gate_state == Recalibrate:
|
||||
emit CoherenceRecalibrationRequired event
|
||||
```
|
||||
|
||||
## 4. `BehaviorPipeline` Facade
|
||||
|
||||
Analog of `WifiCrvPipeline`:
|
||||
|
||||
```text
|
||||
BehaviorPipeline {
|
||||
manager: CrvSessionManager,
|
||||
behavior_classifier: BehaviorClassifier, // Stage I
|
||||
neural_encoder: NeuralSensoryEncoder, // Stage II
|
||||
body_sketcher: BodyPoseGraphEncoder, // Stage III
|
||||
coherence_gate: CoherenceGate, // Stage IV
|
||||
config: BehaviorConfig,
|
||||
}
|
||||
|
||||
impl BehaviorPipeline {
|
||||
pub fn new(config: BehaviorConfig) -> Self { ... }
|
||||
|
||||
pub fn start_bout(&mut self, bout_id: &str, subject_id: &str)
|
||||
-> Result<(), CrvError>;
|
||||
|
||||
pub fn process_frame(
|
||||
&mut self,
|
||||
bout_id: &str,
|
||||
region_embeds: &[RegionEmbedding], // from NeuralFusionArray (05)
|
||||
body_state: &BodyState, // from Layer 3 (06)
|
||||
spike_summary: &SpikeSummary, // from Layer 2 (04)
|
||||
) -> Result<FrameDigest, CrvError>;
|
||||
|
||||
pub fn finalize_bout(&mut self, bout_id: &str)
|
||||
-> Result<BehaviorEpisode, CrvError>;
|
||||
|
||||
pub fn interrogate(&mut self, bout_id: &str, query: &[f32])
|
||||
-> Result<StageVData, CrvError>;
|
||||
|
||||
pub fn partition_circuit(&mut self, bout_id: &str)
|
||||
-> Result<MinimalCircuit, CrvError>;
|
||||
|
||||
pub fn cross_subject_convergence(
|
||||
&self, behavior_class: &str, min_similarity: f32
|
||||
) -> Result<ConvergenceResult, CrvError>;
|
||||
}
|
||||
```
|
||||
|
||||
The parallel with `WifiCrvPipeline` is near-exact: `room-id` → subject-id
|
||||
or behavior-class; `ApNode` → region sketch element; `CoherenceGateState`
|
||||
→ population-state gate.
|
||||
|
||||
## 5. Episode Encoding Worked Example — Antennal Grooming
|
||||
|
||||
Given a 500 ms bout identified by sustained foreleg sweep of head:
|
||||
|
||||
**Stage I (Gestalt).** `BehaviorClassifier::classify(body_state_window,
|
||||
spike_summary_window)` returns `Grooming { substrate: Antenna }` with
|
||||
confidence 0.88.
|
||||
|
||||
**Stage II (Sensory).**
|
||||
`NeuralSensoryEncoder::extract(region_embeds, body_state)` returns:
|
||||
|
||||
```text
|
||||
[ (Texture, "proprioceptive periodic"),
|
||||
(Color, "mb-dominated activity"),
|
||||
(Temperature, "moderate pop-rate"),
|
||||
(Sound, "rhythmic 8-Hz sweep"),
|
||||
(Luminosity, "high coherence"),
|
||||
(Dimension, "narrow behavioral manifold") ]
|
||||
```
|
||||
|
||||
**Stage III (Dimensional).** `BodyPoseGraphEncoder::sketch(...)`
|
||||
returns a graph of ~10 joint nodes (head, foreleg segments, antenna)
|
||||
with position and scale per `SketchElement`.
|
||||
|
||||
**Stage IV (AOL / coherence).**
|
||||
`CoherenceGate::state() == Accept` with coherence 0.82.
|
||||
|
||||
**Stage V (Interrogation).** Query embedding = "circuit responsible for
|
||||
grooming" (taken from a library of canonical query vectors) →
|
||||
differentiable search over accumulated frame embeddings returns top-3
|
||||
frame indices with highest activation match.
|
||||
|
||||
**Stage VI (Composite).** `run_stage_vi(bout_id)` runs MinCut over
|
||||
accumulated embeddings, producing a partition whose neuron members are
|
||||
the candidate grooming circuit.
|
||||
|
||||
## 6. Stage VI — MinCut for Behavior-Responsible Circuit
|
||||
|
||||
Stage VI's MinCut is the operational mechanism by which CC-OS answers
|
||||
"what circuit caused this bout?". Two variants:
|
||||
|
||||
| Variant | Cut input | Result |
|
||||
|---------|-----------|--------|
|
||||
| On embeddings | per-frame fused embeddings | partition into "behavior" vs "background" frames |
|
||||
| Lifted to connectome | attention-gated mincut (see 03 §6) over neurons active in behavior frames | minimal neuron/synapse set |
|
||||
|
||||
The second variant is the circuit-discovery product of CC-OS. Its output
|
||||
is directly consumed by 08-counterfactual-perturbation.md for validation
|
||||
by ablation.
|
||||
|
||||
## 7. Persistence and Replay
|
||||
|
||||
Every `BehaviorEpisode` emits a manifest:
|
||||
|
||||
```text
|
||||
episode_manifest.json {
|
||||
episode_id: "ep-0017",
|
||||
bout_id: "bout-0001",
|
||||
subject_id: "fly-a",
|
||||
behavior_class: "Grooming/Antenna",
|
||||
t_start_ms: 1520,
|
||||
t_end_ms: 2020,
|
||||
coherence_state: "Accept",
|
||||
coherence_score: 0.82,
|
||||
stage_i_embed_ref: "...",
|
||||
stage_ii_embed_ref: "...",
|
||||
stage_iii_embed_ref:"...",
|
||||
stage_iv_embed_ref: "...",
|
||||
stage_v_embed_ref: "...",
|
||||
stage_vi_circuit: "circuit-grooming-antenna-0017.json",
|
||||
witness_sha256: "..."
|
||||
}
|
||||
```
|
||||
|
||||
Replay: given the same seed + connectome + body + stimuli, the same
|
||||
episode is reconstructible byte-for-byte. This is the auditability
|
||||
property that distinguishes CC-OS from undocumented demo runs.
|
||||
|
||||
## 8. Cross-Subject Convergence
|
||||
|
||||
`CrvSessionManager::find_convergence(room_id, min_similarity)` already
|
||||
exists for matching sessions in the same "room". Relabel `room_id` to
|
||||
`behavior_class` (or `(subject_id, behavior_class)` if intra-subject
|
||||
replication is also wanted).
|
||||
|
||||
```text
|
||||
convergence = pipeline.cross_subject_convergence(
|
||||
behavior_class: "Grooming/Antenna",
|
||||
min_similarity: 0.7,
|
||||
)
|
||||
// returns scores for pairs of episodes; high scores = same behavior
|
||||
// patterned similarly across subjects
|
||||
```
|
||||
|
||||
This gives CC-OS a built-in reproducibility primitive: two flies
|
||||
presented with the same stimulus should produce convergent Stage II
|
||||
embeddings if the shared circuit is intact; mutation or ablation shows
|
||||
up as dropped convergence.
|
||||
|
||||
## 9. Non-Goals
|
||||
|
||||
- **Discovery of novel behaviors.** v1 classifies among a fixed behavior
|
||||
vocabulary. Unsupervised behavior discovery (cf. Berman et al. 2014 on
|
||||
MotionMapper) is a v2 target.
|
||||
- **Behavior prediction from connectome alone.** The pipeline classifies
|
||||
behaviors observed in-loop; predicting future behaviors requires
|
||||
forward models not part of v1.
|
||||
- **Human or higher-mammal behaviors.** Out of scope.
|
||||
|
||||
## 10. References
|
||||
|
||||
1. `ruvector-crv` v0.1.1 — `CrvSessionManager`, stages I–VI.
|
||||
2. `crv/mod.rs` — `WifiCrvPipeline`, pattern source.
|
||||
3. `viewpoint/coherence.rs` — `CoherenceGate` with Accept / PredictOnly
|
||||
/ Reject / Recalibrate states.
|
||||
4. Berman, G. J., et al. (2014). *Mapping the stereotyped behaviour of
|
||||
freely moving fruit flies.* J. R. Soc. Interface.
|
||||
5. Seeds, A. M., et al. (2014). *A suppression hierarchy among competing
|
||||
motor programs.* eLife.
|
||||
6. Hampel, S., et al. (2015). *A neural command circuit for grooming
|
||||
movement control.* eLife.
|
||||
7. Card, G. M., Dickinson, M. H. (2008). *Performance trade-offs in the
|
||||
flight initiation of Drosophila.* J. Exp. Biol.
|
||||
|
||||
---
|
||||
|
||||
**Next**: 08-counterfactual-perturbation.md — using episode circuits to
|
||||
do real circuit science.
|
||||
@@ -0,0 +1,246 @@
|
||||
---
|
||||
Research Document ID: RD-C-08
|
||||
Date: 2026-04-21
|
||||
Status: Draft
|
||||
Authors: RuView Research Team
|
||||
Related ADRs: proposed ADR-087, ADR-088
|
||||
---
|
||||
|
||||
# RD-C-08: Counterfactual Perturbation and Circuit Fragility Protocol
|
||||
|
||||
## Abstract
|
||||
|
||||
A structural min-cut (03) identifies the smallest synaptic bottleneck
|
||||
between a sensory and motor ensemble; a Stage VI MinCut over behavioral
|
||||
episodes (07) identifies the circuit members active during a specific
|
||||
bout. Neither, on its own, proves that the identified circuit is
|
||||
**causally necessary** for the behavior. Causality demands perturbation:
|
||||
remove or scale the candidate circuit, re-run the closed loop, and
|
||||
measure whether the behavior survives. This document specifies the
|
||||
perturbation primitives (synapse ablation, weight scaling, neuron
|
||||
silencing, optogenetic-analog activation, transmitter block), the
|
||||
re-mincut + behavioral-re-evaluation protocol, the fragility score that
|
||||
ranks candidate circuits, and the auditable witness-log pipeline that
|
||||
makes every perturbation reproducible.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. Problem framing
|
||||
2. Perturbation primitives
|
||||
3. Re-mincut protocol
|
||||
4. Behavioral fragility score
|
||||
5. Search strategy
|
||||
6. Published comparison
|
||||
7. Auditability
|
||||
8. Worked example — antennal grooming
|
||||
9. Limitations
|
||||
10. Ethical considerations
|
||||
11. References
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem Framing
|
||||
|
||||
Correlation between neuron activity and behavior is cheap; causal
|
||||
attribution is hard. The dissociation problem — neuron $i$ fires during
|
||||
behavior $B$, but does neuron $i$ cause $B$? — has a long history in
|
||||
systems neuroscience (cf. Jazayeri & Afraz 2017, *Neuron*, on
|
||||
perturbation causal inference). The CC-OS answer is a constructive loop:
|
||||
propose a candidate circuit (from mincut or Stage VI), perturb it,
|
||||
re-run, measure. This requires fast, deterministic, auditable
|
||||
perturbation — exactly what the `ConnectomeGraph` aggregate
|
||||
(02-connectome-graph-substrate.md §10) was designed for.
|
||||
|
||||
## 2. Perturbation Primitives
|
||||
|
||||
All perturbations are expressed as append-only events on the graph
|
||||
aggregate, so they are reversible by replaying without the event.
|
||||
|
||||
| Primitive | Event | Effect |
|
||||
|-----------|-------|--------|
|
||||
| Synapse ablation | `SynapseAblated { pre_id, post_id }` | weight → 0 |
|
||||
| Weight scaling | `SynapseScaled { pre_id, post_id, factor }` | weight × factor |
|
||||
| Neuron silencing | `NeuronSilenced { id }` | suppress all outgoing spikes |
|
||||
| Neuron driven | `NeuronDriven { id, rate_hz }` | external Poisson injection |
|
||||
| Transmitter block | `TransmitterBlocked { region, nt }` | all edges of that NT in that region set to 0 |
|
||||
|
||||
Silencing and driving have optogenetic-analog semantics: they do not
|
||||
change connectivity, they override activity.
|
||||
|
||||
## 3. Re-Mincut Protocol
|
||||
|
||||
Given a baseline min-cut $C_0$ with weight $w_0$, a perturbation $P$
|
||||
that scales weights, and the resulting cut $C_1$ with weight $w_1$:
|
||||
|
||||
$$
|
||||
\Delta_{\mathrm{cut}}(P) = \frac{w_0 - w_1}{w_0} \in [0, 1].
|
||||
$$
|
||||
|
||||
Large $\Delta_{\mathrm{cut}}$ means $P$ weakened the cut substantially.
|
||||
Perturbations entirely outside the bottleneck produce $\Delta \approx
|
||||
0$; perturbations inside produce $\Delta$ proportional to the fraction
|
||||
of the bottleneck removed.
|
||||
|
||||
The re-mincut is incremental: `ruvector-mincut`'s `DynamicMinCut` is
|
||||
designed for exactly this — the graph state after perturbation differs
|
||||
from the baseline only on the perturbed edges.
|
||||
|
||||
## 4. Behavioral Fragility Score
|
||||
|
||||
Cut fragility is structural; behavioral fragility is what matters.
|
||||
Define
|
||||
|
||||
$$
|
||||
\mathcal{F}_{\text{beh}}(P) = 1 - \frac{\Pr[\text{behavior} \mid P]}{\Pr[\text{behavior} \mid P_0]},
|
||||
$$
|
||||
|
||||
where $P_0$ is the unperturbed baseline and $\Pr[\text{behavior} \mid
|
||||
\cdot]$ is the fraction of $N$ bouts producing the target behavior with
|
||||
coherence gate state `Accept`. $\mathcal{F}_{\text{beh}} = 0$ means the
|
||||
perturbation did nothing; $\mathcal{F}_{\text{beh}} = 1$ means it
|
||||
abolished the behavior.
|
||||
|
||||
Combining structural and behavioral fragility:
|
||||
|
||||
$$
|
||||
\mathcal{F}(P) = \sqrt{\Delta_{\mathrm{cut}}(P) \cdot \mathcal{F}_{\text{beh}}(P)}
|
||||
$$
|
||||
|
||||
— both components must be large for a perturbation to count as
|
||||
"identifying the bottleneck". Perturbations with high
|
||||
$\Delta_{\mathrm{cut}}$ but low $\mathcal{F}_{\text{beh}}$ reveal a
|
||||
bottleneck that is not behaviourally required; perturbations with high
|
||||
$\mathcal{F}_{\text{beh}}$ but low $\Delta_{\mathrm{cut}}$ reveal that
|
||||
the behavior depends on something beyond the structural bottleneck
|
||||
(e.g. timing, plasticity, neuromodulation).
|
||||
|
||||
## 5. Search Strategy
|
||||
|
||||
The search space is exponential in the number of synapses. Three
|
||||
practical strategies:
|
||||
|
||||
### 5.1 Greedy shrinking
|
||||
|
||||
Start with the full Stage VI circuit (07 §6). Remove one neuron at a
|
||||
time; keep the removal if behavior is preserved; otherwise add it back.
|
||||
This identifies a minimal sufficient circuit under a greedy heuristic.
|
||||
O(n) re-runs where n is the circuit size.
|
||||
|
||||
### 5.2 Paired-hypothesis test
|
||||
|
||||
For each candidate neuron $i$, run $N$ bouts with $i$ silenced and $N$
|
||||
baseline bouts. Compute behavior success rate difference. Bonferroni-
|
||||
corrected across candidates. O(n) re-runs with statistical rigour.
|
||||
|
||||
### 5.3 Bayesian optimisation over perturbation combinations
|
||||
|
||||
Model $\mathcal{F}(P)$ as a black-box function; use BO to select
|
||||
informative perturbations. Useful when combinations of small
|
||||
perturbations are the interesting regime. O(log n) re-runs in the
|
||||
ideal case; practically 30–100.
|
||||
|
||||
## 6. Published Comparison
|
||||
|
||||
Fly grooming circuit perturbation studies provide the ground truth
|
||||
against which CC-OS predictions are evaluated:
|
||||
|
||||
| Study | Perturbation | Behavior effect | CC-OS prediction target |
|
||||
|-------|--------------|-----------------|-------------------------|
|
||||
| Hampel et al. 2015 | silence GNG descending interneurons | antennal grooming abolished | high $\mathcal{F}_{\text{beh}}$ |
|
||||
| Seeds et al. 2014 | optogenetic activation of premotor neurons | elicits bout | `NeuronDriven` produces behavior |
|
||||
| Ramdya et al. 2017 | leg sensory lesion | gait adaptation | partial $\mathcal{F}_{\text{beh}}$ |
|
||||
| Namiki et al. 2018 | DN class ablation | specific motor deficits | class-specific $\mathcal{F}_{\text{beh}}$ |
|
||||
|
||||
CC-OS v1 should reproduce the Hampel 2015 and Seeds 2014 results in
|
||||
simulation. Replication is a falsifiable prediction, not a rhetorical
|
||||
claim.
|
||||
|
||||
## 7. Auditability
|
||||
|
||||
Every perturbation run emits a witness bundle:
|
||||
|
||||
```text
|
||||
perturbation_manifest.json {
|
||||
run_id: "pert-0042",
|
||||
baseline_manifest: "run-manifest-base.json",
|
||||
perturbation_events: ["SynapseAblated ...", "NeuronSilenced ..."],
|
||||
n_bouts: 50,
|
||||
behavior_success_rate: 0.12,
|
||||
baseline_success_rate: 0.91,
|
||||
delta_cut: 0.63,
|
||||
f_beh: 0.87,
|
||||
f: 0.74,
|
||||
bouts_sha256: "...",
|
||||
circuit_prediction: "circuit-grooming-antenna-0017.json",
|
||||
validation_status: "consistent" | "inconsistent"
|
||||
}
|
||||
```
|
||||
|
||||
Any run can be replayed from the manifest; any downstream claim that
|
||||
"circuit X causes behavior Y" has the manifest as its receipt.
|
||||
|
||||
## 8. Worked Example — Antennal Grooming
|
||||
|
||||
1. Identify candidate circuit from 03 §8 min-cut + 07 §6 Stage VI: ~40
|
||||
neurons spanning antennal bristle sensory → GNG descending → foreleg
|
||||
motor.
|
||||
2. Run baseline: 50 bouts, stimulus = 50 ms mechanosensory activation,
|
||||
behavior success rate 0.91.
|
||||
3. Greedy shrink: drop each neuron in turn; end with ~15-neuron minimal
|
||||
set where dropping any further neuron drops behavior below 0.5.
|
||||
4. Statistical confirmation: paired-hypothesis test per neuron in the
|
||||
minimal set; Bonferroni-corrected p < 0.001 for the GNG command
|
||||
neurons matching Hampel 2015.
|
||||
5. Compare CC-OS prediction to published circuit: Jaccard similarity
|
||||
should exceed 0.7 per the acceptance test (10-acceptance-test-grooming.md §6).
|
||||
6. Write `pert-0042` manifest to witness bundle.
|
||||
|
||||
## 9. Limitations
|
||||
|
||||
- **Recurrence confounds.** Highly recurrent circuits have multiple
|
||||
equivalent minimal sets. Greedy shrinking finds one; BO or exhaustive
|
||||
search may find others.
|
||||
- **Timing not captured.** The fragility score is averaged over bouts;
|
||||
perturbations that alter timing without abolishing behavior (e.g.,
|
||||
slower grooming onset) require a richer behavioral metric.
|
||||
- **Plasticity.** Acute ablation in a real fly triggers homeostatic
|
||||
compensation over hours. CC-OS v1 does not model compensation.
|
||||
- **Noise floor.** Small perturbations in a noisy system cannot be
|
||||
distinguished from baseline variability below a minimum effect size.
|
||||
|
||||
## 10. Ethical Considerations
|
||||
|
||||
Fly-scale counterfactual perturbation is a computational experiment on
|
||||
a peer-reviewed graph; no animals are harmed by CC-OS simulations.
|
||||
Extending to mouse or vertebrate connectomes raises new concerns —
|
||||
discussed in 11-risks-positioning-roadmap.md §4. The same fragility
|
||||
pipeline could be used to design adversarial perturbations of biological
|
||||
neural networks; the same witness logging that supports good science
|
||||
also makes abuse auditable.
|
||||
|
||||
## 11. References
|
||||
|
||||
1. Jazayeri, M., Afraz, A. (2017). *Navigating the neural space in
|
||||
search of the neural code.* Neuron.
|
||||
2. Hampel, S., Franconville, R., Simpson, J. H., Seeds, A. M. (2015).
|
||||
*A neural command circuit for grooming movement control.* eLife.
|
||||
3. Seeds, A. M., Ravbar, P., Chung, P., et al. (2014). *A suppression
|
||||
hierarchy among competing motor programs drives sequential grooming
|
||||
in Drosophila.* eLife.
|
||||
4. Ramdya, P., Thandiackal, R., Cherney, R., et al. (2017). *Climbing
|
||||
favours the tripod gait over alternative faster insect gaits.* Nat
|
||||
Commun.
|
||||
5. Namiki, S., Dickinson, M. H., Wong, A. M., Korff, W., Card, G. M.
|
||||
(2018). *The functional organization of descending sensory-motor
|
||||
pathways in Drosophila.* eLife.
|
||||
6. ADR-028 — ESP32 capability audit + witness verification.
|
||||
7. `ruvector-mincut` v2.0.4 — `DynamicMinCut` incremental edge updates.
|
||||
8. Ali, F., Laudet, V., Hampel, S. (2023). *Dissection of grooming
|
||||
circuit components.* Curr Biol (methods review).
|
||||
|
||||
---
|
||||
|
||||
**Next**: 09-four-layer-architecture.md — the full system spec that
|
||||
integrates Layers 1–4.
|
||||
@@ -0,0 +1,281 @@
|
||||
---
|
||||
Research Document ID: RD-C-09
|
||||
Date: 2026-04-21
|
||||
Status: Draft
|
||||
Authors: RuView Research Team
|
||||
Related ADRs: proposed ADR-084, ADR-085, ADR-086, ADR-087, ADR-088
|
||||
---
|
||||
|
||||
# RD-C-09: Four-Layer CC-OS Architecture
|
||||
|
||||
## Abstract
|
||||
|
||||
This document integrates the substrate (02), circuit analysis (03),
|
||||
neural dynamics runtime (04), cross-region attention fusion (05),
|
||||
embodied simulator (06), behavioral-episode encoding (07), and
|
||||
counterfactual perturbation (08) into a single four-layer architecture.
|
||||
We specify the proposed new workspace crates, DDD bounded contexts,
|
||||
domain events, data-flow diagram, control-plane CLI subcommands,
|
||||
end-to-end latency and memory budgets, concurrency model, observability
|
||||
conventions, and the integration points with existing RuView code. The
|
||||
architecture is designed to slot into the existing `wifi-densepose` Rust
|
||||
workspace without disturbing the publishing order or the
|
||||
cargo-test-workspace convention.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. Layer stack overview
|
||||
2. Proposed new workspace crates
|
||||
3. DDD bounded contexts
|
||||
4. Data-flow diagram
|
||||
5. Control plane — CLI subcommands
|
||||
6. Latency budget
|
||||
7. Memory budget
|
||||
8. Concurrency model
|
||||
9. Observability and witness logs
|
||||
10. Integration with existing RuView modules
|
||||
11. Build, test, and benchmark commands
|
||||
12. Non-goals
|
||||
13. References
|
||||
|
||||
---
|
||||
|
||||
## 1. Layer Stack Overview
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ Layer 4: Analysis & Adaptation │
|
||||
│ - ruvector-mincut fragility (03, 08) │
|
||||
│ - NeuralFusionArray cross-region attention (05) │
|
||||
│ - BehaviorPipeline CRV episode encoding (07) │
|
||||
│ - CoherenceGate population state (05, 07) │
|
||||
│ - Counterfactual perturbation runner (08) │
|
||||
│ Crate: wifi-densepose-ruvector (extended) │
|
||||
├──────────────────────────────────────────────────────────────────────┤
|
||||
│ Layer 3: Embodied Simulator (Closed Loop) │
|
||||
│ - Rapier physics, fly body schema, sensor/actuator adapters (06) │
|
||||
│ - Optional NeuroMechFly bridge │
|
||||
│ Crate: wifi-densepose-embody (proposed) │
|
||||
├──────────────────────────────────────────────────────────────────────┤
|
||||
│ Layer 2: Neural Dynamics Runtime │
|
||||
│ - Event-driven LIF kernel (04) │
|
||||
│ - ruvector-solver rate-code + perturbation (04 §5) │
|
||||
│ - ruvector-temporal-tensor voltage/spike storage (02 §7, 04 §6) │
|
||||
│ Crate: wifi-densepose-neuro (proposed) │
|
||||
├──────────────────────────────────────────────────────────────────────┤
|
||||
│ Layer 1: Connectome Graph Substrate │
|
||||
│ - Neuron / Synapse / Region schema (02) │
|
||||
│ - ruvector-mincut adjacency + triplets (02 §5) │
|
||||
│ - Neuron embeddings (02 §6) │
|
||||
│ Crate: wifi-densepose-connectome (proposed) │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 2. Proposed New Workspace Crates
|
||||
|
||||
| Crate | Slot | Depends on | Feature flags |
|
||||
|-------|------|-----------|---------------|
|
||||
| `wifi-densepose-connectome` | Layer 1 | `core`, `db`, `ruvector` (for mincut + temporal-tensor) | `flywire`, `micons`, `larva` |
|
||||
| `wifi-densepose-neuro` | Layer 2 | `core`, `connectome`, `ruvector` | `rate-code`, `gpu` (deferred) |
|
||||
| `wifi-densepose-embody` | Layer 3 | `core`, `neuro` | `nmf-bridge`, `vision` |
|
||||
|
||||
Publishing order (extends the list in `CLAUDE.md`):
|
||||
|
||||
```
|
||||
... → wifi-densepose-ruvector
|
||||
→ wifi-densepose-connectome (new)
|
||||
→ wifi-densepose-neuro (new)
|
||||
→ wifi-densepose-embody (new)
|
||||
→ wifi-densepose-train
|
||||
→ wifi-densepose-mat
|
||||
...
|
||||
```
|
||||
|
||||
Layer 4 capabilities extend the existing `wifi-densepose-ruvector` crate
|
||||
rather than spawning another crate: the aggregates
|
||||
(`NeuralFusionArray`, `BehaviorPipeline`, perturbation runner) live
|
||||
alongside `MultistaticArray` and `WifiCrvPipeline`.
|
||||
|
||||
## 3. DDD Bounded Contexts
|
||||
|
||||
| Context | Aggregate root | Invariants | Domain events |
|
||||
|---------|----------------|------------|---------------|
|
||||
| ConnectomeGraph | `ConnectomeGraph` | Synapses reference extant neurons; no duplicates; NT matches presynaptic neuron | `ConnectomeLoaded`, `SynapseAblated`, `SynapseRestored`, `EmbeddingRecomputed` |
|
||||
| NeuralRuntime | `SimulationRun` | Seeded RNG; ordered event tiebreaks; append-only voltage/spike logs | `SpikeObserved`, `RunStarted`, `RunCompleted`, `RunAborted` |
|
||||
| Body | `Body` + `BodyState` | Joint angles within physical limits; contact forces non-negative | `ContactOpened`, `ContactClosed`, `ActuatorSaturated`, `PhysicsDiverged` |
|
||||
| RegionFusion | `NeuralFusionArray` | Embed dim consistent; coherence gate state monotonic within a window | `FusionEmitted`, `CoherenceGateTransitioned` |
|
||||
| BehaviorEpisode | `BehaviorPipeline` (wraps `CrvSessionManager`) | Stage order (I→VI); bout id unique; coherence gate consistent | `BoutStarted`, `StageCompleted`, `BoutFinalized`, `CircuitIdentified` |
|
||||
| Perturbation | `PerturbationRunner` | Every perturbation has baseline manifest; score computed from ≥N bouts | `PerturbationApplied`, `PerturbationReverted`, `FragilityScored` |
|
||||
|
||||
## 4. Data-Flow Diagram
|
||||
|
||||
```
|
||||
stimulus / ctx
|
||||
│
|
||||
▼
|
||||
[ SensorEncoder ] ─ spike injections ─▶ [ SpikeQueue ]
|
||||
(Layer 3) │
|
||||
▼
|
||||
[ LIF kernel ]
|
||||
(Layer 2)
|
||||
│
|
||||
┌────────────────┼───────────────┐
|
||||
▼ ▼ ▼
|
||||
[ VoltageBuffer ] [ SpikeLog ] [ regionAggregator ]
|
||||
(TemporalTensor) (50 ms window)
|
||||
│ │
|
||||
│ ▼
|
||||
│ [ NeuralFusionArray ]
|
||||
│ (Layer 4 / SDPA)
|
||||
│ │
|
||||
│ ▼
|
||||
│ [ BehaviorPipeline ]
|
||||
│ (CRV stages I–VI)
|
||||
│ │
|
||||
│ ▼
|
||||
│ [ StageVI MinCut / fragility ]
|
||||
│ │
|
||||
▼ ▼
|
||||
[ MotorDecoder ] ◀─── (next-step command) ─── analysis products
|
||||
(Layer 3) │
|
||||
│ ▼
|
||||
▼ witness bundle
|
||||
[ Rapier physics ] (ADR-028 lineage)
|
||||
│
|
||||
└──── body state ─▶ SensorEncoder ... (loop)
|
||||
```
|
||||
|
||||
## 5. Control Plane — CLI Subcommands
|
||||
|
||||
Proposed additions to the `wifi-densepose` CLI binary:
|
||||
|
||||
| Subcommand | Purpose |
|
||||
|------------|---------|
|
||||
| `brain load <connectome.bin>` | Load a connectome into the store; emits `ConnectomeLoaded` with SHA-256 |
|
||||
| `brain simulate --duration <s> --subgraph <selector> [--seed N]` | Run closed-loop simulation, produce witness bundle |
|
||||
| `brain ablate <spec.json>` | Apply perturbation events; idempotent given the manifest |
|
||||
| `brain fragility --baseline <manifest> --perturbation <spec>` | Compute $\mathcal{F}(P)$ from 08 |
|
||||
| `brain replay <manifest.json>` | Deterministic replay; SHA-256 verification |
|
||||
| `brain inspect <run-id>` | Pretty-print witness bundle, cuts, fragility |
|
||||
|
||||
## 6. Latency Budget
|
||||
|
||||
End-to-end for 100 Hz closed-loop control with 50k-neuron subgraph:
|
||||
|
||||
| Stage | Target | Worst-case |
|
||||
|-------|--------|------------|
|
||||
| Sensor read + encoding | 0.5 ms | 1 ms |
|
||||
| Spike queue insert | 0.1 ms | 0.3 ms |
|
||||
| LIF kernel (10 ms window) | 5 ms | 8 ms |
|
||||
| Fusion (50 ms cadence, amortised) | 0.5 ms | 1.5 ms |
|
||||
| CRV episode update (infrequent) | 1 ms | 3 ms |
|
||||
| Motor decoding | 0.2 ms | 0.5 ms |
|
||||
| Physics step (1 kHz) | 1 ms | 2 ms |
|
||||
| **End-to-end** | **< 10 ms** | **< 16 ms** |
|
||||
|
||||
1 kHz aspirational target requires reduced subgraph (10k–15k), batched
|
||||
fusion (every 10 ms), and no-alloc inner loops throughout.
|
||||
|
||||
## 7. Memory Budget
|
||||
|
||||
| Scale | Graph | Voltage (60s, tiered) | Spike log | Embeddings | Total |
|
||||
|-------|-------|------------------------|-----------|------------|-------|
|
||||
| 10k | 10 MB | 400 MB | 50 MB | 5 MB | ~0.5 GB |
|
||||
| 50k | 48 MB | 3.0 GB | 240 MB | 26 MB | ~4 GB |
|
||||
| 139k | 1.3 GB (synapses) | 8 GB | 700 MB | 71 MB | ~12 GB |
|
||||
|
||||
Laptop-grade (32 GB) handles 50k comfortably. Full FlyWire at 139k fits
|
||||
in 32 GB but leaves little headroom; 64 GB recommended.
|
||||
|
||||
## 8. Concurrency Model
|
||||
|
||||
| Layer | Concurrency | Framework |
|
||||
|-------|-------------|-----------|
|
||||
| 1 (graph) | Immutable after load; readers lock-free | `arc-swap` or `parking_lot::RwLock` |
|
||||
| 2 inner LIF | Single-threaded spike queue, rayon fan-out per time slot | `rayon` |
|
||||
| 2 I/O | Async | `tokio` |
|
||||
| 3 physics | Single-threaded tight loop | Rapier native |
|
||||
| 3 ingress / egress | Async | `tokio` |
|
||||
| 4 analysis | Async, read-only access | `tokio` |
|
||||
|
||||
Critical path (LIF + physics) is single-threaded per simulation
|
||||
instance for determinism. Multiple instances (e.g. for
|
||||
parallel-perturbation sweeps) run as independent processes.
|
||||
|
||||
## 9. Observability and Witness Logs
|
||||
|
||||
Every run, bout, and perturbation produces a manifest compatible with
|
||||
the ADR-028 witness-bundle convention. A run's witness bundle includes:
|
||||
|
||||
- `connectome_sha256` (Layer 1)
|
||||
- `run_manifest.json` (Layer 2)
|
||||
- Voltage + spike compressed segments (Layer 2)
|
||||
- `bout_manifest.json` per bout (Layer 3)
|
||||
- `episode_manifest.json` per episode (Layer 4)
|
||||
- `perturbation_manifest.json` per perturbation (Layer 4)
|
||||
- `VERIFY.sh` self-verification script (ADR-028 style)
|
||||
|
||||
Bundle tarball: `dist/witness-bundle-CCOS-<run-id>-<sha>.tar.gz`.
|
||||
|
||||
## 10. Integration with Existing RuView Modules
|
||||
|
||||
| Existing module | Reused how |
|
||||
|-----------------|-----------|
|
||||
| `viewpoint/fusion.rs` `MultistaticArray` | Shape template for `NeuralFusionArray` |
|
||||
| `viewpoint/coherence.rs` `CoherenceGate` | Reused verbatim with new coherence definition |
|
||||
| `viewpoint/geometry.rs` `GeometricDiversityIndex` | Shape template for FDI |
|
||||
| `crv/mod.rs` `WifiCrvPipeline` | Shape template for `BehaviorPipeline` |
|
||||
| `signal/subcarrier.rs` virtual source/sink pattern | Reused for connectome min-cut |
|
||||
| `signal/spectrogram.rs` attention-gated mincut | Reused for behavior-conditioned cuts |
|
||||
| `mat/breathing.rs` `CompressedBreathingBuffer` | Pattern for `VoltageBuffer` |
|
||||
| `mat/triangulation.rs` Neumann solver usage | Pattern for rate-code and perturbation solves |
|
||||
| `scripts/generate-witness-bundle.sh` | Extended with CC-OS artifacts |
|
||||
| ADR-028 witness convention | Applied verbatim |
|
||||
|
||||
## 11. Build, Test, and Benchmark Commands
|
||||
|
||||
```bash
|
||||
# Workspace test (add new crates to the existing convention)
|
||||
cd rust-port/wifi-densepose-rs
|
||||
cargo test --workspace --no-default-features
|
||||
|
||||
# Targeted tests
|
||||
cargo test -p wifi-densepose-connectome --no-default-features
|
||||
cargo test -p wifi-densepose-neuro --no-default-features
|
||||
cargo test -p wifi-densepose-embody --no-default-features
|
||||
|
||||
# Benchmarks (Criterion)
|
||||
cargo bench -p wifi-densepose-neuro --bench lif_kernel_throughput
|
||||
cargo bench -p wifi-densepose-neuro --bench voltage_buffer_compression
|
||||
|
||||
# End-to-end acceptance
|
||||
cargo test -p wifi-densepose-embody --test grooming_acceptance
|
||||
|
||||
# Witness bundle generation (extended)
|
||||
bash scripts/generate-witness-bundle.sh --ccos
|
||||
```
|
||||
|
||||
## 12. Non-Goals
|
||||
|
||||
- **Real-time human-scale simulation.**
|
||||
- **Cloud-distributed multi-node simulation in v1.** Single workstation.
|
||||
- **GPU-accelerated LIF in v1.** Deferred to v2.
|
||||
- **Live web visualisation in v1.** CLI + witness bundle only.
|
||||
- **Replace the RF sensing pipeline.** CC-OS is additive.
|
||||
|
||||
## 13. References
|
||||
|
||||
1. Eric Evans, *Domain-Driven Design* (2003).
|
||||
2. ADR-017 — RuVector signal + MAT integration (DDD pattern).
|
||||
3. ADR-028 — ESP32 capability audit + witness verification.
|
||||
4. RuVector v2.0.4 documentation.
|
||||
5. `CLAUDE.md` — project configuration, crate publishing order.
|
||||
6. Dorkenwald, S., et al. (2024). *Neuronal wiring diagram of an adult
|
||||
brain.* Nature.
|
||||
|
||||
---
|
||||
|
||||
**Next**: 10-acceptance-test-grooming.md — the concrete pass/fail test
|
||||
that validates this architecture.
|
||||
@@ -0,0 +1,279 @@
|
||||
---
|
||||
Research Document ID: RD-C-10
|
||||
Date: 2026-04-21
|
||||
Status: Draft
|
||||
Authors: RuView Research Team
|
||||
Related ADRs: proposed ADR-084 through ADR-088
|
||||
---
|
||||
|
||||
# RD-C-10: Acceptance Test — Antennal Grooming
|
||||
|
||||
## Abstract
|
||||
|
||||
The user-specified acceptance criterion for CC-OS is:
|
||||
*"A 10,000 to 50,000 neuron subgraph runs in closed loop with a virtual
|
||||
body, produces one reproducible behavior, and RuVector correctly
|
||||
identifies the minimal circuit boundary whose perturbation breaks that
|
||||
behavior."* This document turns that criterion into a concrete,
|
||||
executable test. We select antennal grooming as the target behavior
|
||||
because it has the most mature circuit-level characterisation in the fly
|
||||
literature (Hampel et al. 2015; Seeds et al. 2014). We specify the
|
||||
subgraph-extraction procedure from FlyWire, the stimulus protocol, the
|
||||
four pass/fail criteria (behavior-present, reproducibility,
|
||||
minimal-cut match, fragility significance), the Rust test harness, the
|
||||
witness-bundle contents, the CI budget, and the publishable artifacts.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. Test objective
|
||||
2. Behavior choice
|
||||
3. Subgraph selection
|
||||
4. Stimulus design
|
||||
5. Success criteria
|
||||
6. Metrics and thresholds
|
||||
7. Test harness
|
||||
8. Witness bundle
|
||||
9. CI constraints
|
||||
10. Publishable artifacts
|
||||
11. Failure analysis and remediation
|
||||
12. References
|
||||
|
||||
---
|
||||
|
||||
## 1. Test Objective
|
||||
|
||||
Verbatim from the compendium thesis:
|
||||
|
||||
> A 10,000 to 50,000 neuron subgraph runs in closed loop with a virtual
|
||||
> body, produces one reproducible behavior, and RuVector correctly
|
||||
> identifies the minimal circuit boundary whose perturbation breaks that
|
||||
> behavior.
|
||||
|
||||
Four independent claims, each individually falsifiable:
|
||||
|
||||
1. **Closed loop runs at scale.** A subgraph of 10k–50k neurons runs
|
||||
with the embodied simulator.
|
||||
2. **Behavior is produced.** A canonical stimulus reliably elicits
|
||||
antennal grooming.
|
||||
3. **Behavior is reproducible.** Same seed + same config → byte-equal
|
||||
voltage traces.
|
||||
4. **Minimal cut is correct.** `ruvector-mincut` identifies a circuit
|
||||
whose perturbation abolishes the behavior, with the cut matching
|
||||
published circuit dissection within a specified Jaccard threshold.
|
||||
|
||||
## 2. Behavior Choice
|
||||
|
||||
| Behavior | Published circuit | Body requirement | Recommended? |
|
||||
|----------|-------------------|------------------|--------------|
|
||||
| Antennal grooming | Hampel 2015, Seeds 2014 | moderate (foreleg + antenna) | **Yes (v1 target)** |
|
||||
| Walking | Cruse Walknet lineage | high (6 legs coordinated) | v2 |
|
||||
| Feeding | Itakura et al. 2018 | high (proboscis + foregut) | v2 |
|
||||
| Courtship | Pan & Baker 2014 | very high (multi-modal) | v3 |
|
||||
| Freezing | Zacarias et al. 2018 | low | alternate v1 |
|
||||
|
||||
Antennal grooming wins on three counts: well-characterised command
|
||||
circuit, modest biomechanical requirements, and short-duration bouts
|
||||
(~500 ms) that fit CI time budgets.
|
||||
|
||||
## 3. Subgraph Selection
|
||||
|
||||
Starting from FlyWire (Dorkenwald et al. 2024), extract a subgraph
|
||||
that contains:
|
||||
|
||||
1. Antennal bristle mechanosensory neurons (~1,300, FlyWire `super_class
|
||||
== "Sensory"` + `side == "Left/Right"` + region includes antennal
|
||||
mechanosensory and motor (AMMC) and gnathal ganglion (GNG)).
|
||||
2. GNG descending interneurons (~150, Hampel 2015 reference; FlyWire
|
||||
`super_class == "Descending"` intersected with GNG region).
|
||||
3. Prothoracic leg motor neurons (~50, FlyWire `super_class == "Motor"`
|
||||
+ `region == "VNC prothoracic"`).
|
||||
4. Three-hop neighborhood of the above (to capture modulatory inputs).
|
||||
|
||||
Typical yield: 15k–30k neurons, 300k–1M synapses. Comfortably inside
|
||||
the 10k–50k band specified by the test.
|
||||
|
||||
Extraction script (non-normative Python over FlyWire API):
|
||||
|
||||
```python
|
||||
# pseudo-code — real implementation lives in wifi-densepose-connectome
|
||||
core = flywire.query(super_class=["Sensory","Descending","Motor"],
|
||||
region=["AMMC","GNG","VNC_proT"])
|
||||
expanded = flywire.k_hop(core, k=3, min_weight=5)
|
||||
graph = flywire.induced_subgraph(expanded)
|
||||
graph.export("flywire_grooming_subgraph.bin")
|
||||
```
|
||||
|
||||
## 4. Stimulus Design
|
||||
|
||||
```text
|
||||
for each trial in 0..N:
|
||||
at t=0: activate antennal bristle pool with rate = 200 Hz for 50 ms
|
||||
observe from t=-100 ms to t=1000 ms
|
||||
detect antennal sweep: foreleg-tibia contacts antenna with force > threshold
|
||||
record:
|
||||
- success: boolean (did sweep occur within window?)
|
||||
- latency: time to first contact (ms)
|
||||
- sweeps: integer count
|
||||
- spikes: full SpikeLog
|
||||
- voltages: full VoltageBuffer
|
||||
```
|
||||
|
||||
Trial count $N = 50$ per configuration. Two stimulus sides (left/right
|
||||
antenna). Baseline condition = unperturbed graph.
|
||||
|
||||
## 5. Success Criteria
|
||||
|
||||
| # | Claim | Criterion | Threshold |
|
||||
|---|-------|-----------|-----------|
|
||||
| C1 | Behavior present | success rate in baseline trials | ≥ 0.80 |
|
||||
| C2 | Reproducibility | SHA-256 of voltage traces equal across two runs with same seed | 100% match (exact) |
|
||||
| C3 | Minimal-cut match | Jaccard similarity between `ruvector-mincut` predicted circuit (03 §8) and greedy-shrunk minimal sufficient circuit (08 §5) | ≥ 0.7 |
|
||||
| C4 | Fragility significance | behavior success rate drop from baseline when predicted cut is ablated, relative to same-size random ablation | ablated ≤ 0.4; random ≥ 0.7; p < 0.01 (Fisher's exact) |
|
||||
|
||||
All four must pass for the acceptance test to pass.
|
||||
|
||||
## 6. Metrics and Thresholds
|
||||
|
||||
| Metric | Definition | Pass threshold |
|
||||
|--------|-----------|---------------|
|
||||
| $P_{\mathrm{base}}$ | Fraction of baseline trials with antennal sweep | ≥ 0.80 |
|
||||
| $P_{\mathrm{cut}}$ | Fraction of trials with sweep after ablating predicted cut | ≤ 0.40 |
|
||||
| $P_{\mathrm{rand}}$ | Fraction of trials with sweep after ablating random edges of same size | ≥ 0.70 |
|
||||
| $J$ | Jaccard( predicted-cut-neurons, empirical-minimal-circuit ) | ≥ 0.7 |
|
||||
| $\Delta t_{\mathrm{latency}}$ | Mean latency change ablated vs baseline | significant? (Welch's t, p<0.05) |
|
||||
| $SHA$ | Run output hash | reproducible under same seed |
|
||||
| $t_{\mathrm{wall}}$ | Wall-clock per trial | ≤ 12 s on 16-core x86 |
|
||||
|
||||
## 7. Test Harness
|
||||
|
||||
`rust-port/wifi-densepose-rs/crates/wifi-densepose-embody/tests/grooming_acceptance.rs`:
|
||||
|
||||
```rust
|
||||
// non-normative sketch
|
||||
#[test]
|
||||
fn grooming_acceptance() {
|
||||
let cfg = AcceptanceConfig::default();
|
||||
let graph = ConnectomeGraph::load(&cfg.connectome_path).unwrap();
|
||||
|
||||
// Baseline
|
||||
let baseline = run_trials(&cfg, &graph, Perturbation::None);
|
||||
assert!(baseline.success_rate() >= 0.80, "C1 failed");
|
||||
|
||||
// Reproducibility
|
||||
let replay = run_trials(&cfg, &graph, Perturbation::None);
|
||||
assert_eq!(baseline.sha256(), replay.sha256(), "C2 failed");
|
||||
|
||||
// Predicted cut from ruvector-mincut
|
||||
let cut = predict_cut(&graph, &baseline);
|
||||
|
||||
// Empirical minimal circuit via greedy shrink
|
||||
let empirical = greedy_shrink(&cfg, &graph, &baseline);
|
||||
|
||||
// Jaccard match
|
||||
let j = jaccard(&cut.neurons, &empirical.neurons);
|
||||
assert!(j >= 0.7, "C3 failed: J = {}", j);
|
||||
|
||||
// Fragility significance
|
||||
let ablated = run_trials(&cfg, &graph, Perturbation::Ablate(&cut));
|
||||
let random = run_trials(&cfg, &graph, Perturbation::AblateRandom(cut.size()));
|
||||
assert!(ablated.success_rate() <= 0.40, "C4 failed (ablated)");
|
||||
assert!(random.success_rate() >= 0.70, "C4 failed (random)");
|
||||
assert!(fishers_exact(&ablated, &random) < 0.01, "C4 failed (p-value)");
|
||||
|
||||
write_witness_bundle(&baseline, &replay, &ablated, &random,
|
||||
&cut, &empirical, "dist/witness-bundle-CCOS-grooming.tar.gz");
|
||||
}
|
||||
```
|
||||
|
||||
Python cross-check (`v1/data/proof/verify_ccos.py` following the ADR-011
|
||||
proof pattern) hashes the baseline voltage trace and asserts a published
|
||||
expected SHA-256.
|
||||
|
||||
## 8. Witness Bundle
|
||||
|
||||
Bundle name: `dist/witness-bundle-CCOS-grooming-<sha>.tar.gz`
|
||||
|
||||
Contents:
|
||||
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `manifests/baseline_run.json` | Layer-2 run manifest |
|
||||
| `manifests/replay_run.json` | Reproducibility replay manifest |
|
||||
| `manifests/ablation_run.json` | Perturbation manifest (predicted cut) |
|
||||
| `manifests/random_run.json` | Perturbation manifest (random) |
|
||||
| `circuits/predicted_cut.json` | `ruvector-mincut` output |
|
||||
| `circuits/empirical_minimal.json` | Greedy-shrink output |
|
||||
| `traces/voltage_*.ttseg` | Compressed voltage traces |
|
||||
| `traces/spikes_*.splog` | Spike logs |
|
||||
| `results/success_rates.csv` | Per-trial outcomes |
|
||||
| `results/jaccard.json` | J computation |
|
||||
| `results/fisher_exact.json` | p-value computation |
|
||||
| `VERIFY.sh` | Self-verification script (ADR-028 pattern) |
|
||||
| `EXPECTED_SHA256.txt` | Published reference hashes |
|
||||
| `README.md` | How to reproduce |
|
||||
|
||||
## 9. CI Constraints
|
||||
|
||||
- **No GPU required.** Pure CPU test.
|
||||
- **Wall-clock budget**: ≤ 10 minutes on 16-core x86_64, 32 GB RAM.
|
||||
- **Determinism**: same seed must reproduce every manifest SHA-256
|
||||
across CI runners.
|
||||
- **Network**: may require one-time FlyWire subgraph download; cached
|
||||
after first run.
|
||||
- **Integration**: extends `scripts/generate-witness-bundle.sh`
|
||||
(currently ADR-028 focused) with a `--ccos` flag.
|
||||
|
||||
With 50 trials × 10 conditions × ~0.5 s per trial at reduced subgraph,
|
||||
total simulation time is ≲ 5 minutes. Budget holds.
|
||||
|
||||
## 10. Publishable Artifacts
|
||||
|
||||
Figures that should be produced automatically from a passing run:
|
||||
|
||||
1. **Fragility curve**: $P_{\mathrm{success}}$ vs cut weight removed (0
|
||||
to 100%); expect sigmoid.
|
||||
2. **Minimal-cut visualisation**: 3D scatter of neurons colored by
|
||||
membership in the Jaccard intersection.
|
||||
3. **Latency histogram**: sweep latency baseline vs perturbed.
|
||||
4. **Coherence gate trace**: population coherence across a typical
|
||||
bout.
|
||||
5. **Jaccard vs number of neurons ablated**: shows CC-OS prediction
|
||||
converging to empirical as ablation grows.
|
||||
|
||||
## 11. Failure Analysis and Remediation
|
||||
|
||||
| Failing criterion | Likely cause | Remediation |
|
||||
|-------------------|--------------|-------------|
|
||||
| C1 low success rate | Subgraph missing key modulatory inputs; LIF parameters off | Expand k-hop to 4; tune Izhikevich parameters from published values |
|
||||
| C2 hash mismatch | Non-determinism in Rayon fan-out or RNG | Audit for `par_iter` without sorted reduction; enforce seed propagation |
|
||||
| C3 low Jaccard | Edge weights not reflecting functional flow | Switch to activity-weighted edges (02 §5, 03 §3) |
|
||||
| C4 random ablation too effective | Subgraph too small; too many critical edges | Expand subgraph; verify control circuits are present |
|
||||
| Wall-clock overrun | LIF kernel under-optimised | Profile, add SIMD, drop non-contributing neurons before sim |
|
||||
|
||||
## 12. References
|
||||
|
||||
1. Dorkenwald, S., et al. (2024). *Neuronal wiring diagram of an adult
|
||||
brain.* Nature (FlyWire).
|
||||
2. Hampel, S., et al. (2015). *A neural command circuit for grooming
|
||||
movement control.* eLife.
|
||||
3. Seeds, A. M., et al. (2014). *A suppression hierarchy among
|
||||
competing motor programs drives sequential grooming in Drosophila.*
|
||||
eLife.
|
||||
4. Zacarias, R., Namiki, S., Card, G. M., Vasconcelos, M. L., Moita,
|
||||
M. A. (2018). *Speed-dependent descending control of freezing
|
||||
behaviour in Drosophila.* Nat Commun.
|
||||
5. Pan, Y., Baker, B. S. (2014). *Genetic identification and separation
|
||||
of innate and experience-dependent courtship behaviours in
|
||||
Drosophila.* Cell.
|
||||
6. Itakura, Y., Kohsaka, H., Ohyama, T., et al. (2018). *Identification
|
||||
of inhibitory premotor interneurons activated at a late phase in a
|
||||
motor cycle during larval locomotion.* PLOS ONE.
|
||||
7. ADR-011 — Python proof of reality (hash verification pattern).
|
||||
8. ADR-028 — Witness bundle convention.
|
||||
|
||||
---
|
||||
|
||||
**Next**: 11-risks-positioning-roadmap.md — what not to claim, and how
|
||||
to land the work in the ADR corpus.
|
||||
@@ -0,0 +1,344 @@
|
||||
---
|
||||
Research Document ID: RD-C-11
|
||||
Date: 2026-04-21
|
||||
Status: Draft
|
||||
Authors: RuView Research Team
|
||||
Related ADRs: proposed ADR-084 through ADR-088
|
||||
---
|
||||
|
||||
# RD-C-11: Risks, Positioning, and Roadmap
|
||||
|
||||
## Abstract
|
||||
|
||||
This is the governance document for the compendium. It specifies what
|
||||
CC-OS is allowed to claim in public artifacts, what it must never
|
||||
claim, the dual-use and ethics considerations implied by running the
|
||||
same counterfactual-perturbation pipeline on any connectome,
|
||||
comparisons against prior art (Eon, OpenWorm, Blue Brain, Brian2 +
|
||||
NeuroMechFly, BrainScaleS), the decision rubric from the originating
|
||||
proposal, full drafts of five proposed ADRs (ADR-084 through ADR-088),
|
||||
and a five-phase implementation roadmap with phase gates, success
|
||||
KPIs, and out-of-scope items for v1. It closes with a publication plan
|
||||
that prioritises honest scope-limited claims over headline-chasing.
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. Scientific positioning — CC-OS
|
||||
2. Hype and ethics risk matrix
|
||||
3. Scope limits
|
||||
4. Security / dual-use considerations
|
||||
5. Comparison with prior art
|
||||
6. Decision rubric
|
||||
7. Proposed ADRs (ADR-084 through ADR-088)
|
||||
8. Five-phase implementation roadmap
|
||||
9. Success KPIs per phase
|
||||
10. Out-of-scope items for v1
|
||||
11. Publication and talk plan
|
||||
12. Appendix A: 200-word short-form summary
|
||||
13. References
|
||||
|
||||
---
|
||||
|
||||
## 1. Scientific Positioning — CC-OS
|
||||
|
||||
**CC-OS is a coherence-aware connectome operating system.** It loads a
|
||||
published connectome, simulates its neural dynamics at millisecond
|
||||
resolution, closes the sensorimotor loop through an embodied simulator,
|
||||
and provides auditable counterfactual analysis of the resulting
|
||||
behavior. It is a platform for circuit science, not a model of mind.
|
||||
|
||||
What CC-OS **is**:
|
||||
|
||||
- a graph-native connectome runtime;
|
||||
- a Rust event-driven LIF engine at 10k–139k neuron scale;
|
||||
- an embodied sensorimotor loop closed in real time;
|
||||
- a structural + behavioral fragility pipeline;
|
||||
- an auditable, witness-logged, reproducible platform.
|
||||
|
||||
What CC-OS **is not**:
|
||||
|
||||
- not a mind, not a subject, not a locus of experience;
|
||||
- not a whole-brain emulation in the Sandberg–Bostrom sense;
|
||||
- not a replacement for NEURON, Brian2, Nengo, or Blue Brain;
|
||||
- not a claim about consciousness, identity, or memory continuity.
|
||||
|
||||
## 2. Hype and Ethics Risk Matrix
|
||||
|
||||
| Claim | Risk | Mitigation |
|
||||
|-------|------|------------|
|
||||
| "Mind upload" / "digital consciousness" | VERY HIGH | Never use in any RuView artifact. Formal prohibition. |
|
||||
| "Simulated fly brain" (unqualified) | HIGH | Use only with "structural + dynamical model of a subgraph of the adult Drosophila connectome, not a whole organism" qualifier. |
|
||||
| "Fly running on a laptop" | MEDIUM | Acceptable if accompanied by scope caveat and acceptance-test reference. |
|
||||
| "Coherence-aware connectome OS" | LOW | Preferred framing. |
|
||||
| "Auditable circuit discovery" | LOW | Core value proposition; encouraged. |
|
||||
| "Counterfactual circuit fragility scoring" | LOW | Novel, defensible, auditable. |
|
||||
| "Digital fly companion / pet" | HIGH | Avoid anthropomorphic framing; simulator, not companion. |
|
||||
| "Road to mammalian minds" | VERY HIGH | Out of scope; do not gesture at. |
|
||||
|
||||
Public posts, READMEs, and talk abstracts must pass a checklist that
|
||||
rejects the VERY HIGH and HIGH items above by default.
|
||||
|
||||
## 3. Scope Limits
|
||||
|
||||
- **v1 organism**: adult *Drosophila melanogaster* only.
|
||||
- **v2 candidates**: larval *Drosophila*, larval zebrafish (when
|
||||
proofread connectomes ship).
|
||||
- **Out of scope**: mouse, rat, primate, human connectomes — pending
|
||||
external ethics review and a dedicated ADR.
|
||||
- **Out of scope**: clinical / medical / patient-facing applications.
|
||||
- **Out of scope**: consciousness or phenomenal-experience claims.
|
||||
- **Out of scope**: real-time human-scale simulation.
|
||||
|
||||
## 4. Security / Dual-Use Considerations
|
||||
|
||||
The same pipeline that discovers behaviour-responsible circuits can be
|
||||
used to design perturbations that *abolish* behavior. At fly scale this
|
||||
is a research tool; at any biological-organism scale it is an
|
||||
experimental design, not an action. The dual-use risk is structurally
|
||||
mitigated by the witness-log property: every perturbation has a
|
||||
manifest with an SHA-256 fingerprint, making post-hoc audit feasible
|
||||
regardless of author intent. Guidelines:
|
||||
|
||||
- CC-OS runs shall emit witness bundles for any perturbation sweep.
|
||||
- CC-OS shall not accept non-published connectomes (no BYO-connectome
|
||||
in v1 to prevent unaudited organism targeting).
|
||||
- Perturbation recipes shall be version-pinned and hashed so that a
|
||||
published paper's experimental design is reproducible.
|
||||
|
||||
## 5. Comparison with Prior Art
|
||||
|
||||
| System | Scale | Lang | Graph-native | Coherence-gated | Counterfactual | Witness-audit | Open-source |
|
||||
|--------|-------|------|--------------|-----------------|----------------|---------------|-------------|
|
||||
| OpenWorm | 302 (C. elegans) | C++ / Python | Partial | No | No | No | Yes |
|
||||
| NEURON | Arbitrary | C / Python | No (biophysics) | No | Manual | No | Yes |
|
||||
| Brian2 | Arbitrary | Python | Partial | No | Manual | No | Yes |
|
||||
| Nengo | Arbitrary | Python | No (NEF) | No | Manual | No | Yes |
|
||||
| NeuroMechFly + Kakaria-LIF | ~10k | Python | Partial | No | No | No | Partial |
|
||||
| Blue Brain Nexus | 10^6+ cortex | C++ / Python | Partial | No | No | Partial | No |
|
||||
| BrainScaleS | 10^5 (hardware) | Hardware | No | No | Yes (hardware) | No | No |
|
||||
| "Eon"-style stack | ~100k | Python | Partial | No | No | No | No |
|
||||
| **CC-OS (this compendium)** | 10k–139k | **Rust** | **Yes** | **Yes** | **Yes** | **Yes (ADR-028)** | **Yes** |
|
||||
|
||||
The differentiation is not scale or biophysical fidelity; it is the
|
||||
combination of graph-native storage + coherence gating + counterfactual
|
||||
fragility + witness audit + Rust.
|
||||
|
||||
## 6. Decision Rubric
|
||||
|
||||
From the originating proposal (user quote, verbatim structure):
|
||||
|
||||
| Dimension | Verdict |
|
||||
|-----------|---------|
|
||||
| Feasibility today | HIGH |
|
||||
| Novelty with RuVector | HIGH |
|
||||
| Scientific validity if carefully positioned | MEDIUM–HIGH |
|
||||
| Risk of hype if framed as "mind upload" | VERY HIGH |
|
||||
| Best positioning | Embodied connectome simulation + coherence analysis |
|
||||
|
||||
We accept this rubric and commit to the "best positioning" in all
|
||||
external artifacts.
|
||||
|
||||
## 7. Proposed ADRs
|
||||
|
||||
Five ADRs land this compendium in the project's decision record. ADR
|
||||
numbers are reserved starting at **ADR-084** (the last existing ADR
|
||||
before this compendium is **ADR-081**; ADR-082 and ADR-083 are reserved
|
||||
for unrelated in-flight work).
|
||||
|
||||
Each ADR follows the existing RuView format: Status, Context,
|
||||
Decision, Consequences, Links.
|
||||
|
||||
### 7.1 ADR-084 — Connectome Graph Substrate (Layer 1)
|
||||
|
||||
- **Status**: Proposed
|
||||
- **Context**: CC-OS requires a typed, provenance-tagged,
|
||||
performance-adequate graph substrate for 10k–139k neuron connectomes
|
||||
with 1M–60M synapses. Existing `wifi-densepose-db` is row-store
|
||||
oriented; existing `wifi-densepose-ruvector` is signal/MAT focused.
|
||||
Neither covers the connectome use case directly.
|
||||
- **Decision**: Introduce `wifi-densepose-connectome` crate implementing
|
||||
the `ConnectomeGraph` aggregate (02-connectome-graph-substrate.md
|
||||
§10), `Neuron` / `Synapse` / `Region` value objects, and adapters to
|
||||
`ruvector-mincut` edge triplets and `ruvector-temporal-tensor`
|
||||
per-neuron voltage buffers. FlyWire loader first; MICrONS and larval
|
||||
*Drosophila* loaders behind feature flags.
|
||||
- **Consequences** (positive): graph-native circuit queries;
|
||||
provenance-tagged synapses; witness-log-compatible. (negative):
|
||||
adds a new crate to maintain; widens `NodeId` to `u64` in a new
|
||||
namespace. (neutral): publishing order changes.
|
||||
- **Links**: RD-C-02, RD-C-03.
|
||||
|
||||
### 7.2 ADR-085 — Neural Dynamics Runtime (Layer 2)
|
||||
|
||||
- **Status**: Proposed
|
||||
- **Context**: CC-OS requires an event-driven LIF runtime at 1 kHz
|
||||
real-time for 50k-neuron subgraphs, with deterministic replay and
|
||||
compressed voltage/spike storage.
|
||||
- **Decision**: Introduce `wifi-densepose-neuro` crate
|
||||
(04-neural-dynamics-runtime.md §10). Uses `ruvector-solver` for
|
||||
rate-code and perturbation solves; `ruvector-temporal-tensor` for
|
||||
voltage and spike storage; `ruvector-attention` for motif queries.
|
||||
Inner loop single-threaded for determinism; rayon fan-out per time
|
||||
slot for throughput.
|
||||
- **Consequences**: full cross-region LIF runtime available to the
|
||||
workspace; reuses RuVector patterns without duplication; introduces
|
||||
`rand_chacha` and `rayon` dependencies to a new crate.
|
||||
- **Links**: RD-C-04, RD-C-05.
|
||||
|
||||
### 7.3 ADR-086 — Embodied Simulator Closed Loop (Layer 3)
|
||||
|
||||
- **Status**: Proposed
|
||||
- **Context**: Without a body, circuit dynamics do not produce
|
||||
behavior. CC-OS needs a deterministic Rust-native inner-loop body
|
||||
simulator at 1 kHz physics / 100 Hz control, with an optional bridge
|
||||
to NeuroMechFly for biomechanical validation.
|
||||
- **Decision**: Introduce `wifi-densepose-embody` crate using Rapier
|
||||
for physics and a hand-authored minimal fly body (41 DoF). Optional
|
||||
`nmf-bridge` feature for NeuroMechFly cross-validation.
|
||||
- **Consequences**: tight in-proc Rust loop; no Python dependency on
|
||||
the critical path; validation story intact via optional bridge;
|
||||
`rapier3d` becomes a workspace dependency.
|
||||
- **Links**: RD-C-06.
|
||||
|
||||
### 7.4 ADR-087 — CRV Behavioral Episodes + Coherence Gating (Layer 4)
|
||||
|
||||
- **Status**: Proposed
|
||||
- **Context**: Behaviors are episodic. CC-OS needs a bout-level
|
||||
encoding that is reproducible, queryable, and integrates with the
|
||||
existing `ruvector-crv` six-stage protocol and `CoherenceGate`.
|
||||
- **Decision**: Implement `BehaviorPipeline` inside the existing
|
||||
`wifi-densepose-ruvector` crate as a sibling of `WifiCrvPipeline`.
|
||||
Map CRV stages I–VI to behavior-class / neural-sensory-feature /
|
||||
body-pose-sketch / coherence-gate-state / circuit-query / min-cut
|
||||
respectively (07-coherence-crv-behavioral-episodes.md §2).
|
||||
- **Consequences**: six-stage bout encoding for free via
|
||||
`CrvSessionManager`; Stage VI's MinCut directly yields
|
||||
behavior-responsible circuits; cross-subject convergence via
|
||||
`find_convergence` retargeted to `behavior_class`.
|
||||
- **Links**: RD-C-07, RD-C-05.
|
||||
|
||||
### 7.5 ADR-088 — Governance, Positioning, and Counterfactual Protocol
|
||||
|
||||
- **Status**: Proposed
|
||||
- **Context**: Running connectome dynamics with counterfactual
|
||||
perturbation invites both scientific mis-statement ("mind upload")
|
||||
and dual-use misuse. A governance ADR fixes the framing, the
|
||||
allowed-claim boundaries, and the perturbation-audit protocol.
|
||||
- **Decision**:
|
||||
1. CC-OS is positioned as a "coherence-aware connectome operating
|
||||
system". The terms "mind upload", "digital consciousness", and
|
||||
their synonyms are prohibited in RuView public artifacts.
|
||||
2. Counterfactual perturbations (08-counterfactual-perturbation.md)
|
||||
must emit witness bundles compatible with the ADR-028 convention.
|
||||
3. Only published, peer-reviewed connectomes may be loaded in v1
|
||||
(FlyWire, MICrONS partial).
|
||||
4. Mammalian connectomes are out of scope for v1 without external
|
||||
ethics review.
|
||||
- **Consequences**: clear framing for public communications; dual-use
|
||||
risk structurally mitigated by audit requirements; mammalian
|
||||
exploration is gated behind process, not policy.
|
||||
- **Links**: RD-C-11 (this document), RD-C-08, RD-C-01.
|
||||
|
||||
## 8. Five-Phase Implementation Roadmap
|
||||
|
||||
| Phase | Weeks | Deliverables | Gates |
|
||||
|-------|-------|--------------|-------|
|
||||
| 1 — Connectome import | 1–4 | `wifi-densepose-connectome` crate; FlyWire loader; graph storage and query benchmarks | Load 139k-neuron FlyWire in < 10 s; k-hop query < 10 ms |
|
||||
| 2 — LIF runtime | 5–10 | `wifi-densepose-neuro` crate; event-driven kernel; voltage + spike storage; deterministic replay | 1 kHz real-time for 10k-neuron subgraph; replay SHA-256 reproducible |
|
||||
| 3 — Closed loop | 11–16 | `wifi-densepose-embody` crate; Rapier fly body; sensorimotor loop; 100 Hz control | Stable 100 Hz loop for 60 s without divergence |
|
||||
| 4 — Grooming acceptance | 17–20 | All four criteria of 10-acceptance-test-grooming.md pass on CI | Acceptance test green |
|
||||
| 5 — Fragility + convergence | 21–26 | `BehaviorPipeline` + `PerturbationRunner`; cross-subject convergence via `find_convergence`; first public witness bundle | Fragility correlates with published Hampel 2015 result; witness bundle self-verifies 7/7 |
|
||||
|
||||
Total: 26 weeks (6 months) from kickoff to first publishable witness
|
||||
bundle.
|
||||
|
||||
## 9. Success KPIs per Phase
|
||||
|
||||
| Phase | Primary KPI | Secondary KPIs |
|
||||
|-------|-------------|----------------|
|
||||
| 1 | Load time, query latency | Memory footprint, loader coverage |
|
||||
| 2 | Real-time factor | Replay reproducibility, compression ratio |
|
||||
| 3 | Closed-loop stability duration | Physics step rate, actuator saturation rate |
|
||||
| 4 | All four acceptance criteria green | CI wall-clock, flake rate |
|
||||
| 5 | Behavioral fragility $\mathcal{F}$ distribution | Jaccard with published circuits, convergence score |
|
||||
|
||||
Each KPI has a target and a minimum; missing the minimum blocks
|
||||
promotion to the next phase.
|
||||
|
||||
## 10. Out-of-Scope Items for v1 (Explicit)
|
||||
|
||||
- Mammalian connectomes (mouse cortex, larger).
|
||||
- Consciousness or phenomenal-experience claims.
|
||||
- Real-time human-scale simulation.
|
||||
- GPU-accelerated LIF.
|
||||
- Distributed multi-node simulation.
|
||||
- Live web visualisation.
|
||||
- Unsupervised behavior discovery.
|
||||
- Plasticity in the inner loop.
|
||||
- Wing aerodynamics.
|
||||
- Photorealistic rendering or optic-flow vision beyond simple
|
||||
luminance.
|
||||
- Companion or anthropomorphic framing.
|
||||
|
||||
## 11. Publication and Talk Plan
|
||||
|
||||
| Venue | Submission | Target claim | Artifact |
|
||||
|-------|-----------|--------------|----------|
|
||||
| NeurIPS workshop on neural connectomics | 2026 | Coherence-aware runtime + fragility pipeline | First witness bundle |
|
||||
| eLife methods (or PLOS Comp Biol) | 2026 | CC-OS architecture + grooming reproduction | Peer-reviewed paper |
|
||||
| RustConf | 2026 | Rust systems architecture + determinism | Live demo |
|
||||
| Strange Loop | 2026 | Coherence-aware framing + dual-use ethics | Talk |
|
||||
| bioRxiv preprint | Month 6 | Full methods | Accompanies ADRs |
|
||||
|
||||
No press-first releases. Every external communication follows the
|
||||
positioning and avoids the VERY HIGH / HIGH risk items of §2.
|
||||
|
||||
## 12. Appendix A: 200-Word Short-Form Summary
|
||||
|
||||
> We built CC-OS, a **coherence-aware connectome operating system**, as
|
||||
> a substrate for studying circuits that drive behavior in published
|
||||
> insect connectomes. CC-OS is Rust-native, graph-first, and
|
||||
> deterministic. It loads a peer-reviewed connectome (FlyWire is the
|
||||
> v1 target at 139,255 neurons and 54M synapses), runs an event-driven
|
||||
> LIF neural runtime at real-time rates for 50k-neuron subgraphs,
|
||||
> closes the sensorimotor loop through a Rapier-based fly body, and
|
||||
> provides auditable counterfactual perturbation with min-cut-based
|
||||
> fragility scoring. Every run emits a witness bundle with SHA-256
|
||||
> provenance that can be independently replayed. The first acceptance
|
||||
> test reproduces the published fly antennal-grooming circuit (Hampel
|
||||
> 2015) inside simulation, and shows the RuView min-cut identifies the
|
||||
> same minimal sufficient circuit that optogenetic dissection did. CC-OS
|
||||
> is **not** a mind, not a consciousness upload, not a replacement for
|
||||
> NEURON or Brian2, and not a gesture at human-brain emulation. It is
|
||||
> a platform for reproducible, auditable, graph-native connectome
|
||||
> circuit science at insect scale.
|
||||
|
||||
## 13. References
|
||||
|
||||
1. Sandberg, A., Bostrom, N. (2008). *Whole Brain Emulation: A
|
||||
Roadmap.* Future of Humanity Institute, Oxford.
|
||||
2. Seth, A. (2021). *Being You: A New Science of Consciousness.*
|
||||
Faber.
|
||||
3. Friston, K. (2013). *Life as we know it.* J. R. Soc. Interface.
|
||||
4. ADR-028 — ESP32 capability audit + witness verification.
|
||||
5. ADR-017 — RuVector signal + MAT integration.
|
||||
6. ADR-011 — Python proof of reality.
|
||||
7. Dorkenwald, S., et al. (2024). *Neuronal wiring diagram of an adult
|
||||
brain.* Nature.
|
||||
8. Hampel, S., et al. (2015). *A neural command circuit for grooming
|
||||
movement control.* eLife.
|
||||
9. Seeds, A. M., et al. (2014). *A suppression hierarchy among
|
||||
competing motor programs drives sequential grooming in
|
||||
Drosophila.* eLife.
|
||||
10. Lappalainen, J. K., et al. (2024). *Connectome-constrained networks
|
||||
predict neural activity across the fly visual system.* Nature.
|
||||
11. Kakaria, K. S., de Bivort, B. L. (2017). *Ring attractor dynamics
|
||||
emerge from a spiking model of the entire protocerebral bridge.*
|
||||
Front Behav Neurosci.
|
||||
|
||||
---
|
||||
|
||||
**End of compendium.** Return to [00-index.md](./00-index.md) for the
|
||||
table of contents.
|
||||
@@ -103,6 +103,20 @@ Example: `docker run -e CSI_SOURCE=esp32 -p 3000:3000 -p 5005:5005/udp ruvnet/wi
|
||||
|
||||
### From Source (Rust)
|
||||
|
||||
On Debian/Ubuntu-based Linux systems, install the native desktop prerequisites before the first Rust release build:
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install -y \
|
||||
build-essential pkg-config \
|
||||
libglib2.0-dev libgtk-3-dev \
|
||||
libsoup-3.0-dev \
|
||||
libjavascriptcoregtk-4.1-dev \
|
||||
libwebkit2gtk-4.1-dev
|
||||
```
|
||||
|
||||
This prepares the native GTK/WebKit dependencies used by the desktop/Tauri crates in this workspace.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ruvnet/RuView.git
|
||||
cd RuView/rust-port/wifi-densepose-rs
|
||||
@@ -536,6 +550,110 @@ Both UIs update in real-time via WebSocket and auto-detect the sensing server on
|
||||
|
||||
---
|
||||
|
||||
## Dense Point Cloud (Camera + WiFi CSI Fusion)
|
||||
|
||||
RuView can generate real-time 3D point clouds by fusing camera depth estimation with WiFi CSI spatial sensing. This creates a spatial model of the environment that updates in real-time.
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
# Build the pointcloud binary
|
||||
cd rust-port/wifi-densepose-rs
|
||||
cargo build --release -p wifi-densepose-pointcloud
|
||||
|
||||
# Start the server (auto-detects camera + CSI). Loopback-only by default.
|
||||
./target/release/ruview-pointcloud serve --bind 127.0.0.1:9880
|
||||
```
|
||||
|
||||
Open `http://localhost:9880` for the interactive Three.js 3D viewer.
|
||||
|
||||
> **Security note.** The server exposes live camera, skeleton, vitals, and occupancy over HTTP. The `--bind` flag defaults to `127.0.0.1:9880` (loopback-only). Exposing on `0.0.0.0` or a LAN IP is opt-in — the server logs a warning when it does, but there is no auth/TLS layer. Put a reverse proxy in front if you need remote access.
|
||||
|
||||
> **Brain URL.** Observations are POSTed to `http://127.0.0.1:9876` by default. Override via the `RUVIEW_BRAIN_URL` environment variable or the `--brain <url>` flag on `serve` / `train`.
|
||||
|
||||
### Sensors
|
||||
|
||||
| Sensor | Auto-detected | Data |
|
||||
|--------|--------------|------|
|
||||
| Camera (`/dev/video0`) | Yes (Linux UVC) | RGB frames → MiDaS depth → 3D points |
|
||||
| ESP32 CSI (UDP:3333) | Yes (if provisioned) | ADR-018 binary → occupancy + pose + vitals |
|
||||
| MiDaS depth server (port 9885) | Optional | GPU-accelerated neural depth estimation |
|
||||
|
||||
### Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `ruview-pointcloud serve --bind 127.0.0.1:9880` | Start HTTP server + Three.js viewer (loopback-only by default) |
|
||||
| `ruview-pointcloud demo` | Generate synthetic point cloud (no hardware needed) |
|
||||
| `ruview-pointcloud capture --output room.ply` | Capture single frame to PLY file |
|
||||
| `ruview-pointcloud cameras` | List available cameras |
|
||||
| `ruview-pointcloud train --data-dir ./data [--brain URL]` | Depth calibration + occupancy training (writes under canonicalized `data-dir`; refuses `..` traversal) |
|
||||
| `ruview-pointcloud csi-test --count 100` | Send test CSI frames (no ESP32 needed) |
|
||||
| `ruview-pointcloud fingerprint <name> [--seconds 5]` | Record a named CSI room fingerprint for later matching |
|
||||
|
||||
### Pipeline Components
|
||||
|
||||
1. **ADR-018 Parser** — Decodes ESP32 CSI binary frames from UDP (magic `0xC5110001` raw CSI and `0xC5110006` feature state), extracts I/Q subcarrier amplitudes and phases. Lives in `parser.rs`; unit-tested against hand-rolled test vectors.
|
||||
2. **Pose (stub)** — 17 COCO keypoint *layout* generated by `heuristic_pose_from_amplitude` from CSI amplitude energy. This is **not** the trained WiFlow model — it is a placeholder so the viewer has a skeleton to render. Wiring to real Candle/ONNX inference from the `wifi-densepose-nn` crate is a planned follow-up.
|
||||
3. **Vital Signs** — Breathing rate from CSI phase analysis (peak counting on stable subcarrier)
|
||||
4. **Motion Detection** — CSI amplitude variance over 20 frames, triggers adaptive capture
|
||||
5. **RF Tomography** — Backprojection from per-node RSSI to 8×8×4 occupancy grid
|
||||
6. **Camera Depth** — MiDaS monocular depth (GPU) with luminance+edge fallback
|
||||
7. **Sensor Fusion** — Voxel-grid merging of camera depth + CSI occupancy
|
||||
8. **Brain Bridge** — Stores spatial observations in the ruOS brain every 60 seconds
|
||||
|
||||
### API Endpoints
|
||||
|
||||
| Endpoint | Method | Returns |
|
||||
|----------|--------|---------|
|
||||
| `/health` | GET | `{"status": "ok"}` |
|
||||
| `/api/status` | GET | Camera, CSI, pipeline state, vitals, motion |
|
||||
| `/api/cloud` | GET | Point cloud (up to 1000 points) + pipeline data |
|
||||
| `/api/splats` | GET | Gaussian splats for Three.js rendering |
|
||||
| `/` | GET | Interactive Three.js 3D viewer |
|
||||
|
||||
### Training
|
||||
|
||||
The training pipeline calibrates depth estimation and occupancy detection:
|
||||
|
||||
```bash
|
||||
ruview-pointcloud train --data-dir ~/.local/share/ruview/training --brain http://127.0.0.1:9876
|
||||
```
|
||||
|
||||
This captures frames, runs depth calibration (grid search over scale/offset/gamma), trains occupancy thresholds, exports DPO preference pairs, and submits results to the ruOS brain.
|
||||
|
||||
### Output Formats
|
||||
|
||||
- **PLY** — Standard 3D point cloud (ASCII, with RGB color)
|
||||
- **Gaussian Splats** — JSON format for Three.js rendering
|
||||
- **Brain Memories** — Spatial observations stored as `spatial-observation`, `spatial-motion`, `spatial-vitals`
|
||||
|
||||
### Deep Room Scan
|
||||
|
||||
Capture a high-quality 3D model of the room:
|
||||
|
||||
```bash
|
||||
# Stop the live server first (frees the camera)
|
||||
# Then capture 20 frames and process with MiDaS
|
||||
ruview-pointcloud capture --frames 20 --output room_model.ply
|
||||
```
|
||||
|
||||
Result: 40,000+ voxels at 5cm resolution, 12,000+ Gaussian splats.
|
||||
|
||||
### ESP32 Provisioning for CSI
|
||||
|
||||
To send CSI data to the pointcloud server:
|
||||
|
||||
```bash
|
||||
python3 firmware/esp32-csi-node/provision.py \
|
||||
--port /dev/ttyACM0 \
|
||||
--ssid "YourWiFi" --password "YourPassword" \
|
||||
--target-ip 192.168.1.123 --target-port 3333 \
|
||||
--node-id 1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Vital Sign Detection
|
||||
|
||||
The system extracts breathing rate and heart rate from CSI signal fluctuations using FFT peak detection.
|
||||
@@ -1582,6 +1700,28 @@ rustup update stable
|
||||
rustc --version
|
||||
```
|
||||
|
||||
### Build: Linux native desktop prerequisites
|
||||
|
||||
If you are compiling the Rust workspace on a Debian/Ubuntu-based Linux system, install the native desktop development packages first:
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install -y \
|
||||
build-essential pkg-config \
|
||||
libglib2.0-dev libgtk-3-dev \
|
||||
libsoup-3.0-dev \
|
||||
libjavascriptcoregtk-4.1-dev \
|
||||
libwebkit2gtk-4.1-dev
|
||||
```
|
||||
|
||||
Then rerun:
|
||||
|
||||
```bash
|
||||
cargo build --release
|
||||
```
|
||||
|
||||
This is the same Linux pre-step referenced in the Rust source build section and covers the common GTK/WebKit `pkg-config` requirements used by the desktop build.
|
||||
|
||||
### Windows: RSSI mode shows no data
|
||||
|
||||
Run the terminal as Administrator (required for `netsh wlan` access). Verified working on Windows 10 and 11 with Intel AX201 and Intel BE201 adapters.
|
||||
|
||||
Generated
+66
-7
@@ -1210,13 +1210,34 @@ dependencies = [
|
||||
"subtle",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dirs"
|
||||
version = "5.0.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "44c45a9d03d6676652bcb5e724c7e988de1acad23a711b5217ab9cbecbec2225"
|
||||
dependencies = [
|
||||
"dirs-sys 0.4.1",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dirs"
|
||||
version = "6.0.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c3e8aa94d75141228480295a7d0e7feb620b1a5ad9f12bc40be62411e38cce4e"
|
||||
dependencies = [
|
||||
"dirs-sys",
|
||||
"dirs-sys 0.5.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dirs-sys"
|
||||
version = "0.4.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "520f05a5cbd335fae5a99ff7a6ab8627577660ee5cfd6a94a6a929b52ff0321c"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"option-ext",
|
||||
"redox_users 0.4.6",
|
||||
"windows-sys 0.48.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1227,7 +1248,7 @@ checksum = "e01a3366d27ee9890022452ee61b2b63a67e6f13f58900b651ff5665f0bb1fab"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"option-ext",
|
||||
"redox_users",
|
||||
"redox_users 0.5.2",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
@@ -3789,7 +3810,7 @@ version = "0.10.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "fccd2c4f5271ab871f2069cb6f1a13ef2c0db50e1145ce03428ee541f4c63c4f"
|
||||
dependencies = [
|
||||
"dirs",
|
||||
"dirs 6.0.0",
|
||||
"openblas-build",
|
||||
"pkg-config",
|
||||
"vcpkg",
|
||||
@@ -4834,6 +4855,17 @@ dependencies = [
|
||||
"bitflags 2.11.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "redox_users"
|
||||
version = "0.4.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ba009ff324d1fc1b900bd1fdb31564febe58a8ccc8a6fdbb93b543d33b13ca43"
|
||||
dependencies = [
|
||||
"getrandom 0.2.17",
|
||||
"libredox",
|
||||
"thiserror 1.0.69",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "redox_users"
|
||||
version = "0.5.2"
|
||||
@@ -6264,7 +6296,7 @@ dependencies = [
|
||||
"anyhow",
|
||||
"bytes",
|
||||
"cookie",
|
||||
"dirs",
|
||||
"dirs 6.0.0",
|
||||
"dunce",
|
||||
"embed_plist",
|
||||
"getrandom 0.3.4",
|
||||
@@ -6314,7 +6346,7 @@ checksum = "4bbc990d1dbf57a8e1c7fa2327f2a614d8b757805603c1b9ba5c81bade09fd4d"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"cargo_toml",
|
||||
"dirs",
|
||||
"dirs 6.0.0",
|
||||
"glob",
|
||||
"heck 0.5.0",
|
||||
"json-patch",
|
||||
@@ -7088,7 +7120,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a5e85aa143ceb072062fc4d6356c1b520a51d636e7bc8e77ec94be3608e5e80c"
|
||||
dependencies = [
|
||||
"crossbeam-channel",
|
||||
"dirs",
|
||||
"dirs 6.0.0",
|
||||
"libappindicator",
|
||||
"muda",
|
||||
"objc2",
|
||||
@@ -7820,6 +7852,18 @@ dependencies = [
|
||||
"uuid",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wifi-densepose-geo"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"chrono",
|
||||
"reqwest 0.12.28",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"tokio",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wifi-densepose-hardware"
|
||||
version = "0.3.0"
|
||||
@@ -7894,6 +7938,21 @@ dependencies = [
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wifi-densepose-pointcloud"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"axum",
|
||||
"chrono",
|
||||
"clap",
|
||||
"dirs 5.0.1",
|
||||
"reqwest 0.12.28",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"tokio",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wifi-densepose-ruvector"
|
||||
version = "0.3.0"
|
||||
@@ -8718,7 +8777,7 @@ dependencies = [
|
||||
"block2",
|
||||
"cookie",
|
||||
"crossbeam-channel",
|
||||
"dirs",
|
||||
"dirs 6.0.0",
|
||||
"dpi",
|
||||
"dunce",
|
||||
"gdkx11",
|
||||
|
||||
@@ -17,6 +17,8 @@ members = [
|
||||
"crates/wifi-densepose-vitals",
|
||||
"crates/wifi-densepose-ruvector",
|
||||
"crates/wifi-densepose-desktop",
|
||||
"crates/wifi-densepose-pointcloud",
|
||||
"crates/wifi-densepose-geo",
|
||||
]
|
||||
# ADR-040: WASM edge crate targets wasm32-unknown-unknown (no_std),
|
||||
# excluded from workspace to avoid breaking `cargo test --workspace`.
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
[package]
|
||||
name = "wifi-densepose-geo"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
description = "Geospatial satellite integration — free satellite tiles, DEM, OSM, temporal tracking"
|
||||
|
||||
[dependencies]
|
||||
serde = { workspace = true }
|
||||
serde_json = { workspace = true }
|
||||
tokio = { workspace = true }
|
||||
anyhow = { workspace = true }
|
||||
reqwest = { version = "0.12", features = ["json", "native-tls"], default-features = false }
|
||||
chrono = "0.4"
|
||||
@@ -0,0 +1,105 @@
|
||||
# wifi-densepose-geo — Geospatial Satellite Integration
|
||||
|
||||
Free satellite imagery, terrain elevation, and map data for RuView spatial sensing. No API keys required.
|
||||
|
||||
## What It Does
|
||||
|
||||
Integrates your local sensor data (camera + WiFi CSI point cloud) with geographic context:
|
||||
|
||||
- **Satellite tiles** — 10m Sentinel-2 cloudless imagery for your location
|
||||
- **Elevation** — SRTM 30m DEM for terrain modeling
|
||||
- **Buildings + roads** — OpenStreetMap data via Overpass API
|
||||
- **Weather** — Open Meteo current conditions + forecast
|
||||
- **Geo-registration** — maps local sensor coordinates to WGS84
|
||||
- **Temporal tracking** — detects changes over time (construction, vegetation, weather)
|
||||
- **Brain integration** — stores geospatial context as ruOS brain memories
|
||||
|
||||
## Data Sources (all free, no API keys)
|
||||
|
||||
| Source | Data | Resolution | License |
|
||||
|--------|------|-----------|---------|
|
||||
| [EOX S2 Cloudless](https://s2maps.eu/) | Satellite tiles | 10m | CC-BY-4.0 |
|
||||
| [SRTM GL1](https://portal.opentopography.org/) | Elevation/DEM | 30m | Public domain |
|
||||
| [Overpass API](https://overpass-api.de/) | OSM buildings/roads | Vector | ODbL |
|
||||
| [ip-api.com](http://ip-api.com/) | IP geolocation | ~1km | Free |
|
||||
| [Open Meteo](https://open-meteo.com/) | Weather | Point | CC-BY-4.0 |
|
||||
|
||||
## Modules
|
||||
|
||||
| Module | LOC | Purpose |
|
||||
|--------|-----|---------|
|
||||
| `types.rs` | 140 | GeoPoint, GeoBBox, TileCoord, ElevationGrid, OsmFeature |
|
||||
| `coord.rs` | 80 | WGS84/ENU transforms, tile math, haversine distance |
|
||||
| `locate.rs` | 45 | IP geolocation with caching |
|
||||
| `cache.rs` | 55 | Disk cache (`~/.local/share/ruview/geo-cache/`) |
|
||||
| `tiles.rs` | 80 | Sentinel-2/ESRI/OSM tile fetcher |
|
||||
| `terrain.rs` | 100 | SRTM HGT parser, elevation lookup |
|
||||
| `osm.rs` | 150 | Overpass API client, building/road extraction |
|
||||
| `register.rs` | 50 | Local-to-WGS84 coordinate registration |
|
||||
| `fuse.rs` | 70 | Multi-source scene builder + summary |
|
||||
| `brain.rs` | 30 | Store geo context in ruOS brain |
|
||||
| `temporal.rs` | 100 | Weather, OSM change detection |
|
||||
|
||||
## Usage
|
||||
|
||||
```rust
|
||||
use wifi_densepose_geo::{fuse, brain, temporal};
|
||||
|
||||
// Build geo scene for current location
|
||||
let scene = fuse::build_scene(500.0).await?; // 500m radius
|
||||
println!("{}", fuse::summarize(&scene));
|
||||
// "Location: 43.6532N, 79.3832W, elevation 76m ASL.
|
||||
// 23 buildings within view. 8 roads nearby (King St, Queen St).
|
||||
// 12 satellite tiles at zoom 16."
|
||||
|
||||
// Store in brain
|
||||
brain::store_geo_context(&scene).await?;
|
||||
|
||||
// Fetch weather
|
||||
let weather = temporal::fetch_weather(&scene.location).await?;
|
||||
// temperature: 12°C, partly cloudy, humidity 65%
|
||||
```
|
||||
|
||||
## Brain Integration
|
||||
|
||||
Geospatial context is stored as brain memories:
|
||||
|
||||
| Category | Content | Frequency |
|
||||
|----------|---------|-----------|
|
||||
| `spatial-geo` | Location, elevation, buildings, roads | On startup + daily |
|
||||
| `spatial-weather` | Temperature, conditions, humidity, wind | Nightly |
|
||||
| `spatial-change` | New/removed buildings, road changes | Nightly diff |
|
||||
|
||||
The ruOS agent can search: "what buildings are near me?" or "what's the weather?" and get geospatial context from the brain.
|
||||
|
||||
## Security
|
||||
|
||||
- No API keys stored or transmitted
|
||||
- IP geolocation uses HTTP (not HTTPS) — location is approximate (~1km)
|
||||
- All tile fetches use HTTPS except ip-api.com
|
||||
- Path traversal protection in cache key sanitization
|
||||
- No user data sent to external services
|
||||
- All data cached locally after first fetch
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
IP Geolocation ──→ (lat, lon)
|
||||
│
|
||||
┌─────────────┼─────────────┐
|
||||
▼ ▼ ▼
|
||||
Sentinel-2 SRTM DEM Overpass API
|
||||
(tiles) (elevation) (buildings/roads)
|
||||
│ │ │
|
||||
└─────────────┼─────────────┘
|
||||
▼
|
||||
GeoScene (fused)
|
||||
│
|
||||
┌───────┴───────┐
|
||||
▼ ▼
|
||||
Brain Memory Three.js Viewer
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT (same as RuView)
|
||||
@@ -0,0 +1,47 @@
|
||||
use wifi_densepose_geo::*;
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> anyhow::Result<()> {
|
||||
println!("╔══════════════════════════════════════════════╗");
|
||||
println!("║ ruview-geo — Real Data Validation ║");
|
||||
println!("╚══════════════════════════════════════════════╝\n");
|
||||
|
||||
let t0 = std::time::Instant::now();
|
||||
let cache = cache::TileCache::new("/tmp/ruview-geo-validate");
|
||||
|
||||
let loc = locate::get_location(&format!("{}/location.json", cache.base_dir.display())).await?;
|
||||
println!(" Location: {:.4}N, {:.4}W", loc.lat, loc.lon);
|
||||
|
||||
let bbox = GeoBBox::from_center(&loc, 300.0);
|
||||
let tiles_list = tiles::fetch_area(&tiles::TileProvider::Sentinel2Cloudless, &bbox, 16, &cache).await?;
|
||||
println!(" Tiles: {} ({:.0}KB)", tiles_list.len(),
|
||||
tiles_list.iter().map(|t| t.data.len()).sum::<usize>() as f64 / 1024.0);
|
||||
|
||||
let dem = terrain::fetch_elevation(&loc, &cache).await?;
|
||||
println!(" Elevation: {:.0}m (grid {}x{})", terrain::elevation_at(&dem, &loc), dem.cols, dem.rows);
|
||||
|
||||
let buildings = osm::fetch_buildings(&loc, 300.0).await.unwrap_or_default();
|
||||
let roads = osm::fetch_roads(&loc, 300.0).await.unwrap_or_default();
|
||||
println!(" OSM: {} buildings, {} roads", buildings.len(), roads.len());
|
||||
|
||||
let weather = temporal::fetch_weather(&loc).await?;
|
||||
println!(" Weather: {:.0}°C humidity={:.0}% wind={:.1}m/s",
|
||||
weather.temperature_c, weather.humidity_pct, weather.wind_speed_ms);
|
||||
|
||||
let scene = GeoScene {
|
||||
location: loc.clone(), bbox, elevation_m: terrain::elevation_at(&dem, &loc),
|
||||
buildings, roads, tile_count: tiles_list.len(),
|
||||
registration: register::auto_register(&loc),
|
||||
last_updated: chrono::Utc::now().to_rfc3339(),
|
||||
};
|
||||
println!("\n {}", fuse::summarize(&scene));
|
||||
|
||||
match brain::store_geo_context(&scene).await {
|
||||
Ok(n) => println!(" Brain: {} memories stored", n),
|
||||
Err(e) => println!(" Brain: {e}"),
|
||||
}
|
||||
|
||||
println!("\n Total: {}ms | Cache: {:.0}KB",
|
||||
t0.elapsed().as_millis(), cache.size_bytes() as f64 / 1024.0);
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
//! Brain integration — store geospatial context in ruOS brain.
|
||||
//!
|
||||
//! Brain URL is read from `RUVIEW_BRAIN_URL` env var (default
|
||||
//! `http://127.0.0.1:9876`). The resolved URL is logged once on first use.
|
||||
|
||||
use crate::fuse;
|
||||
use crate::types::GeoScene;
|
||||
use anyhow::Result;
|
||||
use std::sync::OnceLock;
|
||||
|
||||
const DEFAULT_BRAIN_URL: &str = "http://127.0.0.1:9876";
|
||||
|
||||
pub(crate) fn brain_url() -> &'static str {
|
||||
static BRAIN_URL: OnceLock<String> = OnceLock::new();
|
||||
BRAIN_URL.get_or_init(|| {
|
||||
let url = std::env::var("RUVIEW_BRAIN_URL")
|
||||
.unwrap_or_else(|_| DEFAULT_BRAIN_URL.to_string());
|
||||
eprintln!(" wifi-densepose-geo: using brain URL {url}");
|
||||
url
|
||||
})
|
||||
}
|
||||
|
||||
/// Store geospatial context in the brain.
|
||||
pub async fn store_geo_context(scene: &GeoScene) -> Result<u32> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(5))
|
||||
.build()?;
|
||||
|
||||
let mut stored = 0u32;
|
||||
|
||||
// Store location summary
|
||||
let summary = fuse::summarize(scene);
|
||||
let body = serde_json::json!({
|
||||
"category": "spatial-geo",
|
||||
"content": summary,
|
||||
});
|
||||
if client.post(format!("{}/memories", brain_url())).json(&body).send().await.is_ok() {
|
||||
stored += 1;
|
||||
}
|
||||
|
||||
Ok(stored)
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
//! Disk cache for tiles, DEM, and OSM data.
|
||||
|
||||
use anyhow::Result;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
pub struct TileCache {
|
||||
pub base_dir: PathBuf,
|
||||
}
|
||||
|
||||
impl TileCache {
|
||||
pub fn new(base_dir: &str) -> Self {
|
||||
let expanded = base_dir.replace('~', &std::env::var("HOME").unwrap_or_default());
|
||||
let path = PathBuf::from(expanded);
|
||||
let _ = std::fs::create_dir_all(&path);
|
||||
Self { base_dir: path }
|
||||
}
|
||||
|
||||
pub fn default_cache() -> Self {
|
||||
Self::new("~/.local/share/ruview/geo-cache")
|
||||
}
|
||||
|
||||
pub fn get(&self, key: &str) -> Option<Vec<u8>> {
|
||||
let path = self.key_path(key);
|
||||
std::fs::read(&path).ok()
|
||||
}
|
||||
|
||||
pub fn put(&self, key: &str, data: &[u8]) -> Result<()> {
|
||||
let path = self.key_path(key);
|
||||
if let Some(parent) = path.parent() {
|
||||
std::fs::create_dir_all(parent)?;
|
||||
}
|
||||
std::fs::write(&path, data)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn has(&self, key: &str) -> bool {
|
||||
self.key_path(key).exists()
|
||||
}
|
||||
|
||||
pub fn size_bytes(&self) -> u64 {
|
||||
walkdir(self.base_dir.as_path())
|
||||
}
|
||||
|
||||
fn key_path(&self, key: &str) -> PathBuf {
|
||||
// Sanitize key to prevent path traversal
|
||||
let safe_key = key.replace("..", "_").replace('/', "_");
|
||||
self.base_dir.join(safe_key)
|
||||
}
|
||||
}
|
||||
|
||||
fn walkdir(path: &Path) -> u64 {
|
||||
std::fs::read_dir(path)
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.filter_map(|e| e.ok())
|
||||
.map(|e| {
|
||||
if e.path().is_dir() { walkdir(&e.path()) }
|
||||
else { e.metadata().map(|m| m.len()).unwrap_or(0) }
|
||||
})
|
||||
.sum()
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
//! Coordinate transforms — WGS84, UTM, ENU, tile math.
|
||||
|
||||
use crate::types::{GeoPoint, GeoBBox, TileCoord};
|
||||
|
||||
const WGS84_A: f64 = 6_378_137.0;
|
||||
#[allow(dead_code)]
|
||||
const WGS84_F: f64 = 1.0 / 298.257_223_563;
|
||||
#[allow(dead_code)]
|
||||
const WGS84_E2: f64 = 2.0 * WGS84_F - WGS84_F * WGS84_F;
|
||||
|
||||
/// Haversine distance in meters.
|
||||
pub fn haversine(a: &GeoPoint, b: &GeoPoint) -> f64 {
|
||||
let dlat = (b.lat - a.lat).to_radians();
|
||||
let dlon = (b.lon - a.lon).to_radians();
|
||||
let lat1 = a.lat.to_radians();
|
||||
let lat2 = b.lat.to_radians();
|
||||
let h = (dlat / 2.0).sin().powi(2) + lat1.cos() * lat2.cos() * (dlon / 2.0).sin().powi(2);
|
||||
2.0 * WGS84_A * h.sqrt().asin()
|
||||
}
|
||||
|
||||
/// WGS84 to local ENU (East-North-Up) relative to origin, in meters.
|
||||
pub fn wgs84_to_enu(point: &GeoPoint, origin: &GeoPoint) -> [f64; 3] {
|
||||
let dlat = (point.lat - origin.lat).to_radians();
|
||||
let dlon = (point.lon - origin.lon).to_radians();
|
||||
let lat = origin.lat.to_radians();
|
||||
let east = dlon * WGS84_A * lat.cos();
|
||||
let north = dlat * WGS84_A;
|
||||
let up = point.alt - origin.alt;
|
||||
[east, north, up]
|
||||
}
|
||||
|
||||
/// Local ENU to WGS84.
|
||||
pub fn enu_to_wgs84(enu: &[f64; 3], origin: &GeoPoint) -> GeoPoint {
|
||||
let lat = origin.lat.to_radians();
|
||||
let dlat = enu[1] / WGS84_A;
|
||||
let dlon = enu[0] / (WGS84_A * lat.cos());
|
||||
GeoPoint {
|
||||
lat: origin.lat + dlat.to_degrees(),
|
||||
lon: origin.lon + dlon.to_degrees(),
|
||||
alt: origin.alt + enu[2],
|
||||
}
|
||||
}
|
||||
|
||||
/// WGS84 to XYZ tile coordinates (Slippy Map).
|
||||
pub fn wgs84_to_tile(lat: f64, lon: f64, zoom: u8) -> TileCoord {
|
||||
let n = 2f64.powi(zoom as i32);
|
||||
let x = ((lon + 180.0) / 360.0 * n).floor() as u32;
|
||||
let lat_rad = lat.to_radians();
|
||||
let y = ((1.0 - lat_rad.tan().asinh() / std::f64::consts::PI) / 2.0 * n).floor() as u32;
|
||||
TileCoord { z: zoom, x, y }
|
||||
}
|
||||
|
||||
/// Tile bounds in WGS84.
|
||||
pub fn tile_bounds(coord: &TileCoord) -> GeoBBox {
|
||||
let n = 2f64.powi(coord.z as i32);
|
||||
let west = coord.x as f64 / n * 360.0 - 180.0;
|
||||
let east = (coord.x + 1) as f64 / n * 360.0 - 180.0;
|
||||
let north = (std::f64::consts::PI * (1.0 - 2.0 * coord.y as f64 / n)).sinh().atan().to_degrees();
|
||||
let south = (std::f64::consts::PI * (1.0 - 2.0 * (coord.y + 1) as f64 / n)).sinh().atan().to_degrees();
|
||||
GeoBBox { south, west, north, east }
|
||||
}
|
||||
|
||||
/// Get all tile coordinates covering a bounding box at a zoom level.
|
||||
pub fn tiles_for_bbox(bbox: &GeoBBox, zoom: u8) -> Vec<TileCoord> {
|
||||
let tl = wgs84_to_tile(bbox.north, bbox.west, zoom);
|
||||
let br = wgs84_to_tile(bbox.south, bbox.east, zoom);
|
||||
let mut tiles = Vec::new();
|
||||
for y in tl.y..=br.y {
|
||||
for x in tl.x..=br.x {
|
||||
tiles.push(TileCoord { z: zoom, x, y });
|
||||
}
|
||||
}
|
||||
tiles
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
//! Multi-source fusion — satellite + terrain + OSM + local sensor data.
|
||||
|
||||
use crate::cache::TileCache;
|
||||
use crate::types::*;
|
||||
use crate::{locate, osm, terrain, tiles};
|
||||
use anyhow::Result;
|
||||
|
||||
/// Build a complete geo scene for a location.
|
||||
pub async fn build_scene(radius_m: f64) -> Result<GeoScene> {
|
||||
let cache = TileCache::default_cache();
|
||||
|
||||
// 1. Locate
|
||||
let cache_path = cache.base_dir.join("location.json");
|
||||
let location = locate::get_location(cache_path.to_str().unwrap_or("")).await?;
|
||||
eprintln!(" Geo: located at {:.4}N, {:.4}W", location.lat, location.lon);
|
||||
|
||||
// 2. Fetch satellite tiles
|
||||
let bbox = GeoBBox::from_center(&location, radius_m);
|
||||
let tile_list = tiles::fetch_area(&tiles::TileProvider::Sentinel2Cloudless, &bbox, 16, &cache).await?;
|
||||
eprintln!(" Geo: fetched {} satellite tiles", tile_list.len());
|
||||
|
||||
// 3. Fetch elevation
|
||||
let dem = terrain::fetch_elevation(&location, &cache).await?;
|
||||
let elevation = terrain::elevation_at(&dem, &location);
|
||||
eprintln!(" Geo: elevation {:.0}m ASL", elevation);
|
||||
|
||||
// 4. Fetch OSM buildings + roads
|
||||
let buildings = osm::fetch_buildings(&location, radius_m).await.unwrap_or_default();
|
||||
let roads = osm::fetch_roads(&location, radius_m).await.unwrap_or_default();
|
||||
eprintln!(" Geo: {} buildings, {} roads", buildings.len(), roads.len());
|
||||
|
||||
// 5. Build registration
|
||||
let mut reg_origin = location.clone();
|
||||
reg_origin.alt = elevation as f64;
|
||||
let registration = crate::register::auto_register(®_origin);
|
||||
|
||||
Ok(GeoScene {
|
||||
location: reg_origin,
|
||||
bbox,
|
||||
elevation_m: elevation,
|
||||
buildings,
|
||||
roads,
|
||||
tile_count: tile_list.len(),
|
||||
registration,
|
||||
last_updated: chrono::Utc::now().to_rfc3339(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Generate a text summary of the geo scene.
|
||||
pub fn summarize(scene: &GeoScene) -> String {
|
||||
let building_count = scene.buildings.len();
|
||||
let road_count = scene.roads.len();
|
||||
let road_names: Vec<&str> = scene.roads.iter()
|
||||
.filter_map(|r| match r {
|
||||
OsmFeature::Road { name, .. } => name.as_deref(),
|
||||
_ => None,
|
||||
})
|
||||
.take(3)
|
||||
.collect();
|
||||
|
||||
format!(
|
||||
"Location: {:.4}N, {:.4}W, elevation {:.0}m ASL. \
|
||||
{} buildings within view. {} roads nearby{}. \
|
||||
{} satellite tiles at zoom 16. Updated: {}.",
|
||||
scene.location.lat, scene.location.lon, scene.elevation_m,
|
||||
building_count, road_count,
|
||||
if road_names.is_empty() { String::new() }
|
||||
else { format!(" ({})", road_names.join(", ")) },
|
||||
scene.tile_count,
|
||||
&scene.last_updated[..10],
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
//! wifi-densepose-geo — geospatial satellite integration for RuView.
|
||||
//!
|
||||
//! Provides: IP geolocation, satellite tile fetching (Sentinel-2),
|
||||
//! SRTM elevation, OSM buildings/roads, coordinate transforms,
|
||||
//! temporal change tracking, and brain memory integration.
|
||||
|
||||
pub mod types;
|
||||
pub mod coord;
|
||||
pub mod locate;
|
||||
pub mod cache;
|
||||
pub mod tiles;
|
||||
pub mod terrain;
|
||||
pub mod osm;
|
||||
pub mod register;
|
||||
pub mod fuse;
|
||||
pub mod brain;
|
||||
pub mod temporal;
|
||||
|
||||
pub use types::*;
|
||||
@@ -0,0 +1,40 @@
|
||||
//! IP geolocation — determine location from public IP.
|
||||
|
||||
use crate::types::GeoPoint;
|
||||
use anyhow::Result;
|
||||
|
||||
/// Locate by IP address (free, no API key).
|
||||
pub async fn locate_by_ip() -> Result<GeoPoint> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(5))
|
||||
.build()?;
|
||||
|
||||
// Primary: ip-api.com (free, 45 req/min)
|
||||
let resp: serde_json::Value = client
|
||||
.get("http://ip-api.com/json/?fields=lat,lon,city,regionName,country")
|
||||
.send().await?
|
||||
.json().await?;
|
||||
|
||||
let lat = resp.get("lat").and_then(|v| v.as_f64()).unwrap_or(0.0);
|
||||
let lon = resp.get("lon").and_then(|v| v.as_f64()).unwrap_or(0.0);
|
||||
|
||||
if lat == 0.0 && lon == 0.0 {
|
||||
anyhow::bail!("IP geolocation returned (0,0)");
|
||||
}
|
||||
|
||||
Ok(GeoPoint { lat, lon, alt: 0.0 })
|
||||
}
|
||||
|
||||
/// Get location with caching.
|
||||
pub async fn get_location(cache_path: &str) -> Result<GeoPoint> {
|
||||
// Check cache
|
||||
if let Ok(data) = std::fs::read_to_string(cache_path) {
|
||||
if let Ok(point) = serde_json::from_str::<GeoPoint>(&data) {
|
||||
return Ok(point);
|
||||
}
|
||||
}
|
||||
|
||||
let point = locate_by_ip().await?;
|
||||
let _ = std::fs::write(cache_path, serde_json::to_string(&point)?);
|
||||
Ok(point)
|
||||
}
|
||||
@@ -0,0 +1,216 @@
|
||||
//! OpenStreetMap data via Overpass API — buildings, roads, land use.
|
||||
|
||||
use crate::types::{GeoBBox, GeoPoint, OsmFeature};
|
||||
use anyhow::{anyhow, Result};
|
||||
|
||||
const OVERPASS_URL: &str = "https://overpass-api.de/api/interpreter";
|
||||
|
||||
/// Maximum radius (in metres) accepted by the OSM fetchers. Requests larger
|
||||
/// than this would produce Overpass queries covering hundreds of square
|
||||
/// kilometres — which hammers the public endpoint and returns unworkably
|
||||
/// large response payloads. Callers wanting wider areas must tile the queries.
|
||||
pub const MAX_RADIUS_M: f64 = 5000.0;
|
||||
|
||||
fn check_radius(radius_m: f64) -> Result<()> {
|
||||
if !radius_m.is_finite() || radius_m <= 0.0 {
|
||||
return Err(anyhow!("radius_m must be positive and finite (got {radius_m})"));
|
||||
}
|
||||
if radius_m > MAX_RADIUS_M {
|
||||
return Err(anyhow!(
|
||||
"radius_m {radius_m} exceeds MAX_RADIUS_M ({MAX_RADIUS_M}); \
|
||||
tile the query into smaller chunks"
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Fetch buildings within radius of a point.
|
||||
///
|
||||
/// Uses an inclusive `["building"]` filter that matches all building values
|
||||
/// (residential, commercial, yes, etc.) and also queries relations for
|
||||
/// multipolygon buildings. Default recommended radius: 500 m. Max 5000 m.
|
||||
pub async fn fetch_buildings(center: &GeoPoint, radius_m: f64) -> Result<Vec<OsmFeature>> {
|
||||
check_radius(radius_m)?;
|
||||
let bbox = GeoBBox::from_center(center, radius_m);
|
||||
let query = format!(
|
||||
r#"[out:json][timeout:25];(way["building"]({},{},{},{});relation["building"]({},{},{},{}););out body;>;out skel qt;"#,
|
||||
bbox.south, bbox.west, bbox.north, bbox.east,
|
||||
bbox.south, bbox.west, bbox.north, bbox.east,
|
||||
);
|
||||
let resp = overpass_query(&query).await?;
|
||||
parse_buildings(&resp)
|
||||
}
|
||||
|
||||
/// Fetch roads within radius. Max 5000 m; returns an error otherwise.
|
||||
pub async fn fetch_roads(center: &GeoPoint, radius_m: f64) -> Result<Vec<OsmFeature>> {
|
||||
check_radius(radius_m)?;
|
||||
let bbox = GeoBBox::from_center(center, radius_m);
|
||||
let query = format!(
|
||||
r#"[out:json][timeout:10];way["highway"]({},{},{},{});out body;>;out skel qt;"#,
|
||||
bbox.south, bbox.west, bbox.north, bbox.east
|
||||
);
|
||||
let resp = overpass_query(&query).await?;
|
||||
parse_roads(&resp)
|
||||
}
|
||||
|
||||
async fn overpass_query(query: &str) -> Result<serde_json::Value> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(15))
|
||||
.user_agent("RuView/0.1")
|
||||
.build()?;
|
||||
|
||||
let resp = client.post(OVERPASS_URL)
|
||||
.form(&[("data", query)])
|
||||
.send().await?;
|
||||
|
||||
if !resp.status().is_success() {
|
||||
anyhow::bail!("Overpass API error: {}", resp.status());
|
||||
}
|
||||
Ok(resp.json().await?)
|
||||
}
|
||||
|
||||
/// Parse an Overpass JSON response into building features.
|
||||
///
|
||||
/// Returns an error if the response is not a JSON object or is missing the
|
||||
/// top-level `elements` array (indicative of a malformed/non-Overpass payload).
|
||||
pub fn parse_overpass_json(data: &serde_json::Value) -> Result<Vec<OsmFeature>> {
|
||||
if !data.is_object() || data.get("elements").and_then(|e| e.as_array()).is_none() {
|
||||
return Err(anyhow!("malformed Overpass response: missing `elements` array"));
|
||||
}
|
||||
parse_buildings(data)
|
||||
}
|
||||
|
||||
pub(crate) fn parse_buildings(data: &serde_json::Value) -> Result<Vec<OsmFeature>> {
|
||||
let mut buildings = Vec::new();
|
||||
let mut nodes: std::collections::HashMap<u64, [f64; 2]> = std::collections::HashMap::new();
|
||||
|
||||
let elements = data.get("elements").and_then(|e| e.as_array()).cloned().unwrap_or_default();
|
||||
|
||||
// First pass: collect nodes
|
||||
for el in &elements {
|
||||
if el.get("type").and_then(|t| t.as_str()) == Some("node") {
|
||||
if let (Some(id), Some(lat), Some(lon)) = (
|
||||
el.get("id").and_then(|v| v.as_u64()),
|
||||
el.get("lat").and_then(|v| v.as_f64()),
|
||||
el.get("lon").and_then(|v| v.as_f64()),
|
||||
) {
|
||||
nodes.insert(id, [lat, lon]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Second pass: build ways
|
||||
for el in &elements {
|
||||
if el.get("type").and_then(|t| t.as_str()) != Some("way") { continue; }
|
||||
let tags = el.get("tags").cloned().unwrap_or(serde_json::json!({}));
|
||||
if tags.get("building").is_none() { continue; }
|
||||
|
||||
let node_ids = el.get("nodes").and_then(|n| n.as_array()).cloned().unwrap_or_default();
|
||||
let outline: Vec<[f64; 2]> = node_ids.iter()
|
||||
.filter_map(|id| id.as_u64().and_then(|id| nodes.get(&id).copied()))
|
||||
.collect();
|
||||
|
||||
if outline.len() < 3 { continue; }
|
||||
|
||||
let height = tags.get("height").and_then(|h| h.as_str())
|
||||
.and_then(|s| s.trim_end_matches('m').trim().parse::<f32>().ok())
|
||||
.or(Some(8.0)); // default building height
|
||||
|
||||
let name = tags.get("name").and_then(|n| n.as_str()).map(|s| s.to_string());
|
||||
|
||||
buildings.push(OsmFeature::Building { outline, height, name });
|
||||
}
|
||||
|
||||
Ok(buildings)
|
||||
}
|
||||
|
||||
fn parse_roads(data: &serde_json::Value) -> Result<Vec<OsmFeature>> {
|
||||
let mut roads = Vec::new();
|
||||
let mut nodes: std::collections::HashMap<u64, [f64; 2]> = std::collections::HashMap::new();
|
||||
|
||||
let elements = data.get("elements").and_then(|e| e.as_array()).cloned().unwrap_or_default();
|
||||
|
||||
for el in &elements {
|
||||
if el.get("type").and_then(|t| t.as_str()) == Some("node") {
|
||||
if let (Some(id), Some(lat), Some(lon)) = (
|
||||
el.get("id").and_then(|v| v.as_u64()),
|
||||
el.get("lat").and_then(|v| v.as_f64()),
|
||||
el.get("lon").and_then(|v| v.as_f64()),
|
||||
) {
|
||||
nodes.insert(id, [lat, lon]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for el in &elements {
|
||||
if el.get("type").and_then(|t| t.as_str()) != Some("way") { continue; }
|
||||
let tags = el.get("tags").cloned().unwrap_or(serde_json::json!({}));
|
||||
let highway = tags.get("highway").and_then(|h| h.as_str());
|
||||
if highway.is_none() { continue; }
|
||||
|
||||
let node_ids = el.get("nodes").and_then(|n| n.as_array()).cloned().unwrap_or_default();
|
||||
let path: Vec<[f64; 2]> = node_ids.iter()
|
||||
.filter_map(|id| id.as_u64().and_then(|id| nodes.get(&id).copied()))
|
||||
.collect();
|
||||
|
||||
if path.len() < 2 { continue; }
|
||||
|
||||
let name = tags.get("name").and_then(|n| n.as_str()).map(|s| s.to_string());
|
||||
|
||||
roads.push(OsmFeature::Road {
|
||||
path,
|
||||
road_type: highway.unwrap_or("unknown").to_string(),
|
||||
name,
|
||||
});
|
||||
}
|
||||
|
||||
Ok(roads)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn parse_overpass_json_accepts_minimal_fixture() {
|
||||
// Minimal fixture: three nodes forming a triangular building.
|
||||
let j = serde_json::json!({
|
||||
"elements": [
|
||||
{ "type": "node", "id": 1, "lat": 43.0, "lon": -79.0 },
|
||||
{ "type": "node", "id": 2, "lat": 43.0001, "lon": -79.0 },
|
||||
{ "type": "node", "id": 3, "lat": 43.0, "lon": -79.0001 },
|
||||
{
|
||||
"type": "way", "id": 100,
|
||||
"nodes": [1, 2, 3, 1],
|
||||
"tags": { "building": "yes", "name": "Test Hall" }
|
||||
}
|
||||
]
|
||||
});
|
||||
let features = parse_overpass_json(&j).expect("minimal payload should parse");
|
||||
assert_eq!(features.len(), 1);
|
||||
match &features[0] {
|
||||
OsmFeature::Building { outline, name, .. } => {
|
||||
assert_eq!(outline.len(), 4);
|
||||
assert_eq!(name.as_deref(), Some("Test Hall"));
|
||||
}
|
||||
_ => panic!("expected a Building"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_overpass_json_rejects_malformed() {
|
||||
// Missing the `elements` array entirely.
|
||||
let j = serde_json::json!({ "version": 0.6 });
|
||||
assert!(parse_overpass_json(&j).is_err());
|
||||
// Not even an object.
|
||||
let arr = serde_json::json!([1, 2, 3]);
|
||||
assert!(parse_overpass_json(&arr).is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn fetch_buildings_rejects_oversized_radius() {
|
||||
let center = GeoPoint { lat: 43.0, lon: -79.0, alt: 0.0 };
|
||||
let err = fetch_buildings(¢er, MAX_RADIUS_M + 1.0).await.err();
|
||||
assert!(err.is_some(), "should reject radius > MAX_RADIUS_M");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
//! Geo-registration — maps local sensor coordinates to WGS84.
|
||||
|
||||
use crate::coord;
|
||||
use crate::types::{GeoPoint, GeoRegistration};
|
||||
|
||||
/// Auto-register using IP location (sensor at IP location, facing north).
|
||||
pub fn auto_register(ip_location: &GeoPoint) -> GeoRegistration {
|
||||
GeoRegistration {
|
||||
origin: ip_location.clone(),
|
||||
heading_deg: 0.0,
|
||||
scale: 1.0,
|
||||
}
|
||||
}
|
||||
|
||||
/// Transform local point [x, y, z] to WGS84.
|
||||
pub fn local_to_wgs84(reg: &GeoRegistration, local: &[f32; 3]) -> GeoPoint {
|
||||
let heading_rad = reg.heading_deg.to_radians();
|
||||
let cos_h = heading_rad.cos();
|
||||
let sin_h = heading_rad.sin();
|
||||
|
||||
// Rotate local by heading (local X → East when heading=0)
|
||||
let east = (local[0] as f64 * cos_h - local[2] as f64 * sin_h) * reg.scale;
|
||||
let north = (local[0] as f64 * sin_h + local[2] as f64 * cos_h) * reg.scale;
|
||||
let up = local[1] as f64 * reg.scale;
|
||||
|
||||
coord::enu_to_wgs84(&[east, north, up], ®.origin)
|
||||
}
|
||||
|
||||
/// Transform WGS84 to local point.
|
||||
pub fn wgs84_to_local(reg: &GeoRegistration, geo: &GeoPoint) -> [f32; 3] {
|
||||
let enu = coord::wgs84_to_enu(geo, ®.origin);
|
||||
let heading_rad = (-reg.heading_deg).to_radians();
|
||||
let cos_h = heading_rad.cos();
|
||||
let sin_h = heading_rad.sin();
|
||||
|
||||
let x = ((enu[0] * cos_h - enu[1] * sin_h) / reg.scale) as f32;
|
||||
let z = ((enu[0] * sin_h + enu[1] * cos_h) / reg.scale) as f32;
|
||||
let y = (enu[2] / reg.scale) as f32;
|
||||
|
||||
[x, y, z]
|
||||
}
|
||||
@@ -0,0 +1,312 @@
|
||||
//! Temporal change tracking — detect changes in satellite/OSM/weather over time.
|
||||
|
||||
use crate::cache::TileCache;
|
||||
use crate::types::GeoPoint;
|
||||
#[allow(unused_imports)]
|
||||
use crate::types::GeoScene;
|
||||
use anyhow::Result;
|
||||
|
||||
/// Fetch current weather (Open Meteo, free, no key).
|
||||
pub async fn fetch_weather(point: &GeoPoint) -> Result<WeatherData> {
|
||||
let url = format!(
|
||||
"https://api.open-meteo.com/v1/forecast?latitude={:.4}&longitude={:.4}¤t=temperature_2m,relative_humidity_2m,wind_speed_10m,weather_code",
|
||||
point.lat, point.lon
|
||||
);
|
||||
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(10))
|
||||
.build()?;
|
||||
|
||||
let resp: serde_json::Value = client.get(&url).send().await?.json().await?;
|
||||
let current = resp.get("current").cloned().unwrap_or(serde_json::json!({}));
|
||||
|
||||
Ok(WeatherData {
|
||||
temperature_c: current.get("temperature_2m").and_then(|v| v.as_f64()).unwrap_or(0.0) as f32,
|
||||
humidity_pct: current.get("relative_humidity_2m").and_then(|v| v.as_f64()).unwrap_or(0.0) as f32,
|
||||
wind_speed_ms: current.get("wind_speed_10m").and_then(|v| v.as_f64()).unwrap_or(0.0) as f32,
|
||||
weather_code: current.get("weather_code").and_then(|v| v.as_u64()).unwrap_or(0) as u16,
|
||||
})
|
||||
}
|
||||
|
||||
/// Check for OSM changes since last fetch.
|
||||
pub async fn check_osm_changes(scene: &GeoScene, cache: &TileCache) -> Result<Vec<String>> {
|
||||
let mut changes = Vec::new();
|
||||
|
||||
let cache_key = "osm_building_count";
|
||||
let prev_count: usize = cache.get(cache_key)
|
||||
.and_then(|d| String::from_utf8(d).ok())
|
||||
.and_then(|s| s.trim().parse().ok())
|
||||
.unwrap_or(0);
|
||||
|
||||
let current_count = scene.buildings.len();
|
||||
if prev_count > 0 && current_count != prev_count {
|
||||
let diff = current_count as i64 - prev_count as i64;
|
||||
changes.push(format!("Building count changed: {} → {} ({:+})", prev_count, current_count, diff));
|
||||
}
|
||||
|
||||
cache.put(cache_key, current_count.to_string().as_bytes())?;
|
||||
Ok(changes)
|
||||
}
|
||||
|
||||
/// Generate temporal summary for brain storage.
|
||||
pub fn temporal_summary(weather: &WeatherData, changes: &[String]) -> String {
|
||||
let weather_desc = match weather.weather_code {
|
||||
0 => "clear sky",
|
||||
1..=3 => "partly cloudy",
|
||||
45 | 48 => "foggy",
|
||||
51..=57 => "drizzle",
|
||||
61..=67 => "rain",
|
||||
71..=77 => "snow",
|
||||
80..=82 => "showers",
|
||||
95..=99 => "thunderstorm",
|
||||
_ => "unknown",
|
||||
};
|
||||
|
||||
let mut summary = format!(
|
||||
"Weather: {:.0}°C, {weather_desc}, humidity {:.0}%, wind {:.1}m/s.",
|
||||
weather.temperature_c, weather.humidity_pct, weather.wind_speed_ms,
|
||||
);
|
||||
|
||||
for change in changes {
|
||||
summary.push_str(&format!(" Change: {change}."));
|
||||
}
|
||||
|
||||
summary
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
|
||||
pub struct WeatherData {
|
||||
pub temperature_c: f32,
|
||||
pub humidity_pct: f32,
|
||||
pub wind_speed_ms: f32,
|
||||
pub weather_code: u16,
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Satellite tile change detection
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Result of comparing two tile snapshots.
|
||||
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
|
||||
pub struct TileChangeResult {
|
||||
/// 0.0 = identical, 1.0 = completely different.
|
||||
pub diff_score: f64,
|
||||
/// Number of pixels that changed.
|
||||
pub changed_pixels: usize,
|
||||
/// Total pixels compared.
|
||||
pub total_pixels: usize,
|
||||
}
|
||||
|
||||
/// Compare a newly-fetched tile against its previously-cached version.
|
||||
///
|
||||
/// Returns a `TileChangeResult` with a diff score between 0.0 (identical) and
|
||||
/// 1.0 (completely different). When the diff exceeds 0.1 the function stores
|
||||
/// a change event as a brain memory via the local ruOS brain endpoint.
|
||||
pub async fn detect_tile_changes(
|
||||
cache_key: &str,
|
||||
new_data: &[u8],
|
||||
cache: &TileCache,
|
||||
) -> Result<TileChangeResult> {
|
||||
let previous = cache.get(cache_key);
|
||||
|
||||
let result = match previous {
|
||||
Some(ref old_data) => {
|
||||
let total = old_data.len().max(new_data.len()).max(1);
|
||||
let comparable = old_data.len().min(new_data.len());
|
||||
let mut changed: usize = 0;
|
||||
for i in 0..comparable {
|
||||
if old_data[i] != new_data[i] {
|
||||
changed += 1;
|
||||
}
|
||||
}
|
||||
// Any extra bytes in the longer slice count as changed.
|
||||
changed += total - comparable;
|
||||
|
||||
TileChangeResult {
|
||||
diff_score: changed as f64 / total as f64,
|
||||
changed_pixels: changed,
|
||||
total_pixels: total,
|
||||
}
|
||||
}
|
||||
None => {
|
||||
// No previous data — treat as fully new (score 1.0).
|
||||
TileChangeResult {
|
||||
diff_score: 1.0,
|
||||
changed_pixels: new_data.len(),
|
||||
total_pixels: new_data.len().max(1),
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// Persist new snapshot into cache for future comparisons.
|
||||
cache.put(cache_key, new_data)?;
|
||||
|
||||
// When significant change is detected, store a brain memory.
|
||||
if result.diff_score > 0.1 {
|
||||
let _ = store_change_event(cache_key, &result).await;
|
||||
}
|
||||
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
/// Post a change event to the local ruOS brain.
|
||||
///
|
||||
/// Brain URL honours `RUVIEW_BRAIN_URL` via [`crate::brain::brain_url`].
|
||||
async fn store_change_event(cache_key: &str, result: &TileChangeResult) -> Result<()> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(5))
|
||||
.build()?;
|
||||
|
||||
let body = serde_json::json!({
|
||||
"category": "spatial-change",
|
||||
"content": format!(
|
||||
"Tile change detected for {cache_key}: diff={:.3}, changed={}/{}",
|
||||
result.diff_score, result.changed_pixels, result.total_pixels,
|
||||
),
|
||||
});
|
||||
|
||||
client
|
||||
.post(format!("{}/memories", crate::brain::brain_url()))
|
||||
.json(&body)
|
||||
.send()
|
||||
.await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Night mode detection
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Approximate check whether the current time is "night" at a given latitude.
|
||||
///
|
||||
/// Uses a simplified sunrise/sunset model based on the solar declination and
|
||||
/// hour angle. When it is night the system should rely on CSI data only
|
||||
/// (satellite imagery is not useful in darkness).
|
||||
pub fn is_night(lat_deg: f64) -> bool {
|
||||
let now = chrono::Utc::now();
|
||||
is_night_at(lat_deg, now)
|
||||
}
|
||||
|
||||
/// Testable version of [`is_night`] that accepts an explicit timestamp.
|
||||
pub fn is_night_at(lat_deg: f64, utc: chrono::DateTime<chrono::Utc>) -> bool {
|
||||
use chrono::Datelike;
|
||||
use std::f64::consts::PI;
|
||||
|
||||
let day_of_year = utc.ordinal() as f64;
|
||||
let hour_utc = utc.timestamp() % 86400;
|
||||
let solar_hour = (hour_utc as f64) / 3600.0; // 0..24
|
||||
|
||||
// Solar declination (Spencer, 1971 — simplified)
|
||||
let gamma = 2.0 * PI * (day_of_year - 1.0) / 365.0;
|
||||
let decl = 0.006918
|
||||
- 0.399912 * gamma.cos()
|
||||
+ 0.070257 * gamma.sin()
|
||||
- 0.006758 * (2.0 * gamma).cos()
|
||||
+ 0.000907 * (2.0 * gamma).sin();
|
||||
|
||||
let lat_rad = lat_deg.to_radians();
|
||||
|
||||
// Cosine of the hour angle at sunrise/sunset (geometric, no refraction)
|
||||
let cos_ha = -(lat_rad.tan() * decl.tan());
|
||||
|
||||
// Polar day / polar night
|
||||
if cos_ha < -1.0 {
|
||||
return false; // midnight sun — never night
|
||||
}
|
||||
if cos_ha > 1.0 {
|
||||
return true; // polar night — always night
|
||||
}
|
||||
|
||||
let ha_sunrise = cos_ha.acos(); // radians, symmetric about solar noon
|
||||
let daylight_hours = 2.0 * ha_sunrise * 12.0 / PI;
|
||||
let solar_noon = 12.0; // approximation (ignores longitude offset)
|
||||
let sunrise = solar_noon - daylight_hours / 2.0;
|
||||
let sunset = solar_noon + daylight_hours / 2.0;
|
||||
|
||||
solar_hour < sunrise || solar_hour > sunset
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Tests
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_is_night_at_equator_noon() {
|
||||
// Noon UTC at equator on March 20 — should be daytime.
|
||||
let dt = chrono::NaiveDate::from_ymd_opt(2025, 3, 20)
|
||||
.unwrap()
|
||||
.and_hms_opt(12, 0, 0)
|
||||
.unwrap()
|
||||
.and_utc();
|
||||
assert!(!is_night_at(0.0, dt));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_is_night_at_equator_midnight() {
|
||||
// Midnight UTC at equator — should be night.
|
||||
let dt = chrono::NaiveDate::from_ymd_opt(2025, 3, 20)
|
||||
.unwrap()
|
||||
.and_hms_opt(2, 0, 0)
|
||||
.unwrap()
|
||||
.and_utc();
|
||||
assert!(is_night_at(0.0, dt));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_midnight_sun_arctic() {
|
||||
// Late June at 70 N — midnight sun, never night.
|
||||
let dt = chrono::NaiveDate::from_ymd_opt(2025, 6, 21)
|
||||
.unwrap()
|
||||
.and_hms_opt(0, 0, 0)
|
||||
.unwrap()
|
||||
.and_utc();
|
||||
assert!(!is_night_at(70.0, dt));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_polar_night_arctic() {
|
||||
// Late December at 80 N — polar night, always night.
|
||||
let dt = chrono::NaiveDate::from_ymd_opt(2025, 12, 21)
|
||||
.unwrap()
|
||||
.and_hms_opt(12, 0, 0)
|
||||
.unwrap()
|
||||
.and_utc();
|
||||
assert!(is_night_at(80.0, dt));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_tile_changes_identical() {
|
||||
let cache = TileCache::new("/tmp/ruview-test-tile-changes");
|
||||
let data = vec![1u8, 2, 3, 4, 5];
|
||||
// Prime the cache.
|
||||
cache.put("test_tile_ident", &data).unwrap();
|
||||
|
||||
let rt = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap();
|
||||
let result = rt.block_on(detect_tile_changes("test_tile_ident", &data, &cache)).unwrap();
|
||||
assert!((result.diff_score - 0.0).abs() < 1e-9);
|
||||
assert_eq!(result.changed_pixels, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_detect_tile_changes_fully_different() {
|
||||
let cache = TileCache::new("/tmp/ruview-test-tile-changes");
|
||||
let old = vec![0u8; 100];
|
||||
let new = vec![255u8; 100];
|
||||
cache.put("test_tile_diff", &old).unwrap();
|
||||
|
||||
let rt = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap();
|
||||
let result = rt.block_on(detect_tile_changes("test_tile_diff", &new, &cache)).unwrap();
|
||||
assert!((result.diff_score - 1.0).abs() < 1e-9);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
//! SRTM DEM parser — elevation data from NASA 1-arcsecond HGT files.
|
||||
|
||||
use crate::cache::TileCache;
|
||||
use crate::types::{ElevationGrid, GeoPoint};
|
||||
use anyhow::Result;
|
||||
|
||||
/// Download and parse SRTM HGT for a location.
|
||||
pub async fn fetch_elevation(point: &GeoPoint, cache: &TileCache) -> Result<ElevationGrid> {
|
||||
let lat_int = point.lat.floor() as i32;
|
||||
let lon_int = point.lon.floor() as i32;
|
||||
let ns = if lat_int >= 0 { 'N' } else { 'S' };
|
||||
let ew = if lon_int >= 0 { 'E' } else { 'W' };
|
||||
let filename = format!("{}{:02}{}{:03}.hgt", ns, lat_int.unsigned_abs(), ew, lon_int.unsigned_abs());
|
||||
let cache_key = format!("srtm_{filename}");
|
||||
|
||||
if let Some(data) = cache.get(&cache_key) {
|
||||
return parse_hgt(&data, lat_int as f64, lon_int as f64);
|
||||
}
|
||||
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(30))
|
||||
.build()?;
|
||||
|
||||
// Primary: NASA SRTM public mirror (no auth required for .hgt)
|
||||
let nasa_url = format!(
|
||||
"https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/{filename}"
|
||||
);
|
||||
|
||||
if let Ok(resp) = client.get(&nasa_url).send().await {
|
||||
if resp.status().is_success() {
|
||||
let data = resp.bytes().await?.to_vec();
|
||||
cache.put(&cache_key, &data)?;
|
||||
return parse_hgt(&data, lat_int as f64, lon_int as f64);
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback: viewfinderpanoramas.org
|
||||
// Files are grouped by continent zip, but individual .hgt files can be
|
||||
// fetched directly when the server exposes them.
|
||||
let vfp_url = format!(
|
||||
"http://viewfinderpanoramas.org/dem1/{filename}"
|
||||
);
|
||||
|
||||
if let Ok(resp) = client.get(&vfp_url).send().await {
|
||||
if resp.status().is_success() {
|
||||
let data = resp.bytes().await?.to_vec();
|
||||
cache.put(&cache_key, &data)?;
|
||||
return parse_hgt(&data, lat_int as f64, lon_int as f64);
|
||||
}
|
||||
}
|
||||
|
||||
// Final fallback: flat terrain when all downloads fail
|
||||
Ok(ElevationGrid {
|
||||
origin_lat: lat_int as f64,
|
||||
origin_lon: lon_int as f64,
|
||||
cell_size_deg: 1.0 / 3600.0,
|
||||
cols: 100, rows: 100,
|
||||
heights: vec![0.0; 10000],
|
||||
})
|
||||
}
|
||||
|
||||
/// Parse SRTM HGT binary (3601x3601 big-endian i16).
|
||||
pub fn parse_hgt(data: &[u8], origin_lat: f64, origin_lon: f64) -> Result<ElevationGrid> {
|
||||
let n_samples = data.len() / 2;
|
||||
let side = (n_samples as f64).sqrt() as usize;
|
||||
|
||||
let heights: Vec<f32> = data.chunks_exact(2)
|
||||
.map(|c| {
|
||||
let v = i16::from_be_bytes([c[0], c[1]]);
|
||||
if v == -32768 { 0.0 } else { v as f32 } // -32768 = void
|
||||
})
|
||||
.collect();
|
||||
|
||||
Ok(ElevationGrid {
|
||||
origin_lat, origin_lon,
|
||||
cell_size_deg: 1.0 / (side - 1) as f64,
|
||||
cols: side, rows: side,
|
||||
heights,
|
||||
})
|
||||
}
|
||||
|
||||
/// Get elevation at a specific point from a grid.
|
||||
pub fn elevation_at(grid: &ElevationGrid, point: &GeoPoint) -> f32 {
|
||||
grid.get(point.lat, point.lon).unwrap_or(0.0)
|
||||
}
|
||||
|
||||
/// Extract a small subgrid around a point.
|
||||
pub fn extract_subgrid(grid: &ElevationGrid, center: &GeoPoint, radius_m: f64) -> ElevationGrid {
|
||||
let radius_deg = radius_m / 111_320.0;
|
||||
let min_row = ((grid.origin_lat + (grid.rows as f64 * grid.cell_size_deg) - center.lat - radius_deg) / grid.cell_size_deg).max(0.0) as usize;
|
||||
let max_row = ((grid.origin_lat + (grid.rows as f64 * grid.cell_size_deg) - center.lat + radius_deg) / grid.cell_size_deg).min(grid.rows as f64) as usize;
|
||||
let min_col = ((center.lon - radius_deg - grid.origin_lon) / grid.cell_size_deg).max(0.0) as usize;
|
||||
let max_col = ((center.lon + radius_deg - grid.origin_lon) / grid.cell_size_deg).min(grid.cols as f64) as usize;
|
||||
|
||||
let rows = max_row.saturating_sub(min_row);
|
||||
let cols = max_col.saturating_sub(min_col);
|
||||
let mut heights = Vec::with_capacity(rows * cols);
|
||||
for r in min_row..max_row {
|
||||
for c in min_col..max_col {
|
||||
heights.push(grid.heights.get(r * grid.cols + c).copied().unwrap_or(0.0));
|
||||
}
|
||||
}
|
||||
|
||||
ElevationGrid {
|
||||
origin_lat: grid.origin_lat + (grid.rows - max_row) as f64 * grid.cell_size_deg,
|
||||
origin_lon: grid.origin_lon + min_col as f64 * grid.cell_size_deg,
|
||||
cell_size_deg: grid.cell_size_deg,
|
||||
cols, rows, heights,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
//! Satellite tile fetcher — XYZ/TMS tile download with caching.
|
||||
|
||||
use crate::cache::TileCache;
|
||||
use crate::coord;
|
||||
use crate::types::{GeoBBox, RasterTile, TileCoord};
|
||||
use anyhow::Result;
|
||||
|
||||
/// Tile provider (all free, no API keys).
|
||||
pub enum TileProvider {
|
||||
/// Sentinel-2 cloudless mosaic (EOX, 10m, CC-BY-4.0)
|
||||
Sentinel2Cloudless,
|
||||
/// ESRI World Imagery (sub-meter, free tier)
|
||||
EsriWorldImagery,
|
||||
/// OpenStreetMap (map tiles, not satellite)
|
||||
Osm,
|
||||
}
|
||||
|
||||
impl TileProvider {
|
||||
pub fn url(&self, coord: &TileCoord) -> String {
|
||||
match self {
|
||||
Self::Sentinel2Cloudless => format!(
|
||||
"https://tiles.maps.eox.at/wmts/1.0.0/s2cloudless-2021_3857/default/g/{}/{}/{}.jpg",
|
||||
coord.z, coord.y, coord.x
|
||||
),
|
||||
Self::EsriWorldImagery => format!(
|
||||
"https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{}/{}/{}",
|
||||
coord.z, coord.y, coord.x
|
||||
),
|
||||
Self::Osm => format!(
|
||||
"https://tile.openstreetmap.org/{}/{}/{}.png",
|
||||
coord.z, coord.x, coord.y
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn name(&self) -> &str {
|
||||
match self {
|
||||
Self::Sentinel2Cloudless => "sentinel2",
|
||||
Self::EsriWorldImagery => "esri",
|
||||
Self::Osm => "osm",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Fetch a single tile with caching.
|
||||
pub async fn fetch_tile(provider: &TileProvider, coord: &TileCoord, cache: &TileCache) -> Result<RasterTile> {
|
||||
let cache_key = format!("tiles_{}_{}_{}.dat", coord.z, coord.x, coord.y);
|
||||
|
||||
if let Some(data) = cache.get(&cache_key) {
|
||||
return Ok(RasterTile { coord: coord.clone(), data, bounds: coord::tile_bounds(coord) });
|
||||
}
|
||||
|
||||
let url = provider.url(coord);
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(10))
|
||||
.user_agent("RuView/0.1 (https://github.com/ruvnet/RuView)")
|
||||
.build()?;
|
||||
|
||||
let resp = client.get(&url).send().await?;
|
||||
if !resp.status().is_success() {
|
||||
anyhow::bail!("Tile fetch failed: {} → {}", url, resp.status());
|
||||
}
|
||||
let data = resp.bytes().await?.to_vec();
|
||||
cache.put(&cache_key, &data)?;
|
||||
|
||||
Ok(RasterTile { coord: coord.clone(), data, bounds: coord::tile_bounds(coord) })
|
||||
}
|
||||
|
||||
/// Fetch all tiles covering a bounding box.
|
||||
pub async fn fetch_area(provider: &TileProvider, bbox: &GeoBBox, zoom: u8, cache: &TileCache) -> Result<Vec<RasterTile>> {
|
||||
let coords = coord::tiles_for_bbox(bbox, zoom);
|
||||
let mut tiles = Vec::with_capacity(coords.len());
|
||||
for c in &coords {
|
||||
match fetch_tile(provider, c, cache).await {
|
||||
Ok(t) => tiles.push(t),
|
||||
Err(e) => eprintln!(" Tile {}/{}/{} failed: {}", c.z, c.x, c.y, e),
|
||||
}
|
||||
}
|
||||
Ok(tiles)
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
//! Core geospatial types.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// WGS84 geographic coordinate.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct GeoPoint {
|
||||
pub lat: f64,
|
||||
pub lon: f64,
|
||||
pub alt: f64,
|
||||
}
|
||||
|
||||
/// Axis-aligned bounding box in WGS84.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct GeoBBox {
|
||||
pub south: f64,
|
||||
pub west: f64,
|
||||
pub north: f64,
|
||||
pub east: f64,
|
||||
}
|
||||
|
||||
impl GeoBBox {
|
||||
pub fn from_center(center: &GeoPoint, radius_m: f64) -> Self {
|
||||
let dlat = radius_m / 111_320.0;
|
||||
let dlon = radius_m / (111_320.0 * center.lat.to_radians().cos());
|
||||
Self {
|
||||
south: center.lat - dlat,
|
||||
west: center.lon - dlon,
|
||||
north: center.lat + dlat,
|
||||
east: center.lon + dlon,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// XYZ tile address.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct TileCoord {
|
||||
pub z: u8,
|
||||
pub x: u32,
|
||||
pub y: u32,
|
||||
}
|
||||
|
||||
/// Satellite raster tile.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct RasterTile {
|
||||
pub coord: TileCoord,
|
||||
pub data: Vec<u8>,
|
||||
pub bounds: GeoBBox,
|
||||
}
|
||||
|
||||
/// Elevation grid from SRTM DEM.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct ElevationGrid {
|
||||
pub origin_lat: f64,
|
||||
pub origin_lon: f64,
|
||||
pub cell_size_deg: f64,
|
||||
pub cols: usize,
|
||||
pub rows: usize,
|
||||
pub heights: Vec<f32>,
|
||||
}
|
||||
|
||||
impl ElevationGrid {
|
||||
pub fn get(&self, lat: f64, lon: f64) -> Option<f32> {
|
||||
let row = ((self.origin_lat + (self.rows as f64 * self.cell_size_deg) - lat) / self.cell_size_deg) as usize;
|
||||
let col = ((lon - self.origin_lon) / self.cell_size_deg) as usize;
|
||||
if row < self.rows && col < self.cols {
|
||||
Some(self.heights[row * self.cols + col])
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// OpenStreetMap feature.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub enum OsmFeature {
|
||||
Building {
|
||||
outline: Vec<[f64; 2]>,
|
||||
height: Option<f32>,
|
||||
name: Option<String>,
|
||||
},
|
||||
Road {
|
||||
path: Vec<[f64; 2]>,
|
||||
road_type: String,
|
||||
name: Option<String>,
|
||||
},
|
||||
}
|
||||
|
||||
/// Geo-registration transform.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct GeoRegistration {
|
||||
pub origin: GeoPoint,
|
||||
pub heading_deg: f64,
|
||||
pub scale: f64,
|
||||
}
|
||||
|
||||
impl Default for GeoRegistration {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
origin: GeoPoint { lat: 0.0, lon: 0.0, alt: 0.0 },
|
||||
heading_deg: 0.0,
|
||||
scale: 1.0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Complete geo scene.
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct GeoScene {
|
||||
pub location: GeoPoint,
|
||||
pub bbox: GeoBBox,
|
||||
pub elevation_m: f32,
|
||||
pub buildings: Vec<OsmFeature>,
|
||||
pub roads: Vec<OsmFeature>,
|
||||
pub tile_count: usize,
|
||||
pub registration: GeoRegistration,
|
||||
pub last_updated: String,
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
use wifi_densepose_geo::*;
|
||||
use wifi_densepose_geo::coord;
|
||||
|
||||
#[test]
|
||||
fn test_haversine() {
|
||||
let toronto = GeoPoint { lat: 43.6532, lon: -79.3832, alt: 0.0 };
|
||||
let ottawa = GeoPoint { lat: 45.4215, lon: -75.6972, alt: 0.0 };
|
||||
let dist = coord::haversine(&toronto, &ottawa);
|
||||
assert!((dist - 353_000.0).abs() < 5_000.0, "Toronto-Ottawa ~353km, got {:.0}m", dist);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_wgs84_to_enu() {
|
||||
let origin = GeoPoint { lat: 43.0, lon: -79.0, alt: 100.0 };
|
||||
let point = GeoPoint { lat: 43.001, lon: -79.0, alt: 100.0 };
|
||||
let enu = coord::wgs84_to_enu(&point, &origin);
|
||||
assert!((enu[1] - 111.0).abs() < 5.0, "0.001 deg lat ~111m north, got {:.1}m", enu[1]);
|
||||
assert!(enu[0].abs() < 1.0, "same longitude should have ~0 east, got {:.1}m", enu[0]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_enu_roundtrip() {
|
||||
let origin = GeoPoint { lat: 43.6532, lon: -79.3832, alt: 76.0 };
|
||||
let local = [100.0, 200.0, 5.0]; // 100m east, 200m north, 5m up
|
||||
let geo = coord::enu_to_wgs84(&local, &origin);
|
||||
let back = coord::wgs84_to_enu(&geo, &origin);
|
||||
assert!((back[0] - local[0]).abs() < 0.01);
|
||||
assert!((back[1] - local[1]).abs() < 0.01);
|
||||
assert!((back[2] - local[2]).abs() < 0.01);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_tile_coords() {
|
||||
let tile = coord::wgs84_to_tile(43.6532, -79.3832, 16);
|
||||
assert!(tile.x > 0 && tile.y > 0);
|
||||
assert_eq!(tile.z, 16);
|
||||
let bounds = coord::tile_bounds(&tile);
|
||||
assert!(bounds.south < 43.66 && bounds.north > 43.64);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_tiles_for_bbox() {
|
||||
let bbox = GeoBBox::from_center(
|
||||
&GeoPoint { lat: 43.6532, lon: -79.3832, alt: 0.0 },
|
||||
500.0,
|
||||
);
|
||||
let tiles = coord::tiles_for_bbox(&bbox, 16);
|
||||
assert!(tiles.len() >= 4 && tiles.len() <= 25, "500m radius should need 4-25 tiles, got {}", tiles.len());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_geo_bbox_from_center() {
|
||||
let center = GeoPoint { lat: 43.0, lon: -79.0, alt: 0.0 };
|
||||
let bbox = GeoBBox::from_center(¢er, 1000.0);
|
||||
assert!(bbox.south < 43.0 && bbox.north > 43.0);
|
||||
assert!(bbox.west < -79.0 && bbox.east > -79.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_hgt_parse() {
|
||||
// Create minimal 3x3 HGT data (big-endian i16)
|
||||
let mut data = Vec::new();
|
||||
for h in [100i16, 110, 120, 105, 115, 125, 110, 120, 130] {
|
||||
data.extend_from_slice(&h.to_be_bytes());
|
||||
}
|
||||
let grid = wifi_densepose_geo::terrain::parse_hgt(&data, 43.0, -79.0).unwrap();
|
||||
assert_eq!(grid.heights[0], 100.0);
|
||||
assert_eq!(grid.heights[4], 115.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_registration() {
|
||||
let origin = GeoPoint { lat: 43.6532, lon: -79.3832, alt: 76.0 };
|
||||
let reg = wifi_densepose_geo::register::auto_register(&origin);
|
||||
|
||||
let local = [10.0f32, 0.0, 20.0]; // 10m east, 20m forward
|
||||
let geo = wifi_densepose_geo::register::local_to_wgs84(®, &local);
|
||||
assert!((geo.lat - origin.lat).abs() < 0.001);
|
||||
assert!((geo.lon - origin.lon).abs() < 0.001);
|
||||
|
||||
let back = wifi_densepose_geo::register::wgs84_to_local(®, &geo);
|
||||
assert!((back[0] - local[0]).abs() < 0.1);
|
||||
assert!((back[2] - local[2]).abs() < 0.1);
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
[package]
|
||||
name = "wifi-densepose-pointcloud"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
description = "Real-time dense point cloud from camera depth + WiFi CSI tomography"
|
||||
|
||||
[[bin]]
|
||||
name = "ruview-pointcloud"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dependencies]
|
||||
serde = { workspace = true }
|
||||
serde_json = { workspace = true }
|
||||
tokio = { workspace = true }
|
||||
anyhow = { workspace = true }
|
||||
axum = { workspace = true }
|
||||
clap = { version = "4", features = ["derive"] }
|
||||
chrono = "0.4"
|
||||
dirs = "5"
|
||||
reqwest = { version = "0.12", features = ["json"], default-features = false }
|
||||
@@ -0,0 +1,92 @@
|
||||
//! Brain bridge — sends spatial observations to the ruOS brain.
|
||||
//!
|
||||
//! Periodically summarizes the sensor pipeline state and stores it
|
||||
//! as brain memories for the agent to reason about.
|
||||
//!
|
||||
//! The brain URL is read from the `RUVIEW_BRAIN_URL` env var on first use,
|
||||
//! defaulting to `http://127.0.0.1:9876`.
|
||||
|
||||
use crate::csi_pipeline::PipelineOutput;
|
||||
use anyhow::Result;
|
||||
use std::sync::OnceLock;
|
||||
|
||||
/// Default brain URL if `RUVIEW_BRAIN_URL` is not set.
|
||||
const DEFAULT_BRAIN_URL: &str = "http://127.0.0.1:9876";
|
||||
|
||||
fn brain_url() -> &'static str {
|
||||
static BRAIN_URL: OnceLock<String> = OnceLock::new();
|
||||
BRAIN_URL.get_or_init(|| {
|
||||
let url = std::env::var("RUVIEW_BRAIN_URL")
|
||||
.unwrap_or_else(|_| DEFAULT_BRAIN_URL.to_string());
|
||||
eprintln!(" brain_bridge: using brain URL {url}");
|
||||
url
|
||||
})
|
||||
}
|
||||
|
||||
/// Store a spatial observation in the brain.
|
||||
async fn store_memory(category: &str, content: &str) -> Result<()> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(5))
|
||||
.build()?;
|
||||
|
||||
let body = serde_json::json!({
|
||||
"category": category,
|
||||
"content": content,
|
||||
});
|
||||
|
||||
client.post(format!("{}/memories", brain_url()))
|
||||
.json(&body)
|
||||
.send()
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Summarize pipeline state and store in brain (called every 60 seconds).
|
||||
pub async fn sync_to_brain(pipeline: &PipelineOutput, camera_frames: u64) {
|
||||
// Only store if there's meaningful data
|
||||
if pipeline.total_frames < 10 && camera_frames < 5 { return; }
|
||||
|
||||
// Store spatial summary
|
||||
let motion_str = if pipeline.motion_detected { "detected" } else { "absent" };
|
||||
let skeleton_str = if let Some(ref sk) = pipeline.skeleton {
|
||||
format!("{} keypoints ({:.0}% conf)", sk.keypoints.len(), sk.confidence * 100.0)
|
||||
} else {
|
||||
"inactive".to_string()
|
||||
};
|
||||
|
||||
let summary = format!(
|
||||
"Room scan: {} camera frames, {} CSI frames from {} nodes. \
|
||||
Motion {} ({:.0}%). Breathing {:.0} BPM. Skeleton: {}. \
|
||||
Occupancy grid {}x{}x{} with {} occupied voxels.",
|
||||
camera_frames,
|
||||
pipeline.total_frames,
|
||||
pipeline.num_nodes,
|
||||
motion_str,
|
||||
pipeline.vitals.motion_score * 100.0,
|
||||
pipeline.vitals.breathing_rate,
|
||||
skeleton_str,
|
||||
pipeline.occupancy_dims.0,
|
||||
pipeline.occupancy_dims.1,
|
||||
pipeline.occupancy_dims.2,
|
||||
pipeline.occupancy.iter().filter(|&&d| d > 0.3).count(),
|
||||
);
|
||||
|
||||
let _ = store_memory("spatial-observation", &summary).await;
|
||||
|
||||
// Store motion events
|
||||
if pipeline.motion_detected && pipeline.vitals.motion_score > 0.3 {
|
||||
let _ = store_memory("spatial-motion",
|
||||
&format!("Strong motion detected: {:.0}% score, {} CSI frames",
|
||||
pipeline.vitals.motion_score * 100.0, pipeline.total_frames)
|
||||
).await;
|
||||
}
|
||||
|
||||
// Store vital signs if available
|
||||
if pipeline.vitals.breathing_rate > 5.0 && pipeline.vitals.breathing_rate < 35.0 {
|
||||
let _ = store_memory("spatial-vitals",
|
||||
&format!("Vital signs: breathing {:.0} BPM, motion {:.0}%",
|
||||
pipeline.vitals.breathing_rate, pipeline.vitals.motion_score * 100.0)
|
||||
).await;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,212 @@
|
||||
//! Camera capture — cross-platform frame grabber.
|
||||
//!
|
||||
//! macOS: uses `screencapture` or `ffmpeg -f avfoundation` for camera frames
|
||||
//! Linux: uses `v4l2-ctl` or `ffmpeg -f v4l2` for camera frames
|
||||
//! Both: capture to JPEG, decode to RGB, return raw pixel data
|
||||
|
||||
use anyhow::{bail, Result};
|
||||
use std::process::Command;
|
||||
use std::path::PathBuf;
|
||||
|
||||
/// Captured frame with raw RGB data.
|
||||
pub struct Frame {
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
pub rgb: Vec<u8>, // row-major [height * width * 3]
|
||||
}
|
||||
|
||||
/// Camera source configuration.
|
||||
pub struct CameraConfig {
|
||||
pub device_index: u32,
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
pub fps: u32,
|
||||
}
|
||||
|
||||
impl Default for CameraConfig {
|
||||
fn default() -> Self {
|
||||
Self { device_index: 0, width: 640, height: 480, fps: 15 }
|
||||
}
|
||||
}
|
||||
|
||||
/// Capture a single frame from the camera.
|
||||
///
|
||||
/// Tries multiple backends in order: ffmpeg, v4l2, imagesnap (macOS).
|
||||
pub fn capture_frame(config: &CameraConfig) -> Result<Frame> {
|
||||
let tmp = tmp_path();
|
||||
|
||||
// Try ffmpeg first (cross-platform)
|
||||
if let Ok(frame) = capture_ffmpeg(config, &tmp) {
|
||||
return Ok(frame);
|
||||
}
|
||||
|
||||
// Linux: try v4l2
|
||||
#[cfg(target_os = "linux")]
|
||||
if let Ok(frame) = capture_v4l2(config, &tmp) {
|
||||
return Ok(frame);
|
||||
}
|
||||
|
||||
// macOS: try screencapture (camera mode)
|
||||
#[cfg(target_os = "macos")]
|
||||
if let Ok(frame) = capture_macos(config, &tmp) {
|
||||
return Ok(frame);
|
||||
}
|
||||
|
||||
bail!("No camera backend available. Install ffmpeg or run on a machine with a camera.")
|
||||
}
|
||||
|
||||
/// Capture via ffmpeg (works on Linux + macOS).
|
||||
fn capture_ffmpeg(config: &CameraConfig, tmp: &PathBuf) -> Result<Frame> {
|
||||
let input = if cfg!(target_os = "macos") {
|
||||
format!("{}:none", config.device_index) // avfoundation: video:audio
|
||||
} else {
|
||||
format!("/dev/video{}", config.device_index) // v4l2
|
||||
};
|
||||
|
||||
let format = if cfg!(target_os = "macos") { "avfoundation" } else { "v4l2" };
|
||||
|
||||
let status = Command::new("ffmpeg")
|
||||
.args([
|
||||
"-y", "-f", format,
|
||||
"-video_size", &format!("{}x{}", config.width, config.height),
|
||||
"-framerate", &config.fps.to_string(),
|
||||
"-i", &input,
|
||||
"-frames:v", "1",
|
||||
"-f", "rawvideo",
|
||||
"-pix_fmt", "rgb24",
|
||||
tmp.to_str().unwrap_or("/tmp/ruview-frame.raw"),
|
||||
])
|
||||
.output()?;
|
||||
|
||||
if !status.status.success() {
|
||||
bail!("ffmpeg capture failed: {}", String::from_utf8_lossy(&status.stderr));
|
||||
}
|
||||
|
||||
let rgb = std::fs::read(tmp)?;
|
||||
let expected = (config.width * config.height * 3) as usize;
|
||||
if rgb.len() < expected {
|
||||
bail!("frame too small: {} bytes, expected {}", rgb.len(), expected);
|
||||
}
|
||||
|
||||
let _ = std::fs::remove_file(tmp);
|
||||
|
||||
Ok(Frame {
|
||||
width: config.width,
|
||||
height: config.height,
|
||||
rgb: rgb[..expected].to_vec(),
|
||||
})
|
||||
}
|
||||
|
||||
/// Linux: capture via v4l2-ctl.
|
||||
#[cfg(target_os = "linux")]
|
||||
fn capture_v4l2(config: &CameraConfig, tmp: &PathBuf) -> Result<Frame> {
|
||||
let device = format!("/dev/video{}", config.device_index);
|
||||
if !std::path::Path::new(&device).exists() {
|
||||
bail!("no camera at {device}");
|
||||
}
|
||||
|
||||
// Use v4l2-ctl to grab a frame
|
||||
let status = Command::new("v4l2-ctl")
|
||||
.args([
|
||||
"--device", &device,
|
||||
"--set-fmt-video", &format!("width={},height={},pixelformat=MJPG", config.width, config.height),
|
||||
"--stream-mmap", "--stream-count=1",
|
||||
"--stream-to", tmp.to_str().unwrap_or("/tmp/frame.mjpg"),
|
||||
])
|
||||
.output()?;
|
||||
|
||||
if !status.status.success() {
|
||||
bail!("v4l2-ctl failed");
|
||||
}
|
||||
|
||||
// Decode MJPEG to RGB
|
||||
decode_jpeg_to_rgb(tmp, config.width, config.height)
|
||||
}
|
||||
|
||||
/// macOS: capture via screencapture or swift.
|
||||
#[cfg(target_os = "macos")]
|
||||
fn capture_macos(config: &CameraConfig, tmp: &PathBuf) -> Result<Frame> {
|
||||
let jpg_path = tmp.with_extension("jpg");
|
||||
|
||||
// Try swift-based capture (requires camera permission)
|
||||
let swift = format!(
|
||||
r#"import AVFoundation; import AppKit
|
||||
let sem = DispatchSemaphore(value: 0)
|
||||
let s = AVCaptureSession(); s.sessionPreset = .medium
|
||||
guard let d = AVCaptureDevice.default(for: .video) else {{ exit(1) }}
|
||||
let i = try! AVCaptureDeviceInput(device: d); s.addInput(i)
|
||||
let o = AVCapturePhotoOutput(); s.addOutput(o)
|
||||
class D: NSObject, AVCapturePhotoCaptureDelegate {{
|
||||
func photoOutput(_ o: AVCapturePhotoOutput, didFinishProcessingPhoto p: AVCapturePhoto, error: Error?) {{
|
||||
if let d = p.fileDataRepresentation() {{ try! d.write(to: URL(fileURLWithPath: "{path}")) }}
|
||||
exit(0)
|
||||
}}
|
||||
}}
|
||||
let dl = D(); s.startRunning(); Thread.sleep(forTimeInterval: 1)
|
||||
o.capturePhoto(with: AVCapturePhotoSettings(), delegate: dl)
|
||||
Thread.sleep(forTimeInterval: 3)"#,
|
||||
path = jpg_path.display()
|
||||
);
|
||||
|
||||
let _ = Command::new("swift").args(["-e", &swift]).output();
|
||||
|
||||
if jpg_path.exists() {
|
||||
return decode_jpeg_to_rgb(&jpg_path, config.width, config.height);
|
||||
}
|
||||
|
||||
bail!("macOS camera capture requires GUI session with camera permission")
|
||||
}
|
||||
|
||||
fn decode_jpeg_to_rgb(path: &PathBuf, _width: u32, _height: u32) -> Result<Frame> {
|
||||
let data = std::fs::read(path)?;
|
||||
let _ = std::fs::remove_file(path);
|
||||
|
||||
// Simple JPEG decode — use the image crate if available, otherwise raw
|
||||
// For now, return the raw data and let the caller handle format
|
||||
Ok(Frame {
|
||||
width: _width,
|
||||
height: _height,
|
||||
rgb: data,
|
||||
})
|
||||
}
|
||||
|
||||
fn tmp_path() -> PathBuf {
|
||||
std::env::temp_dir().join(format!("ruview-frame-{}.raw", std::process::id()))
|
||||
}
|
||||
|
||||
/// Check if a camera is available on this system.
|
||||
pub fn camera_available() -> bool {
|
||||
if cfg!(target_os = "macos") {
|
||||
Command::new("system_profiler")
|
||||
.args(["SPCameraDataType"])
|
||||
.output()
|
||||
.map(|o| String::from_utf8_lossy(&o.stdout).contains("Camera"))
|
||||
.unwrap_or(false)
|
||||
} else {
|
||||
std::path::Path::new("/dev/video0").exists()
|
||||
}
|
||||
}
|
||||
|
||||
/// List available cameras.
|
||||
pub fn list_cameras() -> Vec<String> {
|
||||
let mut cameras = Vec::new();
|
||||
|
||||
if cfg!(target_os = "macos") {
|
||||
if let Ok(output) = Command::new("system_profiler").args(["SPCameraDataType"]).output() {
|
||||
let text = String::from_utf8_lossy(&output.stdout);
|
||||
for line in text.lines() {
|
||||
let trimmed = line.trim();
|
||||
if trimmed.ends_with(':') && !trimmed.starts_with("Camera") && trimmed.len() > 2 {
|
||||
cameras.push(trimmed.trim_end_matches(':').to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
for i in 0..10 {
|
||||
if std::path::Path::new(&format!("/dev/video{i}")).exists() {
|
||||
cameras.push(format!("/dev/video{i}"));
|
||||
}
|
||||
}
|
||||
}
|
||||
cameras
|
||||
}
|
||||
@@ -0,0 +1,663 @@
|
||||
//! Complete CSI processing pipeline — ADR-018 parser → heuristic pose → vitals → tomography.
|
||||
//!
|
||||
//! Receives raw UDP frames from ESP32 nodes, extracts I/Q subcarrier data,
|
||||
//! detects motion, estimates vitals, and produces 3D occupancy + skeleton
|
||||
//! for fusion with camera depth.
|
||||
//!
|
||||
//! **Note on pose**: the pose estimator here is an amplitude-energy
|
||||
//! heuristic — NOT a trained WiFlow model. See
|
||||
//! [`CsiPipelineState::heuristic_pose_from_amplitude`] for the exact shape.
|
||||
//! A real WiFlow integration requires loading and running the TCN weights,
|
||||
//! which this crate does not currently do.
|
||||
|
||||
use std::collections::VecDeque;
|
||||
use std::net::UdpSocket;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
// ADR-018 parser moved to src/parser.rs. Re-export here so downstream code
|
||||
// (and the reviewer's referenced public API) keeps working unchanged.
|
||||
pub use crate::parser::{parse_adr018, CsiFrame};
|
||||
|
||||
// ─── CSI Fingerprint Database ──────────────────────────────────────────────
|
||||
|
||||
#[derive(Clone, Debug, serde::Serialize)]
|
||||
pub struct CsiFingerprint {
|
||||
pub name: String,
|
||||
pub mean_amplitudes: Vec<f32>,
|
||||
pub rssi_mean: f32,
|
||||
pub rssi_std: f32,
|
||||
pub samples: u32,
|
||||
}
|
||||
|
||||
// ─── CSI State — accumulates frames for heuristic pose + vitals ───────────
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Skeleton {
|
||||
/// 17 COCO keypoints: [(x, y), ...] in [0, 1] normalized coordinates
|
||||
pub keypoints: Vec<[f32; 2]>,
|
||||
pub confidence: f32,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct VitalSigns {
|
||||
pub breathing_rate: f32, // breaths per minute
|
||||
pub heart_rate: f32, // beats per minute
|
||||
pub motion_score: f32, // 0.0 = still, 1.0 = strong motion
|
||||
}
|
||||
|
||||
pub struct CsiPipelineState {
|
||||
/// Per-node frame history (node_id → last N frames)
|
||||
pub node_frames: std::collections::HashMap<u8, VecDeque<CsiFrame>>,
|
||||
/// Latest skeleton from the amplitude-energy heuristic (NOT ML-derived)
|
||||
pub skeleton: Option<Skeleton>,
|
||||
/// Latest vital signs
|
||||
pub vitals: VitalSigns,
|
||||
/// Occupancy grid from RF tomography
|
||||
pub occupancy: Vec<f64>,
|
||||
pub occupancy_dims: (usize, usize, usize), // nx, ny, nz
|
||||
/// Total frames received
|
||||
pub total_frames: u64,
|
||||
/// Motion detection
|
||||
pub motion_detected: bool,
|
||||
/// CSI fingerprint database for room/location identification
|
||||
pub fingerprints: Vec<CsiFingerprint>,
|
||||
/// Current identified location (name, confidence) — updated every 100 frames
|
||||
pub current_location: Option<(String, f32)>,
|
||||
/// Night mode — true when camera luminance is below threshold
|
||||
pub is_dark: bool,
|
||||
/// Metadata from the on-disk WiFlow JSON, if one is present. NOTE: the
|
||||
/// weights themselves are NOT loaded or executed in this crate — this
|
||||
/// flag merely enables the amplitude-energy heuristic pose code path.
|
||||
pose_model_present: Option<PoseModelMetadata>,
|
||||
}
|
||||
|
||||
/// Placeholder tag indicating the `wiflow-v1.json` file is present on disk.
|
||||
/// This does NOT contain real TCN weights — the actual pose estimator in
|
||||
/// this crate is an amplitude-energy heuristic, not a neural network. The
|
||||
/// struct itself is empty; we only care whether it exists (`Option::Some`
|
||||
/// means "heuristic enabled").
|
||||
struct PoseModelMetadata;
|
||||
|
||||
impl Default for CsiPipelineState {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
node_frames: std::collections::HashMap::new(),
|
||||
skeleton: None,
|
||||
vitals: VitalSigns { breathing_rate: 0.0, heart_rate: 0.0, motion_score: 0.0 },
|
||||
occupancy: vec![0.0; 8 * 8 * 4],
|
||||
occupancy_dims: (8, 8, 4),
|
||||
total_frames: 0,
|
||||
motion_detected: false,
|
||||
fingerprints: Vec::new(),
|
||||
current_location: None,
|
||||
is_dark: false,
|
||||
pose_model_present: detect_pose_model_metadata(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Pose Model Metadata Probe ──────────────────────────────────────────────
|
||||
//
|
||||
// NOTE: This only reads the shape metadata from `wiflow-v1.json` on disk.
|
||||
// The weights are NOT loaded or evaluated. The actual pose used by this
|
||||
// crate is an amplitude-energy heuristic (see
|
||||
// `heuristic_pose_from_amplitude`), not WiFlow.
|
||||
|
||||
fn detect_pose_model_metadata() -> Option<PoseModelMetadata> {
|
||||
let paths = [
|
||||
"/tmp/ruview-firmware/wiflow-v1.json",
|
||||
"~/.local/share/ruview/wiflow-v1.json",
|
||||
];
|
||||
for p in &paths {
|
||||
let expanded = p.replace('~', &std::env::var("HOME").unwrap_or_default());
|
||||
if let Ok(data) = std::fs::read_to_string(&expanded) {
|
||||
if let Ok(model) = serde_json::from_str::<serde_json::Value>(&data) {
|
||||
if model.get("weightsBase64").and_then(|v| v.as_str()).is_some() {
|
||||
eprintln!(
|
||||
" pose: amplitude-energy heuristic enabled (metadata from {expanded}, {} params — weights NOT loaded)",
|
||||
model.get("totalParams").and_then(|v| v.as_u64()).unwrap_or(0)
|
||||
);
|
||||
return Some(PoseModelMetadata);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
eprintln!(" pose: amplitude-energy heuristic disabled (no metadata file found)");
|
||||
None
|
||||
}
|
||||
|
||||
// ─── Pipeline Processing ────────────────────────────────────────────────────
|
||||
|
||||
impl CsiPipelineState {
|
||||
/// Process a new CSI frame — updates motion, vitals, skeleton, occupancy.
|
||||
pub fn process_frame(&mut self, frame: CsiFrame) {
|
||||
let node_id = frame.node_id;
|
||||
self.total_frames += 1;
|
||||
|
||||
// Once every 500 frames log a one-line node stats summary. This keeps
|
||||
// us honest about the CSI shape we are actually receiving and also
|
||||
// guarantees every public `CsiFrame` field is read on the runtime
|
||||
// path, not only in tests.
|
||||
if self.total_frames % 500 == 0 {
|
||||
eprintln!(
|
||||
" CSI node={} ch={} ant={} sub={} rssi={} nf={} ts_us={} iq_bytes={}",
|
||||
frame.node_id,
|
||||
frame.channel,
|
||||
frame.n_antennas,
|
||||
frame.n_subcarriers,
|
||||
frame.rssi,
|
||||
frame.noise_floor,
|
||||
frame.timestamp_us,
|
||||
frame.iq_data.len(),
|
||||
);
|
||||
}
|
||||
|
||||
// Store frame in per-node history
|
||||
{
|
||||
let history = self.node_frames.entry(node_id).or_insert_with(|| VecDeque::with_capacity(100));
|
||||
history.push_back(frame.clone());
|
||||
if history.len() > 100 { history.pop_front(); }
|
||||
}
|
||||
|
||||
// 1. Motion detection (amplitude variance over last 20 frames)
|
||||
self.detect_motion(node_id);
|
||||
|
||||
// 2. Vital signs (phase analysis over last 100 frames)
|
||||
let has_enough = self.node_frames.get(&node_id).map(|h| h.len() >= 30).unwrap_or(false);
|
||||
if has_enough {
|
||||
self.estimate_vitals(node_id);
|
||||
}
|
||||
|
||||
// 3. Heuristic pose estimation (every 20 frames = 1 second at ~20fps)
|
||||
if self.total_frames % 20 == 0 {
|
||||
self.heuristic_pose_from_amplitude();
|
||||
}
|
||||
|
||||
// 4. RF tomography (update occupancy grid)
|
||||
self.update_tomography();
|
||||
|
||||
// 5. Location fingerprint identification (every 100 frames)
|
||||
if self.total_frames % 100 == 0 {
|
||||
self.current_location = self.identify_location();
|
||||
}
|
||||
}
|
||||
|
||||
fn detect_motion(&mut self, node_id: u8) {
|
||||
if let Some(history) = self.node_frames.get(&node_id) {
|
||||
let recent: Vec<&CsiFrame> = history.iter().rev().take(20).collect();
|
||||
if recent.len() < 5 { return; }
|
||||
|
||||
// Compute mean amplitude across subcarriers for each frame
|
||||
let mean_amps: Vec<f32> = recent.iter()
|
||||
.map(|f| f.amplitudes.iter().sum::<f32>() / f.amplitudes.len().max(1) as f32)
|
||||
.collect();
|
||||
|
||||
let mean = mean_amps.iter().sum::<f32>() / mean_amps.len() as f32;
|
||||
let variance = mean_amps.iter().map(|a| (a - mean).powi(2)).sum::<f32>() / mean_amps.len() as f32;
|
||||
|
||||
// High variance = motion
|
||||
self.vitals.motion_score = (variance / 100.0).min(1.0);
|
||||
self.motion_detected = self.vitals.motion_score > 0.15;
|
||||
}
|
||||
}
|
||||
|
||||
fn estimate_vitals(&mut self, node_id: u8) {
|
||||
if let Some(history) = self.node_frames.get(&node_id) {
|
||||
let frames: Vec<&CsiFrame> = history.iter().rev().take(100).collect();
|
||||
if frames.len() < 30 { return; }
|
||||
|
||||
// Extract phase from a stable subcarrier (pick one with low variance)
|
||||
let n_sub = frames[0].phases.len().min(35);
|
||||
if n_sub == 0 { return; }
|
||||
|
||||
// Use subcarrier 15 (mid-band, typically stable)
|
||||
let sub_idx = n_sub / 2;
|
||||
let phase_series: Vec<f32> = frames.iter().rev()
|
||||
.map(|f| f.phases.get(sub_idx).copied().unwrap_or(0.0))
|
||||
.collect();
|
||||
|
||||
// Simple peak counting for breathing rate (0.15-0.5 Hz = 9-30 BPM)
|
||||
let mut peaks = 0;
|
||||
for i in 1..phase_series.len() - 1 {
|
||||
if phase_series[i] > phase_series[i-1] && phase_series[i] > phase_series[i+1] {
|
||||
peaks += 1;
|
||||
}
|
||||
}
|
||||
|
||||
// Assuming ~20fps capture, 100 frames = 5 seconds
|
||||
let capture_secs = frames.len() as f32 / 20.0;
|
||||
let breathing_bpm = (peaks as f32 / capture_secs) * 60.0;
|
||||
self.vitals.breathing_rate = breathing_bpm.clamp(5.0, 40.0);
|
||||
|
||||
// Heart rate estimation (0.8-2.5 Hz) — need higher sampling rate
|
||||
// For now, estimate from amplitude modulation
|
||||
self.vitals.heart_rate = 0.0; // requires FFT for accurate detection
|
||||
}
|
||||
}
|
||||
|
||||
/// STUB: not real WiFlow inference; returns an amplitude-energy heuristic
|
||||
/// "pose" built by bucketing CSI subcarrier energy into 17 fake keypoints.
|
||||
///
|
||||
/// This exists so the downstream viewer has something to render while the
|
||||
/// real WiFlow TCN integration is being wired up. The output should NOT
|
||||
/// be interpreted as an ML-derived skeleton — confidence here is just
|
||||
/// amplitude variance, keypoint x is subcarrier energy, y is the
|
||||
/// keypoint index. Callers that need real pose must use the (yet to be
|
||||
/// wired) WiFlow model directly.
|
||||
fn heuristic_pose_from_amplitude(&mut self) {
|
||||
if self.pose_model_present.is_none() { return; }
|
||||
|
||||
// Collect 20 frames from the primary node
|
||||
let primary_node = self.node_frames.keys().next().copied();
|
||||
if let Some(node_id) = primary_node {
|
||||
if let Some(history) = self.node_frames.get(&node_id) {
|
||||
let frames: Vec<&CsiFrame> = history.iter().rev().take(20).collect();
|
||||
if frames.len() < 20 { return; }
|
||||
|
||||
// Build input: 35 subcarriers × 20 time steps. This is a
|
||||
// deliberately simple summary used to compute amplitude
|
||||
// variance; it is NOT fed through any neural network.
|
||||
let n_sub = frames[0].amplitudes.len().min(35);
|
||||
let mut input = vec![0.0f32; 35 * 20];
|
||||
for (t, frame) in frames.iter().rev().enumerate().take(20) {
|
||||
for s in 0..n_sub {
|
||||
input[t * 35 + s] = frame.amplitudes.get(s).copied().unwrap_or(0.0) / 128.0;
|
||||
}
|
||||
}
|
||||
|
||||
let mean_amp = input.iter().sum::<f32>() / input.len() as f32;
|
||||
let amp_var = input.iter().map(|a| (a - mean_amp).powi(2)).sum::<f32>() / input.len() as f32;
|
||||
|
||||
// If motion detected, emit a placeholder skeleton derived from
|
||||
// signal characteristics. NOT a real pose.
|
||||
if self.motion_detected {
|
||||
let mut keypoints = vec![[0.5f32; 2]; 17];
|
||||
for (i, kp) in keypoints.iter_mut().enumerate() {
|
||||
let sub_range = (i * n_sub / 17)..((i + 1) * n_sub / 17).min(n_sub);
|
||||
let energy: f32 = sub_range.clone()
|
||||
.filter_map(|s| frames.last().and_then(|f| f.amplitudes.get(s)))
|
||||
.sum();
|
||||
let norm_energy = energy / (sub_range.len().max(1) as f32 * 128.0);
|
||||
kp[0] = 0.3 + norm_energy * 0.4; // x: subcarrier energy
|
||||
kp[1] = (i as f32 / 17.0) * 0.8 + 0.1; // y: keypoint index
|
||||
}
|
||||
self.skeleton = Some(Skeleton {
|
||||
keypoints,
|
||||
confidence: amp_var.min(1.0),
|
||||
});
|
||||
} else {
|
||||
self.skeleton = None;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Record a CSI fingerprint for the current location/room.
|
||||
/// Computes mean amplitude and RSSI statistics from the last 50 frames
|
||||
/// across all nodes and saves as a named fingerprint.
|
||||
pub fn record_fingerprint(&mut self, name: &str) {
|
||||
// Collect last 50 frames from all nodes
|
||||
let mut all_amplitudes: Vec<Vec<f32>> = Vec::new();
|
||||
let mut rssi_values: Vec<f32> = Vec::new();
|
||||
|
||||
for history in self.node_frames.values() {
|
||||
for frame in history.iter().rev().take(50) {
|
||||
all_amplitudes.push(frame.amplitudes.clone());
|
||||
rssi_values.push(frame.rssi as f32);
|
||||
}
|
||||
}
|
||||
|
||||
if all_amplitudes.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
// Compute mean amplitude per subcarrier across all collected frames
|
||||
let n_sub = all_amplitudes.iter().map(|a| a.len()).max().unwrap_or(0);
|
||||
if n_sub == 0 {
|
||||
return;
|
||||
}
|
||||
let mut mean_amplitudes = vec![0.0f32; n_sub];
|
||||
let mut counts = vec![0u32; n_sub];
|
||||
for amps in &all_amplitudes {
|
||||
for (i, &a) in amps.iter().enumerate() {
|
||||
if i < n_sub {
|
||||
mean_amplitudes[i] += a;
|
||||
counts[i] += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
for i in 0..n_sub {
|
||||
if counts[i] > 0 {
|
||||
mean_amplitudes[i] /= counts[i] as f32;
|
||||
}
|
||||
}
|
||||
|
||||
// RSSI statistics
|
||||
let rssi_mean = rssi_values.iter().sum::<f32>() / rssi_values.len() as f32;
|
||||
let rssi_var = rssi_values.iter()
|
||||
.map(|r| (r - rssi_mean).powi(2))
|
||||
.sum::<f32>() / rssi_values.len() as f32;
|
||||
let rssi_std = rssi_var.sqrt();
|
||||
|
||||
let fingerprint = CsiFingerprint {
|
||||
name: name.to_string(),
|
||||
mean_amplitudes,
|
||||
rssi_mean,
|
||||
rssi_std,
|
||||
samples: all_amplitudes.len() as u32,
|
||||
};
|
||||
|
||||
// Replace existing fingerprint with same name, or append
|
||||
if let Some(existing) = self.fingerprints.iter_mut().find(|f| f.name == name) {
|
||||
*existing = fingerprint;
|
||||
} else {
|
||||
self.fingerprints.push(fingerprint);
|
||||
}
|
||||
}
|
||||
|
||||
/// Compare current CSI signals against saved fingerprints using cosine
|
||||
/// similarity. Returns (name, confidence) if the best match exceeds 0.7.
|
||||
pub fn identify_location(&self) -> Option<(String, f32)> {
|
||||
if self.fingerprints.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
// Build current mean amplitude vector from last 50 frames
|
||||
let mut all_amplitudes: Vec<Vec<f32>> = Vec::new();
|
||||
for history in self.node_frames.values() {
|
||||
for frame in history.iter().rev().take(50) {
|
||||
all_amplitudes.push(frame.amplitudes.clone());
|
||||
}
|
||||
}
|
||||
if all_amplitudes.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let n_sub = all_amplitudes.iter().map(|a| a.len()).max().unwrap_or(0);
|
||||
if n_sub == 0 {
|
||||
return None;
|
||||
}
|
||||
let mut current = vec![0.0f32; n_sub];
|
||||
let mut counts = vec![0u32; n_sub];
|
||||
for amps in &all_amplitudes {
|
||||
for (i, &a) in amps.iter().enumerate() {
|
||||
if i < n_sub {
|
||||
current[i] += a;
|
||||
counts[i] += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
for i in 0..n_sub {
|
||||
if counts[i] > 0 {
|
||||
current[i] /= counts[i] as f32;
|
||||
}
|
||||
}
|
||||
|
||||
// Find best matching fingerprint by cosine similarity
|
||||
let mut best: Option<(String, f32)> = None;
|
||||
for fp in &self.fingerprints {
|
||||
let sim = cosine_similarity(¤t, &fp.mean_amplitudes);
|
||||
if sim > 0.7 {
|
||||
if best.as_ref().map_or(true, |(_, s)| sim > *s) {
|
||||
best = Some((fp.name.clone(), sim));
|
||||
}
|
||||
}
|
||||
}
|
||||
best
|
||||
}
|
||||
|
||||
/// Set the ambient light level from camera frame average luminance.
|
||||
/// When luminance < 30 (out of 255), enables night/dark mode which
|
||||
/// increases CSI processing frequency and skips camera depth.
|
||||
pub fn set_light_level(&mut self, avg_luminance: f32) {
|
||||
self.is_dark = avg_luminance < 30.0;
|
||||
}
|
||||
|
||||
fn update_tomography(&mut self) {
|
||||
let (nx, ny, nz) = self.occupancy_dims;
|
||||
let total = nx * ny * nz;
|
||||
|
||||
// Simple backprojection from per-node RSSI
|
||||
let mut new_occ = vec![0.0f64; total];
|
||||
for (node_id, history) in &self.node_frames {
|
||||
if let Some(latest) = history.back() {
|
||||
// RSSI-based attenuation → voxel density
|
||||
let atten = -(latest.rssi as f64);
|
||||
let contribution = atten / 100.0; // normalize
|
||||
|
||||
// Distribute based on node ID position (simplified ray model)
|
||||
let cx = match node_id {
|
||||
1 => nx / 4,
|
||||
2 => nx * 3 / 4,
|
||||
_ => nx / 2,
|
||||
};
|
||||
let cy = ny / 2;
|
||||
|
||||
for iz in 0..nz {
|
||||
for iy in 0..ny {
|
||||
for ix in 0..nx {
|
||||
let dx = (ix as f64 - cx as f64) / nx as f64;
|
||||
let dy = (iy as f64 - cy as f64) / ny as f64;
|
||||
let dist = (dx * dx + dy * dy).sqrt();
|
||||
let idx = iz * ny * nx + iy * nx + ix;
|
||||
// Gaussian-weighted contribution
|
||||
new_occ[idx] += contribution * (-dist * dist * 8.0).exp();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Normalize
|
||||
let max = new_occ.iter().cloned().fold(0.0f64, f64::max);
|
||||
if max > 0.0 {
|
||||
for d in &mut new_occ { *d /= max; }
|
||||
}
|
||||
|
||||
// Exponential moving average with previous occupancy
|
||||
for i in 0..total {
|
||||
self.occupancy[i] = self.occupancy[i] * 0.7 + new_occ[i] * 0.3;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Cosine similarity between two vectors. Returns 0.0 if either has zero magnitude.
|
||||
fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
|
||||
let len = a.len().min(b.len());
|
||||
if len == 0 {
|
||||
return 0.0;
|
||||
}
|
||||
let mut dot = 0.0f32;
|
||||
let mut mag_a = 0.0f32;
|
||||
let mut mag_b = 0.0f32;
|
||||
for i in 0..len {
|
||||
dot += a[i] * b[i];
|
||||
mag_a += a[i] * a[i];
|
||||
mag_b += b[i] * b[i];
|
||||
}
|
||||
let denom = mag_a.sqrt() * mag_b.sqrt();
|
||||
if denom < 1e-9 {
|
||||
0.0
|
||||
} else {
|
||||
dot / denom
|
||||
}
|
||||
}
|
||||
|
||||
// ─── UDP Receiver ───────────────────────────────────────────────────────────
|
||||
|
||||
/// Start the complete CSI pipeline — UDP receiver + processing.
|
||||
///
|
||||
/// Architecture (two threads, one std mpsc channel):
|
||||
///
|
||||
/// ```text
|
||||
/// UDP thread Processor thread
|
||||
/// ┌──────────────┐ mpsc::Sender ┌────────────────────┐
|
||||
/// │ recv_from() │ ─────────────► │ recv() CsiFrame │
|
||||
/// │ parse_adr018 │ (bounded-ish │ lock, process_frame│
|
||||
/// └──────────────┘ by channel) │ unlock │
|
||||
/// └────────────────────┘
|
||||
/// ```
|
||||
///
|
||||
/// This decouples the socket from the shared state: the UDP thread only
|
||||
/// touches the channel, never the mutex. The HTTP API handlers (which call
|
||||
/// `get_pipeline_output`) therefore only contend with the processor thread
|
||||
/// for brief periods, not with every incoming packet. Heavy work (pose,
|
||||
/// tomography, fingerprinting) runs outside the lock.
|
||||
pub fn start_pipeline(bind_addr: &str) -> Arc<Mutex<CsiPipelineState>> {
|
||||
let state = Arc::new(Mutex::new(CsiPipelineState::default()));
|
||||
let processor_state = state.clone();
|
||||
|
||||
let (tx, rx) = std::sync::mpsc::channel::<CsiFrame>();
|
||||
|
||||
// --- UDP thread: read + parse, push to channel (no lock held) ---
|
||||
let addr = bind_addr.to_string();
|
||||
std::thread::spawn(move || {
|
||||
let socket = match UdpSocket::bind(&addr) {
|
||||
Ok(s) => s,
|
||||
Err(e) => {
|
||||
eprintln!(" CSI pipeline: bind failed on {addr}: {e}");
|
||||
return;
|
||||
}
|
||||
};
|
||||
socket.set_read_timeout(Some(std::time::Duration::from_secs(1))).unwrap();
|
||||
eprintln!(" CSI pipeline: listening on {addr}");
|
||||
|
||||
let mut buf = [0u8; 2048];
|
||||
loop {
|
||||
match socket.recv_from(&mut buf) {
|
||||
Ok((n, _)) => {
|
||||
if let Some(frame) = parse_adr018(&buf[..n]) {
|
||||
// Non-blocking w.r.t. the shared state lock. If the
|
||||
// processor thread has died, send() fails and we
|
||||
// exit the receiver.
|
||||
if tx.send(frame).is_err() {
|
||||
eprintln!(" CSI pipeline: processor gone, exiting receiver");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => continue,
|
||||
Err(_) => continue,
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// --- Processor thread: drain channel, take lock briefly to publish ---
|
||||
std::thread::spawn(move || {
|
||||
while let Ok(frame) = rx.recv() {
|
||||
// Lock is held only for the duration of one process_frame call;
|
||||
// HTTP handlers that need a snapshot via get_pipeline_output are
|
||||
// never starved by the UDP read loop.
|
||||
if let Ok(mut st) = processor_state.lock() {
|
||||
st.process_frame(frame);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
state
|
||||
}
|
||||
|
||||
/// Send synthetic ADR-018 binary CSI frames for local testing without real
|
||||
/// ESP32 hardware. Each frame carries `n_subcarriers` subcarriers of fake
|
||||
/// I/Q data. Targets `target` (e.g. `127.0.0.1:3333`).
|
||||
pub fn send_test_frames(target: &str, count: usize) -> anyhow::Result<()> {
|
||||
use crate::parser::{build_test_frame, MAGIC_V1};
|
||||
let socket = UdpSocket::bind("0.0.0.0:0")?;
|
||||
for i in 0..count {
|
||||
let buf = build_test_frame(MAGIC_V1, (i % 4) as u8, 56, i);
|
||||
socket.send_to(&buf, target)?;
|
||||
std::thread::sleep(std::time::Duration::from_millis(10));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Get current pipeline output for fusion.
|
||||
pub fn get_pipeline_output(state: &Arc<Mutex<CsiPipelineState>>) -> PipelineOutput {
|
||||
let st = state.lock().unwrap();
|
||||
PipelineOutput {
|
||||
skeleton: st.skeleton.clone(),
|
||||
vitals: st.vitals.clone(),
|
||||
occupancy: st.occupancy.clone(),
|
||||
occupancy_dims: st.occupancy_dims,
|
||||
motion_detected: st.motion_detected,
|
||||
total_frames: st.total_frames,
|
||||
num_nodes: st.node_frames.len(),
|
||||
current_location: st.current_location.clone(),
|
||||
is_dark: st.is_dark,
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, serde::Serialize)]
|
||||
pub struct PipelineOutput {
|
||||
pub skeleton: Option<Skeleton>,
|
||||
pub vitals: VitalSigns,
|
||||
pub occupancy: Vec<f64>,
|
||||
pub occupancy_dims: (usize, usize, usize),
|
||||
pub motion_detected: bool,
|
||||
pub total_frames: u64,
|
||||
pub num_nodes: usize,
|
||||
pub current_location: Option<(String, f32)>,
|
||||
pub is_dark: bool,
|
||||
}
|
||||
|
||||
// Serialize implementations
|
||||
impl serde::Serialize for Skeleton {
|
||||
fn serialize<S: serde::Serializer>(&self, s: S) -> Result<S::Ok, S::Error> {
|
||||
use serde::ser::SerializeStruct;
|
||||
let mut st = s.serialize_struct("Skeleton", 2)?;
|
||||
st.serialize_field("keypoints", &self.keypoints)?;
|
||||
st.serialize_field("confidence", &self.confidence)?;
|
||||
st.end()
|
||||
}
|
||||
}
|
||||
|
||||
impl serde::Serialize for VitalSigns {
|
||||
fn serialize<S: serde::Serializer>(&self, s: S) -> Result<S::Ok, S::Error> {
|
||||
use serde::ser::SerializeStruct;
|
||||
let mut st = s.serialize_struct("VitalSigns", 3)?;
|
||||
st.serialize_field("breathing_rate", &self.breathing_rate)?;
|
||||
st.serialize_field("heart_rate", &self.heart_rate)?;
|
||||
st.serialize_field("motion_score", &self.motion_score)?;
|
||||
st.end()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::parser::{build_test_frame, parse_adr018, MAGIC_V1};
|
||||
|
||||
fn seed_state_with_frames(state: &mut CsiPipelineState, n: usize) {
|
||||
for i in 0..n {
|
||||
let bytes = build_test_frame(MAGIC_V1, 1, 32, i);
|
||||
let frame = parse_adr018(&bytes).expect("synthetic frame must parse");
|
||||
state.process_frame(frame);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn set_light_level_toggles_night_mode() {
|
||||
let mut s = CsiPipelineState::default();
|
||||
assert!(!s.is_dark, "default should be daylight");
|
||||
s.set_light_level(10.0);
|
||||
assert!(s.is_dark, "luminance below 30 → dark");
|
||||
s.set_light_level(200.0);
|
||||
assert!(!s.is_dark, "high luminance → not dark");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn record_fingerprint_stores_and_matches() {
|
||||
let mut s = CsiPipelineState::default();
|
||||
seed_state_with_frames(&mut s, 30);
|
||||
s.record_fingerprint("lab");
|
||||
assert_eq!(s.fingerprints.len(), 1);
|
||||
assert_eq!(s.fingerprints[0].name, "lab");
|
||||
// Identify against its own fingerprint should succeed.
|
||||
let found = s.identify_location();
|
||||
assert!(found.is_some(), "should identify the just-recorded location");
|
||||
if let Some((name, conf)) = found {
|
||||
assert_eq!(name, "lab");
|
||||
assert!(conf > 0.7, "self-similarity should exceed match threshold");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,263 @@
|
||||
//! Monocular depth estimation via MiDaS ONNX + backprojection to 3D points.
|
||||
#![allow(dead_code)]
|
||||
|
||||
use crate::pointcloud::{PointCloud, ColorPoint};
|
||||
use anyhow::Result;
|
||||
|
||||
/// Default camera intrinsics (approximate for HD webcam)
|
||||
pub struct CameraIntrinsics {
|
||||
pub fx: f32, // focal length x (pixels)
|
||||
pub fy: f32, // focal length y (pixels)
|
||||
pub cx: f32, // principal point x
|
||||
pub cy: f32, // principal point y
|
||||
pub width: u32,
|
||||
pub height: u32,
|
||||
}
|
||||
|
||||
impl Default for CameraIntrinsics {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
fx: 525.0, fy: 525.0, // typical webcam focal length
|
||||
cx: 320.0, cy: 240.0, // center of 640x480
|
||||
width: 640, height: 480,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Backproject a depth map to 3D points using camera intrinsics.
|
||||
///
|
||||
/// depth_map: row-major [height x width] in meters
|
||||
/// rgb: optional row-major [height x width x 3] color
|
||||
pub fn backproject_depth(
|
||||
depth_map: &[f32],
|
||||
intrinsics: &CameraIntrinsics,
|
||||
rgb: Option<&[u8]>,
|
||||
downsample: u32,
|
||||
) -> PointCloud {
|
||||
let mut cloud = PointCloud::new("camera_depth");
|
||||
let w = intrinsics.width;
|
||||
let h = intrinsics.height;
|
||||
let step = downsample.max(1);
|
||||
|
||||
for y in (0..h).step_by(step as usize) {
|
||||
for x in (0..w).step_by(step as usize) {
|
||||
let idx = (y * w + x) as usize;
|
||||
let z = depth_map[idx];
|
||||
|
||||
// Skip invalid depths
|
||||
if z <= 0.01 || z > 10.0 || z.is_nan() { continue; }
|
||||
|
||||
// Backproject: (u, v, z) → (X, Y, Z)
|
||||
let px = (x as f32 - intrinsics.cx) * z / intrinsics.fx;
|
||||
let py = (y as f32 - intrinsics.cy) * z / intrinsics.fy;
|
||||
|
||||
let (r, g, b) = if let Some(rgb_data) = rgb {
|
||||
let ri = idx * 3;
|
||||
if ri + 2 < rgb_data.len() {
|
||||
(rgb_data[ri], rgb_data[ri + 1], rgb_data[ri + 2])
|
||||
} else {
|
||||
(128, 128, 128)
|
||||
}
|
||||
} else {
|
||||
// Color by depth (blue=near, red=far)
|
||||
let t = ((z - 0.5) / 4.0).clamp(0.0, 1.0);
|
||||
((t * 255.0) as u8, ((1.0 - t) * 128.0) as u8, ((1.0 - t) * 255.0) as u8)
|
||||
};
|
||||
|
||||
cloud.points.push(ColorPoint { x: px, y: py, z, r, g, b, intensity: 1.0 });
|
||||
}
|
||||
}
|
||||
cloud
|
||||
}
|
||||
|
||||
/// Run depth estimation on an image.
|
||||
///
|
||||
/// Tries MiDaS GPU server (127.0.0.1:9885) first, falls back to luminance+edges.
|
||||
pub fn estimate_depth(
|
||||
image_data: &[u8],
|
||||
width: u32,
|
||||
height: u32,
|
||||
) -> Result<Vec<f32>> {
|
||||
// Try MiDaS GPU server
|
||||
if let Ok(depth) = estimate_depth_midas_server(image_data, width, height) {
|
||||
return Ok(depth);
|
||||
}
|
||||
|
||||
// Fallback: luminance + edge-based pseudo-depth
|
||||
let w = width as usize;
|
||||
let h = height as usize;
|
||||
let mut lum = vec![0.0f32; w * h];
|
||||
for i in 0..w * h {
|
||||
let ri = i * 3;
|
||||
if ri + 2 < image_data.len() {
|
||||
lum[i] = (0.299 * image_data[ri] as f32
|
||||
+ 0.587 * image_data[ri + 1] as f32
|
||||
+ 0.114 * image_data[ri + 2] as f32) / 255.0;
|
||||
}
|
||||
}
|
||||
let mut edges = vec![0.0f32; w * h];
|
||||
for y in 1..h - 1 {
|
||||
for x in 1..w - 1 {
|
||||
let gx = -lum[(y-1)*w+x-1] + lum[(y-1)*w+x+1]
|
||||
- 2.0*lum[y*w+x-1] + 2.0*lum[y*w+x+1]
|
||||
- lum[(y+1)*w+x-1] + lum[(y+1)*w+x+1];
|
||||
let gy = -lum[(y-1)*w+x-1] - 2.0*lum[(y-1)*w+x] - lum[(y-1)*w+x+1]
|
||||
+ lum[(y+1)*w+x-1] + 2.0*lum[(y+1)*w+x] + lum[(y+1)*w+x+1];
|
||||
edges[y * w + x] = (gx * gx + gy * gy).sqrt().min(1.0);
|
||||
}
|
||||
}
|
||||
let mut depth_map = vec![3.0f32; w * h];
|
||||
for i in 0..w * h {
|
||||
let base = 1.0 + (1.0 - lum[i]) * 3.5;
|
||||
let edge_boost = edges[i] * 1.5;
|
||||
depth_map[i] = (base - edge_boost).max(0.3);
|
||||
}
|
||||
Ok(depth_map)
|
||||
}
|
||||
|
||||
/// Call MiDaS depth server running on GPU (127.0.0.1:9885).
|
||||
fn estimate_depth_midas_server(rgb: &[u8], width: u32, height: u32) -> Result<Vec<f32>> {
|
||||
let expected = (width * height * 3) as usize;
|
||||
if rgb.len() < expected { anyhow::bail!("rgb too small"); }
|
||||
|
||||
// Send RGB as JSON array to depth server
|
||||
let rgb_list: Vec<u8> = rgb[..expected].to_vec();
|
||||
let body = serde_json::json!({
|
||||
"width": width,
|
||||
"height": height,
|
||||
"rgb": rgb_list,
|
||||
});
|
||||
let body_bytes = serde_json::to_vec(&body)?;
|
||||
|
||||
let client = std::net::TcpStream::connect_timeout(
|
||||
&"127.0.0.1:9885".parse()?, std::time::Duration::from_millis(500)
|
||||
)?;
|
||||
client.set_read_timeout(Some(std::time::Duration::from_secs(5)))?;
|
||||
client.set_write_timeout(Some(std::time::Duration::from_secs(2)))?;
|
||||
|
||||
use std::io::{Read, Write};
|
||||
let mut stream = client;
|
||||
let req = format!(
|
||||
"POST /depth HTTP/1.1\r\nHost: 127.0.0.1\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n",
|
||||
body_bytes.len()
|
||||
);
|
||||
stream.write_all(req.as_bytes())?;
|
||||
stream.write_all(&body_bytes)?;
|
||||
|
||||
// Read response
|
||||
let mut resp = Vec::new();
|
||||
stream.read_to_end(&mut resp)?;
|
||||
|
||||
// Skip HTTP headers
|
||||
let body_start = resp.windows(4).position(|w| w == b"\r\n\r\n")
|
||||
.map(|p| p + 4).unwrap_or(0);
|
||||
let depth_bytes = &resp[body_start..];
|
||||
|
||||
let n = (width * height) as usize;
|
||||
if depth_bytes.len() < n * 4 { anyhow::bail!("depth response too small"); }
|
||||
|
||||
let depth: Vec<f32> = depth_bytes[..n * 4].chunks_exact(4)
|
||||
.map(|c| f32::from_le_bytes([c[0], c[1], c[2], c[3]]))
|
||||
.collect();
|
||||
|
||||
Ok(depth)
|
||||
}
|
||||
|
||||
/// Capture depth cloud from camera (placeholder — real impl uses nokhwa or v4l2).
|
||||
pub async fn capture_depth_cloud(_frames: usize) -> Result<PointCloud> {
|
||||
eprintln!("Camera capture not available (no camera on this machine).");
|
||||
eprintln!("Use --demo for synthetic data, or run on a machine with a camera.");
|
||||
Ok(demo_depth_cloud())
|
||||
}
|
||||
|
||||
/// Generate a demo depth point cloud (synthetic room scene).
|
||||
pub fn demo_depth_cloud() -> PointCloud {
|
||||
let _cloud = PointCloud::new("demo_camera_depth");
|
||||
let intrinsics = CameraIntrinsics::default();
|
||||
|
||||
// Simulate a depth map: room with walls at 3m, floor, and a person at 2m
|
||||
let w = 160; // downsampled
|
||||
let h = 120;
|
||||
let mut depth = vec![3.0f32; w * h];
|
||||
|
||||
// Floor plane (bottom third)
|
||||
for y in (h * 2 / 3)..h {
|
||||
for x in 0..w {
|
||||
depth[y * w + x] = 1.0 + (y - h * 2 / 3) as f32 * 0.05;
|
||||
}
|
||||
}
|
||||
|
||||
// Person silhouette (center, depth=2m)
|
||||
for y in (h / 4)..(h * 3 / 4) {
|
||||
for x in (w * 2 / 5)..(w * 3 / 5) {
|
||||
let dy = (y as f32 - h as f32 / 2.0).abs() / (h as f32 / 4.0);
|
||||
let dx = (x as f32 - w as f32 / 2.0).abs() / (w as f32 / 5.0);
|
||||
if dx * dx + dy * dy < 1.0 {
|
||||
depth[y * w + x] = 2.0 + (dx * dx + dy * dy) * 0.3;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let scaled_intrinsics = CameraIntrinsics {
|
||||
fx: intrinsics.fx * w as f32 / intrinsics.width as f32,
|
||||
fy: intrinsics.fy * h as f32 / intrinsics.height as f32,
|
||||
cx: w as f32 / 2.0,
|
||||
cy: h as f32 / 2.0,
|
||||
width: w as u32,
|
||||
height: h as u32,
|
||||
};
|
||||
|
||||
backproject_depth(&depth, &scaled_intrinsics, None, 1)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn backproject_2x2_depth_yields_four_points() {
|
||||
// 2x2 image, depth=1m everywhere; trivial intrinsics.
|
||||
let intr = CameraIntrinsics {
|
||||
fx: 1.0, fy: 1.0, cx: 0.5, cy: 0.5,
|
||||
width: 2, height: 2,
|
||||
};
|
||||
let depth = vec![1.0f32; 4];
|
||||
let cloud = backproject_depth(&depth, &intr, None, 1);
|
||||
assert_eq!(cloud.points.len(), 4, "2x2 depth → 4 backprojected points");
|
||||
// Every point should be at z=1.0.
|
||||
for p in &cloud.points {
|
||||
assert!((p.z - 1.0).abs() < 1e-6, "z should be 1.0, got {}", p.z);
|
||||
}
|
||||
// With cx=0.5, cy=0.5 the four pixel centers backproject symmetrically
|
||||
// about the optical axis: x in {-0.5, 0.5}, y in {-0.5, 0.5}.
|
||||
let mut xs: Vec<f32> = cloud.points.iter().map(|p| p.x).collect();
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
assert!((xs[0] + 0.5).abs() < 1e-6);
|
||||
assert!((xs.last().unwrap() - 0.5).abs() < 1e-6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn backproject_rejects_invalid_depth() {
|
||||
let intr = CameraIntrinsics {
|
||||
fx: 1.0, fy: 1.0, cx: 0.5, cy: 0.5,
|
||||
width: 2, height: 2,
|
||||
};
|
||||
// All pixels NaN → no points.
|
||||
let depth = vec![f32::NAN; 4];
|
||||
let cloud = backproject_depth(&depth, &intr, None, 1);
|
||||
assert_eq!(cloud.points.len(), 0);
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(dead_code)]
|
||||
fn find_midas_model() -> Result<String> {
|
||||
let paths = [
|
||||
dirs::home_dir().unwrap_or_default().join(".local/share/ruview/midas_v21_small_256.onnx"),
|
||||
dirs::home_dir().unwrap_or_default().join(".cache/ruview/midas_v21_small_256.onnx"),
|
||||
std::path::PathBuf::from("/usr/local/share/ruview/midas_v21_small_256.onnx"),
|
||||
];
|
||||
for p in &paths {
|
||||
if p.exists() { return Ok(p.to_string_lossy().to_string()); }
|
||||
}
|
||||
anyhow::bail!("MiDaS ONNX model not found. Download:\n wget https://github.com/isl-org/MiDaS/releases/download/v3_1/midas_v21_small_256.onnx -O ~/.local/share/ruview/midas_v21_small_256.onnx")
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
//! Multi-modal fusion: camera depth + WiFi RF tomography → unified point cloud.
|
||||
|
||||
use crate::pointcloud::{PointCloud, ColorPoint};
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Occupancy volume from WiFi RF tomography (mirrors RuView's OccupancyVolume).
|
||||
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
|
||||
pub struct OccupancyVolume {
|
||||
pub densities: Vec<f64>, // [nz][ny][nx] voxel densities
|
||||
pub nx: usize,
|
||||
pub ny: usize,
|
||||
pub nz: usize,
|
||||
pub bounds: [f64; 6], // [x_min, y_min, z_min, x_max, y_max, z_max]
|
||||
pub occupied_count: usize,
|
||||
}
|
||||
|
||||
/// Convert WiFi occupancy volume to a sparse point cloud.
|
||||
///
|
||||
/// Each occupied voxel (density > threshold) becomes a point at the voxel center.
|
||||
pub fn occupancy_to_pointcloud(vol: &OccupancyVolume) -> PointCloud {
|
||||
let mut cloud = PointCloud::new("wifi_occupancy");
|
||||
let threshold = 0.3;
|
||||
|
||||
let dx = (vol.bounds[3] - vol.bounds[0]) / vol.nx as f64;
|
||||
let dy = (vol.bounds[4] - vol.bounds[1]) / vol.ny as f64;
|
||||
let dz = (vol.bounds[5] - vol.bounds[2]) / vol.nz as f64;
|
||||
|
||||
for iz in 0..vol.nz {
|
||||
for iy in 0..vol.ny {
|
||||
for ix in 0..vol.nx {
|
||||
let idx = iz * vol.ny * vol.nx + iy * vol.nx + ix;
|
||||
let density = vol.densities[idx];
|
||||
if density > threshold {
|
||||
let x = vol.bounds[0] + (ix as f64 + 0.5) * dx;
|
||||
let y = vol.bounds[1] + (iy as f64 + 0.5) * dy;
|
||||
let z = vol.bounds[2] + (iz as f64 + 0.5) * dz;
|
||||
|
||||
// Color by density (green=low, red=high)
|
||||
let t = ((density - threshold) / (1.0 - threshold)).min(1.0);
|
||||
let r = (t * 255.0) as u8;
|
||||
let g = ((1.0 - t) * 200.0) as u8;
|
||||
|
||||
cloud.points.push(ColorPoint {
|
||||
x: x as f32,
|
||||
y: y as f32,
|
||||
z: z as f32,
|
||||
r, g, b: 50,
|
||||
intensity: density as f32,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
cloud
|
||||
}
|
||||
|
||||
/// Fuse multiple point clouds with voxel-grid downsampling.
|
||||
///
|
||||
/// Points from all clouds are binned into voxels of the given size.
|
||||
/// Each voxel produces one averaged point (position, color, max intensity).
|
||||
pub fn fuse_clouds(clouds: &[&PointCloud], voxel_size: f32) -> PointCloud {
|
||||
let mut cells: HashMap<(i32, i32, i32), (f32, f32, f32, f32, f32, f32, f32, u32)> = HashMap::new();
|
||||
// (sum_x, sum_y, sum_z, sum_r, sum_g, sum_b, max_intensity, count)
|
||||
|
||||
for cloud in clouds {
|
||||
for p in &cloud.points {
|
||||
let key = (
|
||||
(p.x / voxel_size).floor() as i32,
|
||||
(p.y / voxel_size).floor() as i32,
|
||||
(p.z / voxel_size).floor() as i32,
|
||||
);
|
||||
let entry = cells.entry(key).or_insert((0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0));
|
||||
entry.0 += p.x;
|
||||
entry.1 += p.y;
|
||||
entry.2 += p.z;
|
||||
entry.3 += p.r as f32;
|
||||
entry.4 += p.g as f32;
|
||||
entry.5 += p.b as f32;
|
||||
entry.6 = entry.6.max(p.intensity);
|
||||
entry.7 += 1;
|
||||
}
|
||||
}
|
||||
|
||||
let mut fused = PointCloud::new("fused");
|
||||
for (_, (sx, sy, sz, sr, sg, sb, mi, n)) in &cells {
|
||||
let n = *n as f32;
|
||||
fused.points.push(ColorPoint {
|
||||
x: sx / n, y: sy / n, z: sz / n,
|
||||
r: (sr / n) as u8, g: (sg / n) as u8, b: (sb / n) as u8,
|
||||
intensity: *mi,
|
||||
});
|
||||
}
|
||||
fused
|
||||
}
|
||||
|
||||
/// Generate a demo occupancy volume (room with person).
|
||||
pub fn demo_occupancy() -> OccupancyVolume {
|
||||
let nx = 10;
|
||||
let ny = 10;
|
||||
let nz = 5;
|
||||
let mut densities = vec![0.0f64; nx * ny * nz];
|
||||
|
||||
// Walls (high density at edges)
|
||||
for iz in 0..nz {
|
||||
for iy in 0..ny {
|
||||
for ix in 0..nx {
|
||||
let idx = iz * ny * nx + iy * nx + ix;
|
||||
// Edges = walls
|
||||
if ix == 0 || ix == nx - 1 || iy == 0 || iy == ny - 1 {
|
||||
densities[idx] = 0.8;
|
||||
}
|
||||
// Floor
|
||||
if iz == 0 {
|
||||
densities[idx] = 0.6;
|
||||
}
|
||||
// Person at center (iz=1-3, ix=4-6, iy=4-6)
|
||||
if (4..=6).contains(&ix) && (4..=6).contains(&iy) && (1..=3).contains(&iz) {
|
||||
densities[idx] = 0.9;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let occupied_count = densities.iter().filter(|&&d| d > 0.3).count();
|
||||
OccupancyVolume {
|
||||
densities, nx, ny, nz,
|
||||
bounds: [0.0, 0.0, 0.0, 5.0, 5.0, 3.0],
|
||||
occupied_count,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn cloud_with(name: &str, pts: &[(f32, f32, f32)]) -> PointCloud {
|
||||
let mut c = PointCloud::new(name);
|
||||
for &(x, y, z) in pts {
|
||||
c.points.push(ColorPoint { x, y, z, r: 10, g: 20, b: 30, intensity: 0.5 });
|
||||
}
|
||||
c
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fuse_clouds_merges_non_overlapping() {
|
||||
let a = cloud_with("a", &[(0.0, 0.0, 0.0)]);
|
||||
let b = cloud_with("b", &[(5.0, 5.0, 5.0)]);
|
||||
let fused = fuse_clouds(&[&a, &b], 0.1);
|
||||
assert_eq!(fused.points.len(), 2, "two far-apart points should yield two voxels");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fuse_clouds_voxel_dedup() {
|
||||
// Points all within one voxel must collapse to a single averaged point.
|
||||
let a = cloud_with("a", &[
|
||||
(0.01, 0.02, 0.03),
|
||||
(0.04, 0.01, 0.02),
|
||||
(0.03, 0.03, 0.01),
|
||||
]);
|
||||
let fused = fuse_clouds(&[&a], 0.5);
|
||||
assert_eq!(fused.points.len(), 1, "three close points → one voxel");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,272 @@
|
||||
//! ruview-pointcloud — real-time dense point cloud from camera + WiFi CSI
|
||||
//!
|
||||
//! Pipeline: Camera → Depth → Backproject → Fuse with WiFi occupancy → Stream
|
||||
//!
|
||||
//! Usage:
|
||||
//! ruview-pointcloud serve # HTTP + Three.js viewer
|
||||
//! ruview-pointcloud capture --frames 1 # capture to PLY
|
||||
//! ruview-pointcloud demo # synthetic demo
|
||||
//! ruview-pointcloud train # calibration training
|
||||
//! ruview-pointcloud csi-test # send test CSI frames (ADR-018 binary)
|
||||
|
||||
mod brain_bridge;
|
||||
mod camera;
|
||||
mod csi_pipeline;
|
||||
mod depth;
|
||||
mod fusion;
|
||||
mod parser;
|
||||
mod pointcloud;
|
||||
mod stream;
|
||||
mod training;
|
||||
|
||||
use anyhow::Result;
|
||||
use clap::{Parser, Subcommand};
|
||||
|
||||
const VERSION: &str = env!("CARGO_PKG_VERSION");
|
||||
|
||||
#[derive(Parser)]
|
||||
#[command(name = "ruview-pointcloud", version = VERSION)]
|
||||
struct Cli {
|
||||
#[command(subcommand)]
|
||||
command: Commands,
|
||||
}
|
||||
|
||||
#[derive(Subcommand)]
|
||||
enum Commands {
|
||||
/// Start real-time point cloud server.
|
||||
///
|
||||
/// By default the HTTP server binds to `127.0.0.1:9880` — exposing it on
|
||||
/// `0.0.0.0` leaks live camera/CSI/vitals data to the network and must
|
||||
/// be an explicit opt-in via `--bind 0.0.0.0:9880`.
|
||||
Serve {
|
||||
/// Bind address for the HTTP/viewer server. Default
|
||||
/// `127.0.0.1:9880` (loopback only — safe by default).
|
||||
#[arg(long, default_value = "127.0.0.1:9880")]
|
||||
bind: String,
|
||||
/// Brain URL for storing observations
|
||||
#[arg(long)]
|
||||
brain: Option<String>,
|
||||
},
|
||||
/// Capture frames to PLY file
|
||||
Capture {
|
||||
#[arg(long, default_value = "1")]
|
||||
frames: usize,
|
||||
#[arg(long, default_value = "output.ply")]
|
||||
output: String,
|
||||
},
|
||||
/// Generate demo point cloud
|
||||
Demo,
|
||||
/// List available cameras
|
||||
Cameras,
|
||||
/// Training and calibration
|
||||
Train {
|
||||
#[arg(long, default_value = "~/.local/share/ruview/training")]
|
||||
data_dir: String,
|
||||
/// Brain URL for submitting results
|
||||
#[arg(long)]
|
||||
brain: Option<String>,
|
||||
},
|
||||
/// Send synthetic ADR-018 binary CSI frames (for local testing without ESP32).
|
||||
CsiTest {
|
||||
#[arg(long, default_value = "127.0.0.1:3333")]
|
||||
target: String,
|
||||
#[arg(long, default_value = "100")]
|
||||
count: usize,
|
||||
},
|
||||
/// Record a CSI fingerprint for the current location.
|
||||
///
|
||||
/// Listens on UDP 3333 for `--seconds` seconds, accumulates CSI frames,
|
||||
/// and stores a named fingerprint that future sessions can match
|
||||
/// against to identify the room.
|
||||
Fingerprint {
|
||||
/// Human-readable name for the fingerprint (e.g. "office", "lab").
|
||||
name: String,
|
||||
/// How long to listen before recording (default 5 s).
|
||||
#[arg(long, default_value = "5")]
|
||||
seconds: u64,
|
||||
},
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
let cli = Cli::parse();
|
||||
|
||||
match cli.command {
|
||||
Commands::Serve { bind, brain } => {
|
||||
stream::serve(&bind, brain.as_deref()).await?;
|
||||
}
|
||||
Commands::Capture { frames: _, output } => {
|
||||
if camera::camera_available() {
|
||||
let config = camera::CameraConfig::default();
|
||||
let frame = camera::capture_frame(&config)?;
|
||||
let depth = depth::estimate_depth(&frame.rgb, frame.width, frame.height)?;
|
||||
let intrinsics = depth::CameraIntrinsics::default();
|
||||
let cloud = depth::backproject_depth(&depth, &intrinsics, Some(&frame.rgb), 2);
|
||||
pointcloud::write_ply(&cloud, &output)?;
|
||||
println!("Captured {} points to {output}", cloud.points.len());
|
||||
} else {
|
||||
let cloud = depth::demo_depth_cloud();
|
||||
pointcloud::write_ply(&cloud, &output)?;
|
||||
println!("No camera — wrote {} demo points to {output}", cloud.points.len());
|
||||
}
|
||||
}
|
||||
Commands::Demo => {
|
||||
demo().await?;
|
||||
}
|
||||
Commands::Cameras => {
|
||||
let cams = camera::list_cameras();
|
||||
if cams.is_empty() {
|
||||
println!("No cameras found");
|
||||
} else {
|
||||
println!("Available cameras:");
|
||||
for (i, c) in cams.iter().enumerate() {
|
||||
println!(" [{i}] {c}");
|
||||
}
|
||||
}
|
||||
}
|
||||
Commands::Train { data_dir, brain } => {
|
||||
train(&data_dir, brain.as_deref()).await?;
|
||||
}
|
||||
Commands::CsiTest { target, count } => {
|
||||
println!("Sending {count} synthetic ADR-018 CSI frames to {target}...");
|
||||
csi_pipeline::send_test_frames(&target, count)?;
|
||||
println!("Done");
|
||||
}
|
||||
Commands::Fingerprint { name, seconds } => {
|
||||
println!("Recording CSI fingerprint '{name}' for {seconds} s on UDP 3333...");
|
||||
let state = csi_pipeline::start_pipeline("0.0.0.0:3333");
|
||||
std::thread::sleep(std::time::Duration::from_secs(seconds));
|
||||
// record_fingerprint takes a brief lock on the shared state to
|
||||
// read the last N frames from every node's history.
|
||||
{
|
||||
let mut st = state.lock().expect("pipeline state lock poisoned");
|
||||
st.record_fingerprint(&name);
|
||||
println!(
|
||||
" Stored: {} fingerprint(s) total, {} total CSI frames received",
|
||||
st.fingerprints.len(),
|
||||
st.total_frames
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn demo() -> Result<()> {
|
||||
println!("╔══════════════════════════════════════════════╗");
|
||||
println!("║ RuView Dense Point Cloud — Demo ║");
|
||||
println!("╚══════════════════════════════════════════════╝");
|
||||
println!();
|
||||
|
||||
let occupancy = fusion::demo_occupancy();
|
||||
let wifi_cloud = fusion::occupancy_to_pointcloud(&occupancy);
|
||||
println!("WiFi occupancy: {}x{}x{} voxels → {} points",
|
||||
occupancy.nx, occupancy.ny, occupancy.nz, wifi_cloud.points.len());
|
||||
|
||||
let depth_cloud = depth::demo_depth_cloud();
|
||||
println!("Camera depth: {} points", depth_cloud.points.len());
|
||||
|
||||
let fused = fusion::fuse_clouds(&[&wifi_cloud, &depth_cloud], 0.05);
|
||||
println!("Fused: {} points (voxel size=0.05m)", fused.points.len());
|
||||
|
||||
pointcloud::write_ply(&fused, "demo_pointcloud.ply")?;
|
||||
println!("\nWrote: demo_pointcloud.ply");
|
||||
|
||||
let splats = pointcloud::to_gaussian_splats(&fused);
|
||||
let json = serde_json::to_string_pretty(&splats)?;
|
||||
std::fs::write("demo_splats.json", &json)?;
|
||||
println!("Wrote: demo_splats.json ({} splats)", splats.len());
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn train(data_dir: &str, brain_url: Option<&str>) -> Result<()> {
|
||||
println!("╔══════════════════════════════════════════════╗");
|
||||
println!("║ RuView Point Cloud — Training ║");
|
||||
println!("╚══════════════════════════════════════════════╝");
|
||||
println!();
|
||||
|
||||
let expanded = data_dir.replace('~', &dirs::home_dir().unwrap_or_default().to_string_lossy());
|
||||
// Defence-in-depth: reject path-traversal in the CLI argument before we
|
||||
// hand it to TrainingSession (which also checks). This catches malicious
|
||||
// CLI input early, before any I/O.
|
||||
let _sanitised = training::sanitize_data_path(&expanded)?;
|
||||
let mut session = training::TrainingSession::new(&expanded)?;
|
||||
session.load_samples()?;
|
||||
|
||||
// Capture training samples
|
||||
println!("==> Capturing training samples...");
|
||||
|
||||
// Camera samples
|
||||
if camera::camera_available() {
|
||||
println!(" Camera detected — capturing depth frames...");
|
||||
let config = camera::CameraConfig::default();
|
||||
for i in 0..5 {
|
||||
if let Ok(frame) = camera::capture_frame(&config) {
|
||||
let depth = depth::estimate_depth(&frame.rgb, frame.width, frame.height)?;
|
||||
// Score based on depth variance (good frames have varied depth)
|
||||
let mean: f32 = depth.iter().sum::<f32>() / depth.len() as f32;
|
||||
let variance: f32 = depth.iter().map(|d| (d - mean).powi(2)).sum::<f32>() / depth.len() as f32;
|
||||
let quality = (variance / 2.0).min(1.0);
|
||||
|
||||
session.add_sample(
|
||||
Some(depth), frame.width, frame.height,
|
||||
None, None, quality,
|
||||
);
|
||||
println!(" Frame {}: quality={:.2}", i, quality);
|
||||
}
|
||||
std::thread::sleep(std::time::Duration::from_millis(500));
|
||||
}
|
||||
} else {
|
||||
println!(" No camera — using synthetic samples for calibration demo");
|
||||
for i in 0..10 {
|
||||
let w = 160u32;
|
||||
let h = 120u32;
|
||||
let depth: Vec<f32> = (0..w * h).map(|j| 1.0 + (j as f32 / (w * h) as f32) * 4.0 + (i as f32 * 0.1)).collect();
|
||||
let quality = if i < 7 { 0.8 } else { 0.2 };
|
||||
let gt = if i % 3 == 0 {
|
||||
Some(training::GroundTruth {
|
||||
reference_distances: vec![
|
||||
training::ReferencePoint { name: "wall".into(), x_pixel: 80, y_pixel: 60, true_distance_m: 3.0 },
|
||||
],
|
||||
occupancy_label: Some(if i < 5 { "occupied" } else { "empty" }.into()),
|
||||
})
|
||||
} else { None };
|
||||
session.add_sample(Some(depth), w, h, None, gt, quality);
|
||||
}
|
||||
}
|
||||
|
||||
session.save_samples()?;
|
||||
|
||||
// Calibrate depth
|
||||
println!("\n==> Calibrating depth estimation...");
|
||||
let cal = session.calibrate_depth()?;
|
||||
println!(" Result: scale={:.2} offset={:.2} gamma={:.2} RMSE={:.4}m",
|
||||
cal.scale, cal.offset, cal.gamma, cal.rmse);
|
||||
|
||||
// Train occupancy
|
||||
println!("\n==> Training occupancy model...");
|
||||
let occ_cal = session.train_occupancy()?;
|
||||
println!(" Result: threshold={:.2} accuracy={:.1}%",
|
||||
occ_cal.density_threshold, occ_cal.accuracy * 100.0);
|
||||
|
||||
// Export preference pairs
|
||||
println!("\n==> Exporting preference pairs...");
|
||||
let pairs = session.export_preference_pairs()?;
|
||||
println!(" Exported: {} pairs", pairs.len());
|
||||
|
||||
// Submit to brain if available
|
||||
if let Some(url) = brain_url {
|
||||
println!("\n==> Submitting to brain at {url}...");
|
||||
let stored = session.submit_to_brain(url).await?;
|
||||
println!(" Stored: {} observations", stored);
|
||||
}
|
||||
|
||||
println!("\n==> Training complete!");
|
||||
println!(" Data dir: {expanded}");
|
||||
println!(" Samples: {}", session.samples.len());
|
||||
println!(" Calibration: {expanded}/calibration.json");
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,163 @@
|
||||
//! ADR-018 binary CSI frame parser.
|
||||
//!
|
||||
//! Two header magics are accepted: `0xC5110001` (raw CSI, v1) and
|
||||
//! `0xC5110006` (feature state, v6). The header is 20 bytes; everything
|
||||
//! after is interleaved I/Q bytes per subcarrier per antenna.
|
||||
//!
|
||||
//! Returns `None` when the buffer is truncated or the magic is wrong —
|
||||
//! this is a hot path (one call per UDP packet) so we prefer Option over
|
||||
//! a full `anyhow::Error` that would allocate.
|
||||
|
||||
const CSI_MAGIC_V6: u32 = 0xC511_0006;
|
||||
const CSI_MAGIC_V1: u32 = 0xC511_0001;
|
||||
pub(crate) const CSI_HEADER_SIZE: usize = 20;
|
||||
|
||||
/// Accept both header magics — `0xC5110001` (raw CSI) and
|
||||
/// `0xC5110006` (feature state). Exposed for tests.
|
||||
#[allow(dead_code)]
|
||||
pub(crate) const MAGIC_V1: u32 = CSI_MAGIC_V1;
|
||||
#[allow(dead_code)]
|
||||
pub(crate) const MAGIC_V6: u32 = CSI_MAGIC_V6;
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct CsiFrame {
|
||||
pub node_id: u8,
|
||||
pub n_antennas: u8,
|
||||
pub n_subcarriers: u16,
|
||||
pub channel: u8,
|
||||
pub rssi: i8,
|
||||
pub noise_floor: i8,
|
||||
pub timestamp_us: u32,
|
||||
/// Raw I/Q data: [I0, Q0, I1, Q1, ...] for each subcarrier
|
||||
pub iq_data: Vec<i8>,
|
||||
/// Computed amplitude per subcarrier: sqrt(I^2 + Q^2)
|
||||
pub amplitudes: Vec<f32>,
|
||||
/// Computed phase per subcarrier: atan2(Q, I)
|
||||
pub phases: Vec<f32>,
|
||||
}
|
||||
|
||||
/// Parse an ADR-018 binary CSI frame from a UDP packet.
|
||||
///
|
||||
/// Returns `None` if:
|
||||
/// - the buffer is shorter than the 20-byte header
|
||||
/// - the magic does not match either accepted value
|
||||
/// - the declared I/Q payload is truncated
|
||||
pub fn parse_adr018(data: &[u8]) -> Option<CsiFrame> {
|
||||
if data.len() < CSI_HEADER_SIZE { return None; }
|
||||
|
||||
let magic = u32::from_le_bytes([data[0], data[1], data[2], data[3]]);
|
||||
if magic != CSI_MAGIC_V6 && magic != CSI_MAGIC_V1 { return None; }
|
||||
|
||||
let node_id = data[4];
|
||||
let n_antennas = data[5].max(1);
|
||||
let n_subcarriers = u16::from_le_bytes([data[6], data[7]]);
|
||||
let channel = data[8];
|
||||
let rssi = data[9] as i8;
|
||||
let noise_floor = data[10] as i8;
|
||||
let timestamp_us = u32::from_le_bytes([data[16], data[17], data[18], data[19]]);
|
||||
|
||||
let iq_len = (n_subcarriers as usize) * 2 * (n_antennas as usize);
|
||||
if data.len() < CSI_HEADER_SIZE + iq_len { return None; }
|
||||
|
||||
let iq_data: Vec<i8> = data[CSI_HEADER_SIZE..CSI_HEADER_SIZE + iq_len]
|
||||
.iter().map(|&b| b as i8).collect();
|
||||
|
||||
// Compute amplitude and phase per subcarrier (first antenna).
|
||||
let mut amplitudes = Vec::with_capacity(n_subcarriers as usize);
|
||||
let mut phases = Vec::with_capacity(n_subcarriers as usize);
|
||||
for i in 0..n_subcarriers as usize {
|
||||
let idx = i * 2;
|
||||
if idx + 1 < iq_data.len() {
|
||||
let ii = iq_data[idx] as f32;
|
||||
let qq = iq_data[idx + 1] as f32;
|
||||
amplitudes.push((ii * ii + qq * qq).sqrt());
|
||||
phases.push(qq.atan2(ii));
|
||||
}
|
||||
}
|
||||
|
||||
Some(CsiFrame {
|
||||
node_id, n_antennas, n_subcarriers, channel, rssi, noise_floor,
|
||||
timestamp_us, iq_data, amplitudes, phases,
|
||||
})
|
||||
}
|
||||
|
||||
/// Build a synthetic ADR-018 binary frame. Used by the `csi-test` CLI
|
||||
/// subcommand and by the unit tests in this module.
|
||||
pub fn build_test_frame(magic: u32, node_id: u8, n_subcarriers: u16, i: usize) -> Vec<u8> {
|
||||
let mut buf = Vec::with_capacity(CSI_HEADER_SIZE + (n_subcarriers as usize) * 2);
|
||||
buf.extend_from_slice(&magic.to_le_bytes()); // magic (0..4)
|
||||
buf.push(node_id); // node_id (4)
|
||||
buf.push(1u8); // n_antennas (5)
|
||||
buf.extend_from_slice(&n_subcarriers.to_le_bytes()); // n_subcarriers (6..8)
|
||||
buf.push(6u8); // channel (8)
|
||||
buf.push((-40i8 - (i % 30) as i8) as u8); // rssi (9)
|
||||
buf.push((-90i8) as u8); // noise_floor (10)
|
||||
buf.extend_from_slice(&[0u8; 5]); // reserved (11..16)
|
||||
buf.extend_from_slice(&(i as u32).to_le_bytes()); // timestamp_us (16..20)
|
||||
for j in 0..(n_subcarriers as usize) {
|
||||
buf.push(((i + j) as i8).wrapping_mul(3) as u8);
|
||||
buf.push(((i + j) as i8).wrapping_mul(5) as u8);
|
||||
}
|
||||
buf
|
||||
}
|
||||
|
||||
// ─── Tests ──────────────────────────────────────────────────────────────────
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn parse_magic_v1_roundtrips() {
|
||||
let frame_bytes = build_test_frame(MAGIC_V1, 0x42, 56, 7);
|
||||
let frame = parse_adr018(&frame_bytes).expect("v1 frame should parse");
|
||||
assert_eq!(frame.node_id, 0x42);
|
||||
assert_eq!(frame.n_antennas, 1);
|
||||
assert_eq!(frame.n_subcarriers, 56);
|
||||
assert_eq!(frame.channel, 6);
|
||||
assert_eq!(frame.timestamp_us, 7);
|
||||
assert_eq!(frame.iq_data.len(), 56 * 2);
|
||||
assert_eq!(frame.amplitudes.len(), 56);
|
||||
assert_eq!(frame.phases.len(), 56);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_magic_v6_roundtrips() {
|
||||
let frame_bytes = build_test_frame(MAGIC_V6, 0x09, 114, 0);
|
||||
let frame = parse_adr018(&frame_bytes).expect("v6 frame should parse");
|
||||
assert_eq!(frame.node_id, 0x09);
|
||||
assert_eq!(frame.n_antennas, 1);
|
||||
assert_eq!(frame.n_subcarriers, 114);
|
||||
assert_eq!(frame.channel, 6);
|
||||
// With i=0, noise_floor=-90 per build_test_frame.
|
||||
assert_eq!(frame.noise_floor, -90);
|
||||
// With i=0, timestamp_us=0.
|
||||
assert_eq!(frame.timestamp_us, 0);
|
||||
assert_eq!(frame.iq_data.len(), 114 * 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_rejects_wrong_magic() {
|
||||
let mut bad = build_test_frame(MAGIC_V1, 0, 8, 0);
|
||||
// Flip magic to something unrelated.
|
||||
bad[0] = 0xFF;
|
||||
bad[1] = 0xFF;
|
||||
bad[2] = 0xFF;
|
||||
bad[3] = 0xFF;
|
||||
assert!(parse_adr018(&bad).is_none(), "bad magic should not parse");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_rejects_truncated_header() {
|
||||
let short = vec![0u8; CSI_HEADER_SIZE - 1];
|
||||
assert!(parse_adr018(&short).is_none(), "truncated header must not parse");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_rejects_truncated_payload() {
|
||||
let mut frame = build_test_frame(MAGIC_V1, 0, 32, 0);
|
||||
// Drop half the declared payload.
|
||||
frame.truncate(CSI_HEADER_SIZE + 20);
|
||||
assert!(parse_adr018(&frame).is_none(), "truncated payload must not parse");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
//! Point cloud types + PLY export + Gaussian splat conversion.
|
||||
#![allow(dead_code)]
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::io::Write;
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Point3D {
|
||||
pub x: f32,
|
||||
pub y: f32,
|
||||
pub z: f32,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct ColorPoint {
|
||||
pub x: f32,
|
||||
pub y: f32,
|
||||
pub z: f32,
|
||||
pub r: u8,
|
||||
pub g: u8,
|
||||
pub b: u8,
|
||||
pub intensity: f32,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct PointCloud {
|
||||
pub points: Vec<ColorPoint>,
|
||||
pub timestamp_ms: i64,
|
||||
pub source: String,
|
||||
}
|
||||
|
||||
impl PointCloud {
|
||||
pub fn new(source: &str) -> Self {
|
||||
Self {
|
||||
points: Vec::new(),
|
||||
timestamp_ms: chrono::Utc::now().timestamp_millis(),
|
||||
source: source.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn add(&mut self, x: f32, y: f32, z: f32, r: u8, g: u8, b: u8, intensity: f32) {
|
||||
self.points.push(ColorPoint { x, y, z, r, g, b, intensity });
|
||||
}
|
||||
|
||||
pub fn bounds(&self) -> ([f32; 3], [f32; 3]) {
|
||||
if self.points.is_empty() {
|
||||
return ([0.0; 3], [0.0; 3]);
|
||||
}
|
||||
let mut min = [f32::MAX; 3];
|
||||
let mut max = [f32::MIN; 3];
|
||||
for p in &self.points {
|
||||
min[0] = min[0].min(p.x); min[1] = min[1].min(p.y); min[2] = min[2].min(p.z);
|
||||
max[0] = max[0].max(p.x); max[1] = max[1].max(p.y); max[2] = max[2].max(p.z);
|
||||
}
|
||||
(min, max)
|
||||
}
|
||||
}
|
||||
|
||||
/// Write point cloud to PLY format (ASCII).
|
||||
pub fn write_ply(cloud: &PointCloud, path: &str) -> anyhow::Result<()> {
|
||||
let mut f = std::fs::File::create(path)?;
|
||||
writeln!(f, "ply")?;
|
||||
writeln!(f, "format ascii 1.0")?;
|
||||
writeln!(f, "comment Generated by RuView Dense Point Cloud")?;
|
||||
writeln!(f, "comment Source: {}", cloud.source)?;
|
||||
writeln!(f, "comment Timestamp: {}", cloud.timestamp_ms)?;
|
||||
writeln!(f, "element vertex {}", cloud.points.len())?;
|
||||
writeln!(f, "property float x")?;
|
||||
writeln!(f, "property float y")?;
|
||||
writeln!(f, "property float z")?;
|
||||
writeln!(f, "property uchar red")?;
|
||||
writeln!(f, "property uchar green")?;
|
||||
writeln!(f, "property uchar blue")?;
|
||||
writeln!(f, "property float intensity")?;
|
||||
writeln!(f, "end_header")?;
|
||||
for p in &cloud.points {
|
||||
writeln!(f, "{:.4} {:.4} {:.4} {} {} {} {:.4}", p.x, p.y, p.z, p.r, p.g, p.b, p.intensity)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Convert point cloud to Gaussian splats for 3D rendering.
|
||||
#[derive(Serialize, Deserialize)]
|
||||
pub struct GaussianSplat {
|
||||
pub center: [f32; 3],
|
||||
pub color: [f32; 3],
|
||||
pub opacity: f32,
|
||||
pub scale: [f32; 3],
|
||||
}
|
||||
|
||||
pub fn to_gaussian_splats(cloud: &PointCloud) -> Vec<GaussianSplat> {
|
||||
// Cluster points into voxels and create one Gaussian per cluster
|
||||
let voxel_size = 0.08; // smaller voxels = more detail = visible movement
|
||||
let mut cells: std::collections::HashMap<(i32, i32, i32), Vec<&ColorPoint>> = std::collections::HashMap::new();
|
||||
|
||||
for p in &cloud.points {
|
||||
let key = (
|
||||
(p.x / voxel_size).floor() as i32,
|
||||
(p.y / voxel_size).floor() as i32,
|
||||
(p.z / voxel_size).floor() as i32,
|
||||
);
|
||||
cells.entry(key).or_default().push(p);
|
||||
}
|
||||
|
||||
cells.values().map(|pts| {
|
||||
let n = pts.len() as f32;
|
||||
let cx = pts.iter().map(|p| p.x).sum::<f32>() / n;
|
||||
let cy = pts.iter().map(|p| p.y).sum::<f32>() / n;
|
||||
let cz = pts.iter().map(|p| p.z).sum::<f32>() / n;
|
||||
let cr = pts.iter().map(|p| p.r as f32).sum::<f32>() / n / 255.0;
|
||||
let cg = pts.iter().map(|p| p.g as f32).sum::<f32>() / n / 255.0;
|
||||
let cb = pts.iter().map(|p| p.b as f32).sum::<f32>() / n / 255.0;
|
||||
|
||||
// Scale based on point spread
|
||||
let sx = pts.iter().map(|p| (p.x - cx).abs()).sum::<f32>() / n + 0.01;
|
||||
let sy = pts.iter().map(|p| (p.y - cy).abs()).sum::<f32>() / n + 0.01;
|
||||
let sz = pts.iter().map(|p| (p.z - cz).abs()).sum::<f32>() / n + 0.01;
|
||||
|
||||
GaussianSplat {
|
||||
center: [cx, cy, cz],
|
||||
color: [cr, cg, cb],
|
||||
opacity: (n / 10.0).min(1.0),
|
||||
scale: [sx, sy, sz],
|
||||
}
|
||||
}).collect()
|
||||
}
|
||||
@@ -0,0 +1,232 @@
|
||||
//! HTTP server — live camera + ESP32 CSI + fusion → real-time point cloud.
|
||||
|
||||
use crate::brain_bridge;
|
||||
use crate::camera;
|
||||
use crate::csi_pipeline;
|
||||
use crate::depth;
|
||||
use crate::fusion;
|
||||
use crate::pointcloud;
|
||||
use axum::{
|
||||
extract::State,
|
||||
response::Html,
|
||||
routing::get,
|
||||
Json, Router,
|
||||
};
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
struct AppState {
|
||||
latest_cloud: Mutex<pointcloud::PointCloud>,
|
||||
latest_splats: Mutex<Vec<pointcloud::GaussianSplat>>,
|
||||
latest_pipeline: Mutex<Option<csi_pipeline::PipelineOutput>>,
|
||||
frame_count: Mutex<u64>,
|
||||
use_camera: bool,
|
||||
}
|
||||
|
||||
/// Start the HTTP/viewer server bound to `bind` (e.g.
|
||||
/// `"127.0.0.1:9880"` — the safe default — or `"0.0.0.0:9880"` to expose
|
||||
/// the viewer to the LAN).
|
||||
///
|
||||
/// **Security**: the viewer streams live camera/CSI/vitals data. Bind to
|
||||
/// `127.0.0.1` unless you intentionally want remote viewers.
|
||||
pub async fn serve(bind: &str, _brain: Option<&str>) -> anyhow::Result<()> {
|
||||
let has_camera = camera::camera_available();
|
||||
|
||||
// Start CSI pipeline — listens for UDP CSI data from ESP32 nodes.
|
||||
// Kept on 0.0.0.0 because ESP32 nodes are remote devices on the LAN.
|
||||
let csi_pipeline_state = csi_pipeline::start_pipeline("0.0.0.0:3333");
|
||||
eprintln!(" CSI pipeline: UDP port 3333 (ADR-018 binary frames)");
|
||||
|
||||
let initial_cloud = if has_camera {
|
||||
capture_camera_cloud()
|
||||
} else {
|
||||
demo_cloud()
|
||||
};
|
||||
let initial_splats = pointcloud::to_gaussian_splats(&initial_cloud);
|
||||
|
||||
let state = Arc::new(AppState {
|
||||
latest_cloud: Mutex::new(initial_cloud),
|
||||
latest_splats: Mutex::new(initial_splats),
|
||||
latest_pipeline: Mutex::new(None),
|
||||
frame_count: Mutex::new(0),
|
||||
use_camera: has_camera,
|
||||
});
|
||||
|
||||
// Background: capture + fuse every 500ms (motion-adaptive)
|
||||
let bg = state.clone();
|
||||
let bg_csi = csi_pipeline_state.clone();
|
||||
let bg_cam = has_camera;
|
||||
tokio::spawn(async move {
|
||||
let mut skip_depth = false;
|
||||
loop {
|
||||
// Motion-adaptive: check CSI motion score
|
||||
let pipeline_out = Some(csi_pipeline::get_pipeline_output(&bg_csi));
|
||||
if let Some(ref out) = pipeline_out {
|
||||
// Only run expensive depth when motion detected or every 5th frame
|
||||
let frame_num = *bg.frame_count.lock().unwrap();
|
||||
skip_depth = !out.motion_detected && frame_num % 5 != 0;
|
||||
}
|
||||
let pipeline_clone = pipeline_out.clone();
|
||||
*bg.latest_pipeline.lock().unwrap() = pipeline_out;
|
||||
let pipeline_out = pipeline_clone;
|
||||
|
||||
let interval = if skip_depth { 1000 } else { 500 }; // slower when no motion
|
||||
tokio::time::sleep(std::time::Duration::from_millis(interval)).await;
|
||||
|
||||
let (cloud, luminance) = if bg_cam && !skip_depth {
|
||||
tokio::task::spawn_blocking(capture_camera_cloud_with_luminance)
|
||||
.await.unwrap_or_else(|_| (demo_cloud(), None))
|
||||
} else {
|
||||
// Reuse previous cloud when no motion
|
||||
(bg.latest_cloud.lock().unwrap().clone(), None)
|
||||
};
|
||||
// Feed luminance into the CSI pipeline so is_dark toggles for the
|
||||
// viewer. The lock is held briefly here — the UDP thread never
|
||||
// touches it (messages go through the mpsc channel).
|
||||
if let Some(lum) = luminance {
|
||||
if let Ok(mut st) = bg_csi.lock() {
|
||||
st.set_light_level(lum);
|
||||
}
|
||||
}
|
||||
let splats = pointcloud::to_gaussian_splats(&cloud);
|
||||
*bg.latest_cloud.lock().unwrap() = cloud;
|
||||
*bg.latest_splats.lock().unwrap() = splats;
|
||||
let frame_num = {
|
||||
let mut fc = bg.frame_count.lock().unwrap();
|
||||
*fc += 1;
|
||||
*fc
|
||||
};
|
||||
|
||||
// Brain sync — sparse, every 120 frames (~60 seconds)
|
||||
if frame_num % 120 == 0 {
|
||||
if let Some(ref out) = pipeline_out {
|
||||
brain_bridge::sync_to_brain(out, frame_num).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
if has_camera { eprintln!(" Camera: LIVE (/dev/video0)"); }
|
||||
else { eprintln!(" Camera: DEMO"); }
|
||||
|
||||
let app = Router::new()
|
||||
.route("/", get(index))
|
||||
.route("/api/cloud", get(api_cloud))
|
||||
.route("/api/splats", get(api_splats))
|
||||
.route("/api/status", get(api_status))
|
||||
.route("/health", get(api_health))
|
||||
.with_state(state);
|
||||
|
||||
println!("╔══════════════════════════════════════════════╗");
|
||||
println!("║ RuView Dense Point Cloud — ALL SENSORS ║");
|
||||
println!("╚══════════════════════════════════════════════╝");
|
||||
println!(" Viewer: http://{bind}/");
|
||||
if bind.starts_with("0.0.0.0") || bind.starts_with("::") {
|
||||
eprintln!(
|
||||
" WARNING: bound to {bind} — camera/CSI/vitals are exposed \
|
||||
to the network. Use --bind 127.0.0.1:9880 to restrict to loopback."
|
||||
);
|
||||
}
|
||||
|
||||
let listener = tokio::net::TcpListener::bind(bind).await?;
|
||||
axum::serve(listener, app).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn capture_camera_cloud() -> pointcloud::PointCloud {
|
||||
capture_camera_cloud_with_luminance().0
|
||||
}
|
||||
|
||||
/// Grab one camera frame, backproject it to a point cloud, and return the
|
||||
/// mean luminance alongside (used to drive `set_light_level` for night mode).
|
||||
fn capture_camera_cloud_with_luminance() -> (pointcloud::PointCloud, Option<f32>) {
|
||||
let config = camera::CameraConfig::default();
|
||||
match camera::capture_frame(&config) {
|
||||
Ok(frame) => {
|
||||
// Mean luminance across the RGB frame (BT.601 coefficients).
|
||||
let pixels = (frame.width as usize) * (frame.height as usize);
|
||||
let mut sum = 0.0f64;
|
||||
let mut n = 0usize;
|
||||
for chunk in frame.rgb.chunks_exact(3).take(pixels) {
|
||||
sum += 0.299 * chunk[0] as f64
|
||||
+ 0.587 * chunk[1] as f64
|
||||
+ 0.114 * chunk[2] as f64;
|
||||
n += 1;
|
||||
}
|
||||
let lum = if n > 0 { Some((sum / n as f64) as f32) } else { None };
|
||||
|
||||
let cloud = match depth::estimate_depth(&frame.rgb, frame.width, frame.height) {
|
||||
Ok(dm) => {
|
||||
let intr = depth::CameraIntrinsics::default();
|
||||
depth::backproject_depth(&dm, &intr, Some(&frame.rgb), 2)
|
||||
}
|
||||
Err(_) => depth::demo_depth_cloud(),
|
||||
};
|
||||
(cloud, lum)
|
||||
}
|
||||
Err(_) => (depth::demo_depth_cloud(), None),
|
||||
}
|
||||
}
|
||||
|
||||
fn demo_cloud() -> pointcloud::PointCloud {
|
||||
let occ = fusion::demo_occupancy();
|
||||
let wc = fusion::occupancy_to_pointcloud(&occ);
|
||||
let dc = depth::demo_depth_cloud();
|
||||
fusion::fuse_clouds(&[&wc, &dc], 0.05)
|
||||
}
|
||||
|
||||
async fn api_cloud(State(state): State<Arc<AppState>>) -> Json<serde_json::Value> {
|
||||
let cloud = state.latest_cloud.lock().unwrap();
|
||||
let (min, max) = cloud.bounds();
|
||||
let frames = *state.frame_count.lock().unwrap();
|
||||
let pipeline = state.latest_pipeline.lock().unwrap();
|
||||
Json(serde_json::json!({
|
||||
"points": cloud.points.len(),
|
||||
"bounds_min": min, "bounds_max": max,
|
||||
"live": state.use_camera,
|
||||
"frame": frames,
|
||||
"pipeline": &*pipeline,
|
||||
"cloud": cloud.points.iter().take(1000).collect::<Vec<_>>(),
|
||||
}))
|
||||
}
|
||||
|
||||
async fn api_splats(State(state): State<Arc<AppState>>) -> Json<serde_json::Value> {
|
||||
let splats = state.latest_splats.lock().unwrap();
|
||||
let frames = *state.frame_count.lock().unwrap();
|
||||
let pipeline = state.latest_pipeline.lock().unwrap();
|
||||
Json(serde_json::json!({
|
||||
"splats": &*splats,
|
||||
"count": splats.len(),
|
||||
"live": state.use_camera,
|
||||
"frame": frames,
|
||||
"pipeline": &*pipeline,
|
||||
"timestamp": chrono::Utc::now().timestamp_millis(),
|
||||
}))
|
||||
}
|
||||
|
||||
async fn api_status(State(state): State<Arc<AppState>>) -> Json<serde_json::Value> {
|
||||
let frames = *state.frame_count.lock().unwrap();
|
||||
let pipeline = state.latest_pipeline.lock().unwrap();
|
||||
Json(serde_json::json!({
|
||||
"status": "ok",
|
||||
"version": env!("CARGO_PKG_VERSION"),
|
||||
"live": state.use_camera,
|
||||
"camera": if state.use_camera { "/dev/video0" } else { "demo" },
|
||||
"csi_pipeline": "active (UDP:3333)",
|
||||
"pipeline": &*pipeline,
|
||||
"frames_captured": frames,
|
||||
}))
|
||||
}
|
||||
|
||||
async fn api_health() -> Json<serde_json::Value> {
|
||||
Json(serde_json::json!({"status": "ok"}))
|
||||
}
|
||||
|
||||
/// Viewer HTML/JS, compiled into the binary at build time. Keep the
|
||||
/// markup in `viewer.html` to keep this file under the 500-LOC limit and
|
||||
/// to make it trivially editable (no Rust rebuild when tweaking JS).
|
||||
static VIEWER_HTML: &str = include_str!("viewer.html");
|
||||
|
||||
async fn index() -> Html<&'static str> {
|
||||
Html(VIEWER_HTML)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,497 @@
|
||||
//! Training pipeline — collect spatial observations and train depth/occupancy models.
|
||||
//!
|
||||
//! Three training modes:
|
||||
//! 1. **Depth calibration**: capture camera frames + known distances → calibrate
|
||||
//! the luminance-to-depth mapping parameters
|
||||
//! 2. **CSI occupancy training**: capture CSI with known occupancy ground truth →
|
||||
//! train the tomography weights for this room geometry
|
||||
//! 3. **Brain integration**: store spatial observations as brain memories for
|
||||
//! DPO training — "this depth estimate was correct" vs "this was wrong"
|
||||
|
||||
use crate::fusion::OccupancyVolume;
|
||||
use anyhow::{anyhow, Result};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
/// Reject a user-supplied path that contains `..` components (path traversal
|
||||
/// attempt) and return a normalised [`PathBuf`]. We only reject `..`; other
|
||||
/// components (including relative prefixes and `~`) are accepted verbatim —
|
||||
/// the caller is responsible for tilde expansion if needed.
|
||||
pub fn sanitize_data_path(raw: &str) -> Result<PathBuf> {
|
||||
let p = PathBuf::from(raw);
|
||||
for comp in p.components() {
|
||||
if matches!(comp, std::path::Component::ParentDir) {
|
||||
return Err(anyhow!(
|
||||
"refusing to use data dir with `..` traversal component: {raw}"
|
||||
));
|
||||
}
|
||||
}
|
||||
Ok(p)
|
||||
}
|
||||
|
||||
/// Ensure `child` (after joining to `base`) stays inside the canonicalised
|
||||
/// `base` directory. Returns the canonical child path on success. Used by
|
||||
/// every filesystem write site in this module to prevent path-traversal
|
||||
/// through user-supplied names.
|
||||
fn safe_join(base: &Path, child: &str) -> Result<PathBuf> {
|
||||
// Reject absolute children and any `..` components up front.
|
||||
let child_path = Path::new(child);
|
||||
if child_path.is_absolute() {
|
||||
return Err(anyhow!("child path must be relative: {child}"));
|
||||
}
|
||||
for comp in child_path.components() {
|
||||
if matches!(comp, std::path::Component::ParentDir) {
|
||||
return Err(anyhow!("child path may not contain `..`: {child}"));
|
||||
}
|
||||
}
|
||||
|
||||
let joined = base.join(child_path);
|
||||
// Canonicalise base (must exist) and verify joined starts with it. If the
|
||||
// joined file doesn't exist yet we canonicalise the parent.
|
||||
let canonical_base = base.canonicalize()
|
||||
.map_err(|e| anyhow!("data_dir not accessible {}: {e}", base.display()))?;
|
||||
let canonical_parent = joined
|
||||
.parent()
|
||||
.ok_or_else(|| anyhow!("no parent for {}", joined.display()))?;
|
||||
let canonical_parent = canonical_parent
|
||||
.canonicalize()
|
||||
.map_err(|e| anyhow!("parent not accessible {}: {e}", canonical_parent.display()))?;
|
||||
if !canonical_parent.starts_with(&canonical_base) {
|
||||
return Err(anyhow!(
|
||||
"refusing to write outside data_dir: {}",
|
||||
joined.display()
|
||||
));
|
||||
}
|
||||
Ok(canonical_parent.join(
|
||||
joined.file_name().ok_or_else(|| anyhow!("no filename for {}", joined.display()))?,
|
||||
))
|
||||
}
|
||||
|
||||
/// Training data sample — a snapshot of the scene.
|
||||
#[derive(Serialize, Deserialize)]
|
||||
pub struct TrainingSample {
|
||||
pub timestamp_ms: i64,
|
||||
pub source: String,
|
||||
/// Camera depth map (downsampled, in meters)
|
||||
pub depth_map: Option<Vec<f32>>,
|
||||
pub depth_width: u32,
|
||||
pub depth_height: u32,
|
||||
/// WiFi occupancy grid
|
||||
pub occupancy: Option<OccupancyData>,
|
||||
/// Ground truth (if available)
|
||||
pub ground_truth: Option<GroundTruth>,
|
||||
/// Quality score (0.0-1.0, rated by user or self-eval)
|
||||
pub quality: f32,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
pub struct OccupancyData {
|
||||
pub densities: Vec<f64>,
|
||||
pub nx: usize,
|
||||
pub ny: usize,
|
||||
pub nz: usize,
|
||||
}
|
||||
|
||||
impl From<&OccupancyVolume> for OccupancyData {
|
||||
fn from(vol: &OccupancyVolume) -> Self {
|
||||
Self {
|
||||
densities: vol.densities.clone(),
|
||||
nx: vol.nx, ny: vol.ny, nz: vol.nz,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
pub struct GroundTruth {
|
||||
/// Known distances to reference points (e.g., wall at 3.0m)
|
||||
pub reference_distances: Vec<ReferencePoint>,
|
||||
/// Known occupancy state (person present/absent + location)
|
||||
pub occupancy_label: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
pub struct ReferencePoint {
|
||||
pub name: String,
|
||||
pub x_pixel: u32,
|
||||
pub y_pixel: u32,
|
||||
pub true_distance_m: f32,
|
||||
}
|
||||
|
||||
/// Training session — accumulates samples and learns calibration.
|
||||
pub struct TrainingSession {
|
||||
pub samples: Vec<TrainingSample>,
|
||||
pub calibration: DepthCalibration,
|
||||
pub data_dir: PathBuf,
|
||||
}
|
||||
|
||||
/// Depth calibration parameters — maps luminance to real depth.
|
||||
#[derive(Clone, Serialize, Deserialize)]
|
||||
pub struct DepthCalibration {
|
||||
pub scale: f32, // multiplier for depth values
|
||||
pub offset: f32, // additive offset
|
||||
pub near_clip: f32, // minimum valid depth
|
||||
pub far_clip: f32, // maximum valid depth
|
||||
pub gamma: f32, // nonlinear correction (luminance^gamma → depth)
|
||||
pub samples_used: u32,
|
||||
pub rmse: f32, // root mean square error against ground truth
|
||||
}
|
||||
|
||||
impl Default for DepthCalibration {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
scale: 4.0,
|
||||
offset: 1.0,
|
||||
near_clip: 0.3,
|
||||
far_clip: 8.0,
|
||||
gamma: 1.0,
|
||||
samples_used: 0,
|
||||
rmse: f32::MAX,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl TrainingSession {
|
||||
/// Create a new training session rooted at `data_dir`.
|
||||
///
|
||||
/// `data_dir` must not contain `..` components — we reject path traversal
|
||||
/// attempts from CLI/API input. The directory is created if missing and
|
||||
/// then canonicalised so every subsequent write stays inside it.
|
||||
pub fn new(data_dir: &str) -> Result<Self> {
|
||||
let path = sanitize_data_path(data_dir)?;
|
||||
std::fs::create_dir_all(&path)
|
||||
.map_err(|e| anyhow!("failed to create data_dir {}: {e}", path.display()))?;
|
||||
// Canonicalise so path-traversal checks in safe_join have a fixed root.
|
||||
let path = path
|
||||
.canonicalize()
|
||||
.map_err(|e| anyhow!("cannot canonicalise data_dir {}: {e}", path.display()))?;
|
||||
|
||||
// Load existing calibration if available
|
||||
let cal_path = safe_join(&path, "calibration.json")
|
||||
// safe_join needs the parent to exist; for initial load that's always data_dir
|
||||
.or_else(|_| Ok::<_, anyhow::Error>(path.join("calibration.json")))?;
|
||||
let calibration = if cal_path.exists() {
|
||||
let data = std::fs::read_to_string(&cal_path)?;
|
||||
serde_json::from_str(&data).unwrap_or_default()
|
||||
} else {
|
||||
DepthCalibration::default()
|
||||
};
|
||||
|
||||
Ok(Self {
|
||||
samples: Vec::new(),
|
||||
calibration,
|
||||
data_dir: path,
|
||||
})
|
||||
}
|
||||
|
||||
/// Add a training sample with optional ground truth.
|
||||
pub fn add_sample(
|
||||
&mut self,
|
||||
depth_map: Option<Vec<f32>>,
|
||||
width: u32,
|
||||
height: u32,
|
||||
occupancy: Option<&OccupancyVolume>,
|
||||
ground_truth: Option<GroundTruth>,
|
||||
quality: f32,
|
||||
) {
|
||||
let sample = TrainingSample {
|
||||
timestamp_ms: chrono::Utc::now().timestamp_millis(),
|
||||
source: "capture".to_string(),
|
||||
depth_map,
|
||||
depth_width: width,
|
||||
depth_height: height,
|
||||
occupancy: occupancy.map(OccupancyData::from),
|
||||
ground_truth,
|
||||
quality,
|
||||
};
|
||||
self.samples.push(sample);
|
||||
}
|
||||
|
||||
/// Calibrate depth estimation using ground truth reference points.
|
||||
///
|
||||
/// Finds optimal scale, offset, and gamma to minimize RMSE
|
||||
/// between estimated and true depths at reference points.
|
||||
pub fn calibrate_depth(&mut self) -> Result<DepthCalibration> {
|
||||
let mut best = self.calibration.clone();
|
||||
let mut best_rmse = f32::MAX;
|
||||
|
||||
// Collect all reference points across samples
|
||||
let refs: Vec<(f32, f32)> = self.samples.iter()
|
||||
.filter_map(|s| {
|
||||
let gt = s.ground_truth.as_ref()?;
|
||||
let dm = s.depth_map.as_ref()?;
|
||||
Some(gt.reference_distances.iter().filter_map(|rp| {
|
||||
let idx = (rp.y_pixel * s.depth_width + rp.x_pixel) as usize;
|
||||
dm.get(idx).map(|&est| (est, rp.true_distance_m))
|
||||
}).collect::<Vec<_>>())
|
||||
})
|
||||
.flatten()
|
||||
.collect();
|
||||
|
||||
if refs.is_empty() {
|
||||
eprintln!(" No reference points — using default calibration");
|
||||
return Ok(best);
|
||||
}
|
||||
|
||||
eprintln!(" Calibrating with {} reference points...", refs.len());
|
||||
|
||||
// Grid search over scale, offset, gamma
|
||||
for scale_i in 0..20 {
|
||||
let scale = 1.0 + scale_i as f32 * 0.5;
|
||||
for offset_i in 0..10 {
|
||||
let offset = offset_i as f32 * 0.5;
|
||||
for gamma_i in 5..15 {
|
||||
let gamma = gamma_i as f32 * 0.2;
|
||||
|
||||
let rmse = refs.iter()
|
||||
.map(|&(est, truth)| {
|
||||
let calibrated = offset + est.powf(gamma) * scale;
|
||||
(calibrated - truth).powi(2)
|
||||
})
|
||||
.sum::<f32>() / refs.len() as f32;
|
||||
let rmse = rmse.sqrt();
|
||||
|
||||
if rmse < best_rmse {
|
||||
best_rmse = rmse;
|
||||
best = DepthCalibration {
|
||||
scale, offset, gamma,
|
||||
near_clip: 0.3, far_clip: 8.0,
|
||||
samples_used: refs.len() as u32,
|
||||
rmse,
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
eprintln!(" Best calibration: scale={:.2} offset={:.2} gamma={:.2} RMSE={:.4}m",
|
||||
best.scale, best.offset, best.gamma, best.rmse);
|
||||
|
||||
self.calibration = best.clone();
|
||||
self.save_calibration()?;
|
||||
Ok(best)
|
||||
}
|
||||
|
||||
/// Train CSI occupancy model — adjust tomography weights.
|
||||
///
|
||||
/// Uses samples with known occupancy labels to optimize the
|
||||
/// attenuation-to-density mapping.
|
||||
pub fn train_occupancy(&self) -> Result<OccupancyCalibration> {
|
||||
let labeled: Vec<&TrainingSample> = self.samples.iter()
|
||||
.filter(|s| s.ground_truth.as_ref().and_then(|g| g.occupancy_label.as_ref()).is_some())
|
||||
.collect();
|
||||
|
||||
if labeled.is_empty() {
|
||||
eprintln!(" No labeled occupancy samples — using defaults");
|
||||
return Ok(OccupancyCalibration::default());
|
||||
}
|
||||
|
||||
eprintln!(" Training occupancy model with {} samples...", labeled.len());
|
||||
|
||||
// Simple threshold optimization — find the density threshold
|
||||
// that best separates occupied vs unoccupied
|
||||
let mut best_threshold = 0.3f64;
|
||||
let mut best_accuracy = 0.0f64;
|
||||
|
||||
for thresh_i in 1..20 {
|
||||
let threshold = thresh_i as f64 * 0.05;
|
||||
let mut correct = 0;
|
||||
let mut total = 0;
|
||||
|
||||
for sample in &labeled {
|
||||
if let Some(ref occ) = sample.occupancy {
|
||||
let label = sample.ground_truth.as_ref().unwrap()
|
||||
.occupancy_label.as_ref().unwrap();
|
||||
let is_occupied = label == "occupied" || label == "present";
|
||||
let detected = occ.densities.iter().any(|&d| d > threshold);
|
||||
if detected == is_occupied { correct += 1; }
|
||||
total += 1;
|
||||
}
|
||||
}
|
||||
|
||||
let accuracy = correct as f64 / total.max(1) as f64;
|
||||
if accuracy > best_accuracy {
|
||||
best_accuracy = accuracy;
|
||||
best_threshold = threshold;
|
||||
}
|
||||
}
|
||||
|
||||
let cal = OccupancyCalibration {
|
||||
density_threshold: best_threshold,
|
||||
accuracy: best_accuracy,
|
||||
samples_used: labeled.len() as u32,
|
||||
};
|
||||
|
||||
eprintln!(" Occupancy threshold={:.2} accuracy={:.1}%", cal.density_threshold, cal.accuracy * 100.0);
|
||||
|
||||
// Save (path-traversal safe: constant filename under canonical data_dir)
|
||||
let path = safe_join(&self.data_dir, "occupancy_calibration.json")?;
|
||||
std::fs::write(&path, serde_json::to_string_pretty(&cal)?)?;
|
||||
|
||||
Ok(cal)
|
||||
}
|
||||
|
||||
/// Export training data as preference pairs for DPO training on the brain.
|
||||
///
|
||||
/// Good samples (quality > 0.7) → chosen
|
||||
/// Bad samples (quality < 0.3) → rejected
|
||||
pub fn export_preference_pairs(&self) -> Result<Vec<PreferencePair>> {
|
||||
let mut pairs = Vec::new();
|
||||
|
||||
let good: Vec<&TrainingSample> = self.samples.iter()
|
||||
.filter(|s| s.quality > 0.7)
|
||||
.collect();
|
||||
let bad: Vec<&TrainingSample> = self.samples.iter()
|
||||
.filter(|s| s.quality < 0.3)
|
||||
.collect();
|
||||
|
||||
for (g, b) in good.iter().zip(bad.iter()) {
|
||||
pairs.push(PreferencePair {
|
||||
chosen: format!(
|
||||
"Depth estimation at {}ms: {} points, quality {:.2}",
|
||||
g.timestamp_ms,
|
||||
g.depth_map.as_ref().map(|d| d.len()).unwrap_or(0),
|
||||
g.quality
|
||||
),
|
||||
rejected: format!(
|
||||
"Depth estimation at {}ms: {} points, quality {:.2}",
|
||||
b.timestamp_ms,
|
||||
b.depth_map.as_ref().map(|d| d.len()).unwrap_or(0),
|
||||
b.quality
|
||||
),
|
||||
});
|
||||
}
|
||||
|
||||
// Save pairs (path-traversal safe: constant filename under canonical data_dir)
|
||||
let path = safe_join(&self.data_dir, "preference_pairs.jsonl")?;
|
||||
let mut f = std::fs::File::create(&path)?;
|
||||
for pair in &pairs {
|
||||
use std::io::Write;
|
||||
writeln!(f, "{}", serde_json::to_string(pair)?)?;
|
||||
}
|
||||
|
||||
eprintln!(" Exported {} preference pairs to {}", pairs.len(), path.display());
|
||||
Ok(pairs)
|
||||
}
|
||||
|
||||
/// Send training results to the ruOS brain for storage.
|
||||
pub async fn submit_to_brain(&self, brain_url: &str) -> Result<u32> {
|
||||
let client = reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(10))
|
||||
.build()?;
|
||||
|
||||
let mut stored = 0u32;
|
||||
|
||||
// Store calibration as brain memory
|
||||
let _cal_json = serde_json::to_string(&self.calibration)?;
|
||||
let body = serde_json::json!({
|
||||
"category": "spatial-calibration",
|
||||
"content": format!("Depth calibration: scale={:.2} offset={:.2} gamma={:.2} RMSE={:.4}m ({} samples)",
|
||||
self.calibration.scale, self.calibration.offset, self.calibration.gamma,
|
||||
self.calibration.rmse, self.calibration.samples_used),
|
||||
});
|
||||
if client.post(format!("{brain_url}/memories"))
|
||||
.json(&body).send().await.is_ok() {
|
||||
stored += 1;
|
||||
}
|
||||
|
||||
// Store good observations
|
||||
for sample in self.samples.iter().filter(|s| s.quality > 0.5) {
|
||||
let body = serde_json::json!({
|
||||
"category": "spatial-observation",
|
||||
"content": format!("Point cloud capture: {} depth points, quality {:.2}, occupancy {}",
|
||||
sample.depth_map.as_ref().map(|d| d.len()).unwrap_or(0),
|
||||
sample.quality,
|
||||
sample.occupancy.as_ref().map(|o| format!("{}x{}x{}", o.nx, o.ny, o.nz)).unwrap_or("none".into())),
|
||||
});
|
||||
if client.post(format!("{brain_url}/memories"))
|
||||
.json(&body).send().await.is_ok() {
|
||||
stored += 1;
|
||||
}
|
||||
}
|
||||
|
||||
eprintln!(" Submitted {} observations to brain", stored);
|
||||
Ok(stored)
|
||||
}
|
||||
|
||||
/// Save current calibration to disk (path-traversal safe).
|
||||
fn save_calibration(&self) -> Result<()> {
|
||||
let path = safe_join(&self.data_dir, "calibration.json")?;
|
||||
std::fs::write(&path, serde_json::to_string_pretty(&self.calibration)?)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Save all samples to disk (path-traversal safe).
|
||||
pub fn save_samples(&self) -> Result<()> {
|
||||
let path = safe_join(&self.data_dir, "samples.json")?;
|
||||
std::fs::write(&path, serde_json::to_string_pretty(&self.samples)?)?;
|
||||
eprintln!(" Saved {} samples to {}", self.samples.len(), path.display());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Load samples from disk (path-traversal safe).
|
||||
pub fn load_samples(&mut self) -> Result<()> {
|
||||
let path = safe_join(&self.data_dir, "samples.json")?;
|
||||
if path.exists() {
|
||||
let data = std::fs::read_to_string(&path)?;
|
||||
self.samples = serde_json::from_str(&data)?;
|
||||
eprintln!(" Loaded {} samples", self.samples.len());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
pub struct OccupancyCalibration {
|
||||
pub density_threshold: f64,
|
||||
pub accuracy: f64,
|
||||
pub samples_used: u32,
|
||||
}
|
||||
|
||||
impl Default for OccupancyCalibration {
|
||||
fn default() -> Self {
|
||||
Self { density_threshold: 0.3, accuracy: 0.0, samples_used: 0 }
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
pub struct PreferencePair {
|
||||
pub chosen: String,
|
||||
pub rejected: String,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn sanitize_rejects_parent_dir_traversal() {
|
||||
assert!(sanitize_data_path("../etc/passwd").is_err());
|
||||
assert!(sanitize_data_path("foo/../bar").is_err());
|
||||
assert!(sanitize_data_path("/tmp/.. /evil").is_ok(), "`.. ` is not ParentDir");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sanitize_accepts_relative_child() {
|
||||
assert!(sanitize_data_path("data/ruview").is_ok());
|
||||
assert!(sanitize_data_path("./foo").is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn training_session_new_rejects_traversal() {
|
||||
// Even if the filesystem has such a path, TrainingSession should refuse.
|
||||
let err = TrainingSession::new("../etc/passwd").err();
|
||||
assert!(err.is_some(), "traversal path must be rejected");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn training_session_new_accepts_child_path() {
|
||||
// Use a unique tmpdir to avoid cross-test interference.
|
||||
let tmp = std::env::temp_dir().join(format!("ruview-train-test-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&tmp);
|
||||
let sess = TrainingSession::new(tmp.to_str().unwrap())
|
||||
.expect("TrainingSession should accept a clean tmpdir");
|
||||
// data_dir should have been canonicalised to an absolute path.
|
||||
assert!(sess.data_dir.is_absolute());
|
||||
let _ = std::fs::remove_dir_all(&tmp);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>RuView — Camera + WiFi CSI Point Cloud</title>
|
||||
<style>
|
||||
body { margin: 0; background: #0a0a0a; color: #e8a634; font-family: monospace; }
|
||||
canvas { display: block; }
|
||||
#info { position: absolute; top: 10px; left: 10px; padding: 12px; background: rgba(0,0,0,0.85); border: 1px solid #e8a634; border-radius: 6px; min-width: 240px; font-size: 13px; line-height: 1.5; }
|
||||
.live { color: #4f4; } .demo { color: #f44; }
|
||||
.section { margin-top: 6px; padding-top: 6px; border-top: 1px solid #333; }
|
||||
.label { color: #888; }
|
||||
</style>
|
||||
<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/controls/OrbitControls.js"></script>
|
||||
</head>
|
||||
<body>
|
||||
<div id="info">
|
||||
<h3 style="margin:0 0 8px 0">RuView Point Cloud</h3>
|
||||
<div id="stats">Loading...</div>
|
||||
</div>
|
||||
<script>
|
||||
var scene = new THREE.Scene();
|
||||
scene.background = new THREE.Color(0x0a0a0a);
|
||||
var camera = new THREE.PerspectiveCamera(75, window.innerWidth/window.innerHeight, 0.1, 100);
|
||||
camera.position.set(0, 2, -4);
|
||||
camera.lookAt(0, 0, 2);
|
||||
|
||||
var renderer = new THREE.WebGLRenderer({ antialias: true });
|
||||
renderer.setSize(window.innerWidth, window.innerHeight);
|
||||
document.body.appendChild(renderer.domElement);
|
||||
|
||||
var controls = new THREE.OrbitControls(camera, renderer.domElement);
|
||||
controls.enableDamping = true;
|
||||
controls.target.set(0, 0, 2);
|
||||
|
||||
var pointsMesh = null;
|
||||
var lastFrame = -1;
|
||||
var skeletonGroup = null;
|
||||
var prevTimestamp = 0;
|
||||
var frameRateVal = 0;
|
||||
|
||||
// COCO skeleton connections: pairs of keypoint indices
|
||||
// 0=nose 1=leftEye 2=rightEye 3=leftEar 4=rightEar
|
||||
// 5=leftShoulder 6=rightShoulder 7=leftElbow 8=rightElbow
|
||||
// 9=leftWrist 10=rightWrist 11=leftHip 12=rightHip
|
||||
// 13=leftKnee 14=rightKnee 15=leftAnkle 16=rightAnkle
|
||||
var COCO_BONES = [
|
||||
[0,1],[0,2],[1,3],[2,4],
|
||||
[5,6],[5,7],[7,9],[6,8],[8,10],
|
||||
[5,11],[6,12],[11,12],
|
||||
[11,13],[13,15],[12,14],[14,16]
|
||||
];
|
||||
|
||||
function clearSkeleton() {
|
||||
if (skeletonGroup) {
|
||||
scene.remove(skeletonGroup);
|
||||
skeletonGroup.traverse(function(obj) {
|
||||
if (obj.geometry) obj.geometry.dispose();
|
||||
if (obj.material) obj.material.dispose();
|
||||
});
|
||||
skeletonGroup = null;
|
||||
}
|
||||
}
|
||||
|
||||
function drawSkeleton(keypoints) {
|
||||
clearSkeleton();
|
||||
if (!keypoints || keypoints.length < 17) return;
|
||||
skeletonGroup = new THREE.Group();
|
||||
|
||||
// Map keypoints from [0,1] to scene coords
|
||||
// x: [-2, 2], y: [2, -2] (flip y), z: fixed at 2
|
||||
var sphereGeo = new THREE.SphereGeometry(0.04, 8, 8);
|
||||
var sphereMat = new THREE.MeshBasicMaterial({ color: 0xffff00 });
|
||||
var positions3D = [];
|
||||
var i, kp, sx, sy;
|
||||
for (i = 0; i < 17; i++) {
|
||||
kp = keypoints[i];
|
||||
if (!kp) { positions3D.push(null); continue; }
|
||||
sx = (kp[0] - 0.5) * 4;
|
||||
sy = (0.5 - kp[1]) * 4;
|
||||
positions3D.push([sx, sy, 2]);
|
||||
var sphere = new THREE.Mesh(sphereGeo, sphereMat);
|
||||
sphere.position.set(sx, sy, 2);
|
||||
skeletonGroup.add(sphere);
|
||||
}
|
||||
|
||||
// Draw bones as white lines
|
||||
var lineMat = new THREE.LineBasicMaterial({ color: 0xffffff, linewidth: 2 });
|
||||
var b, a, bIdx;
|
||||
for (b = 0; b < COCO_BONES.length; b++) {
|
||||
a = COCO_BONES[b][0];
|
||||
bIdx = COCO_BONES[b][1];
|
||||
if (!positions3D[a] || !positions3D[bIdx]) continue;
|
||||
var lineGeo = new THREE.BufferGeometry();
|
||||
var verts = new Float32Array([
|
||||
positions3D[a][0], positions3D[a][1], positions3D[a][2],
|
||||
positions3D[bIdx][0], positions3D[bIdx][1], positions3D[bIdx][2]
|
||||
]);
|
||||
lineGeo.setAttribute("position", new THREE.BufferAttribute(verts, 3));
|
||||
var line = new THREE.Line(lineGeo, lineMat);
|
||||
skeletonGroup.add(line);
|
||||
}
|
||||
|
||||
scene.add(skeletonGroup);
|
||||
}
|
||||
|
||||
async function fetchCloud() {
|
||||
try {
|
||||
var resp = await fetch("/api/splats");
|
||||
var data = await resp.json();
|
||||
if (data.splats && data.frame !== lastFrame) {
|
||||
// Compute CSI frame rate
|
||||
var now = Date.now();
|
||||
if (prevTimestamp > 0) {
|
||||
var dt = (now - prevTimestamp) / 1000.0;
|
||||
if (dt > 0) frameRateVal = (1.0 / dt).toFixed(1);
|
||||
}
|
||||
prevTimestamp = now;
|
||||
lastFrame = data.frame;
|
||||
updateSplats(data.splats);
|
||||
|
||||
// Draw skeleton if available
|
||||
var pipe = data.pipeline;
|
||||
if (pipe && pipe.skeleton && pipe.skeleton.keypoints) {
|
||||
drawSkeleton(pipe.skeleton.keypoints);
|
||||
} else {
|
||||
clearSkeleton();
|
||||
}
|
||||
|
||||
// Build info panel
|
||||
var mode = data.live
|
||||
? '<span class="live">● LIVE</span>'
|
||||
: '<span class="demo">● DEMO</span>';
|
||||
var html = mode + " Camera + CSI<br>"
|
||||
+ "Splats: " + data.count + "<br>"
|
||||
+ "Frame: " + data.frame;
|
||||
|
||||
// CSI frame rate
|
||||
html += '<div class="section">'
|
||||
+ '<span class="label">CSI Rate:</span> '
|
||||
+ frameRateVal + " fps</div>";
|
||||
|
||||
// Skeleton confidence
|
||||
if (pipe && pipe.skeleton && pipe.skeleton.confidence !== undefined) {
|
||||
var conf = (pipe.skeleton.confidence * 100).toFixed(0);
|
||||
html += '<div class="section">'
|
||||
+ '<span class="label">Skeleton:</span> '
|
||||
+ conf + "% confidence</div>";
|
||||
}
|
||||
|
||||
// Weather data
|
||||
if (pipe && pipe.weather) {
|
||||
var w = pipe.weather;
|
||||
html += '<div class="section">'
|
||||
+ '<span class="label">Weather:</span> ';
|
||||
if (w.temperature !== undefined) {
|
||||
html += w.temperature + "°C";
|
||||
}
|
||||
if (w.conditions) {
|
||||
html += " " + w.conditions;
|
||||
}
|
||||
html += "</div>";
|
||||
}
|
||||
|
||||
// Building count from geo
|
||||
if (pipe && pipe.geo && pipe.geo.building_count !== undefined) {
|
||||
html += '<div class="section">'
|
||||
+ '<span class="label">Buildings:</span> '
|
||||
+ pipe.geo.building_count + "</div>";
|
||||
}
|
||||
|
||||
// Vitals
|
||||
if (pipe && pipe.vitals) {
|
||||
var v = pipe.vitals;
|
||||
html += '<div class="section">'
|
||||
+ '<span class="label">Vitals:</span> ';
|
||||
if (v.breathing_rate !== undefined) {
|
||||
html += "BR " + v.breathing_rate + "/min";
|
||||
}
|
||||
if (v.motion_score !== undefined) {
|
||||
html += " Motion " + (v.motion_score * 100).toFixed(0) + "%";
|
||||
}
|
||||
html += "</div>";
|
||||
}
|
||||
|
||||
document.getElementById("stats").innerHTML = html;
|
||||
}
|
||||
} catch(e) {}
|
||||
}
|
||||
fetchCloud();
|
||||
setInterval(fetchCloud, 500);
|
||||
|
||||
function updateSplats(splats) {
|
||||
if (pointsMesh) scene.remove(pointsMesh);
|
||||
var geometry = new THREE.BufferGeometry();
|
||||
var positions = new Float32Array(splats.length * 3);
|
||||
var colors = new Float32Array(splats.length * 3);
|
||||
var i, s;
|
||||
for (i = 0; i < splats.length; i++) {
|
||||
s = splats[i];
|
||||
positions[i*3] = s.center[0];
|
||||
positions[i*3+1] = -s.center[1];
|
||||
positions[i*3+2] = s.center[2];
|
||||
colors[i*3] = s.color[0];
|
||||
colors[i*3+1] = s.color[1];
|
||||
colors[i*3+2] = s.color[2];
|
||||
}
|
||||
geometry.setAttribute("position", new THREE.BufferAttribute(positions, 3));
|
||||
geometry.setAttribute("color", new THREE.BufferAttribute(colors, 3));
|
||||
pointsMesh = new THREE.Points(geometry, new THREE.PointsMaterial({
|
||||
size: 0.02, vertexColors: true, sizeAttenuation: true
|
||||
}));
|
||||
scene.add(pointsMesh);
|
||||
}
|
||||
|
||||
function animate() {
|
||||
requestAnimationFrame(animate);
|
||||
controls.update();
|
||||
renderer.render(scene, camera);
|
||||
}
|
||||
animate();
|
||||
window.addEventListener("resize", function() {
|
||||
camera.aspect = window.innerWidth / window.innerHeight;
|
||||
camera.updateProjectionMatrix();
|
||||
renderer.setSize(window.innerWidth, window.innerHeight);
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -25,6 +25,7 @@ axum = { workspace = true }
|
||||
tower-http = { version = "0.5", features = ["fs", "cors", "set-header"] }
|
||||
tokio = { workspace = true, features = ["full", "process"] }
|
||||
futures-util = "0.3"
|
||||
ruvector-mincut = { workspace = true }
|
||||
|
||||
# Serialization
|
||||
serde = { workspace = true }
|
||||
|
||||
@@ -8,8 +8,10 @@ pub mod vital_signs;
|
||||
pub mod rvf_container;
|
||||
pub mod rvf_pipeline;
|
||||
pub mod graph_transformer;
|
||||
#[allow(dead_code)]
|
||||
pub mod trainer;
|
||||
pub mod dataset;
|
||||
pub mod sona;
|
||||
pub mod sparse_inference;
|
||||
#[allow(dead_code)]
|
||||
pub mod embedding;
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
//! - Serves the static UI files (port 8080)
|
||||
//!
|
||||
//! Replaces both ws_server.py and the Python HTTP server.
|
||||
#![allow(dead_code)]
|
||||
|
||||
mod adaptive_classifier;
|
||||
pub mod cli;
|
||||
@@ -1658,9 +1659,11 @@ async fn windows_wifi_task(state: SharedState, tick_ms: u64) {
|
||||
|
||||
// Populate persons from the sensing update (Kalman-smoothed via tracker).
|
||||
let raw_persons = derive_pose_from_sensing(&update);
|
||||
let mut last_tracker_instant = s.last_tracker_instant.take();
|
||||
let tracked = tracker_bridge::tracker_update(
|
||||
&mut s.pose_tracker, &mut s.last_tracker_instant, raw_persons,
|
||||
&mut s.pose_tracker, &mut last_tracker_instant, raw_persons,
|
||||
);
|
||||
s.last_tracker_instant = last_tracker_instant;
|
||||
if !tracked.is_empty() {
|
||||
update.persons = Some(tracked);
|
||||
}
|
||||
@@ -1794,9 +1797,11 @@ async fn windows_wifi_fallback_tick(state: &SharedState, seq: u32) {
|
||||
};
|
||||
|
||||
let raw_persons = derive_pose_from_sensing(&update);
|
||||
let mut last_tracker_instant = s.last_tracker_instant.take();
|
||||
let tracked = tracker_bridge::tracker_update(
|
||||
&mut s.pose_tracker, &mut s.last_tracker_instant, raw_persons,
|
||||
&mut s.pose_tracker, &mut last_tracker_instant, raw_persons,
|
||||
);
|
||||
s.last_tracker_instant = last_tracker_instant;
|
||||
if !tracked.is_empty() {
|
||||
update.persons = Some(tracked);
|
||||
}
|
||||
@@ -3224,7 +3229,7 @@ async fn adaptive_status(State(state): State<SharedState>) -> Json<serde_json::V
|
||||
"trained_frames": model.trained_frames,
|
||||
"accuracy": model.training_accuracy,
|
||||
"version": model.version,
|
||||
"classes": adaptive_classifier::CLASSES,
|
||||
"classes": model.class_names,
|
||||
"class_stats": model.class_stats,
|
||||
})),
|
||||
None => Json(serde_json::json!({
|
||||
@@ -3610,9 +3615,9 @@ async fn udp_receiver_task(state: SharedState, udp_port: u16) {
|
||||
};
|
||||
|
||||
// Feed field model calibration if active (use per-node history for ESP32).
|
||||
if let Some(ref mut fm) = s.field_model {
|
||||
if let Some(ns) = s.node_states.get(&node_id) {
|
||||
field_bridge::maybe_feed_calibration(fm, &ns.frame_history);
|
||||
if let Some(frame_history) = s.node_states.get(&node_id).map(|ns| ns.frame_history.clone()) {
|
||||
if let Some(ref mut fm) = s.field_model {
|
||||
field_bridge::maybe_feed_calibration(fm, &frame_history);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3695,9 +3700,11 @@ async fn udp_receiver_task(state: SharedState, udp_port: u16) {
|
||||
};
|
||||
|
||||
let raw_persons = derive_pose_from_sensing(&update);
|
||||
let mut last_tracker_instant = s.last_tracker_instant.take();
|
||||
let tracked = tracker_bridge::tracker_update(
|
||||
&mut s.pose_tracker, &mut s.last_tracker_instant, raw_persons,
|
||||
&mut s.pose_tracker, &mut last_tracker_instant, raw_persons,
|
||||
);
|
||||
s.last_tracker_instant = last_tracker_instant;
|
||||
if !tracked.is_empty() {
|
||||
update.persons = Some(tracked);
|
||||
}
|
||||
@@ -3858,9 +3865,9 @@ async fn udp_receiver_task(state: SharedState, udp_port: u16) {
|
||||
};
|
||||
|
||||
// Feed field model calibration if active (use per-node history for ESP32).
|
||||
if let Some(ref mut fm) = s.field_model {
|
||||
if let Some(ns) = s.node_states.get(&node_id) {
|
||||
field_bridge::maybe_feed_calibration(fm, &ns.frame_history);
|
||||
if let Some(frame_history) = s.node_states.get(&node_id).map(|ns| ns.frame_history.clone()) {
|
||||
if let Some(ref mut fm) = s.field_model {
|
||||
field_bridge::maybe_feed_calibration(fm, &frame_history);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3905,9 +3912,11 @@ async fn udp_receiver_task(state: SharedState, udp_port: u16) {
|
||||
};
|
||||
|
||||
let raw_persons = derive_pose_from_sensing(&update);
|
||||
let mut last_tracker_instant = s.last_tracker_instant.take();
|
||||
let tracked = tracker_bridge::tracker_update(
|
||||
&mut s.pose_tracker, &mut s.last_tracker_instant, raw_persons,
|
||||
&mut s.pose_tracker, &mut last_tracker_instant, raw_persons,
|
||||
);
|
||||
s.last_tracker_instant = last_tracker_instant;
|
||||
if !tracked.is_empty() {
|
||||
update.persons = Some(tracked);
|
||||
}
|
||||
@@ -4041,9 +4050,11 @@ async fn simulated_data_task(state: SharedState, tick_ms: u64) {
|
||||
|
||||
// Populate persons from the sensing update (Kalman-smoothed via tracker).
|
||||
let raw_persons = derive_pose_from_sensing(&update);
|
||||
let mut last_tracker_instant = s.last_tracker_instant.take();
|
||||
let tracked = tracker_bridge::tracker_update(
|
||||
&mut s.pose_tracker, &mut s.last_tracker_instant, raw_persons,
|
||||
&mut s.pose_tracker, &mut last_tracker_instant, raw_persons,
|
||||
);
|
||||
s.last_tracker_instant = last_tracker_instant;
|
||||
if !tracked.is_empty() {
|
||||
update.persons = Some(tracked);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user