ADR-271: Depth Anything Rust Camera-Depth Backend

Status: Accepted — phase 1 implemented
Date: 2026-07-20
Deciders: ruv
Tags: rust, depth-anything, ggml, camera, point-cloud, ffi, model-licensing

Context

The wifi-densepose-pointcloud crate fuses camera-derived depth with WiFi CSI occupancy. Its current camera depth path calls an optional loopback MiDaS service and otherwise produces a luminance/edge heuristic. The fallback is deterministic and useful for demonstrations, but it is not a learned or metrically trustworthy depth estimator. It also provides neither confidence nor camera pose.

mudler/depth-anything.cpp is a C++17/ggml inference engine for Depth Anything 2 and 3. It offers quantized GGUF models, CPU/CUDA/Metal/Vulkan execution, and a flat C ABI that returns dense depth, confidence, intrinsics, extrinsics, and a metric-output flag. Reimplementing the transformer, DPT heads, quantization, and ggml backends in Rust would duplicate a large parity-sensitive codebase and make RuView responsible for maintaining several hardware backends.

The upstream ABI currently accepts image paths rather than in-memory RGB frames. RuView captures RGB8 buffers directly, so phase 1 needs a bounded file bridge until an upstream RGB-buffer ABI is available.

Model licensing is not uniform. The upstream engine is MIT, while official DA3 weights include Apache-2.0 and CC-BY-NC-4.0 variants. A permissively licensed runtime must not make non-commercial weights look generally redistributable.

Decision

RuView will integrate Depth Anything as an optional native backend behind a safe Rust boundary, not as a pure-Rust model rewrite and not as a required dependency of camera-free sensing.

Load the upstream shared library dynamically at runtime. The RuView build remains Rust-only and succeeds without CMake, ggml, a model, or a camera.
Isolate the ABI and all unsafe operations in depth_anything.rs. Keep the native library alive for the lifetime of its function pointers and context, serialize calls through Rust mutable ownership, copy native output into Rust vectors, and release every upstream allocation through its matching function.
Require C ABI version 4. Fail closed on any other version.
Configure the backend with environment variables:
- RUVIEW_DEPTH_BACKEND=auto|depth-anything|midas|heuristic
- RUVIEW_DA_LIBRARY, RUVIEW_DA_MODEL, RUVIEW_DA_MODEL_LICENSE
- optional RUVIEW_DA_MODEL_SHA256 and RUVIEW_DA_THREADS
- RUVIEW_DA_ALLOW_NONCOMMERCIAL=1 for an explicit local-only override
Permit Apache-2.0 weights by default. Reject unknown licenses and reject CC-BY-NC-4.0 unless the explicit non-commercial override is present. RuView will not bundle model weights.
When a SHA-256 is configured, verify it before native model loading.
Bridge RGB8 frames through a unique, create-new PPM file with an RAII cleanup guard. Never reuse a predictable existing file. Replace this bridge when the upstream project exposes an in-memory RGB ABI.
Accept only metric Depth Anything output in the point-cloud fusion path. Relative depth must not be labelled or fused as metres.
Preserve the existing MiDaS and heuristic paths. auto attempts configured Depth Anything first, then MiDaS, then the heuristic. An explicitly selected Depth Anything backend reports configuration/load/inference errors instead of silently claiming that heuristic output came from the model.
Carry backend identity, metric status, confidence, and model-provided camera geometry into the Rust DepthEstimate. Confidence becomes point intensity; low-confidence pixels are excluded before voxel fusion.
Camera depth remains an optional point-cloud feature. It does not enter the RuView/RuField camera-free sensing proof path.
Run the native context on a dedicated 16 MiB-stack worker. On Windows, the validated upstream ABI-v4 build currently access-violates while destroying a successfully used DA2 context; RuView therefore retains its process-singleton context and DLL until OS process teardown. Other platforms call da_capi_free.

Consequences

Positive

RuView gains local metric monocular depth without Python or PyTorch inference.
The Rust application owns configuration, policy, provenance, memory safety, fallback behavior, and multimodal fusion.
Confidence and calibrated intrinsics improve point-cloud quality over the current fixed-intrinsics heuristic.
Dynamic loading avoids a mandatory C++ toolchain and preserves portable default builds.
Explicit licensing and digest gates make model provenance auditable.

Negative

Production Depth Anything use still requires building or obtaining a compatible native library and separately obtaining model weights.
The phase-1 PPM bridge performs bounded temporary-file I/O per inferred frame.
The upstream Windows build needs CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS=ON, and its current DA2 destructor requires a process-lifetime context workaround.
Metric models are substantially larger than the current heuristic and are not suitable for ESP32 firmware.
GPU backend quality and performance remain an upstream responsibility.

Neutral

MiDaS compatibility and the deterministic heuristic remain available.
Motion gating in the point-cloud server continues to limit expensive inference.
This decision does not authorize redistribution of third-party model weights.

Validation and acceptance gates

Default builds and tests pass with no native library installed.
License policy accepts Apache-2.0, rejects unknown licenses, and gates CC-BY-NC-4.0 behind an explicit override.
Digest mismatch prevents model loading.
The PPM bridge validates buffer dimensions, uses create-new semantics, and removes its file on success and failure.
A mock ABI proves version gating, native-buffer copying/freeing, metric enforcement, confidence propagation, and context destruction.
Backprojection consumes model dimensions/intrinsics and confidence without out-of-bounds access.
cargo test -p wifi-densepose-pointcloud and Clippy pass.
A real upstream Windows/MSVC ABI-v4 build passes depth-check --abi-only and synthetic inference with the Apache-2.0 DA2 metric Hypersim small Q4_K model (decd99f5c756b564aae5ca1a1612f896e7f76889060e1d25ba610549bbc39b52). Hardware camera validation remains pending receipt of the target hardware.

Alternatives considered

Pure Rust DA3 rewrite

Rejected for phase 1. It would duplicate ggml kernels, quantization, model conversion, and multiple GPU backends while delaying usable integration.

Statically vendor depth-anything.cpp and ggml

Rejected as the default. It would force the C++ build and backend dependency tree onto every RuView build, including camera-free and ESP32-oriented workflows.

Continue with only the MiDaS service

Rejected as the sole production path because it requires an external process and does not provide the same confidence, pose, quantization, or native deployment surface.

Treat relative depth as metres

Rejected because it creates geometrically false point clouds and contaminates WiFi/mmWave fusion.

7.4 KiB Raw Blame History