mirror of
https://github.com/ruvnet/RuView
synced 2026-07-21 17:13:19 +00:00
rUv
42dcf49f4d
* fix(signal): circular phase variance for ghost-tap guard (ADR-154 §7.4 #1) `phase_variance` computed a LINEAR sample variance over phase angles that wrap at ±π, so a tightly-clustered set straddling the branch cut reported spuriously HIGH dispersion — false-tripping the `> TAU` ghost-tap guard on real, tightly-clustered CIR taps. Replace with Mardia's circular variance V = 1 − R̄, bounded [0,1] and invariant to where the cluster sits on the circle. Re-derive the guard against the bounded metric via a named const `GHOST_TAP_CIRCULAR_VARIANCE_MAX` (the old TAU-scaled threshold is meaningless on [0,1]). Grade: metric fix MEASURED; threshold value DATA-GATED — a clean single-path ramp also sweeps the circle, so V alone cannot separate clean from unsanitized without labelled frames. Conservative default (0.99) errs toward never false-rejecting, strictly more permissive at the wrap boundary than the buggy linear guard. Fails-on-old test: `phase_variance_circular_not_fooled_by_branch_cut` — inlines the old linear variance to show it exceeds TAU on wrap-straddling phases while circular V≈0 and the guard no longer trips. Plus `phase_variance_circular_is_bounded_and_extremal` (V∈[0,1], V≈0 identical, V≈1 uniform). cargo test -p wifi-densepose-signal --no-default-features --features cir --lib → 432 passed, 0 failed. Co-Authored-By: claude-flow <ruv@ruv.net> * fix(signal): pin Welford n=0/n=1 finiteness guard (ADR-154 §7.4 #10) The shared `WelfordStats` (field_model.rs, used by longitudinal.rs and others) relies on `count < 2` guards in `variance`/`sample_variance`/`std_dev`/ `z_score` to stay finite at the boundaries. The guards existed but the n=0 boundary was UNTESTED — exactly the §4 divide-by-(n−1) family the ADR groups this with. Add `welford_finite_at_n0_and_n1` asserting every statistic is finite and returns the documented sentinel (0.0) at n=0 and n=1, plus load-bearing doc comments on the two guards. Fails-on-old proof: with the `sample_variance` guard removed, the test FAILS with "attempt to subtract with overflow" at the `(self.count - 1)` underflow (0usize − 1); `variance` would similarly yield 0.0/0.0 = NaN. The guard is restored; the test pins it so a future regression is caught. Grade: MEASURED (boundary finiteness is asserted; the guard is the §4-family fix made testable). cargo test -p wifi-densepose-signal --no-default-features --lib field_model → 22 passed, 0 failed. Co-Authored-By: claude-flow <ruv@ruv.net> * refactor(signal): de-magic adversarial thresholds + boundary tests (ADR-154 §7.4 #13) Lift the bare numeric literals buried in `check`/`check_consistency` into named, documented module consts (FIELD_MODEL_GINI_VIOLATION=0.8, ENERGY_RATIO_HIGH_VIOLATION=2.0, ENERGY_RATIO_LOW_VIOLATION=0.1, CONSISTENCY_ACTIVE_FRACTION_OF_MEAN=0.1, SCORE_W_* weights). VALUES UNCHANGED — each const equals the original literal; only names + pinning tests are new. Grade: DATA-GATED. The operating values stay empirical (defensible values need labelled spoofed/clean CSI — Wi-Spoof, §6.2/§7.3). The de-magicking + characterization tests are MEASURED: `tuning_consts_unchanged_from_literals`, `energy_ratio_high_boundary`, `energy_ratio_low_boundary`, `field_model_gini_boundary`, `consistency_active_fraction_boundary` pin the decision boundaries at/just-below/just-above each threshold, so a future data-driven retune is a visible, tested change. Fails-on-change proof: bumping ENERGY_RATIO_HIGH_VIOLATION 2.0→3.0 makes `energy_ratio_high_boundary` FAIL (restored). Operating values explicitly NOT changed. cargo test -p wifi-densepose-signal --no-default-features --lib ruvsense::adversarial → 20 passed, 0 failed. Co-Authored-By: claude-flow <ruv@ruv.net> * refactor(signal): de-magic coherence drift/gate thresholds (ADR-154 §7.4 #9) Lift the bare detection literals in `coherence.rs::classify_drift` (DRIFT_STABLE_SCORE=0.85, DRIFT_STEP_CHANGE_MAX_STALE=10) and the `coherence_gate.rs` Default impl (DEFAULT_ACCEPT_THRESHOLD=0.85, DEFAULT_REJECT_THRESHOLD=0.5, DEFAULT_MAX_STALE_FRAMES=200, DEFAULT_PREDICT_ONLY_NOISE=3.0) into named, documented consts. VALUES UNCHANGED. The gate already exposed these via GatePolicyConfig (config seam); this names + pins the defaults. Grade: DATA-GATED. Operating values stay empirical (defensible Z-score thresholds need labelled stable/drifting coherence traces). De-magicking + boundary tests are MEASURED: `classify_drift_stable_score_boundary`, `classify_drift_stale_count_boundary` pin the at/just-below/just-above decisions; `drift_consts_unchanged_from_literals` / `gate_default_consts_unchanged_from_literals` pin the values. Operating values explicitly NOT changed. cargo test -p wifi-densepose-signal --no-default-features --lib ruvsense::coherence → 40 passed, 0 failed. Co-Authored-By: claude-flow <ruv@ruv.net> * docs(adr-154): mark §7.4 P1 backlog cleared — Milestone-1 (#1,#10 RESOLVED; #9,#13 DATA-GATED) Update ADR-154 §7.4 backlog rows #1, #9, #10, #13 with commit refs + grades, the §7.4 intro count (four P1 items cleared, ~41 P2/P3 remain), the Horizon-ledger one-liner (Milestone-1 DONE), and the §8 honest-limits #1 line (metric now correct; threshold still DATA-GATED). Add CHANGELOG [Unreleased] entry. Grades: #1 RESOLVED (MEASURED metric / DATA-GATED threshold), #10 RESOLVED (MEASURED), #9 & #13 RESOLVED-PARTIAL (DATA-GATED — de-magicked + boundary tested, operating values unchanged). Validation: cargo test --workspace --no-default-features → 2057 passed, 0 failed; wifi-densepose-signal lib → 442 passed (no-default + --features cir); python archive/v1/data/proof/verify.py → VERDICT: PASS, hash f8e76f21…46f7a UNCHANGED (CIR ghost-tap guard is not on the deterministic proof path). Co-Authored-By: claude-flow <ruv@ruv.net> * fix(sensing-server): stop leaking internal errors in HTTP responses (ADR-080 #2) Six handlers in `main.rs` serialized the internal error `Display` straight into the JSON response body, leaking server internals to any client (ADR-080 finding #2, CWE-209; reframed onto the Rust boundary by ADR-164 G11): - edge_registry_endpoint: a panicked spawn_blocking `JoinError` ("task … panicked") in a 500, and the raw upstream error in a 503 - delete_model / delete_recording / start_recording: std::io::Error strings carrying OS detail / filesystem paths - calibration_start / calibration_stop: the FieldModel error chain New `error_response` module: `internal_error` / `internal_error_json` / `upstream_unavailable` log the full detail server-side only (tagged with a correlation id) and return a generic body (`{"error":"internal_error","correlation_id":…}`) — no `panicked`, no file paths, no Debug chain. The correlation id lets an operator join a client report to the exact server log line without ever shipping the detail. Pinned by 5 error_response tests, incl. a leak-substring guard (internal_error_body_does_not_leak_detail) verified to FAIL on the reverted old body (returns the panic message / path / "os error"). The HOMECORE sweep (ADR-161) covered homecore-server, not this crate. Co-Authored-By: claude-flow <ruv@ruv.net> * test(sensing-server): pin XFF-immunity + no-query-token (ADR-080 #1, #3) Findings #1 (XFF-spoofing bypass) and #3 (JWT-in-URL, CWE-598) were logged against the Python v1 API but are VERIFIED ABSENT on the current Rust sensing-server, so they get regression tests rather than redundant fixes: - #1 XFF: there is no IP-based rate-limiter or IP-allowlist to bypass, and neither security middleware reads a forwarded header. Added bearer_auth::xff_header_never_affects_auth_decision (spoofed X-Forwarded-For never flips a 401<->200 decision) and host_validation::forwarded_headers_never_bypass_host_allowlist (spoofed X-Forwarded-Host: localhost never lets Host: evil.com past the allowlist). - #3 JWT-in-URL: require_bearer reads the token only from the Authorization header; WS handlers take no query token; the sole Query extractor (EdgeRegistryParams) is a non-secret refresh flag. Added bearer_auth::query_string_token_is_never_accepted — ?token= / ?access_token= in the URL never authenticates (stays 401) while the header path still 200s. Verified to FAIL when a query-token path is injected into require_bearer. Co-Authored-By: claude-flow <ruv@ruv.net> * docs(adr-080): mark P0 security findings #1-#3 RESOLVED; close ADR-164 G11 - ADR-080: Status note + per-finding closure (#1 XFF and #3 JWT-in-URL verified absent + regression-pinned; #2 leaked errors fixed via the error_response module). Records the v1-vs-Rust boundary distinction explicitly: v1 paths remain archived; this closure governs the shipped Rust sensing-server. - ADR-164: Gap Register G11 and the Open/Gated Backlog entry marked RESOLVED with the fix + branch reference. - CHANGELOG: [Unreleased] -> ### Security entry covering all three findings. Co-Authored-By: claude-flow <ruv@ruv.net> * docs(adr): renumber 6 displaced ADRs to resolve duplicate-number collisions (ADR-164 G1) Resolves the 5 duplicate ADR numbers (6 displaced files) flagged by ADR-164 Gap Register item G1. Canonical keeper per number = first file committed at that number (date tie-broken by inbound cross-reference count / parent-appendix relationship). Displaced files renumbered to the next free numbers (166-171): 050 keeps provisioning-tool-enhancements (5 refs vs 1) -> ADR-166-quality-engineering-security-hardening 052 keeps tauri-desktop-frontend (parent ADR) -> ADR-167-ddd-bounded-contexts (its appendix) 147 keeps nvidia-cosmos/OccWorld (the actual ADR, has Status header) -> ADR-168-benchmark-proof (proof companion, no Status) -> ADR-169-adam-mode-light-theme (was untracked) 148 keeps drone-swarm-control-system (committed #862) -> ADR-170-yoga-mode-pose-system (was untracked) 149 keeps public-community-leaderboard-huggingface (committed 16:47 vs 17:38) -> ADR-171-swarm-benchmarking-evaluation-methodology Updates in-file `# ADR-NNN` headers and intra-file self-references (yoga-modes * docs(adr): repoint inbound cross-references to renumbered ADRs (166-171) Follow-up to the ADR renumbering (ADR-164 G1). Updates every inbound reference that pointed at a displaced ADR, disambiguating shared numbers by title/slug so only references to the DISPLACED topic move and keeper references stay put. ADR-168 (was 147 benchmark-proof): README, CHANGELOG, user-guide, proof-of-capabilities, research docs 00/03 — all path/label refs updated. ADR-169 (was 147 adam-mode) / ADR-170 (was 148 yoga-mode): docs/adr/README index. ADR-171 (was 149 swarm-benchmarking): all ruview-swarm eval code+docs (Cargo.toml, evals/, eval_swarm.rs, metrics/mod/report/runner.rs), research doc 03 (every §-ref matched ADR-171 sections, not AetherArena), 00-system-review, series README, CHANGELOG, and ADR-148's forward/"open issues" pointers. ADR-166 (was 050 quality-engineering / security-hardening): disambiguated from the ADR-050 provisioning KEEPER by topic. The HMAC/secure_tdm, directory-traversal, bind-address, and OTA-PSK-auth references in code comments (wifi-densepose-hardware Cargo.toml + secure_tdm.rs, sensing-server main.rs) and in ADR-052-tauri / ADR-167 all describe the security-hardening ADR -> ADR-166. ADR-167 (was 052 ddd-appendix): inbound appendix references. Index/registry updates: docs/adr/README.md, gap-analysis/census.md (rows + header count), gap-analysis/lens-findings.md (collision table marked RESOLVED), and ADR-164 Gap Register G1 marked RESOLVED with the full renumber map. Keeper references deliberately untouched: all ADR-147 OccWorld code, all ADR-148 drone-swarm code/docs, all ADR-149 AetherArena refs (incl. ADR-150's SSL/resampling refs, which ADR-150 explicitly binds to the AetherArena benchmark), ADR-050 provisioning refs, ADR-052 tauri refs. The frozen GitHub blob URLs in docs/adr/.issue-177-body.md (pinned to an old branch) are left as historical. Comment-only code edits; no behavior change. wifi-densepose-hardware compiles clean; the sensing-server build's sole blocker is the pre-existing upstream midstreamer-temporal-compare@0.2.1 registry crate, unrelated to these edits. Co-Authored-By: claude-flow <ruv@ruv.net>
258 lines
14 KiB
Markdown
258 lines
14 KiB
Markdown
# ADR-171: Drone Swarm Benchmarking & Evaluation Methodology — Metrics, Leaderboards, and Statistical Rigor
|
||
|
||
| Field | Value |
|
||
|------------|-----------------------------------------------------------------------------------------|
|
||
| Status | Accepted (peer-reviewed 2026-05-30) |
|
||
| Date | 2026-05-30 |
|
||
| Deciders | ruv |
|
||
| Relates to | ADR-148 (ruview-swarm), ADR-147 (OccWorld), ADR-146 (RF encoder), ADR-028 (witness) |
|
||
|
||
> Companion to ADR-148. ADR-148 shipped the swarm and 5 criterion micro-benchmarks
|
||
> plus a `SotaComparison` against Wi2SAR. This ADR defines **how we evaluate the swarm
|
||
> rigorously** — what metrics, what statistics, what baselines, and an honest account
|
||
> of which external leaderboards do and do not apply.
|
||
|
||
---
|
||
|
||
## 1. Context
|
||
|
||
ADR-148's `ruview-swarm` reports performance via five `criterion` micro-benchmarks and a
|
||
single `SotaComparison` (localization 1.732 m vs Wi2SAR 5 m; coverage ~223 s vs 810 s).
|
||
These numbers are **internally valid but insufficient as scientific claims**:
|
||
|
||
- The criterion figures (3.3 µs MARL inference, 43 µs RRT-APF, 54 ns fusion, 248 µs PPO
|
||
step) measure **wall-clock latency**, not policy quality or coverage/localization quality.
|
||
- The 1.732 m localization comes from a **single synthetic geometry** (3 drones at 120°
|
||
around a known point), not a distribution of victim positions under realistic noise.
|
||
- The 223 s coverage is an **analytic estimate** (`estimate_coverage_time_secs()`), not an
|
||
episode rollout.
|
||
- All numbers are **single-run point estimates**. The MARL reproducibility literature
|
||
(Henderson 2018; Agarwal 2021; Gorsane 2022) shows single/few-seed point estimates
|
||
routinely flip algorithm rankings and overstate gains.
|
||
|
||
We need a defined, reproducible evaluation methodology before any "beats SOTA" claim can
|
||
survive external review, and an honest position on external leaderboards.
|
||
|
||
---
|
||
|
||
## 2. Decision
|
||
|
||
Adopt a two-tier evaluation methodology:
|
||
|
||
1. **Micro-benchmarks (criterion)** — keep for compute-latency regression gating only.
|
||
Explicitly labeled as latency, never as quality.
|
||
2. **Domain evaluation harness** — a seeded, multi-run, statistically-reported harness
|
||
producing SAR metrics (localization CEP, coverage, detection rate) and MARL metrics
|
||
(IQM return, probability-of-improvement) over **≥10 seeds with 95% stratified-bootstrap
|
||
confidence intervals**, against **≥3 baselines**, following the Agarwal/Gorsane standard.
|
||
|
||
Do **not** claim leaderboard standing — no public leaderboard accepts drone-swarm CSI-SAR
|
||
submissions. Comparisons to Wi2SAR are **paper-to-paper**, labeled as such, acknowledging
|
||
the sensing-modality difference (RSS bearing vs CSI multi-view fusion).
|
||
|
||
---
|
||
|
||
## 3. External Leaderboard Landscape — Honest Assessment
|
||
|
||
**There is no public, externally-administered leaderboard that accepts a drone-swarm,
|
||
CSI-based, multi-view SAR system.** This is a research niche; comparison is paper-to-paper.
|
||
The adjacent options and their fit:
|
||
|
||
| Benchmark / Leaderboard | Domain | Live submission? | Fit for ruview-swarm |
|
||
|-------------------------|--------|------------------|----------------------|
|
||
| **Wi2SAR** (arxiv 2604.09115) | Drone WiFi SAR | No (paper) | **Direct baseline** — paper-to-paper only; RSS bearing ≠ CSI fusion |
|
||
| **MARL4DRP** (Springer 2023) | Drone routing MARL | No | Closest drone-MARL benchmark; would need a routing→coverage adapter |
|
||
| **CSI-Bench** (NeurIPS 2025) | Static WiFi sensing | Splits + paper baselines | Adjacent (localization task) but no moving-sensor/multi-view fusion |
|
||
| **SMAC / SMACv2** | StarCraft cooperative MARL | No live LB | Structural analogy (CTDE) only; combat task, not coverage |
|
||
| **PettingZoo MPE** (Simple Spread) | 2D cooperative particles | No | Cheap MARL **correctness check**, no physics/CSI |
|
||
| **Melting Pot** | Social-dynamics MARL | Closed (NeurIPS '24) | Not applicable |
|
||
| **MAMuJoCo / Hanabi / GRF / Overcooked** | Various cooperative MARL | No live LB | Not applicable |
|
||
| **OmniDrones / gym-pybullet-drones / Pegasus** | Drone-control sim platforms | No (platforms) | **Training infrastructure**, not leaderboards; no CSI layer |
|
||
|
||
**Conclusion:** We will (a) keep Wi2SAR as the cited paper baseline, (b) optionally build a
|
||
MARL4DRP/MPE adapter to post a recognized cooperative-MARL number (tangential to SAR), and
|
||
(c) **not** represent any internal number as a leaderboard placement.
|
||
|
||
---
|
||
|
||
## 4. Evaluation Metrics
|
||
|
||
### 4.1 SAR Domain Metrics (primary — comparable to Wi2SAR)
|
||
|
||
| Metric | Definition | Reporting |
|
||
|--------|-----------|-----------|
|
||
| Localization CEP50 | Median horizontal error, fused victim position vs ground truth | m, 95% CI |
|
||
| Localization CEP95 | 95th-percentile horizontal error | m |
|
||
| **GDOP** | Geometric Dilution of Precision of the contributing-drone constellation at detection time | dimensionless (tracked per detection) |
|
||
| Coverage rate @ T | Fraction of area scanned ≥1× within T=240 s | %, 95% CI |
|
||
| Coverage time to 95% | Time to scan 95% of bounded area | s, mean ± CI |
|
||
| Time-to-first-detection | Mission start → first confident detection (conf > 0.85) | s, 95% CI |
|
||
| Detection rate | P(detected \| victim present) per mission | %, 95% CI |
|
||
| False-alarm rate | P(confident detection \| no victim) | %, 95% CI |
|
||
| Collision rate | Collisions (d < 1.5 m) per mission | count/mission |
|
||
| Overlap ratio | Fraction of path re-covering scanned cells | % |
|
||
|
||
### 4.2 MARL Policy-Quality Metrics
|
||
|
||
| Metric | Definition |
|
||
|--------|-----------|
|
||
| IQM episodic return | Interquartile mean over 10 seeds × 50 eval episodes (Agarwal 2021) |
|
||
| Probability of improvement | P(MAPPO return > IPPO return) on a random episode |
|
||
| Optimality gap | Expected gap to a defined reference performance |
|
||
| Performance profile | Fraction of (seed, episode) with localization error < τ, plotted vs τ ∈ [0,10] m |
|
||
| Sample efficiency | Return vs training steps (curve, not point) |
|
||
|
||
### 4.3 Micro-benchmarks (criterion — latency only)
|
||
|
||
Retained from ADR-148, **labeled as compute latency, not quality**:
|
||
`marl_actor_inference` 3.3 µs · `rrt_apf_100iter` 43 µs · `multiview_fusion_3drones` 54 ns ·
|
||
`demo_coverage_estimate` 100 ps · `ppo_update_64transitions` 248 µs. Purpose: prove the
|
||
control loop has no compute bottleneck (all ≪ the 10 ms / 100 Hz budget) and gate
|
||
performance regressions. They are **not** evidence of policy or localization quality.
|
||
|
||
---
|
||
|
||
## 5. Statistical Protocol (Agarwal 2021 / Gorsane 2022)
|
||
|
||
| Requirement | Standard adopted |
|
||
|-------------|------------------|
|
||
| Seeds per condition | **≥10** training runs from distinct seeds |
|
||
| Evaluation episodes | 50 fixed, versioned episodes per trained policy (10 victim layouts × 5 CSI-noise levels) |
|
||
| Aggregate metric | **IQM** (not mean, not median) + performance profiles |
|
||
| Confidence intervals | **95% stratified bootstrap**, 1,000 resamples |
|
||
| Baselines (≥3) | Random walk (lower bound), Boustrophedon+manual-triangulation (heuristic), IPPO (no shared critic) |
|
||
| Reproducibility | Versioned YAML config (drone count, area, victims, CSI σ amplitude / κ phase, wind, packet loss) + all seeds committed with results |
|
||
|
||
Rationale: Henderson et al. (2018) found ≤5-seed point estimates flip rankings; Agarwal et
|
||
al. (2021, NeurIPS Outstanding Paper) show IQM needs ~10 runs for the statistical power that
|
||
the median needs ~200 runs for; Gorsane et al. (2022) made ≥10 seeds + IQM + stratified CIs
|
||
the cooperative-MARL standard. `rliable` (google-research/rliable) is the reference impl.
|
||
|
||
---
|
||
|
||
## 6. Reproducibility Harness (`evals/`)
|
||
|
||
A new evaluation harness (separate from criterion micro-benchmarks):
|
||
|
||
1. **Seeded episodes** — every episode, noise perturbation, and training run seeded from a
|
||
versioned config; seeds committed with results (no `Date.now()`/unseeded RNG).
|
||
2. **Per-episode logging** — coverage %, localization error, GDOP, time-to-first-detection,
|
||
collisions, detection binary → JSONL (reuses the ADR-148 telemetry schema).
|
||
3. **Aggregation** — IQM ± 95% stratified-bootstrap CI across the 10-seed × 50-episode matrix.
|
||
4. **Baseline sweep** — random / boustrophedon-heuristic / IPPO / MAPPO, so
|
||
probability-of-improvement and performance profiles are computable.
|
||
5. **Output** — committed `evals/RESULTS.md`: a reproducible internal leaderboard ranking
|
||
our 6 flight patterns × learning patterns on the SAR metrics, plus the Wi2SAR paper row.
|
||
|
||
This `RESULTS.md` is the **real, defensible "leaderboard" for this system** — patterns ranked
|
||
against each other and the cited baseline, reproducibly, with CIs.
|
||
|
||
### 6.1 Dual-stage pipeline (compute-cost mitigation)
|
||
|
||
The full matrix is **10 seeds × 50 episodes × ≥4 conditions = ≥2,000 rollouts per policy**.
|
||
Running each rollout against the OccWorld 3D prior (ADR-147, ~375 ms/inference) would melt
|
||
the L4 / RTX 5080 budget. Split evaluation into two stages:
|
||
|
||
- **Stage 1 — Kinematic (fast, full matrix).** Stripped vector environment; OccWorld paths
|
||
pre-cached or treated as static analytical volumes. Produces episodic **return, IQM,
|
||
sample-efficiency curves, coverage %, GDOP, localization error** over the full 10-seed matrix.
|
||
- **Stage 2 — High-fidelity physics (sub-sampled).** Take the **3 median seeds** (by Stage-1
|
||
IQM) into Gazebo + PX4 SITL with full CSI phase/amplitude noise. Extracts **false-alarm
|
||
rate** and **collision rate** under realistic dynamics (heading-rate limits, APF repulsion,
|
||
motor response) that the kinematic sim omits.
|
||
|
||
Stage 1 is CI-runnable today; Stage 2 requires the Gazebo/PX4 SITL bring-up (follow-on).
|
||
|
||
### 6.2 Noise sweep (coherence-gate threshold)
|
||
|
||
The config generator systematically varies the two CSI noise parameters:
|
||
- **σ** — Gaussian amplitude noise (CSI magnitude)
|
||
- **κ** — von Mises phase concentration (lower κ = noisier phase)
|
||
|
||
Sweeping (σ, κ) isolates the exact environmental threshold where `CrossViewpointAttention`
|
||
(ADR-016) drops out of its coherence gate (`coherence_gate.rs` Accept → PredictOnly/Reject,
|
||
ADR-135). This finds the operating envelope, not just a single-point accuracy.
|
||
|
||
### 6.3 GDOP tracking
|
||
|
||
Localization accuracy is meaningless without the constellation geometry that produced it.
|
||
The harness records **GDOP** per detection: 3 drones in a ~120° constellation give the
|
||
√3 ≈ 1.73× CRLB improvement; 3 **collinear** drones degrade toward the single-view
|
||
Cramer-Rao limit (~2.9 m). Reporting localization error **stratified by GDOP band** prevents
|
||
the headline number from being a best-case geometric artifact.
|
||
|
||
---
|
||
|
||
## 7. Evidence Grading of Current ADR-148 Numbers
|
||
|
||
| Claim | Grade | Why |
|
||
|-------|-------|-----|
|
||
| criterion latencies (3.3 µs / 43 µs / 54 ns / 248 µs) | **High** | Deterministic compute, hardware-specific, reproducible |
|
||
| Wi2SAR baseline (5 m, 160k m²/13.5 min) | **High** | Published field trial, open source |
|
||
| 1.732 m 3-view localization | **Low–Medium** | Single synthetic geometry; no noise distribution; CRLB predicts ~2.9 m for N=3 |
|
||
| 223 s 4-drone coverage | **Low** | Analytic estimate, not an episode rollout |
|
||
| "beats SOTA" | **Directional only** | Valid as paper-to-paper direction; not leaderboard, not multi-seed |
|
||
|
||
The √N multi-view scaling claim is theoretically sound (CRLB: σ ∝ 1/√(N·SNR); N=3 → √3 ≈
|
||
1.73× improvement), but the measured 1.732 m must be reproduced over a victim-position and
|
||
noise distribution before it is defensible.
|
||
|
||
---
|
||
|
||
## 8. Consequences
|
||
|
||
### Positive
|
||
- Converts scattered numbers into a reproducible, statistically-honest evaluation.
|
||
- The `RESULTS.md` internal leaderboard ranks the 6 flight × 4 learning patterns fairly.
|
||
- Aligns with the recognized MARL evaluation standard (IQM + stratified CIs + ≥10 seeds).
|
||
- Honest external-leaderboard position avoids overclaiming.
|
||
|
||
### Costs / Risks
|
||
- ≥10 seeds × 50 episodes × N patterns × N baselines is a real compute cost — this is where
|
||
the ADR-148 GCP L4 / local RTX 5080 training budget is actually spent.
|
||
- Requires the MARL policy to be **trained to convergence** first (the ADR-148 5-episode CPU
|
||
run shows decreasing value_loss, not convergence).
|
||
- Coverage/localization must move from analytic estimate / synthetic geometry to **episode
|
||
rollouts under realistic CSI noise** before headline numbers are republished.
|
||
|
||
### Open issues → follow-on work
|
||
1. Train MAPPO/IPPO to convergence (M4 follow-on) before running the eval harness.
|
||
2. Build the seeded `evals/` harness + `RESULTS.md` generator.
|
||
3. Optional: MARL4DRP or MPE Simple-Spread adapter for a recognized cooperative-MARL number.
|
||
4. Re-state ADR-148 §14 headline numbers with CIs once the harness has run.
|
||
|
||
---
|
||
|
||
## 9. Research Notes & References
|
||
|
||
Compiled by `ruflo-goals:deep-researcher` (2026-05-30). Full landscape in the agent record.
|
||
|
||
**MARL evaluation rigor**
|
||
- Henderson et al., "Deep RL That Matters", arxiv 1709.06560 — ≤5-seed estimates flip rankings
|
||
- Agarwal et al., "Deep RL at the Edge of the Statistical Precipice", NeurIPS 2021, arxiv 2108.13264 — IQM, performance profiles, stratified bootstrap; `rliable`
|
||
- Gorsane et al., "Standardised Evaluation Protocol for Cooperative MARL", NeurIPS 2022, arxiv 2209.10485 — ≥10 seeds + IQM standard
|
||
- BenchMARL, arxiv 2312.01472 — operationalizes the above
|
||
|
||
**Cooperative-MARL benchmarks**
|
||
- SMACv2, arxiv 2212.07489 · PettingZoo MPE (Farama) · Melting Pot (DeepMind, NeurIPS 2024 contest) · MAMuJoCo (Gymnasium-Robotics) · MARL4DRP, Springer 2023 (closest drone-MARL)
|
||
|
||
**Drone-sim platforms**
|
||
- gym-pybullet-drones, arxiv 2103.02142 · OmniDrones, IEEE RA-L 2024 · Pegasus, arxiv 2307.05263 · Flightmare (IROS 2021) · AirSim (discontinued 2022) · Crazyswarm2
|
||
|
||
**SAR / coverage / CSI sensing**
|
||
- Wi2SAR, arxiv 2604.09115 (direct baseline: 5 m, 160k m²/13.5 min, 18.4° median AoA)
|
||
- CSI-Bench, NeurIPS 2025, arxiv 2505.21866 (461 h WiFi sensing, localization task)
|
||
- Coverage path planning, PMC9571681 (boustrophedon ~5% faster than spiral)
|
||
- Bio-inspired SAR, Nature s41598-025-33223-z (PSO > Levy/ACO on exploration score)
|
||
- CRLB for CSI localization, IEEE 8110647 (σ ∝ 1/√(N·SNR))
|
||
|
||
**Tooling**
|
||
- criterion.rs known limitations — wall-clock only, not algorithmic quality
|
||
- rliable, github.com/google-research/rliable
|
||
|
||
---
|
||
|
||
*ADR authored with research support from `ruflo-goals:deep-researcher` (2026-05-30).
|
||
Companion to ADR-148. Defines the evaluation methodology that the ADR-148 headline
|
||
numbers must satisfy before being republished as defensible claims.*
|