mirror of
https://github.com/ruvnet/RuView
synced 2026-07-25 17:51:48 +00:00
ADR-152: WiFi-Pose SOTA 2026 intake — WiFlow-STD benchmark, Rust integrations, ADR-153 802.11bf layer, efficiency frontier (#1008)
* feat(calibration): NodeGeometry transceiver-geometry recording (ADR-152 §2.1.1) PerceptAlign-motivated geometry capture at enrollment: per-node optional records (position, antenna orientation, inter-node distances, acquisition method) — recorded when known, never required. Event-sourced via EnrollmentEvent::GeometryRecorded (latest recording wins); persisted on SpecialistBank with serde defaults so pre-ADR-152 bank JSON loads cleanly (fixture-proven, and geometry-free banks serialize byte-shape-identical to the old schema); threaded through MultiNodeMixture as data only — the learned geometry embeddings and algorithmic fusion use are §2.1.2, deliberately deferred until the ADR-151 P6 LoRA heads exist. Geometry recorded from now on means banks captured today remain usable for layout-conditioned training later — you can't retroactively add geometry to data you didn't record. 8 new tests (3 geometry, 2 anchor, 2 bank, 1 multistatic) + full-loop extension (2-node geometry, one tape-measured + one unknown, surviving the bank JSON round-trip the runtime loads from). 50/50 calibration (both feature configs) + 23 CLI tests green. Co-Authored-By: RuFlo <ruv@ruv.net> * feat(training): two-checkerboard camera↔room calibration for ADR-079 labels (ADR-152 §2.1.3) Defends the camera-supervised pipeline against PerceptAlign's "coordinate overfitting": MediaPipe keypoints were emitted in raw camera coordinates with no shared frame and no transceiver-geometry metadata — the exact label shape that memorizes deployment layout and collapses cross-layout. - scripts/calibrate-camera-room.py + calibration_lib.py: OpenCV two-checkerboard calibration → versioned bundle JSON (intrinsics, camera→room extrinsics, checkerboard spec, transceiver geometry, sha256 calibration_id). Intrinsics resolve from file > cache > multi-view computation > loud-warning 2-view fallback. - collect-ground-truth.py --calibration <bundle>: every sample gains keypoints_room (unit bearing rays from the camera center in the room frame — documented projective alignment; raw image coords preserved so training chooses), camera_origin_room, calibration_id, and the transceiver geometry stamp. Without the flag, output is byte-identical to before (tested) + a one-line ADR-152 warning. Design finding (recorded for ADR-152): a single planar checkerboard's corner grid is centrosymmetric — the reversed corner ordering fits a ghost camera pose with IDENTICAL reprojection error, so per-board flip disambiguation is mathematically ill-posed. solve_two_board_extrinsics solves the joint wall+floor set over all 4 flip combinations, where the minimum is unique — an independent reason the TWO-checkerboard method is required, beyond what PerceptAlign states. 15 headless pytest tests green (synthetic corners: extrinsics recovery incl. ghost resolution, bundle round-trip + hash stability, ray transforms w/ distortion + cross-resolution, no-calibration byte identity). Co-Authored-By: RuFlo <ruv@ruv.net> * feat(benchmarks): WiFlow-STD reproduction harness + measurement (a) results (ADR-152 §2.2) Shipped checkpoint REFUTED (0.08% PCK@20, wrong keypoint normalization); 6 reproducibility defects documented (broken imports, corrupted dataset tail with float32-max garbage that NaN-poisons fp16 BatchNorm, unreachable test phase). After repairs, retraining with upstream defaults reproduces 96.09% PCK@20 full-test / 96.61% corruption-free (published 97.25%) on RTX 5080. Claims graded MEASURED-EQUIVALENT; 2.23M params + ~0.055 GFLOPs verified. Third-party code/weights/data stay out of tree (gitignored). Co-Authored-By: claude-flow <ruv@ruv.net> * feat: ADR-152 Rust integrations + ADR-153 802.11bf protocol model - calibration: GeometryEmbedding — 32-slot permutation-invariant NodeGeometry featurization for future LoRA-head conditioning (ADR-152 §2.1.2); derived SpecialistBank::geometry_embedding() accessor; 59 tests - train: MaePretrainConfig + patchify/random-mask with UNSW measured recipe (80% masking, (30,3) patches; ADR-152 §2.3, arXiv 2511.18792); strict no-truncate/no-NaN policy; proptest properties - train: WiFlowStdModel — tch-gated port of the verified ~96%-PCK@20 WiFlow-STD architecture (ADR-152 §2.2 beyond-SOTA); ungated param formula pinned to 2,225,042; 15/17-keypoint support; 239 crate tests - hardware: ieee80211bf forward-compatibility protocol model (ADR-153): SpecProfile gates, SensingCapabilities negotiation, required ConsentMode, session FSM, SensingTransport + SimTransport + OpportunisticCsiBridge; full acceptance checklist covered; 156+4 tests - deps: ruvector bumps per ADR-152 §2.6 survey (mincut/solver 2.0.6, attention 2.1.0, gnn 2.2.0); vendor/ruvector synced to a083bd77f - docs: ADR-153 accepted; ADR-152 §2.2 status, §2.4 amendment, §2.6 added Workspace: 162 test suites green (--no-default-features); Python proof PASS. Known pre-existing flake: homecore-api env_empty_falls_back_to_defaults (unserialized env-var mutation) — untouched, follow-up. Co-Authored-By: claude-flow <ruv@ruv.net> * docs: CHANGELOG + CLAUDE.md entries for ADR-152 integrations and ADR-153 Co-Authored-By: claude-flow <ruv@ruv.net> * fix(train): repair tch-backend bit-rot — gated path compiles and tests run again Mechanical API refresh against current tch: Vec::from(Tensor) -> try_from (+ explicit flatten), numel() usize cast, Rem/div ops -> remainder() / divide_scalar_mode(floor) — the latter fixed a silent true-division bug in heatmap argmax decoding; clamp(1.0, f64::MAX) -> clamp_min (torch 2.x scalar overflow panic); petgraph EdgeRef import; missing EvalMetrics and verify_checkpoint_dir APIs that tests documented. wiflow_std roundtrip test uses safetensors (.pt _save_parameters roundtrip broken in torch 2.11 Windows). Gated: 349 passed (incl. all 20 wiflow_std); ungated: unchanged. Known pre-existing: gaussian-heatmap convention mismatch (2 tests), proof seed race under parallel threads — documented, deliberate follow-ups. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(train): WiFlow-STD PyTorch->tch weight import + numerical parity proof export_to_safetensors.py maps the retrained checkpoint (295 tensors -> 248 mapped, param sum exactly 2,225,042; num_batches_tracked dropped) into a tch-loadable safetensors plus a deterministic parity fixture. Gated #[ignore] integration test loads it strictly and asserts forward-pass agreement: max abs diff 1.192e-7 on the seed-42 fixture. dump_variable_names test makes the tch name layout authoritative. Zero architecture discrepancies found. Co-Authored-By: claude-flow <ruv@ruv.net> * fix: workflow-review findings — BN gamma init, ThresholdParams serde, init docs Concurrent validation workflow (2 review lanes + adversarial verification, 13 agents): 5 confirmed findings, 3 refuted. Fixes: - wiflow_std: pin BatchNorm gamma to 1.0 (tch default draws Uniform(0,1) — silently halves activations in from-scratch training; loaded checkpoints unaffected, parity re-verified after the change) - wiflow_std: document the conv-init divergences vs the reference's effective kaiming_normal(fan_out) re-init (from-scratch dynamics only) - ieee80211bf: ThresholdParams deserialization validates via try_from so the <=100 invariant holds for untrusted payloads (+ rejection test) Benchmarks (release, ruvzen): GeometryEmbedding 1.84us/call (542k/s), MAE tokenization 7.38us/window (135k/s), 802.11bf FSM 8.9M events/s — nothing suspicious. Co-Authored-By: claude-flow <ruv@ruv.net> * docs(adr): ADR-152 §2.1.4 gate resolved — PerceptAlign repo MIT, dataset on HF Co-Authored-By: claude-flow <ruv@ruv.net> * feat(benchmarks): edge optimization measured + measurement (b) blocked + 92.9% retraction Edge optimization (ADR-152 optimize track): ONNX Runtime fp32 is the CPU latency win (3.2 ms/window, ~3.4x faster than torch, parity 2.4e-7); ORT dynamic int8 reaches 2.44 MB (paper's ~2.2 MB claim plausible only via conv-capable toolchains; -0.16pt PCK@20, +18% MPJPE, 2x slower); torch dynamic quant converts 0% of this conv-only model; fp16 halves storage free but is slower on CPU. Measurement (b) BLOCKED-ON-DATA: only 1,077 paired ESP32 windows exist (stop rule <2k). Forensic recheck of the surviving April holdout RETRACTS the ADR-079 '92.9% PCK@20' figure: constant-output model, absolute (not torso) threshold, 69 near-static frames — mean predictor scores 100% under that protocol; torso-PCK@20 is 19.1%. Corroborates PR #535. Stale citations removed from user-guide, readme-details, ADR-152 §2.1.3; no-citation rule extended to ADR-079 accuracy claims. Unblock: >=2k-window multi-pose paired session + torso-PCK re-baseline. Co-Authored-By: claude-flow <ruv@ruv.net> * docs(user-guide): corrected camera-supervised collection tutorial Step 0 CSI-rate check + session-length math (window yield = frames/20 — the May session's 8x under-delivery was a ~12 Hz CSI rate, not an aligner bug); two-checkerboard calibration step (ADR-152 §2.1.3); pose-variety and confidence guidance; torso-normalized PCK + temporal-split + pred-variance eval protocol (lessons from the 92.9% retraction); scale presets re-keyed to realistic window counts. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(benchmarks): static PTQ int8 (calibrated) results + overnight capture script Conv-only static QDQ beats dynamic int8 on accuracy (PCK@20 96.61-96.63% vs 96.52%, MPJPE +10% vs +18% over fp32) at ~equal size/latency; all-ops QDQ strictly worse (int8 activations through attention glue). Entropy calibration verified bit-identical to MinMax on this data. Deployment: ONNX fp32 for speed (3.2ms), static conv-only QDQ for smallest (2.53MB). Also: scripts/overnight-empty-capture.py — segmented UDP CSI recorder for empty-room baselines (no glob collisions, detach-safe). Co-Authored-By: claude-flow <ruv@ruv.net> * feat(benchmarks): measurement (b) MEASURED — optimization transfer only, mean-pose baseline wins WiFlow-STD fine-tuned on 2,046 fresh single-room ESP32 paired windows (temporal 70/15/15, 70->540 adapter, K=17): pretrained-init 65% PCK@20 vs scratch 0% (optimization transfer) but frozen-trunk ~0% (no feature transfer), and NOTHING beats the mean-pose baseline (95.9% PCK@20 — single subject, near-static normalized coords). Honesty gates held: pred std 0.0113 (non-constant model) but mean-baseline dominance means no citable CSI->pose capability from this data. ADR-152 open question 1 answered partially; definitive answer needs multi-subject/position data. Two new aligner findings: heterogeneous csi_shape with silent zero-padding (~20%), and extractCsiMatrix's transposed shape label (frame-major data, [nSc, nFrames] label) — fixes pending. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(benchmarks): efficiency sweep MEASURED — half model dominates full reference Compact WiFlow-STD variants on the same data/split/protocol: half (843,834 params, 0.38x) strictly dominates the 2.23M reference (PCK@20 96.62 vs 96.61, PCK@50 99.47 vs 99.11, MPJPE 0.00898 vs 0.0094) — the published architecture is over-parameterized for its own benchmark. quarter (338k) 96.05%; tiny (56,290 params, 1/39.5) holds 94.11% — a ~220KB fp32 edge candidate. In-domain caveats recorded; cross-domain untested. Co-Authored-By: claude-flow <ruv@ruv.net> * feat(train): compact WiFlow-STD presets in Rust + tiny edge artifact (ADR-152) WiFlowStdConfig gains half()/quarter()/tiny() mirroring the overnight sweep exactly: TcnGroupsMode (Fixed/Gcd/Depthwise), input_pw_groups, derived stride schedule and decoder-mid (all default to upstream behavior; legacy serde JSON unaffected). Param formulas pin to trained ground truth first try: 843,834 / 338,600 / 56,290; default 2,225,042 pin and 1.192e-7 parity unchanged. 248 tests green. Tiny edge artifact (tiny_edge_bench.py): ONNX fp32 = 295 KB, 0.66 ms/win (~1,500/s CPU), 94.11% PCK@20 (matches sweep clean-test exactly; parity 1.49e-7). Static int8 is a bad trade at this scale (-1.43pt, +19% MPJPE, -16% size, slower) — recorded as negative result. Export note: width-16 breaks AdaptiveAvgPool((15,1)) TorchScript export; replaced by exact mean+matmul equivalent, proven by parity. Co-Authored-By: claude-flow <ruv@ruv.net> * fix: resolve all 10 confirmed code-review findings (7-angle review, 20/20 verified) wiflow_std: min_feature_width (default 15) replaces the keypoints->stride coupling — for_keypoints(17) now provably builds the trained [2,2,2,2] graph and pools 15->17, matching the validated Python protocol (pinned by tests); param_count() total on invalid configs; random_mask returns Result and rejects non-finite/out-of-range ratios; trainer checkpoints switched to safetensors (.pt VarStore roundtrip broken on Windows torch 2.11). ieee80211bf: SBP proxy now re-triggers instances and relays reports via Action::RelaySbpReport -> SensingFrame::SbpReport (clients consume via their existing path); missed_instances reset on success = consecutive semantics; SessionTable gains a guarded SBP entry point + unknown-id drop counter; initiator-role sessions reject inbound setup/SBP requests (RejectedNotSupported) closing the idle hijack; StartSetup/StartSbp outside Idle return InvalidStateForCommand; SBP validation unified through evaluate_setup with a 1:1 SetupStatus->SbpStatus mapping. events.rs split out to honor the 500-line cap. calibration/cli: enrollment geometry now actually reaches trained banks — both production call sites attach .with_geometry; --geometry flag on train-room and POST /enroll/geometry + train-body geometry on calibrate-serve give production a recording surface; geometry-free banks log the ADR-152 §2.1.2 note. benchmarks: corruption masks committed as ground truth (unregenerable after in-place cleaning; verified bit-identical regeneration from the pristine copy) + generate_corruption_masks.py producer; _bench_common.py dedups the 5x-copied shim/evaluate/seed/remap (post-refactor PCK@20 re-verified equal to the last digit); remote scripts get the mmap patch; tiny_edge --calib validated multiple-of-64; onnx_bench --help no longer executes (and overwrote) the export — artifact restored byte-exact. Workspace: 2,963 tests passed, 0 failed; Python proof PASS. Co-Authored-By: claude-flow <ruv@ruv.net> * ci: build workspace tests without debuginfo — runner disk exhaustion The combined 38-crate debug target exceeds the GitHub runner's disk ('final link failed: No space left on device'); the same tree measured 151GB locally with full debuginfo. CARGO_PROFILE_{DEV,TEST}_DEBUG=0 shrinks the target ~5-10x; debuginfo serves no purpose in CI test runs. Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
rUv
@@ -0,0 +1,26 @@
|
||||
# Upstream clone (WiFlow-STD, DY2434) -- never commit third-party code/weights
|
||||
upstream/
|
||||
|
||||
# Local python env
|
||||
.venv/
|
||||
|
||||
# Downloaded data / artifacts
|
||||
data/
|
||||
downloads/
|
||||
*.pth
|
||||
*.pt
|
||||
*.npy
|
||||
*.npz
|
||||
*.zip
|
||||
*.mat
|
||||
*.safetensors
|
||||
results/parity_fixture.json
|
||||
__pycache__/
|
||||
*.onnx
|
||||
|
||||
# Committed ground truth: corruption masks for the pristine Kaggle download.
|
||||
# remote/clean_v2.py zeroes the corrupted source windows IN PLACE, so these
|
||||
# masks CANNOT be regenerated from a cleaned copy (generate_corruption_masks.py
|
||||
# documents the criteria and reproduces them only from a fresh download).
|
||||
!results/nan_windows_mask.npy
|
||||
!results/big_windows_mask.npy
|
||||
@@ -0,0 +1,486 @@
|
||||
# WiFlow-STD (DY2434) Benchmark Results — ADR-152 §2.2
|
||||
|
||||
Upstream: <https://github.com/DY2434/WiFlow-WiFi-Pose-Estimation-with-Spatio-Temporal-Decoupling>
|
||||
pinned at `06899d29` (2026-04-05), Apache-2.0. Dataset: Kaggle `kaka2434/wiflow-dataset`
|
||||
(12.8 GB archive → 15.5 GB extracted; 360,000 windows of 540×20 CSI + 15-keypoint 2D labels).
|
||||
|
||||
Published claims (README "Setting 1"): PCK@20 97.25%, PCK@30 98.63%, PCK@40 99.16%,
|
||||
PCK@50 99.48%, MPJPE 0.007 m, 2.23M params, 0.07 GFLOPs.
|
||||
|
||||
## Measurement (a): their model on their data
|
||||
|
||||
### Artifact verification (MEASURED, 2026-06-10, this repo `eval_repro.py`)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Parameter count | **2,225,042 (2.23M) — matches claim** |
|
||||
| FLOPs (torch profiler, batch 1) | ~0.055 GFLOPs — consistent with 0.07B claim |
|
||||
| CPU latency (Windows box, torch 2.12 CPU) | 13.2 ms/window @ batch 1 (76/s); 2.48 ms/sample @ batch 64 (403/s) |
|
||||
| Checkpoint load | `weights_only=True` (no pickle code execution) |
|
||||
|
||||
### Released checkpoint does NOT reproduce the claims — REFUTED as shipped
|
||||
|
||||
Running the released `best_pose_model.pth` through the released code on the released
|
||||
dataset with the released split procedure (seed-42 file-level 70/15/15; 54,000 test
|
||||
samples) yields:
|
||||
|
||||
| Metric | Published | Measured (shipped checkpoint) |
|
||||
|---|---|---|
|
||||
| PCK@20 | 97.25% | **0.08%** |
|
||||
| PCK@30 | 98.63% | 0.78% |
|
||||
| PCK@40 | 99.16% | 5.53% |
|
||||
| PCK@50 | 99.48% | 15.42% |
|
||||
| MPJPE | 0.007 | **NaN** (dataset contains NaN CSI windows) |
|
||||
|
||||
Raw output: `results/repro_a.json`.
|
||||
|
||||
Diagnostics (on 2,000 NaN-free windows from the first files of the dataset, i.e.
|
||||
mostly would-be *training* data — so this is not a split mismatch):
|
||||
|
||||
- Predictions correlate with targets (Pearson r ≈ 0.76) — the checkpoint is a trained
|
||||
model, but in a **different keypoint normalization/order** than the released data.
|
||||
- Best-case post-hoc global per-axis affine correction: PCK@20 ≈ 20%.
|
||||
- Best-case per-keypoint affine correction (15×2 fitted transforms — generous
|
||||
cheating): PCK@20 ≈ 72%, still far below 97.25%.
|
||||
- Pred↔target keypoint correspondence matrix is degenerate (multiple predicted
|
||||
keypoints best-match the same target joint) — keypoint convention mismatch.
|
||||
|
||||
### Reproducibility defects in the released artifacts
|
||||
|
||||
1. `models/__init__.py` imports `TemporalConvNet`, which `models/tcn.py` does not
|
||||
define — **the published code does not import/run as-is**.
|
||||
2. The released root checkpoint uses pre-rename module names (`att.*`, `final_conv.*`)
|
||||
vs the published code (`attention.*`, `decoder.*`) — same shapes/param count, but
|
||||
confirms the checkpoint predates the published code.
|
||||
3. The second shipped checkpoint (`cross_dataset_test/WiFlow/best_pose_model.pth`) is
|
||||
a **different architecture** (342-channel input = MM-Fi layout, 3 TCN layers,
|
||||
3-channel/3D decoder) — not usable on their own dataset.
|
||||
4. `run.py` ignores `--data_dir` and hardcodes `../preprocessed_csi_data`.
|
||||
5. The released dataset's final 13 files (indices 487–499; 9,072 windows, 2.52%)
|
||||
are corrupted: NaN values plus garbage amplitudes up to 3.4e38 (float32 max) in
|
||||
data that is otherwise [0,1]-normalized. Upstream code has no NaN/inf handling;
|
||||
training as published on this download diverges — the first corrupted batch
|
||||
overflows fp16 autocast and permanently poisons BatchNorm running statistics
|
||||
(GradScaler step-skipping does not protect BN). The authors' training curves
|
||||
show normal convergence, so their local data evidently differed from the
|
||||
Kaggle upload. Window masks: `results/nan_windows_mask.npy`,
|
||||
`results/big_windows_mask.npy`.
|
||||
|
||||
### Reproducing the corruption masks
|
||||
|
||||
The two mask files (9,070 NaN/Inf windows, 9,072 with |amplitude| > 1.5;
|
||||
union 9,072, all in dataset files 487–499) are **committed ground truth**
|
||||
(gitignore-negated, ~352 KB each). They can only be regenerated from a
|
||||
**pristine** Kaggle download: `remote/clean_v2.py` repairs the dataset by
|
||||
zeroing the corrupted windows in place, after which the corruption evidence
|
||||
is gone and a rescan returns all-False. `generate_corruption_masks.py`
|
||||
re-derives them (chunked scan, criteria: any non-finite value OR
|
||||
max |finite| > 1.5 per 540×20 window) and refuses to write all-False masks,
|
||||
which indicate a cleaned copy. Verified 2026-06-11: a regeneration from the
|
||||
local pristine download is bit-identical to the committed masks.
|
||||
|
||||
### Retraining result (MEASURED, 2026-06-10): claims APPROXIMATELY REPRODUCED
|
||||
|
||||
Since the shipped checkpoint is unusable, measurement (a) fell back to retraining
|
||||
with upstream code + defaults (seed 42, batch 64, early-stopped at epoch 41 of 50,
|
||||
best epoch 36, ~75 s/epoch) on ruvultra (RTX 5080). Deviations, all forced and
|
||||
documented: one-line fix for defect (1); torch 2.x+cu128 instead of pinned 2.3.1
|
||||
(Blackwell sm_120 unsupported); the 9,072 corrupted windows (defect 5) zeroed
|
||||
entirely — without this the published pipeline produces NaN from epoch 1 (observed).
|
||||
Scripts mirrored in `remote/`; raw metrics in `results/eval_retrained.json`.
|
||||
|
||||
| Metric | Published | Retrained (full test, 54,000) | Retrained (corruption-free, 52,560) |
|
||||
|---|---|---|---|
|
||||
| PCK@20 | 97.25% | **96.09%** | **96.61%** |
|
||||
| PCK@30 | 98.63% | 97.89% | 98.23% |
|
||||
| PCK@40 | 99.16% | 98.58% | 98.79% |
|
||||
| PCK@50 | 99.48% | 98.99% | 99.11% |
|
||||
| MPJPE | 0.007 | 0.0098 | 0.0094 |
|
||||
|
||||
Within ~0.6–1.2 PCK points of every published figure (single run, corrupted train
|
||||
windows zeroed, different torch/GPU). **Verdict: the accuracy claims are credible
|
||||
and approximately reproducible — but only after repairing the released dataset and
|
||||
code.** Val best: PCK@20 96.99%, MPJPE 0.0086 (epoch 36).
|
||||
|
||||
One more defect found during the run:
|
||||
|
||||
6. `train.py` calls `plot_training_history`, which is not defined anywhere — the
|
||||
built-in post-training test evaluation is unreachable as published (crashes
|
||||
with NameError after training completes).
|
||||
|
||||
## ADR-152 §2.2 citation rule
|
||||
|
||||
Evidence grade for the WiFlow-STD accuracy claims after measurement (a):
|
||||
**MEASURED-EQUIVALENT (96.1–96.6% PCK@20 reproduced by retraining; shipped
|
||||
checkpoint REFUTED; dataset/code require repairs)**. RuView docs may cite
|
||||
"~96% PCK@20 (our reproduction)" — still **not comparable** to our 17-keypoint
|
||||
ESP32 numbers (different hardware, 5 subjects, in-domain random split,
|
||||
15 keypoints).
|
||||
|
||||
## Edge optimization (measured)
|
||||
|
||||
ADR-152 "optimize beyond SOTA" track, 2026-06-10, this Windows box (Windows 11,
|
||||
16 torch threads, torch 2.12.0+cpu, onnxruntime 1.26.0). Subject: the retrained
|
||||
checkpoint `results/retrained_best_pose_model.pth` (2,225,042 fp32 params).
|
||||
Scripts: `quantize_bench.py`, `onnx_bench.py`, `eval_ort_accuracy.py`.
|
||||
Raw numbers: `results/edge_optimization.json`.
|
||||
|
||||
Accuracy is on a **10,000-window seed-42 random subset** of the corruption-free
|
||||
test split (same seed-42 file-level 70/15/15 split as `eval_repro.py`; 54,000
|
||||
test windows, 1,440 corrupted excluded via `results/nan_windows_mask.npy` |
|
||||
`results/big_windows_mask.npy`, leaving 52,560; subset drawn with
|
||||
`np.random.default_rng(42)`). The fp32 subset PCK@20 (96.68%) matches the full
|
||||
clean-test figure (96.61%), so the subset is representative.
|
||||
|
||||
Latency is CPU ms/window, median of repeated runs, 3 interleaved repetitions
|
||||
per variant (medians below; run-to-run spread on this box is large, roughly
|
||||
±20-40% at batch 1 — reps are in the JSON).
|
||||
|
||||
| Variant | Disk size | Batch 1 (ms/win) | Batch 64 (ms/win) | PCK@20 | PCK@50 | MPJPE |
|
||||
|---|---|---|---|---|---|---|
|
||||
| torch fp32 (baseline) | 9.07 MB | 11.0 | 2.27 | 96.68% | 99.15% | 0.00936 |
|
||||
| torch fp16 (`.half()`) | **4.58 MB** | 24.3 | 2.42 | 96.68% | 99.15% | 0.00946 |
|
||||
| torch int8 dynamic | 9.07 MB (unchanged) | 15.6 | 2.06 | 96.68% (identical) | 99.15% | 0.00936 |
|
||||
| ONNX fp32 (onnxruntime) | 8.97 MB | **3.2** | **2.0** | 96.68% | 99.15% | 0.00936 |
|
||||
| ONNX int8 (ORT dynamic, supplementary) | **2.44 MB** | 6.5 | 5.8 | 96.52% | 99.15% | 0.01108 |
|
||||
|
||||
Findings:
|
||||
|
||||
- **torch dynamic INT8 quantizes nothing on this model.** The architecture has
|
||||
**zero `nn.Linear` layers** — it is entirely Conv1d (21) + Conv2d (22) +
|
||||
BatchNorm. `torch.ao.quantization.quantize_dynamic` (requested over
|
||||
`{Linear, Conv1d, Conv2d}`) converted **0 modules / 0.0% of params**: dynamic
|
||||
quantization only has kernels for Linear/RNN-family modules and silently
|
||||
skips convolutions. The "int8" model is bit-identical to fp32 (same outputs,
|
||||
same 9.07 MB). Conv quantization would require static (PTQ) quantization
|
||||
with calibration — out of scope here; the ORT dynamic path below is the
|
||||
honest int8 datapoint.
|
||||
- **fp16 halves size for free accuracy-wise** (PCK@20 −0.005 pt, MPJPE
|
||||
+0.0001) but is *slower* on CPU at batch 1 (~2.2×) — torch CPU fp16 conv
|
||||
kernels are emulated. fp16 is a storage/transport format here, not a CPU
|
||||
runtime win.
|
||||
- **ONNX Runtime is the real batch-1 latency win: ~3.4× faster than torch**
|
||||
(3.2 vs 11.0 ms/window) at identical accuracy (parity 2.4e-7).
|
||||
|
||||
### Verdict on the paper's "~2.2 MB int8" claim
|
||||
|
||||
**Plausible but not free, and unreachable by the obvious PyTorch route.**
|
||||
2,225,042 params × 1 byte ≈ 2.2 MB assumes *every* parameter quantizes.
|
||||
PyTorch dynamic quantization — the one-liner most readers would reach for —
|
||||
yields **9.07 MB (0% quantized)** because the model has no Linear layers.
|
||||
ONNX Runtime dynamic quantization, which does have int8 conv weight support,
|
||||
gets **2.44 MB** (close to the claim; the overhead is BatchNorm params/buffers
|
||||
and quantization scales kept in fp32) at a measurable accuracy cost:
|
||||
PCK@20 96.68 → 96.52% (−0.16 pt) and MPJPE 0.00936 → 0.01108 (+18%), and
|
||||
~2× slower inference than ONNX fp32 (ConvInteger kernels). The paper does not
|
||||
state a method or an int8 accuracy; treat "2.2 MB" as a weight-arithmetic
|
||||
estimate, achievable in practice only via conv-capable quantization toolchains
|
||||
and with a small accuracy penalty.
|
||||
|
||||
### ONNX export status
|
||||
|
||||
**Works.** Exported via the TorchScript exporter (`dynamo=False`), opset 17,
|
||||
with a dynamic batch axis — `results/retrained_fp32_dynamic.onnx` (8.97 MB),
|
||||
verified to run at batch 1/2/64. The axial attention's
|
||||
`view(N*W, C, H)` reshape traced correctly (sizes recorded as graph ops, not
|
||||
baked constants). The dynamo exporter also captures the graph but crashed on
|
||||
this box writing a ✅ to a cp1252 console (cosmetic Windows encoding issue, not
|
||||
a model blocker). Parity vs torch on the stored fixture
|
||||
(`results/parity_fixture.npz`, batch 2, seed 42): **max abs diff 2.4e-7 —
|
||||
PASS** (< 1e-4). ORT-quantized int8 model: `results/retrained_int8_ort_dynamic.onnx`.
|
||||
|
||||
### Static PTQ (calibrated) — follow-up
|
||||
|
||||
Follow-up to the dynamic-int8 row above (2026-06-10, same box, onnxruntime
|
||||
1.26.0): ONNX Runtime **static** post-training quantization
|
||||
(`quantize_static`, QDQ format, per-channel int8 weights + int8 activations)
|
||||
of the same fp32 export, calibrated on **corruption-free TRAINING-split
|
||||
windows only** (seed-42 file-level split, same masks; 1,000 windows for
|
||||
MinMax, 512 for the histogram calibrators; never test windows). Scopes:
|
||||
"conv-only" (`op_types_to_quantize=["Conv"]` — the attention path exports as
|
||||
Einsum/Softmax, which ORT never quantizes anyway, so "all-ops" additionally
|
||||
quantizes the elementwise Mul/Sigmoid/Add/AveragePool glue). Accuracy on the
|
||||
identical 10k-window seed-42 corruption-free test subset; latency median of
|
||||
3 interleaved reps (fp32/dynamic re-benched in-session as references).
|
||||
Script: `static_ptq_bench.py`; raw: `results/edge_optimization.json`
|
||||
(`onnx_static_ptq`).
|
||||
|
||||
| Variant | Disk size | Batch 1 (ms/win) | Batch 64 (ms/win) | PCK@20 | PCK@50 | MPJPE |
|
||||
|---|---|---|---|---|---|---|
|
||||
| ONNX fp32 (reference) | 8.97 MB | 2.5 | 1.9 | 96.68% | 99.15% | 0.00936 |
|
||||
| ORT dynamic int8 (baseline) | **2.44 MB** | 5.7 | 4.6 | 96.52% | 99.15% | 0.01108 |
|
||||
| static QDQ **Percentile(99.99) conv-only** | 2.53 MB | 5.3 | 4.7 | 96.61% | 99.16% | **0.01031** |
|
||||
| static QDQ MinMax conv-only | 2.53 MB | 5.2 | 3.3 | **96.63%** | 99.19% | 0.01084 |
|
||||
| static QDQ Entropy conv-only | 2.53 MB | 5.2 | 3.1 | 96.60% | 99.19% | 0.01078 |
|
||||
| static QDQ MinMax all-ops | 2.60 MB | 6.5 | 3.9 | 95.45% | 99.14% | 0.01486 |
|
||||
| static QDQ Entropy all-ops | 2.60 MB | 5.7 | 4.1 | 95.30% | 99.13% | 0.01510 |
|
||||
| static QDQ Percentile all-ops | 2.60 MB | 5.3 | 4.3 | 96.39% | 99.17% | 0.01218 |
|
||||
|
||||
**Verdict: static PTQ (conv-only) is the new best int8 point on accuracy —
|
||||
but only modestly, and it does not fix int8's latency penalty.**
|
||||
|
||||
- **Accuracy: beats dynamic.** All three conv-only calibrations land at
|
||||
PCK@20 96.60–96.63% (vs dynamic 96.52%, fp32 96.68% — recovers ~⅔ of the
|
||||
dynamic gap) and MPJPE 0.0103–0.0108 (vs dynamic 0.01108). Best MPJPE:
|
||||
Percentile conv-only, +10% over fp32 instead of dynamic's +18%.
|
||||
- **Size: slightly worse.** 2.53 MB vs 2.44 MB (+3.6%) — QDQ nodes and
|
||||
per-channel scales cost a little; BatchNorm stays fp32 in both (the 12 BNs
|
||||
follow Slice/Einsum/Reshape, never Conv, so they cannot be folded).
|
||||
- **Latency: a wash vs dynamic, still ~2× slower than ONNX fp32 at batch 1.**
|
||||
Batch-1 medians 5.2–5.3 vs dynamic 5.7 ms/win in-session — within this
|
||||
box's ±20–40% noise. Batch 64 leans static (3.1–3.3 for MinMax/Entropy
|
||||
conv-only vs 4.6), same caveat.
|
||||
- **All-ops QDQ is strictly worse**: up to −1.4 pt PCK@20 and +60% MPJPE for
|
||||
zero size/latency benefit — int8 activations through the elementwise glue
|
||||
around the attention blocks is where the damage is. Conv-only is the right
|
||||
scope.
|
||||
- Negative result worth recording: **Entropy calibration is a no-op here** —
|
||||
on an identical calibration set it selects full-range thresholds
|
||||
bit-identical to MinMax (all 247 scales equal; verified on a 64-window
|
||||
smoke set). Also, ORT 1.26's `CalibMaxIntermediateOutputs` raises a
|
||||
spurious "No data is collected" when the batch count divides the chunk
|
||||
size (worked around in the script).
|
||||
|
||||
Deployment guidance: need speed → ONNX fp32 (3.2 ms b1). Need int8 weights
|
||||
for size → static QDQ conv-only (Percentile or MinMax,
|
||||
`results/retrained_int8_static_percentile_conv.onnx`), which strictly
|
||||
dominates dynamic int8 on accuracy at ~equal latency and +0.09 MB.
|
||||
|
||||
## Efficiency sweep (MEASURED, overnight 2026-06-10/11)
|
||||
|
||||
ADR-152 beyond-SOTA track: compact purpose-built variants of the WiFlow-STD
|
||||
architecture, trained from scratch on the same cleaned dataset, identical
|
||||
seed-42 file-level split, loss and protocol as the measurement-(a) reference
|
||||
(fp32, batch 64, ≤50 epochs, patience 5; RTX 5080, ~22–29 min/variant).
|
||||
Variant transforms are pure channel/group/stride scalings of an
|
||||
architecture-exact parameterized model (validated: reproduces 2,225,042 params
|
||||
at the reference config). Scripts: `remote/sweep/`; raw:
|
||||
`results/efficiency_sweep.jsonl`; checkpoints `results/{half,quarter,tiny}_best.pth`
|
||||
(gitignored).
|
||||
|
||||
| Variant | Params | vs 2.23M | Clean-test PCK@20 | PCK@50 | MPJPE | Best epoch |
|
||||
|---|---|---|---|---|---|---|
|
||||
| full (reference, meas. a) | 2,225,042 | 1× | 96.61% | 99.11% | 0.0094 | 36 |
|
||||
| **half** | **843,834** | **0.38×** | **96.62%** | **99.47%** | **0.00898** | 23 |
|
||||
| quarter | 338,600 | 0.15× | 96.05% | 99.43% | 0.00928 | 50 |
|
||||
| tiny | 56,290 | 0.025× | 94.11% | 99.36% | 0.0125 | 47 |
|
||||
|
||||
Findings:
|
||||
|
||||
- **The half model (843k params) strictly dominates the full reference** on
|
||||
this dataset — equal PCK@20, better PCK@50 and MPJPE, converges in fewer
|
||||
epochs. The published 2.23M architecture is over-parameterized for its own
|
||||
benchmark.
|
||||
- **tiny (56k params, 1/39.5) holds 94.11% PCK@20** — a ~220 KB fp32 /
|
||||
~60 KB int8-class model in reach of severely constrained edge targets,
|
||||
at −2.5 pt from the full reference.
|
||||
- Caveats: in-domain (5-subject random-file split) like every number on this
|
||||
dataset; single run per variant; corruption-free test subset (52,560).
|
||||
Cross-domain behavior of compact variants is untested — ADR-150's evidence
|
||||
says capacity *hurts* cross-subject, so the compact end may generalize no
|
||||
worse, but that is a hypothesis, not a measurement.
|
||||
|
||||
### Compact-variant edge artifacts (MEASURED, 2026-06-11)
|
||||
|
||||
Edge pipeline for the **tiny** checkpoint (56,290 params), same machinery and
|
||||
protocol as the full-model edge rows above (this Windows box, torch
|
||||
2.12.0+cpu, onnxruntime 1.26.0; dynamic-batch opset-17 TorchScript export;
|
||||
static QDQ **Percentile(99.99) conv-only** int8 calibrated on **512**
|
||||
corruption-free TRAIN-split windows; accuracy on the identical 10k-window
|
||||
seed-42 clean test subset; latency = median ms/window over 3 interleaved
|
||||
reps, with the full-model fp32/int8 sessions interleaved as same-session
|
||||
references). Script: `tiny_edge_bench.py`; raw:
|
||||
`results/edge_optimization.json` (`tiny_variant`). Torch-vs-ORT parity on the
|
||||
stored fixture input: **max abs diff 1.5e-7 — PASS** (< 1e-4). The tiny fp32
|
||||
subset PCK@20 (94.11%) matches the full clean-test sweep figure (94.11%)
|
||||
exactly, so the subset remains representative.
|
||||
|
||||
Two forced deviations, both recorded in the JSON:
|
||||
|
||||
1. **Adaptive-pool export rewrite.** tiny's derived stride schedule
|
||||
`[2,1,1,1]` leaves feature width 16, and the TorchScript exporter rejects
|
||||
`AdaptiveAvgPool2d((15,1))` when 15 is not a factor of the input height
|
||||
(the full model never hit this — its width was exactly 15). Since the
|
||||
pool over a fixed-size map is a fixed linear operator, the export wrapper
|
||||
replaces it with `mean(-1)` (W axis, a factor) + a constant averaging
|
||||
matmul using PyTorch's exact bin rule; the parity check (vs the original
|
||||
torch model with the real pool) proves exactness.
|
||||
2. **Calibration count 512, not "~500"**: ORT 1.26's histogram collector
|
||||
`np.asarray()`'s the per-batch maxima, so the calibration count must be a
|
||||
multiple of the 64-window calibration batch or the ragged last batch
|
||||
crashes it (the earlier static-PTQ run dodged this by using exactly 512).
|
||||
|
||||
| Variant | Disk size | Batch 1 (ms/win) | Batch 64 (ms/win) | PCK@20 | PCK@50 | MPJPE |
|
||||
|---|---|---|---|---|---|---|
|
||||
| full ONNX fp32 (same-session ref) | 8.97 MB | 2.27 | 1.42 | 96.68% | 99.15% | 0.00936 |
|
||||
| full static QDQ Percentile conv-only (same-session ref) | 2.53 MB | 5.53 | 3.82 | 96.61% | 99.16% | 0.01031 |
|
||||
| **tiny ONNX fp32** | **0.295 MB** | **0.66** | **0.24** | **94.11%** | 99.37% | 0.01253 |
|
||||
| tiny static QDQ Percentile conv-only | 0.248 MB | 0.85 | 1.03 | 92.68% | 99.33% | 0.01491 |
|
||||
|
||||
(tiny torch `.pth` checkpoint for reference: 0.34 MB on disk; 56,290 fp32
|
||||
params ≈ 225 KB of weights.)
|
||||
|
||||
Findings:
|
||||
|
||||
- **The smallest deployable WiFlow-class model is the tiny ONNX fp32
|
||||
artifact: ~295 KB on disk, 0.66 ms/window batch-1 CPU (~1,500 windows/s),
|
||||
94.1% PCK@20** — 30× smaller and ~3.4× faster (in-session) than the full
|
||||
ONNX fp32 model for −2.6 pt PCK@20.
|
||||
- **int8 is a bad trade at this scale.** Static QDQ conv-only — the recipe
|
||||
that cost the full model only 0.07 pt — costs tiny **−1.43 pt** PCK@20
|
||||
(94.11 → 92.68%) and +19% MPJPE, saves only 47 KB (−16%; QDQ scales and
|
||||
the fp32 BN/attention glue are proportionally larger in a small graph),
|
||||
and is *slower* than tiny fp32 (0.85 vs 0.66 ms b1; 1.03 vs 0.24 ms b64 —
|
||||
QDQ kernel overhead dominates when the convs are this small). A 56k-param
|
||||
model has little redundancy left to absorb weight+activation rounding.
|
||||
- Deployment guidance, compact edition: ship tiny as **ONNX fp32** — at
|
||||
295 KB the int8 size saving solves no real constraint and costs accuracy
|
||||
and speed. If ~250 KB vs ~295 KB ever matters, weight-only quantization
|
||||
would be the thing to try next, not QDQ.
|
||||
|
||||
## Measurement (b): BLOCKED-ON-DATA (attempted 2026-06-10)
|
||||
|
||||
The fine-tune-on-ESP32 measurement stopped at dataset characterization, per the
|
||||
pre-registered stop rule (<2,000 paired windows). Findings (MEASURED):
|
||||
|
||||
- **Only one trainable paired dataset exists**: `ruvultra:~/work/cog-pose-train/paired.jsonl`
|
||||
— 1,077 windows (one subject, one room, one 29.9-min session, single node;
|
||||
CSI [56, 20]; 17 COCO keypoints, MediaPipe confidence mean 0.44 — only 264
|
||||
windows pass ADR-079's own conf>0.5 training filter). Prior measured attempts
|
||||
on this exact set: 0–3% torso-PCK@20 (temporal splits, three independent
|
||||
pipelines). Fine-tuning a 2.23M-param model on ~860 train windows would
|
||||
measure memorization, not transfer.
|
||||
- **The April session behind the old "92.9% PCK@20" claim is lost** (345
|
||||
samples, 35 subcarriers; raw CSI gone from ruvzen/ruvultra/cognitum-v0; only
|
||||
a 69-sample predictions+GT holdout survives at `models/wiflow-real/eval-holdout.jsonl`).
|
||||
- **Forensic recheck of that holdout RETRACTS the 92.9% figure**: the trainer's
|
||||
`pck()` used an absolute 0.2 image-unit threshold (not torso-normalized) and
|
||||
the model output a **constant pose** (pred std 0.0000 across 69 near-static
|
||||
frames; a mean predictor scores 100% under the same protocol). The
|
||||
torso-normalized PCK@20 on the same holdout is 19.1%. This corroborates the
|
||||
2026-05-11 audit retraction (CHANGELOG, PR #535); stale doc citations were
|
||||
removed 2026-06-10 (user-guide, readme-details, ADR-152 §2.1.3). The §2.2
|
||||
no-citation rule now applies to ADR-079 accuracy claims.
|
||||
|
||||
Unblock criteria: a paired collection session of ≥2k windows (≈35+ min at the
|
||||
observed stride; multi-pose, conf>0.5, ideally with the §2.1.3 two-checkerboard
|
||||
calibration), plus a re-baselined our-pipeline number under torso-PCK@20 on the
|
||||
same split. WiFlow-STD assets stand ready on ruvultra (`~/wiflow-std-bench/`).
|
||||
Also worth investigating: ADR-079's protocol predicts ~9k windows per 30 min;
|
||||
the May session under-delivered ~8× (aligner drop rate?).
|
||||
|
||||
## Measurement (b) (MEASURED 2026-06-10/11)
|
||||
|
||||
The data baseline unblocked: the 2026-06-10 22:10–22:40 collection session produced
|
||||
**2,046 paired windows** (`ruvultra:~/wiflow-std-bench/paired-20260610.jsonl`; ONE
|
||||
subject, ONE room, ONE ESP32 node, varied poses: walk/raise/squat/kick/wave/turn/
|
||||
jump/sit; aligner `scripts/align-ground-truth.js`, non-overlapping 20-frame windows
|
||||
~0.42 s; 17 COCO keypoints in normalized [0,1] camera coords; MediaPipe confidence
|
||||
mean 0.802, min 0.692 — all windows pass the conf>0.5 filter). The −4 h timestamp
|
||||
bug and the empty-frame confidence-dilution aligner findings are recorded
|
||||
separately; results only here. Trained on ruvultra (RTX 5080, torch 2.11+cu128,
|
||||
fp32, batch 32, GPU shared with the efficiency sweep). Scripts mirrored in
|
||||
`remote/measb/`; raw metrics + full training curves in `results/measurement_b.json`.
|
||||
|
||||
### Two new aligner/dataset findings (forced deviations, MEASURED)
|
||||
|
||||
1. **`csi_shape` is heterogeneous, not [70, 20]**: 1,347× [70,20], 284× [134,20],
|
||||
243× [26,20], 130× [12,20], 42× [20,20]. The ESP32 stream emits mixed frame
|
||||
types and `extractCsiMatrix` stamps each window's subcarrier count from
|
||||
`window[0].subcarriers`, zero-padding/truncating the other frames — even
|
||||
native-70 windows contain ~20.4% internally zero-padded short frames
|
||||
(subcarriers 40–69 all-zero). Handling: the primary suite ("all 2,046")
|
||||
linearly resamples every frame's subcarrier axis to 70 bins (identity for
|
||||
native-70 frames) so the pre-registered n and split sizes hold; a secondary
|
||||
suite restricts to the 1,347 native [70,20] windows as a homogeneity check.
|
||||
2. **Aligner layout bug**: `extractCsiMatrix` fills `matrix[f * nSc + s]`
|
||||
(frame-major) but declares `shape: [nSc, nFrames]` — the stored shape label is
|
||||
transposed relative to the data. Confirmed by coherent per-frame zero-tails;
|
||||
corrected on load (`reshape(nFrames, nSc).T`).
|
||||
|
||||
### Protocol (pre-registered, followed)
|
||||
|
||||
Temporal split, no shuffling across time: first 70% train (1,432), next 15% val
|
||||
(307), last 15% test (307); seed 42 elsewhere. Model: learned 1×1 Conv1d 70→540
|
||||
adapter prepended to the upstream WiFlow-STD trunk; K=17 via the parameter-free
|
||||
adaptive pool (`AdaptiveAvgPool2d((17,1))` — pretrained weights load strict for
|
||||
any K). CSI normalized by the TRAIN-split p99 amplitude (129.7 all / 130.9
|
||||
native-70), clipped to [0,1]. Three runs, ≤60 epochs, early-stop patience 8 on
|
||||
val MPJPE, AdamW (adapter lr 1e-4; pretrained trunk lr 1e-5, 10× lower; scratch
|
||||
all 1e-4), fp32. Pretrained init = the measurement-(a) **retrained** checkpoint
|
||||
(`upstream/test/best_pose_model.pth`, ~96% PCK@20 on WiFlow data; the
|
||||
`att.`/`final_conv.` key remap from `eval_repro.py` applied defensively — a no-op,
|
||||
that checkpoint already uses post-rename keys). Frozen-trunk run: trunk
|
||||
`requires_grad=False` **and** held in `.eval()` so BatchNorm running stats cannot
|
||||
drift — a pure transfer probe; only the 70→540 adapter (38,340 params) trains.
|
||||
|
||||
PCK is torso-normalized with **torso = ‖l_shoulder(5) − l_hip(11)‖** (upstream
|
||||
`calculate_pck` math — per-frame norm clamped at 0.01, mean over keypoints ×
|
||||
frames — but upstream's `NECK_IDX/PELVIS_IDX = 2, 12` is a 15-keypoint
|
||||
convention; on 17-kp COCO those indices are right_eye/right_hip, so the indices
|
||||
were replaced, not the math). MPJPE is in normalized image units (not meters).
|
||||
|
||||
### Results — primary suite, all 2,046 windows (test = last 307)
|
||||
|
||||
| Run | PCK@10 | PCK@20 | PCK@30 | PCK@40 | PCK@50 | MPJPE | pred std | best ep |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| **mean-pose baseline** (honesty bar) | **73.1%** | **95.9%** | **98.7%** | 99.3% | 99.3% | **0.0148** | 0 (by constr.) | — |
|
||||
| (i) pretrained-init, full fine-tune | 26.0% | 65.0% | 88.0% | 96.4% | 98.9% | 0.0313 | 0.0113 | 58/60 |
|
||||
| (ii) scratch | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.2554 | 0.0002 | 4 (stop @13) |
|
||||
| (iii) frozen-trunk (adapter only) | 0.0% | 0.0% | 0.2% | 3.2% | 14.4% | 0.1260 | 0.0073 | 59/60 |
|
||||
|
||||
Secondary suite (native [70,20] windows only, n=1,347, test=202) reproduces the
|
||||
same ordering: mean-baseline 96.0% / pretrained 67.1% / scratch 0.0% /
|
||||
frozen-trunk 0.0% PCK@20 (MPJPE 0.0153 / 0.0318 / 0.2236 / 0.1343) — the
|
||||
subcarrier-resampling choice does not change any conclusion.
|
||||
|
||||
### Interpretation
|
||||
|
||||
- **Did pretraining-transfer happen? Partially — as optimization transfer, not
|
||||
feature transfer, and not past the honesty bar.**
|
||||
- *Pretrained vs scratch*: dramatic (65.0% vs 0.0% PCK@20). The pretrained init
|
||||
is the only configuration that trains at all under the pre-registered budget.
|
||||
- *Frozen-trunk*: near-zero (0.0% PCK@20, 14.4% @50). WiFlow-STD's frozen
|
||||
features do **not** transfer to our ESP32 domain through a linear subcarrier
|
||||
adapter — the pretrained benefit is a well-conditioned initialization (incl.
|
||||
calibrated BN/output scales), not reusable CSI→pose features.
|
||||
- *Everything vs mean-pose baseline*: **no run beats it.** A constant
|
||||
train-mean pose scores 95.9% torso-PCK@20 / 0.0148 MPJPE on this test split,
|
||||
because a single subject in one camera frame barely moves in normalized
|
||||
coordinates. The fine-tuned model is a real, non-constant model
|
||||
(pred std 0.0113 > 0 — passes the constant-pose detector that retracted the
|
||||
old 92.9% figure) but its deviations from the mean hurt: it fits train-period
|
||||
temporal dynamics that do not generalize across the temporal split.
|
||||
- **Verdict for ADR-152 §2.2(b): fine-tuning WiFlow-STD on this dataset does not
|
||||
demonstrate CSI→pose signal beyond the mean pose.** Until a model beats the
|
||||
mean-pose baseline on a temporal split, no PCK number from this line may be
|
||||
cited as pose-estimation capability.
|
||||
|
||||
### Caveats (honest, pre-registered)
|
||||
|
||||
- Single subject, single room, single session (30 min), single ESP32 node —
|
||||
in-domain temporal split only; nothing here speaks to cross-room or
|
||||
cross-subject generalization.
|
||||
- 2k windows vs the 360k-window WiFlow-STD corpus — **NOT comparable** to the
|
||||
~96% in-domain measurement-(a) number, and the published 97.25% even less so.
|
||||
- The scratch run's total collapse (it cannot even reach the mean pose; its
|
||||
output BatchNorm/SiLU head must learn output scale from random init at lr 1e-4)
|
||||
is an optimization outcome under the fixed budget, not proof the architecture
|
||||
cannot learn from scratch — the pretrained-vs-scratch gap partially reflects
|
||||
this conditioning advantage.
|
||||
- Mixed-subcarrier frames (finding 1) mean even the "clean" windows carry ~20%
|
||||
zero-padded frames; collection-side frame-type filtering should precede the
|
||||
next session.
|
||||
- Mean-baseline PCK is inflated by low pose variance relative to torso size
|
||||
(~0.2–0.3 image units); PCK@10 (73.1%) shows the same ceiling effect at a
|
||||
stricter threshold — the bar is the bar, but a livelier dataset would lower it.
|
||||
|
||||
## Pending
|
||||
|
||||
- (b) fine-tune on our ESP32 17-keypoint eval set — **MEASURED 2026-06-10/11**,
|
||||
see above: no run beats the mean-pose baseline; pretraining transfers as
|
||||
optimization aid only.
|
||||
- (c) our internal WiFlow on their dataset (15-keypoint subset mapping) — also
|
||||
affected: there is currently no validated internal pose model to compare
|
||||
(the 92.9% artifact is retracted; the MM-Fi SOTA models in ADR-150 §3 are a
|
||||
different input domain).
|
||||
@@ -0,0 +1,200 @@
|
||||
"""Shared infrastructure for the LOCAL wiflow-std benchmark scripts (ADR-152).
|
||||
|
||||
This module is the single canonical implementation of the helpers that were
|
||||
previously copy-pasted across eval_repro.py / quantize_bench.py /
|
||||
onnx_bench.py / eval_ort_accuracy.py / export_to_safetensors.py:
|
||||
|
||||
- ``import_upstream()`` -- sys.path setup + the models-package stub that
|
||||
works around the upstream import bug, plus the >1GB np.load mmap patch
|
||||
- ``install_np_load_mmap_patch()`` -- the mmap patch on its own
|
||||
- ``remap_legacy_keys()`` / ``load_remapped_state()`` -- checkpoint
|
||||
key remap for the pre-rename released checkpoint
|
||||
- ``load_wiflow_model()`` -- WiFlowPoseModel from a checkpoint, eval mode
|
||||
- ``set_seed()`` -- mirrors upstream run.py seeding exactly
|
||||
- ``evaluate()`` -- THE canonical batch-weighted PCK/MPJPE evaluation loop
|
||||
(thresholds 0.1-0.5, upstream utils/metrics.py math); accepts either a
|
||||
torch nn.Module or an onnxruntime InferenceSession
|
||||
|
||||
The scripts under remote/ deploy to ruvultra as standalone single files and
|
||||
therefore intentionally inline private copies of these helpers; when editing
|
||||
them, treat this module as the reference implementation and keep the copies
|
||||
in sync.
|
||||
"""
|
||||
|
||||
import os
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
import types
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
UPSTREAM = os.path.join(HERE, "upstream")
|
||||
RESULTS = os.path.join(HERE, "results")
|
||||
|
||||
DEFAULT_THRESHOLDS = (0.1, 0.2, 0.3, 0.4, 0.5)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# >1GB np.load mmap patch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# csi_windows.npy is ~13 GB; mmap large arrays instead of loading into RAM
|
||||
# (loading it eagerly needs ~15 GB).
|
||||
_np_load = np.load
|
||||
|
||||
|
||||
def _np_load_mmap(path, *a, **kw):
|
||||
if (isinstance(path, str) and path.endswith(".npy")
|
||||
and os.path.getsize(path) > 1 << 30 and "mmap_mode" not in kw):
|
||||
kw["mmap_mode"] = "r"
|
||||
return _np_load(path, *a, **kw)
|
||||
|
||||
|
||||
def install_np_load_mmap_patch():
|
||||
"""Globally patch np.load so .npy files >1GB are mmap'd read-only.
|
||||
|
||||
Idempotent. Patching the numpy module attribute is equivalent to the
|
||||
historical ``upstream_dataset.np.load = _np_load_mmap`` (dataset.np IS
|
||||
the numpy module), but works regardless of import order.
|
||||
"""
|
||||
np.load = _np_load_mmap
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# upstream import shim
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def import_upstream(mmap_patch=True):
|
||||
"""Make the upstream WiFlow-STD clone importable; returns its path.
|
||||
|
||||
Upstream bug: models/__init__.py imports TemporalConvNet, which
|
||||
models/tcn.py does not define -- the package fails to import as
|
||||
published. Register a stub package so the broken __init__ never
|
||||
executes; submodules (models.pose_model etc.) still resolve via
|
||||
__path__. Idempotent.
|
||||
"""
|
||||
if UPSTREAM not in sys.path:
|
||||
sys.path.insert(0, UPSTREAM)
|
||||
if "models" not in sys.modules:
|
||||
_models_pkg = types.ModuleType("models")
|
||||
_models_pkg.__path__ = [os.path.join(UPSTREAM, "models")]
|
||||
sys.modules["models"] = _models_pkg
|
||||
if mmap_patch:
|
||||
install_np_load_mmap_patch()
|
||||
return UPSTREAM
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# checkpoint loading
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# The released checkpoint predates the published code: modules were renamed
|
||||
# att -> attention, final_conv -> decoder (param count identical, 2.23M).
|
||||
LEGACY_RENAMES = {"att.": "attention.", "final_conv.": "decoder."}
|
||||
|
||||
|
||||
def remap_legacy_keys(state):
|
||||
"""Remap pre-rename state_dict keys; no-op for already-new-style keys."""
|
||||
return {next((new + k[len(old):] for old, new in LEGACY_RENAMES.items()
|
||||
if k.startswith(old)), k): v
|
||||
for k, v in state.items()}
|
||||
|
||||
|
||||
def load_remapped_state(path, map_location="cpu"):
|
||||
"""torch.load (weights_only) + legacy key remap."""
|
||||
state = torch.load(path, map_location=map_location, weights_only=True)
|
||||
return remap_legacy_keys(state)
|
||||
|
||||
|
||||
def load_wiflow_model(checkpoint, map_location="cpu", dropout=0.5):
|
||||
"""Full-size WiFlowPoseModel from a checkpoint, strict load, eval mode."""
|
||||
import_upstream()
|
||||
from models.pose_model import WiFlowPoseModel
|
||||
model = WiFlowPoseModel(dropout=dropout)
|
||||
model.load_state_dict(load_remapped_state(checkpoint, map_location),
|
||||
strict=True)
|
||||
model.eval()
|
||||
return model
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# seeding
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def set_seed(seed=42):
|
||||
# mirror upstream run.py exactly
|
||||
random.seed(seed)
|
||||
np.random.seed(seed)
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed(seed)
|
||||
torch.cuda.manual_seed_all(seed)
|
||||
torch.backends.cudnn.deterministic = True
|
||||
torch.backends.cudnn.benchmark = False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# THE canonical evaluation loop
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def evaluate(model, loader, device=None, dtype=None, label="",
|
||||
thresholds=DEFAULT_THRESHOLDS, progress_every=50):
|
||||
"""Batch-weighted PCK/MPJPE over a DataLoader (upstream metrics math).
|
||||
|
||||
``model`` may be a torch nn.Module (optionally evaluated on ``device``
|
||||
with inputs cast to ``dtype``) or an onnxruntime InferenceSession.
|
||||
Per-threshold PCK values are independent in upstream calculate_pck, so
|
||||
evaluating a superset of thresholds never changes any individual value.
|
||||
|
||||
Returns {"samples", "mpjpe", "pck@10".."pck@50", "wall_seconds"}.
|
||||
"""
|
||||
import_upstream()
|
||||
from utils.metrics import calculate_mpjpe, calculate_pck
|
||||
|
||||
is_ort = hasattr(model, "get_inputs") # onnxruntime InferenceSession
|
||||
if is_ort:
|
||||
inp = model.get_inputs()[0].name
|
||||
|
||||
def forward(bx):
|
||||
return torch.from_numpy(model.run(None, {inp: bx.numpy()})[0])
|
||||
else:
|
||||
model.eval()
|
||||
|
||||
def forward(bx):
|
||||
if device is not None:
|
||||
bx = bx.to(device)
|
||||
if dtype is not None:
|
||||
bx = bx.to(dtype)
|
||||
return model(bx).float()
|
||||
|
||||
thresholds = list(thresholds)
|
||||
totals = {t: 0.0 for t in thresholds}
|
||||
total_mpe, n = 0.0, 0
|
||||
t0 = time.time()
|
||||
with torch.no_grad():
|
||||
for batch_idx, (bx, by) in enumerate(loader):
|
||||
out = forward(bx)
|
||||
if device is not None and not is_ort:
|
||||
by = by.to(device)
|
||||
mpe = calculate_mpjpe(out, by)
|
||||
pck = calculate_pck(out, by, thresholds=thresholds)
|
||||
bs = by.size(0)
|
||||
total_mpe += mpe * bs
|
||||
for t in totals:
|
||||
totals[t] += pck[t] * bs
|
||||
n += bs
|
||||
if batch_idx % progress_every == 0:
|
||||
tag = f"[{label}] " if label else ""
|
||||
pck20 = totals.get(0.2)
|
||||
pck20_str = f"pck20={pck20 / n:.4f} " if pck20 is not None else ""
|
||||
print(f" {tag}batch {batch_idx}: n={n} {pck20_str}"
|
||||
f"mpjpe={total_mpe / n:.4f} ({time.time() - t0:.0f}s)",
|
||||
flush=True)
|
||||
return {
|
||||
"samples": n,
|
||||
"mpjpe": total_mpe / n,
|
||||
**{f"pck@{int(t * 100)}": totals[t] / n for t in thresholds},
|
||||
"wall_seconds": time.time() - t0,
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
"""ADR-152 edge optimization: accuracy of the ONNX fp32 and ORT-dynamic-int8
|
||||
models on the same corruption-free 10k test subset used by quantize_bench.py.
|
||||
|
||||
The torch dynamic-int8 path quantizes nothing (no nn.Linear in the model), so
|
||||
the only real int8 datapoint for the paper's "~2.2 MB int8" claim is the
|
||||
onnxruntime dynamically quantized model -- this script measures what that
|
||||
quantization costs in PCK/MPJPE.
|
||||
|
||||
Usage:
|
||||
.venv/Scripts/python.exe eval_ort_accuracy.py \
|
||||
--data-dir <preprocessed_csi_data> [--subset 10000]
|
||||
|
||||
Writes/merges into results/edge_optimization.json under key "onnx_accuracy".
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
from _bench_common import RESULTS, evaluate # noqa: E402
|
||||
from quantize_bench import build_test_subset # noqa: E402 (sets up upstream imports)
|
||||
|
||||
|
||||
def evaluate_ort(sess, loader, label):
|
||||
"""ORT-session evaluation via the canonical _bench_common.evaluate loop."""
|
||||
return evaluate(sess, loader, label=label)
|
||||
|
||||
|
||||
def main():
|
||||
import onnxruntime as ort
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--data-dir", default=os.path.join(
|
||||
os.path.expanduser("~"), ".cache", "kagglehub", "datasets", "kaka2434",
|
||||
"wiflow-dataset", "versions", "1", "preprocessed_csi_data"))
|
||||
parser.add_argument("--subset", type=int, default=10000)
|
||||
parser.add_argument("--out", default=os.path.join(RESULTS, "edge_optimization.json"))
|
||||
args = parser.parse_args()
|
||||
|
||||
loader, _n_clean = build_test_subset(args.data_dir, args.subset)
|
||||
results = {}
|
||||
for label, fname in (("onnx_fp32", "retrained_fp32_dynamic.onnx"),
|
||||
("onnx_int8_ort_dynamic", "retrained_int8_ort_dynamic.onnx")):
|
||||
path = os.path.join(RESULTS, fname)
|
||||
if not os.path.exists(path):
|
||||
results[label] = {"error": f"{fname} not found; run onnx_bench.py first"}
|
||||
continue
|
||||
sess = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
|
||||
print(f"=== accuracy: {label} ({fname}) ===")
|
||||
results[label] = evaluate_ort(sess, loader, label)
|
||||
print(json.dumps(results[label], indent=2))
|
||||
|
||||
merged = {}
|
||||
if os.path.exists(args.out):
|
||||
with open(args.out) as f:
|
||||
merged = json.load(f)
|
||||
merged["onnx_accuracy"] = results
|
||||
with open(args.out, "w") as f:
|
||||
json.dump(merged, f, indent=2)
|
||||
print(f"wrote {args.out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,102 @@
|
||||
"""ADR-152 §2.2 measurement (a): reproduce WiFlow-STD (DY2434) published test metrics.
|
||||
|
||||
Runs the released pretrained checkpoint (upstream/best_pose_model.pth) against the
|
||||
released Kaggle dataset (kaka2434/wiflow-dataset) using the upstream code path:
|
||||
identical dataset class, identical file-level 70/15/15 split at seed 42, identical
|
||||
PCK/MPJPE implementations (utils/metrics.py).
|
||||
|
||||
Published claims (README, "Setting 1 random split"):
|
||||
PCK@20 97.25% | PCK@30 98.63% | PCK@40 99.16% | PCK@50 99.48% | MPJPE 0.007 m
|
||||
|
||||
Usage:
|
||||
.venv/Scripts/python.exe eval_repro.py --data-dir <dir containing csi_windows.npy>
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
import torch
|
||||
from torch.utils.data import DataLoader
|
||||
|
||||
from _bench_common import (UPSTREAM, evaluate, import_upstream,
|
||||
load_remapped_state, set_seed)
|
||||
|
||||
import_upstream() # sys.path + models stub + >1GB np.load mmap patch
|
||||
|
||||
from dataset import PreprocessedCSIKeypointsDataset, create_preprocessed_train_val_test_loaders # noqa: E402
|
||||
from models.pose_model import WiFlowPoseModel # noqa: E402
|
||||
|
||||
|
||||
def find_data_dir(root):
|
||||
for dirpath, _dirnames, filenames in os.walk(root):
|
||||
if "csi_windows.npy" in filenames:
|
||||
return dirpath
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--data-dir", required=True,
|
||||
help="Directory containing csi_windows.npy (searched recursively)")
|
||||
parser.add_argument("--checkpoint", default=os.path.join(UPSTREAM, "best_pose_model.pth"))
|
||||
parser.add_argument("--batch-size", type=int, default=64)
|
||||
parser.add_argument("--out", default=os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
||||
"results", "repro_a.json"))
|
||||
args = parser.parse_args()
|
||||
|
||||
data_dir = args.data_dir
|
||||
if not os.path.exists(os.path.join(data_dir, "csi_windows.npy")):
|
||||
located = find_data_dir(data_dir)
|
||||
if located is None:
|
||||
sys.exit(f"csi_windows.npy not found under {data_dir}")
|
||||
data_dir = located
|
||||
print(f"data dir: {data_dir}")
|
||||
|
||||
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||
print(f"device: {device}, torch {torch.__version__}")
|
||||
|
||||
set_seed(42)
|
||||
|
||||
dataset = PreprocessedCSIKeypointsDataset(
|
||||
data_dir=data_dir, keypoint_scale=1000.0, enable_temporal_clean=True)
|
||||
|
||||
# split must match upstream: file-level shuffle at random_seed=42, 70/15/15
|
||||
_train_loader, _val_loader, test_loader = create_preprocessed_train_val_test_loaders(
|
||||
dataset=dataset, batch_size=args.batch_size, num_workers=0, random_seed=42)
|
||||
|
||||
model = WiFlowPoseModel(dropout=0.5).to(device)
|
||||
# released checkpoint predates the published code: modules were renamed
|
||||
# att -> attention, final_conv -> decoder (param count identical, 2.23M)
|
||||
state = load_remapped_state(args.checkpoint, map_location=device)
|
||||
model.load_state_dict(state, strict=True)
|
||||
n_params = sum(p.numel() for p in model.parameters())
|
||||
print(f"checkpoint: {args.checkpoint} ({n_params/1e6:.2f}M params)")
|
||||
|
||||
# upstream also evaluates with drop_last=True; we report the full test set
|
||||
# (drop_last=False) and the drop_last variant for exact comparability
|
||||
results = {"published": {"pck@20": 0.9725, "pck@30": 0.9863, "pck@40": 0.9916,
|
||||
"pck@50": 0.9948, "mpjpe": 0.007},
|
||||
"params_millions": n_params / 1e6,
|
||||
"data_dir": data_dir,
|
||||
"device": str(device)}
|
||||
|
||||
print("=== test set (full, drop_last=False) ===")
|
||||
results["test_full"] = evaluate(model, test_loader, device=device)
|
||||
print(json.dumps(results["test_full"], indent=2))
|
||||
|
||||
test_loader_dl = DataLoader(test_loader.dataset, batch_size=args.batch_size,
|
||||
shuffle=False, drop_last=True)
|
||||
print("=== test set (drop_last=True, as upstream train.py) ===")
|
||||
results["test_drop_last"] = evaluate(model, test_loader_dl, device=device)
|
||||
print(json.dumps(results["test_drop_last"], indent=2))
|
||||
|
||||
os.makedirs(os.path.dirname(args.out), exist_ok=True)
|
||||
with open(args.out, "w") as f:
|
||||
json.dump(results, f, indent=2)
|
||||
print(f"wrote {args.out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,174 @@
|
||||
"""ADR-152 §2.2: export the retrained WiFlow-STD PyTorch checkpoint to
|
||||
safetensors with tch-rs (VarStore) variable names, plus a numerical-parity
|
||||
fixture for the Rust port.
|
||||
|
||||
Outputs (all under results/, gitignored):
|
||||
retrained_wiflow_std.safetensors -- 248 f32 tensors named exactly as the
|
||||
Rust WiFlowStdModel VarStore expects
|
||||
(see wiflow_std/model.rs
|
||||
`dump_variable_names` for the
|
||||
authoritative name dump)
|
||||
parity_fixture.npz -- deterministic input (seed 42,
|
||||
shape (2, 540, 20), uniform [0,1]) and
|
||||
the Python model's eval-mode output
|
||||
parity_fixture.json -- same data as flattened f32 lists, for
|
||||
the dependency-free Rust test
|
||||
(tests/test_wiflow_std_parity.rs)
|
||||
|
||||
PyTorch -> tch key mapping (derived from the VarStore dump, not guessed):
|
||||
|
||||
tcn.network.{i}.conv1_group.weight -> tcn{i}.conv1_group.weight
|
||||
tcn.network.{i}.bn*_{group,pw}.<leaf> -> tcn{i}.bn*_{group,pw}.<leaf>
|
||||
tcn.network.{i}.downsample.0.weight -> tcn{i}.ds_conv.weight
|
||||
tcn.network.{i}.downsample.1.<leaf> -> tcn{i}.ds_bn.<leaf>
|
||||
up.block.{0,1,4,5,8,9}.<leaf> -> conv_in.{conv1,bn1,conv2,bn2,conv3,bn3}.<leaf>
|
||||
up.downsample.{0,1}.<leaf> -> conv_in.{ds_conv,ds_bn}.<leaf>
|
||||
residual_blocks.{i}.block.{...}.<leaf> -> conv{i}.{conv1..bn3}.<leaf>
|
||||
residual_blocks.{i}.downsample.{0,1} -> conv{i}.{ds_conv,ds_bn}
|
||||
attention.{width,height}_axis.qkv_transform.weight
|
||||
-> attention.{width,height}.qkv.weight
|
||||
attention.{width,height}_axis.bn_* -> attention.{width,height}.bn_*
|
||||
decoder.{0,1,3,4}.<leaf> -> {dec_conv1,dec_bn1,dec_conv2,dec_bn2}.<leaf>
|
||||
*.num_batches_tracked -> dropped (tch BatchNorm has no such buffer)
|
||||
|
||||
Legacy upstream names (att. -> attention., final_conv. -> decoder.) are
|
||||
remapped first, exactly as eval_repro.py does for the released checkpoint.
|
||||
|
||||
Usage:
|
||||
.venv/Scripts/python.exe export_to_safetensors.py
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from safetensors.torch import save_file
|
||||
|
||||
from _bench_common import RESULTS, import_upstream, remap_legacy_keys
|
||||
|
||||
import_upstream() # sys.path + models stub
|
||||
|
||||
from models.pose_model import WiFlowPoseModel # noqa: E402
|
||||
|
||||
CHECKPOINT = os.path.join(RESULTS, "retrained_best_pose_model.pth")
|
||||
|
||||
# Sequential index -> tch sub-name inside one ConvBlock1/AsymmetricConvBlock:
|
||||
# [Conv2d(0), BN(1), SiLU(2), Dropout2d(3), Conv2d(4), BN(5), SiLU(6),
|
||||
# Dropout2d(7), Conv2d(8), BN(9)]
|
||||
_BLOCK_IDX = {"0": "conv1", "1": "bn1", "4": "conv2", "5": "bn2",
|
||||
"8": "conv3", "9": "bn3"}
|
||||
_DS_IDX = {"0": "ds_conv", "1": "ds_bn"}
|
||||
_DECODER_IDX = {"0": "dec_conv1", "1": "dec_bn1", "3": "dec_conv2",
|
||||
"4": "dec_bn2"}
|
||||
|
||||
|
||||
def _conv_block(new_prefix: str, rest: str) -> str:
|
||||
m = re.fullmatch(r"block\.(\d+)\.(.+)", rest)
|
||||
if m:
|
||||
return f"{new_prefix}.{_BLOCK_IDX[m.group(1)]}.{m.group(2)}"
|
||||
m = re.fullmatch(r"downsample\.(\d+)\.(.+)", rest)
|
||||
if m:
|
||||
return f"{new_prefix}.{_DS_IDX[m.group(1)]}.{m.group(2)}"
|
||||
raise KeyError(f"unmapped conv-block key: {new_prefix} / {rest}")
|
||||
|
||||
|
||||
def map_key(key: str) -> str:
|
||||
"""Map one PyTorch state_dict key to the tch VarStore name."""
|
||||
m = re.fullmatch(r"tcn\.network\.(\d+)\.(.+)", key)
|
||||
if m:
|
||||
i, rest = m.groups()
|
||||
rest = (rest.replace("downsample.0.", "ds_conv.")
|
||||
.replace("downsample.1.", "ds_bn."))
|
||||
return f"tcn{i}.{rest}"
|
||||
|
||||
m = re.fullmatch(r"up\.(.+)", key)
|
||||
if m:
|
||||
return _conv_block("conv_in", m.group(1))
|
||||
|
||||
m = re.fullmatch(r"residual_blocks\.(\d+)\.(.+)", key)
|
||||
if m:
|
||||
return _conv_block(f"conv{m.group(1)}", m.group(2))
|
||||
|
||||
m = re.fullmatch(r"attention\.(width|height)_axis\.(.+)", key)
|
||||
if m:
|
||||
axis, rest = m.groups()
|
||||
rest = rest.replace("qkv_transform.", "qkv.")
|
||||
return f"attention.{axis}.{rest}"
|
||||
|
||||
m = re.fullmatch(r"decoder\.(\d+)\.(.+)", key)
|
||||
if m:
|
||||
return f"{_DECODER_IDX[m.group(1)]}.{m.group(2)}"
|
||||
|
||||
raise KeyError(f"unmapped checkpoint key: {key}")
|
||||
|
||||
|
||||
def main():
|
||||
state = torch.load(CHECKPOINT, map_location="cpu", weights_only=True)
|
||||
if not isinstance(state, dict) or "tcn.network.0.conv1_group.weight" not in {
|
||||
k for k in state
|
||||
} | {k.replace("att.", "attention.") for k in state}:
|
||||
# tolerate trainer wrappers like {"model_state_dict": ...}
|
||||
for wrapper in ("model_state_dict", "state_dict", "model"):
|
||||
if isinstance(state, dict) and wrapper in state:
|
||||
state = state[wrapper]
|
||||
break
|
||||
|
||||
# Legacy upstream names predate the published code (_bench_common).
|
||||
state = remap_legacy_keys(state)
|
||||
|
||||
mapped = {}
|
||||
dropped = 0
|
||||
for k, v in state.items():
|
||||
if k.endswith("num_batches_tracked"):
|
||||
dropped += 1
|
||||
continue
|
||||
tch_key = map_key(k)
|
||||
if tch_key in mapped:
|
||||
raise KeyError(f"duplicate mapped key: {k} -> {tch_key}")
|
||||
mapped[tch_key] = v.detach().to(torch.float32).contiguous()
|
||||
|
||||
n_params = sum(v.numel() for k, v in mapped.items()
|
||||
if "running_" not in k)
|
||||
print(f"checkpoint tensors: {len(state)} "
|
||||
f"(dropped {dropped} num_batches_tracked)")
|
||||
print(f"mapped tensors: {len(mapped)}, "
|
||||
f"non-buffer params: {n_params/1e6:.6f}M")
|
||||
assert len(mapped) == 248, f"expected 248 tch variables, got {len(mapped)}"
|
||||
assert n_params == 2_225_042, f"param count mismatch: {n_params}"
|
||||
|
||||
st_path = os.path.join(RESULTS, "retrained_wiflow_std.safetensors")
|
||||
save_file(mapped, st_path)
|
||||
print(f"wrote {st_path}")
|
||||
|
||||
# ---- parity fixture --------------------------------------------------
|
||||
model = WiFlowPoseModel(dropout=0.5)
|
||||
model.load_state_dict(state, strict=True)
|
||||
model.eval()
|
||||
|
||||
gen = torch.Generator().manual_seed(42)
|
||||
x = torch.rand(2, 540, 20, generator=gen, dtype=torch.float32)
|
||||
with torch.no_grad():
|
||||
y = model(x)
|
||||
print(f"fixture input {tuple(x.shape)} -> output {tuple(y.shape)}, "
|
||||
f"output range [{y.min().item():.6f}, {y.max().item():.6f}]")
|
||||
|
||||
np.savez(os.path.join(RESULTS, "parity_fixture.npz"),
|
||||
input=x.numpy(), output=y.numpy())
|
||||
fixture = {
|
||||
"seed": 42,
|
||||
"input_shape": list(x.shape),
|
||||
"input": x.flatten().tolist(),
|
||||
"output_shape": list(y.shape),
|
||||
"output": y.flatten().tolist(),
|
||||
}
|
||||
json_path = os.path.join(RESULTS, "parity_fixture.json")
|
||||
with open(json_path, "w") as f:
|
||||
json.dump(fixture, f)
|
||||
print(f"wrote {os.path.join(RESULTS, 'parity_fixture.npz')}")
|
||||
print(f"wrote {json_path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,148 @@
|
||||
"""Regenerate results/nan_windows_mask.npy + results/big_windows_mask.npy by
|
||||
scanning a PRISTINE kagglehub download of the WiFlow-STD dataset
|
||||
(kaka2434/wiflow-dataset v1, csi_windows.npy, 360,000 windows of 540x20).
|
||||
|
||||
============================ READ THIS FIRST ===============================
|
||||
This script MUST be run against an UNCLEANED copy of the dataset.
|
||||
|
||||
remote/clean_v2.py (and its predecessor clean_nan.py) repair the dataset by
|
||||
zeroing the corrupted windows IN PLACE, with no backup. A cleaned copy
|
||||
contains no non-finite values and no out-of-range amplitudes, so on a cleaned
|
||||
copy this scan produces ALL-FALSE masks -- silently wrong ground truth. The
|
||||
script errors out loudly in that case (see the sanity check in main()).
|
||||
|
||||
That irreversibility is exactly why the two committed mask files under
|
||||
results/ (gitignore-negated) are the canonical ground truth: once a download
|
||||
has been cleaned, the masks can NEVER be regenerated from it. Only run this
|
||||
on a fresh `kagglehub.dataset_download("kaka2434/wiflow-dataset")`.
|
||||
============================================================================
|
||||
|
||||
Criteria (per window; mirrors the original 2026-06-10 scan and the
|
||||
remote/clean_v2.py repair criteria):
|
||||
|
||||
nan mask: any non-finite value (NaN/Inf) anywhere in the 540x20 window
|
||||
big mask: max |finite value| > 1.5 (the data is otherwise [0,1]-normalized;
|
||||
the corrupted files contain garbage up to 3.4e38, float32 max)
|
||||
|
||||
Expected result on the pristine Kaggle download (RESULTS.md defect 5):
|
||||
nan: 9,070 True | big: 9,072 True | union: 9,072 -- all windows in dataset
|
||||
files 487-499 (the final 13 files), window indices 350,922-359,999.
|
||||
|
||||
Usage:
|
||||
PYTHONUTF8=1 .venv/Scripts/python.exe generate_corruption_masks.py \
|
||||
[--data-dir <dir containing csi_windows.npy>] [--out-dir results]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
RESULTS = os.path.join(HERE, "results")
|
||||
|
||||
EXPECTED = {"nan": 9070, "big": 9072, "union": 9072,
|
||||
"files": (487, 499), "windows": (350922, 359999)}
|
||||
|
||||
|
||||
def scan(csi_path, chunk=4000):
|
||||
"""Chunked scan of the (mmap'd) windows array; returns (nan_mask, big_mask)."""
|
||||
csi = np.load(csi_path, mmap_mode="r")
|
||||
n = len(csi)
|
||||
nan_mask = np.zeros(n, dtype=bool)
|
||||
big_mask = np.zeros(n, dtype=bool)
|
||||
for i in range(0, n, chunk):
|
||||
block = np.asarray(csi[i:i + chunk])
|
||||
finite = np.isfinite(block)
|
||||
nan_mask[i:i + chunk] = (~finite).any(axis=(1, 2))
|
||||
big_mask[i:i + chunk] = (
|
||||
np.abs(np.where(finite, block, 0)).max(axis=(1, 2)) > 1.5)
|
||||
if (i // chunk) % 10 == 0:
|
||||
print(f" scanned {min(i + chunk, n):,}/{n:,} windows "
|
||||
f"(nan={int(nan_mask.sum()):,} big={int(big_mask.sum()):,})",
|
||||
flush=True)
|
||||
return nan_mask, big_mask
|
||||
|
||||
|
||||
def describe_files(data_dir, mask):
|
||||
"""Map marked windows to dataset file indices via window_info.npz."""
|
||||
info = os.path.join(data_dir, "window_info.npz")
|
||||
if not os.path.exists(info):
|
||||
return None
|
||||
w2f = np.load(info)["window_to_file"]
|
||||
return np.unique(w2f[mask])
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Regenerate the corruption masks from a PRISTINE "
|
||||
"(uncleaned) kagglehub download. See module docstring.")
|
||||
parser.add_argument("--data-dir", default=os.path.join(
|
||||
os.path.expanduser("~"), ".cache", "kagglehub", "datasets", "kaka2434",
|
||||
"wiflow-dataset", "versions", "1", "preprocessed_csi_data"),
|
||||
help="Directory containing csi_windows.npy (PRISTINE copy)")
|
||||
parser.add_argument("--out-dir", default=RESULTS,
|
||||
help="Where to write the two .npy masks")
|
||||
parser.add_argument("--chunk", type=int, default=4000,
|
||||
help="Windows per scan chunk (memory/speed tradeoff)")
|
||||
args = parser.parse_args()
|
||||
|
||||
csi_path = os.path.join(args.data_dir, "csi_windows.npy")
|
||||
if not os.path.exists(csi_path):
|
||||
sys.exit(f"csi_windows.npy not found in {args.data_dir}")
|
||||
|
||||
print(f"scanning {csi_path} (chunk={args.chunk}) ...")
|
||||
nan_mask, big_mask = scan(csi_path, args.chunk)
|
||||
union = nan_mask | big_mask
|
||||
print(f"nan: {int(nan_mask.sum()):,} | big: {int(big_mask.sum()):,} | "
|
||||
f"union: {int(union.sum()):,} of {len(union):,} windows")
|
||||
|
||||
# ---- sanity check: an all-False result means a CLEANED copy ------------
|
||||
if not union.any():
|
||||
sys.exit(
|
||||
"ERROR: scan found ZERO corrupted windows.\n"
|
||||
"\n"
|
||||
"The pristine Kaggle download (kaka2434/wiflow-dataset v1) is "
|
||||
"known to contain\n"
|
||||
"9,072 corrupted windows (NaN/Inf + amplitudes up to 3.4e38) in "
|
||||
"dataset files\n"
|
||||
"487-499 (RESULTS.md, reproducibility defect 5). Finding none "
|
||||
"means this copy\n"
|
||||
"has almost certainly already been repaired by remote/clean_v2.py "
|
||||
"(or clean_nan.py),\n"
|
||||
"which zeroes the corrupted windows IN PLACE -- after that the "
|
||||
"corruption evidence\n"
|
||||
"is gone and the masks CANNOT be regenerated from this copy.\n"
|
||||
"\n"
|
||||
"Refusing to overwrite the committed ground-truth masks with "
|
||||
"all-False ones.\n"
|
||||
"Re-download the dataset (kagglehub.dataset_download("
|
||||
"'kaka2434/wiflow-dataset'))\n"
|
||||
"and point --data-dir at the fresh, uncleaned copy.")
|
||||
|
||||
files = describe_files(args.data_dir, union)
|
||||
if files is not None:
|
||||
print(f"marked windows span dataset files {files.min()}-{files.max()}: "
|
||||
f"{files.tolist()}")
|
||||
lo, hi = EXPECTED["files"]
|
||||
if files.min() != lo or files.max() != hi:
|
||||
print(f"WARNING: expected marked files exactly {lo}-{hi} "
|
||||
f"(the pristine v1 download); got {files.min()}-{files.max()}. "
|
||||
f"Different dataset version, or a partially cleaned copy?")
|
||||
for name, mask, exp in (("nan", nan_mask, EXPECTED["nan"]),
|
||||
("big", big_mask, EXPECTED["big"])):
|
||||
if int(mask.sum()) != exp:
|
||||
print(f"WARNING: {name} mask has {int(mask.sum()):,} True windows; "
|
||||
f"the pristine v1 download yields {exp:,}.")
|
||||
|
||||
os.makedirs(args.out_dir, exist_ok=True)
|
||||
for name, mask in (("nan_windows_mask.npy", nan_mask),
|
||||
("big_windows_mask.npy", big_mask)):
|
||||
out = os.path.join(args.out_dir, name)
|
||||
np.save(out, mask)
|
||||
print(f"wrote {out} ({int(mask.sum()):,} True)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,220 @@
|
||||
"""ADR-152 edge optimization: ONNX export + onnxruntime CPU benchmark for the
|
||||
retrained WiFlow-STD checkpoint.
|
||||
|
||||
- Exports fp32 to ONNX. The axial attention reshapes with python ints taken
|
||||
from tensor.size() (view(N*W, C, H)), so a traced graph bakes the batch
|
||||
size; we first try a dynamic-batch export and verify it actually works at
|
||||
batch sizes 1/2/64 -- if not, we fall back to fixed-batch exports.
|
||||
- Verifies output parity vs torch on the stored fixture
|
||||
(results/parity_fixture.npz, batch 2, seed 42): max abs diff < 1e-4.
|
||||
- Measures onnxruntime CPU latency at batch 1 and 64 (median of N runs).
|
||||
- Supplementary: onnxruntime dynamic int8 quantization of the exported model
|
||||
(weight size datapoint for the paper's "~2.2 MB int8" claim).
|
||||
|
||||
Usage:
|
||||
.venv/Scripts/python.exe onnx_bench.py
|
||||
|
||||
Writes/merges into results/edge_optimization.json under key "onnx".
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import statistics
|
||||
import time
|
||||
import traceback
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from _bench_common import RESULTS, import_upstream, load_wiflow_model
|
||||
|
||||
import_upstream() # sys.path + models stub + >1GB np.load mmap patch
|
||||
|
||||
CHECKPOINT = os.path.join(RESULTS, "retrained_best_pose_model.pth")
|
||||
OUT_JSON = os.path.join(RESULTS, "edge_optimization.json")
|
||||
|
||||
|
||||
def load_fp32_model():
|
||||
return load_wiflow_model(CHECKPOINT)
|
||||
|
||||
|
||||
def try_export(model, path, batch, dynamic, opset=17):
|
||||
"""Returns (ok, exporter_used, error)."""
|
||||
x = torch.rand(batch, 540, 20)
|
||||
attempts = []
|
||||
if dynamic:
|
||||
attempts.append(("dynamo", dict(dynamo=True,
|
||||
dynamic_shapes={"x": {0: "batch"}})))
|
||||
attempts.append(("torchscript", dict(dynamo=False,
|
||||
dynamic_axes={"input": {0: "batch"},
|
||||
"output": {0: "batch"}})))
|
||||
else:
|
||||
attempts.append(("torchscript", dict(dynamo=False)))
|
||||
attempts.append(("dynamo", dict(dynamo=True)))
|
||||
last_err = None
|
||||
for name, kw in attempts:
|
||||
try:
|
||||
with torch.no_grad():
|
||||
torch.onnx.export(model, (x,), path, opset_version=opset,
|
||||
input_names=["input"], output_names=["output"],
|
||||
**kw)
|
||||
return True, name, None
|
||||
except Exception as e: # noqa: BLE001
|
||||
last_err = f"{name}: {type(e).__name__}: {e}"
|
||||
traceback.print_exc()
|
||||
return False, None, last_err
|
||||
|
||||
|
||||
def ort_session(path):
|
||||
import onnxruntime as ort
|
||||
return ort.InferenceSession(path, providers=["CPUExecutionProvider"])
|
||||
|
||||
|
||||
def ort_run(sess, x):
|
||||
inp = sess.get_inputs()[0].name
|
||||
return sess.run(None, {inp: x})[0]
|
||||
|
||||
|
||||
def bench_ort(sess, batch, n_runs):
|
||||
rng = np.random.default_rng(123)
|
||||
x = rng.random((batch, 540, 20), dtype=np.float32)
|
||||
for _ in range(max(5, n_runs // 10)):
|
||||
ort_run(sess, x)
|
||||
times = []
|
||||
for _ in range(n_runs):
|
||||
t0 = time.perf_counter()
|
||||
ort_run(sess, x)
|
||||
times.append(time.perf_counter() - t0)
|
||||
med = statistics.median(times)
|
||||
return {
|
||||
"batch_size": batch,
|
||||
"runs": n_runs,
|
||||
"median_ms_per_batch": med * 1e3,
|
||||
"median_ms_per_window": med * 1e3 / batch,
|
||||
"windows_per_second": batch / med,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(
|
||||
description="ONNX export + onnxruntime CPU benchmark for the "
|
||||
"retrained WiFlow-STD checkpoint (no options; see "
|
||||
"module docstring). NB: the published "
|
||||
"retrained_fp32_dynamic.onnx came from the TorchScript "
|
||||
"exporter; on newer torch the dynamo attempt may succeed "
|
||||
"first and produce a different (external-data) artifact.")
|
||||
parser.parse_args()
|
||||
|
||||
import onnxruntime
|
||||
model = load_fp32_model()
|
||||
results = {
|
||||
"env": {
|
||||
"torch": torch.__version__,
|
||||
"onnxruntime": onnxruntime.__version__,
|
||||
"platform": platform.platform(),
|
||||
},
|
||||
}
|
||||
|
||||
fixture = np.load(os.path.join(RESULTS, "parity_fixture.npz"))
|
||||
fx, fy = fixture["input"], fixture["output"] # (2,540,20) -> (2,15,2)
|
||||
|
||||
# ---- export: dynamic batch first, fall back to fixed --------------------
|
||||
dyn_path = os.path.join(RESULTS, "retrained_fp32_dynamic.onnx")
|
||||
ok, exporter, err = try_export(model, dyn_path, batch=2, dynamic=True)
|
||||
dynamic_works = False
|
||||
if ok:
|
||||
# verify the dynamic graph really runs at other batch sizes
|
||||
try:
|
||||
sess = ort_session(dyn_path)
|
||||
for b in (1, 2, 64):
|
||||
y = ort_run(sess, np.zeros((b, 540, 20), dtype=np.float32))
|
||||
assert y.shape == (b, 15, 2), y.shape
|
||||
dynamic_works = True
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"dynamic-batch model does not generalize: {e}")
|
||||
|
||||
sessions = {}
|
||||
if dynamic_works:
|
||||
results["export"] = {"mode": "dynamic-batch", "exporter": exporter,
|
||||
"file": os.path.basename(dyn_path),
|
||||
"size_mb": os.path.getsize(dyn_path) / 1e6}
|
||||
sess = ort_session(dyn_path)
|
||||
sessions = {1: sess, 2: sess, 64: sess}
|
||||
print(f"dynamic-batch export OK via {exporter}")
|
||||
else:
|
||||
results["export"] = {"mode": "fixed-batch", "fallback_reason": err,
|
||||
"files": {}}
|
||||
for b in (1, 2, 64):
|
||||
p = os.path.join(RESULTS, f"retrained_fp32_b{b}.onnx")
|
||||
ok, exporter, err = try_export(model, p, batch=b, dynamic=False)
|
||||
if not ok:
|
||||
results["export"]["files"][str(b)] = {"error": err}
|
||||
print(f"EXPORT FAILED at batch {b}: {err}")
|
||||
continue
|
||||
results["export"]["files"][str(b)] = {
|
||||
"exporter": exporter, "file": os.path.basename(p),
|
||||
"size_mb": os.path.getsize(p) / 1e6}
|
||||
sessions[b] = ort_session(p)
|
||||
print(f"fixed-batch {b} export OK via {exporter}")
|
||||
|
||||
# ---- parity vs torch on the fixture -------------------------------------
|
||||
if 2 in sessions:
|
||||
y_ort = ort_run(sessions[2], fx)
|
||||
with torch.no_grad():
|
||||
y_torch = model(torch.from_numpy(fx)).numpy()
|
||||
results["parity"] = {
|
||||
"fixture": "results/parity_fixture.npz (batch 2, seed 42)",
|
||||
"max_abs_diff_vs_stored_fixture": float(np.abs(y_ort - fy).max()),
|
||||
"max_abs_diff_vs_torch_now": float(np.abs(y_ort - y_torch).max()),
|
||||
"pass_lt_1e-4": bool(np.abs(y_ort - y_torch).max() < 1e-4),
|
||||
}
|
||||
print("parity:", json.dumps(results["parity"], indent=2))
|
||||
|
||||
# ---- latency -------------------------------------------------------------
|
||||
results["latency"] = {}
|
||||
if 1 in sessions:
|
||||
results["latency"]["batch1"] = bench_ort(sessions[1], 1, 100)
|
||||
print(f"ORT batch 1: {results['latency']['batch1']['median_ms_per_window']:.2f} ms/window")
|
||||
if 64 in sessions:
|
||||
results["latency"]["batch64"] = bench_ort(sessions[64], 64, 30)
|
||||
print(f"ORT batch 64: {results['latency']['batch64']['median_ms_per_window']:.3f} ms/window")
|
||||
|
||||
# ---- supplementary: ORT dynamic int8 (size datapoint for the 2.2MB claim)
|
||||
src = (dyn_path if dynamic_works
|
||||
else os.path.join(RESULTS, "retrained_fp32_b1.onnx"))
|
||||
if os.path.exists(src):
|
||||
try:
|
||||
from onnxruntime.quantization import QuantType, quantize_dynamic
|
||||
q_path = os.path.join(RESULTS, "retrained_int8_ort_dynamic.onnx")
|
||||
quantize_dynamic(src, q_path, weight_type=QuantType.QInt8)
|
||||
entry = {"file": os.path.basename(q_path),
|
||||
"size_mb": os.path.getsize(q_path) / 1e6}
|
||||
try:
|
||||
qs = ort_session(q_path)
|
||||
yq = ort_run(qs, fx[:1] if not dynamic_works else fx)
|
||||
ref = fy[:1] if not dynamic_works else fy
|
||||
entry["runs"] = True
|
||||
entry["max_abs_diff_vs_fp32_fixture"] = float(np.abs(yq - ref).max())
|
||||
except Exception as e: # noqa: BLE001
|
||||
entry["runs"] = False
|
||||
entry["run_error"] = f"{type(e).__name__}: {e}"
|
||||
results["ort_int8_dynamic_supplementary"] = entry
|
||||
print("ORT int8:", json.dumps(entry, indent=2))
|
||||
except Exception as e: # noqa: BLE001
|
||||
results["ort_int8_dynamic_supplementary"] = {
|
||||
"error": f"{type(e).__name__}: {e}"}
|
||||
|
||||
merged = {}
|
||||
if os.path.exists(OUT_JSON):
|
||||
with open(OUT_JSON) as f:
|
||||
merged = json.load(f)
|
||||
merged["onnx"] = results
|
||||
with open(OUT_JSON, "w") as f:
|
||||
json.dump(merged, f, indent=2)
|
||||
print(f"wrote {OUT_JSON}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,228 @@
|
||||
"""ADR-152 "optimize beyond SOTA": edge-optimization benchmark for the
|
||||
retrained WiFlow-STD checkpoint (results/retrained_best_pose_model.pth,
|
||||
~96% PCK@20, fp32 params 2,225,042).
|
||||
|
||||
Measures, for fp32 / fp16 / dynamic-int8 torch variants:
|
||||
(a) serialized state_dict size on disk,
|
||||
(b) CPU inference latency per window at batch 1 and batch 64
|
||||
(median of repeated runs, this Windows box),
|
||||
(c) accuracy (PCK@20/50 + MPJPE, upstream metrics) on a corruption-free
|
||||
random subset of the seed-42 file-level 70/15/15 test split
|
||||
(same split as eval_repro.py; corrupted windows 487-499 excluded via
|
||||
results/nan_windows_mask.npy | results/big_windows_mask.npy).
|
||||
|
||||
Also verifies the paper's "~2.2 MB int8" size claim: reports which layer
|
||||
types torch dynamic quantization actually converts (the model contains NO
|
||||
nn.Linear -- it is Conv1d/Conv2d/BatchNorm only) and the real on-disk size.
|
||||
|
||||
Usage:
|
||||
.venv/Scripts/python.exe quantize_bench.py \
|
||||
--data-dir C:/Users/ruv/.cache/kagglehub/datasets/kaka2434/wiflow-dataset/versions/1/preprocessed_csi_data \
|
||||
[--subset 10000] [--skip-accuracy]
|
||||
|
||||
Writes/merges into results/edge_optimization.json under key "torch".
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import statistics
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
from torch.utils.data import DataLoader
|
||||
|
||||
from _bench_common import HERE, RESULTS, evaluate, import_upstream, load_wiflow_model
|
||||
|
||||
import_upstream() # sys.path + models stub + >1GB np.load mmap patch
|
||||
|
||||
from dataset import ( # noqa: E402
|
||||
PreprocessedCSIKeypointsDataset,
|
||||
create_preprocessed_train_val_test_loaders,
|
||||
)
|
||||
|
||||
CHECKPOINT = os.path.join(RESULTS, "retrained_best_pose_model.pth")
|
||||
|
||||
|
||||
def load_fp32_model():
|
||||
# legacy upstream key remap inside is a harmless no-op on this checkpoint
|
||||
return load_wiflow_model(CHECKPOINT)
|
||||
|
||||
|
||||
def state_dict_size_bytes(model, path):
|
||||
torch.save(model.state_dict(), path)
|
||||
return os.path.getsize(path)
|
||||
|
||||
|
||||
def bench_latency(model, batch_size, n_runs, dtype=torch.float32):
|
||||
gen = torch.Generator().manual_seed(123)
|
||||
x = torch.rand(batch_size, 540, 20, generator=gen).to(dtype)
|
||||
with torch.no_grad():
|
||||
for _ in range(max(5, n_runs // 10)): # warmup
|
||||
model(x)
|
||||
times = []
|
||||
for _ in range(n_runs):
|
||||
t0 = time.perf_counter()
|
||||
model(x)
|
||||
times.append(time.perf_counter() - t0)
|
||||
med = statistics.median(times)
|
||||
return {
|
||||
"batch_size": batch_size,
|
||||
"runs": n_runs,
|
||||
"median_ms_per_batch": med * 1e3,
|
||||
"median_ms_per_window": med * 1e3 / batch_size,
|
||||
"windows_per_second": batch_size / med,
|
||||
}
|
||||
|
||||
|
||||
def build_test_subset(data_dir, subset_size, batch_size=64):
|
||||
"""Seed-42 file-level 70/15/15 test split (exactly as eval_repro.py),
|
||||
minus corrupted windows, then a seed-42 random subset."""
|
||||
dataset = PreprocessedCSIKeypointsDataset(
|
||||
data_dir=data_dir, keypoint_scale=1000.0, enable_temporal_clean=True)
|
||||
_tr, _va, test_loader = create_preprocessed_train_val_test_loaders(
|
||||
dataset=dataset, batch_size=batch_size, num_workers=0, random_seed=42)
|
||||
test_indices = np.asarray(test_loader.dataset.indices)
|
||||
|
||||
corrupted = (np.load(os.path.join(RESULTS, "nan_windows_mask.npy"))
|
||||
| np.load(os.path.join(RESULTS, "big_windows_mask.npy")))
|
||||
clean = test_indices[~corrupted[test_indices]]
|
||||
print(f"test split: {len(test_indices)} windows, "
|
||||
f"{len(test_indices) - len(clean)} corrupted excluded, "
|
||||
f"{len(clean)} clean")
|
||||
|
||||
if subset_size and subset_size < len(clean):
|
||||
rng = np.random.default_rng(42)
|
||||
clean = np.sort(rng.choice(clean, size=subset_size, replace=False))
|
||||
subset = torch.utils.data.Subset(dataset, clean.tolist())
|
||||
loader = DataLoader(subset, batch_size=batch_size, shuffle=False,
|
||||
num_workers=0)
|
||||
return loader, len(clean)
|
||||
|
||||
|
||||
def quantize_int8_dynamic(fp32_model):
|
||||
"""torch.ao.quantization.quantize_dynamic on Linear/Conv where supported.
|
||||
Returns (model, report) where report documents what actually quantized."""
|
||||
qmodel = torch.ao.quantization.quantize_dynamic(
|
||||
fp32_model, {nn.Linear, nn.Conv1d, nn.Conv2d}, dtype=torch.qint8)
|
||||
|
||||
quantized, total_params, quant_params = [], 0, 0
|
||||
for name, mod in qmodel.named_modules():
|
||||
cls = type(mod).__module__ + "." + type(mod).__name__
|
||||
if "quantized" in cls:
|
||||
w = mod.weight() if callable(getattr(mod, "weight", None)) else None
|
||||
numel = w.numel() if w is not None else 0
|
||||
quant_params += numel
|
||||
quantized.append({"module": name, "class": cls, "params": numel})
|
||||
for p in fp32_model.parameters():
|
||||
total_params += p.numel()
|
||||
|
||||
n_linear = sum(isinstance(m, nn.Linear) for m in fp32_model.modules())
|
||||
n_conv1d = sum(isinstance(m, nn.Conv1d) for m in fp32_model.modules())
|
||||
n_conv2d = sum(isinstance(m, nn.Conv2d) for m in fp32_model.modules())
|
||||
report = {
|
||||
"eligible_module_counts": {
|
||||
"nn.Linear": n_linear, "nn.Conv1d": n_conv1d, "nn.Conv2d": n_conv2d},
|
||||
"modules_actually_quantized": quantized,
|
||||
"n_modules_quantized": len(quantized),
|
||||
"params_total": total_params,
|
||||
"params_quantized": quant_params,
|
||||
"params_quantized_fraction": quant_params / total_params,
|
||||
}
|
||||
return qmodel, report
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--data-dir", default=os.path.join(
|
||||
os.path.expanduser("~"), ".cache", "kagglehub", "datasets", "kaka2434",
|
||||
"wiflow-dataset", "versions", "1", "preprocessed_csi_data"))
|
||||
parser.add_argument("--subset", type=int, default=10000)
|
||||
parser.add_argument("--runs-b1", type=int, default=100)
|
||||
parser.add_argument("--runs-b64", type=int, default=30)
|
||||
parser.add_argument("--skip-accuracy", action="store_true")
|
||||
parser.add_argument("--out", default=os.path.join(RESULTS, "edge_optimization.json"))
|
||||
args = parser.parse_args()
|
||||
|
||||
torch.manual_seed(42)
|
||||
results = {
|
||||
"env": {
|
||||
"torch": torch.__version__,
|
||||
"platform": platform.platform(),
|
||||
"processor": platform.processor(),
|
||||
"num_threads": torch.get_num_threads(),
|
||||
"checkpoint": os.path.relpath(CHECKPOINT, HERE),
|
||||
},
|
||||
"variants": {},
|
||||
}
|
||||
|
||||
# ---- build variants ---------------------------------------------------
|
||||
fp32 = load_fp32_model()
|
||||
n_params = sum(p.numel() for p in fp32.parameters())
|
||||
results["env"]["params"] = n_params
|
||||
print(f"fp32 model: {n_params:,} params")
|
||||
|
||||
fp16 = load_fp32_model().half()
|
||||
|
||||
int8, q_report = quantize_int8_dynamic(load_fp32_model())
|
||||
results["int8_dynamic_quant_report"] = q_report
|
||||
print(f"int8 dynamic: {q_report['n_modules_quantized']} modules quantized, "
|
||||
f"{q_report['params_quantized_fraction']*100:.1f}% of params")
|
||||
|
||||
variants = {
|
||||
"fp32": (fp32, torch.float32, "retrained_fp32_resaved.pth"),
|
||||
"fp16": (fp16, torch.float16, "retrained_fp16.pth"),
|
||||
"int8_dynamic": (int8, torch.float32, "retrained_int8_dynamic.pth"),
|
||||
}
|
||||
|
||||
# ---- (a) size + (b) latency -------------------------------------------
|
||||
for name, (model, dtype, fname) in variants.items():
|
||||
path = os.path.join(RESULTS, fname)
|
||||
size = state_dict_size_bytes(model, path)
|
||||
print(f"\n=== {name}: {size/1e6:.3f} MB on disk ({fname}) ===")
|
||||
lat1 = bench_latency(model, 1, args.runs_b1, dtype)
|
||||
lat64 = bench_latency(model, 64, args.runs_b64, dtype)
|
||||
print(f" batch 1: {lat1['median_ms_per_window']:.2f} ms/window "
|
||||
f"({lat1['windows_per_second']:.0f}/s)")
|
||||
print(f" batch 64: {lat64['median_ms_per_window']:.3f} ms/window "
|
||||
f"({lat64['windows_per_second']:.0f}/s)")
|
||||
results["variants"][name] = {
|
||||
"file": fname,
|
||||
"size_bytes": size,
|
||||
"size_mb": size / 1e6,
|
||||
"latency_batch1": lat1,
|
||||
"latency_batch64": lat64,
|
||||
}
|
||||
|
||||
# ---- (c) accuracy ------------------------------------------------------
|
||||
if not args.skip_accuracy:
|
||||
loader, n_clean = build_test_subset(args.data_dir, args.subset)
|
||||
results["accuracy_subset"] = {
|
||||
"description": "seed-42 file-level 70/15/15 test split, corrupted "
|
||||
"windows (files 487-499) excluded, seed-42 random "
|
||||
"subset",
|
||||
"subset_size": min(args.subset, n_clean) if args.subset else n_clean,
|
||||
"clean_test_total": n_clean,
|
||||
}
|
||||
for name, (model, dtype, _f) in variants.items():
|
||||
print(f"\n=== accuracy: {name} ===")
|
||||
results["variants"][name]["accuracy"] = evaluate(
|
||||
model, loader, dtype=dtype, label=name)
|
||||
print(json.dumps(results["variants"][name]["accuracy"], indent=2))
|
||||
|
||||
# ---- merge into edge_optimization.json ---------------------------------
|
||||
merged = {}
|
||||
if os.path.exists(args.out):
|
||||
with open(args.out) as f:
|
||||
merged = json.load(f)
|
||||
merged["torch"] = results
|
||||
with open(args.out, "w") as f:
|
||||
json.dump(merged, f, indent=2)
|
||||
print(f"\nwrote {args.out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,14 @@
|
||||
import numpy as np, os
|
||||
d = os.path.expanduser('~/wiflow-std-bench/preprocessed_csi_data')
|
||||
csi = np.load(os.path.join(d, 'csi_windows.npy'), mmap_mode='r+')
|
||||
zeroed = 0
|
||||
chunk = 4000
|
||||
for i in range(0, len(csi), chunk):
|
||||
block = csi[i:i+chunk]
|
||||
finite = np.isfinite(block)
|
||||
bad = (~finite).any(axis=(1, 2)) | (np.abs(np.where(finite, block, 0)).max(axis=(1, 2)) > 1.5)
|
||||
if bad.any():
|
||||
block[bad] = 0.0
|
||||
zeroed += int(bad.sum())
|
||||
csi.flush()
|
||||
print(f'zeroed {zeroed} corrupted windows entirely')
|
||||
@@ -0,0 +1,112 @@
|
||||
"""Evaluate the retrained WiFlow-STD checkpoint (ADR-152 §2.2a fallback).
|
||||
|
||||
Scores the model produced by run.py (train_output/best_pose_model.pth or similar)
|
||||
on the seed-42 test split: full test set AND NaN-free subset (excluding windows
|
||||
that were zero-filled by clean_nan.py — file indices 487-499).
|
||||
|
||||
NOTE: deployed to ruvultra (~/wiflow-std-bench) as a standalone single file,
|
||||
so it deliberately inlines its helpers. The reference implementations (upstream
|
||||
import shim, >1GB np.load mmap patch, key-remap loader, canonical evaluate
|
||||
loop) live in benchmarks/wiflow-std/_bench_common.py — keep copies in sync.
|
||||
"""
|
||||
import json, os, random, sys
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from torch.utils.data import DataLoader, Subset
|
||||
|
||||
# csi_windows.npy is ~13 GB; mmap large arrays instead of eagerly loading
|
||||
# ~15 GB into RAM (same patch as _bench_common._np_load_mmap).
|
||||
_np_load = np.load
|
||||
|
||||
|
||||
def _np_load_mmap(path, *a, **kw):
|
||||
if (isinstance(path, str) and path.endswith('.npy')
|
||||
and os.path.getsize(path) > 1 << 30 and 'mmap_mode' not in kw):
|
||||
kw['mmap_mode'] = 'r'
|
||||
return _np_load(path, *a, **kw)
|
||||
|
||||
|
||||
np.load = _np_load_mmap
|
||||
|
||||
sys.path.insert(0, os.path.expanduser('~/wiflow-std-bench/upstream'))
|
||||
from dataset import PreprocessedCSIKeypointsDataset, create_preprocessed_train_val_test_loaders
|
||||
from models.pose_model import WiFlowPoseModel
|
||||
from utils.metrics import calculate_pck, calculate_mpjpe
|
||||
|
||||
|
||||
def find_checkpoint():
|
||||
cands = []
|
||||
for root, _, files in os.walk(os.path.expanduser('~/wiflow-std-bench/train_output')):
|
||||
for f in files:
|
||||
if f.endswith('.pth'):
|
||||
cands.append(os.path.join(root, f))
|
||||
# also upstream/test default output dir
|
||||
for root, _, files in os.walk(os.path.expanduser('~/wiflow-std-bench/upstream')):
|
||||
for f in files:
|
||||
if f.endswith('.pth') and 'best' in f and 'cross_dataset' not in root:
|
||||
p = os.path.join(root, f)
|
||||
if os.path.getmtime(p) > os.path.getmtime(os.path.expanduser('~/wiflow-std-bench/train.log')) - 86400 * 2:
|
||||
cands.append(p)
|
||||
cands = [c for c in cands if not c.endswith('upstream/best_pose_model.pth')]
|
||||
if not cands:
|
||||
sys.exit('no retrained checkpoint found')
|
||||
return max(cands, key=os.path.getmtime)
|
||||
|
||||
|
||||
def evaluate(model, loader, device):
|
||||
model.eval()
|
||||
totals = {t: 0.0 for t in (0.1, 0.2, 0.3, 0.4, 0.5)}
|
||||
total_mpe, n = 0.0, 0
|
||||
with torch.no_grad():
|
||||
for bx, by in loader:
|
||||
bx, by = bx.to(device), by.to(device)
|
||||
out = model(bx)
|
||||
bs = by.size(0)
|
||||
total_mpe += calculate_mpjpe(out, by) * bs
|
||||
pck = calculate_pck(out, by, thresholds=list(totals))
|
||||
for t in totals:
|
||||
totals[t] += pck[t] * bs
|
||||
n += bs
|
||||
return {'samples': n, 'mpjpe': total_mpe / n,
|
||||
**{f'pck@{int(t*100)}': totals[t] / n for t in totals}}
|
||||
|
||||
|
||||
random.seed(42); np.random.seed(42); torch.manual_seed(42)
|
||||
torch.cuda.manual_seed_all(42)
|
||||
torch.backends.cudnn.deterministic = True
|
||||
|
||||
d = os.path.expanduser('~/wiflow-std-bench/preprocessed_csi_data')
|
||||
dataset = PreprocessedCSIKeypointsDataset(data_dir=d, keypoint_scale=1000.0,
|
||||
enable_temporal_clean=True)
|
||||
_, _, test_loader = create_preprocessed_train_val_test_loaders(
|
||||
dataset=dataset, batch_size=256, num_workers=2, random_seed=42)
|
||||
|
||||
device = torch.device('cuda')
|
||||
ckpt = find_checkpoint()
|
||||
print('checkpoint:', ckpt)
|
||||
model = WiFlowPoseModel(dropout=0.5).to(device)
|
||||
state = torch.load(ckpt, map_location=device, weights_only=True)
|
||||
renames = {'att.': 'attention.', 'final_conv.': 'decoder.'}
|
||||
state = {next((new + k[len(old):] for old, new in renames.items()
|
||||
if k.startswith(old)), k): v for k, v in state.items()}
|
||||
model.load_state_dict(state, strict=True)
|
||||
|
||||
results = {'checkpoint': ckpt}
|
||||
print('=== full test set ===')
|
||||
results['test_full'] = evaluate(model, test_loader, device)
|
||||
print(json.dumps(results['test_full'], indent=2))
|
||||
|
||||
# NaN-free subset: exclude windows from corrupted files 487-499
|
||||
test_subset = test_loader.dataset # Subset(dataset, test_indices)
|
||||
w2f = dataset.window_to_file
|
||||
clean_idx = [i for i in test_subset.indices if w2f[i] < 487]
|
||||
print(f'=== NaN-free test subset ({len(clean_idx)} of {len(test_subset.indices)}) ===')
|
||||
clean_loader = DataLoader(Subset(dataset, clean_idx), batch_size=256, shuffle=False)
|
||||
results['test_clean'] = evaluate(model, clean_loader, device)
|
||||
print(json.dumps(results['test_clean'], indent=2))
|
||||
|
||||
out = os.path.expanduser('~/wiflow-std-bench/eval_retrained.json')
|
||||
with open(out, 'w') as f:
|
||||
json.dump(results, f, indent=2)
|
||||
print('wrote', out)
|
||||
@@ -0,0 +1,374 @@
|
||||
"""ADR-152 SS2.2 measurement (b): WiFlow-STD fine-tuned on our fresh ESP32 paired dataset.
|
||||
|
||||
Dataset: ~/wiflow-std-bench/paired-20260610.jsonl -- 2,046 paired windows collected
|
||||
2026-06-10 22:10-22:40 (ONE subject, ONE room, ONE ESP32 node, varied poses).
|
||||
Per record: csi = flat float32 list, csi_shape, kp = 17 COCO [x, y] normalized [0,1]
|
||||
camera coords, conf (MediaPipe mean confidence, all > 0.5 in this set), ts_start/ts_end.
|
||||
Aligner: scripts/align-ground-truth.js, non-overlapping 20-frame windows (~0.42 s each).
|
||||
|
||||
Dataset findings (MEASURED on this file, 2026-06-10):
|
||||
- csi_shape is HETEROGENEOUS, not uniformly [70, 20]: 1,347x [70,20], 284x [134,20],
|
||||
243x [26,20], 130x [12,20], 42x [20,20]. The ESP32 stream emits mixed frame types
|
||||
and the aligner stamps each window's subcarrier count from frame[0]
|
||||
(extractCsiMatrix: nSc = window[0].subcarriers), zero-padding/truncating the rest.
|
||||
Even native-70 windows contain ~20.4% internally zero-padded short frames
|
||||
(subcarriers 40..69 all-zero for those frames).
|
||||
- LAYOUT BUG: the aligner fills matrix[f * nSc + s] (frame-major) but declares
|
||||
shape [nSc, nFrames]. The true layout is (frame, subcarrier); we reshape
|
||||
(nFrames, nSc) and transpose. Confirmed by coherent per-frame zero-tails.
|
||||
- Handling here (primary suite, "all2046"): every frame's subcarrier axis is
|
||||
linearly resampled to 70 bins (np.interp over a normalized index domain;
|
||||
identity for native-70 frames) so the pre-registered n=2,046 and split sizes
|
||||
hold. Secondary suite ("native70") restricts to the 1,347 native [70,20]
|
||||
windows (temporal 70/15/15 of those) as a homogeneity robustness check.
|
||||
|
||||
Pre-registered protocol (followed exactly):
|
||||
1. TEMPORAL split (records are time-sorted; asserted): first 70% train (1,432),
|
||||
next 15% val (307), last 15% test (307). No shuffling across time. Seed 42
|
||||
for everything else.
|
||||
2. Model: upstream WiFlow-STD trunk (WiFlowPoseModel) with a learned 1x1 Conv1d
|
||||
projection 70->540 prepended, and K=17 via the parameter-free adaptive pool
|
||||
(AdaptiveAvgPool2d((17, 1)) instead of (15, 1)) -- pretrained weights load
|
||||
for any K. CSI normalization: divide by the TRAIN-split 99th-percentile
|
||||
amplitude, clip to [0, 1] (documented in output JSON).
|
||||
3. Three runs, <=60 epochs, early-stop patience 8 on val MPJPE, batch 32,
|
||||
AdamW, fp32 (no autocast):
|
||||
(i) pretrained-init: trunk init from upstream/test/best_pose_model.pth
|
||||
(the measurement-(a) retrained checkpoint, ~96% PCK@20 on WiFlow data;
|
||||
key remap att.->attention. / final_conv.->decoder. applied defensively
|
||||
as in eval_repro.py -- a no-op for this checkpoint, which already uses
|
||||
the new names). Discriminative lr: adapter 1e-4, trunk 1e-5.
|
||||
(ii) scratch: same architecture, random init, all params lr 1e-4.
|
||||
(iii) frozen-trunk: pretrained trunk frozen (requires_grad=False AND held in
|
||||
.eval() so BatchNorm running stats cannot drift -- pure transfer probe);
|
||||
only the 70->540 adapter trains, lr 1e-4.
|
||||
4. Metrics on the temporal TEST split: torso-normalized PCK@10/20/30/40/50 and
|
||||
MPJPE. Upstream utils/metrics.py calculate_pck(use_torso_norm=True) hardcodes
|
||||
NECK_IDX/PELVIS_IDX = 2, 12 -- a 15-keypoint convention that is WRONG for our
|
||||
17 COCO keypoints (2 = right_eye, 12 = right_hip). We therefore reimplement the
|
||||
identical math (per-frame norm distance, clamp min 0.01, mean over all
|
||||
keypoints x frames) with torso = ||l_shoulder(5) - l_hip(11)||.
|
||||
Also reported: prediction std across test frames (constant-pose detector;
|
||||
must be > 0) and the mean-pose-predictor baseline (train-split mean pose
|
||||
evaluated on test -- the honesty bar).
|
||||
|
||||
Usage (on ruvultra):
|
||||
nice -n 10 nohup ~/wiflow-std-bench/venv/bin/python train_measb.py > train_measb.log 2>&1 &
|
||||
|
||||
NOTE: deployed to ruvultra as a standalone single file, so it deliberately
|
||||
inlines its helpers. The reference implementations (upstream import shim,
|
||||
np.load mmap patch, key-remap loader, canonical evaluate loop) live in
|
||||
benchmarks/wiflow-std/_bench_common.py — keep copies in sync.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
BENCH = os.path.expanduser("~/wiflow-std-bench")
|
||||
UPSTREAM = os.path.join(BENCH, "upstream")
|
||||
MEASB = os.path.join(BENCH, "measb")
|
||||
DATA = os.path.join(BENCH, "paired-20260610.jsonl")
|
||||
CHECKPOINT = os.path.join(UPSTREAM, "test", "best_pose_model.pth")
|
||||
|
||||
sys.path.insert(0, UPSTREAM)
|
||||
|
||||
# Upstream defect (1): models/__init__.py imports a name tcn.py does not define.
|
||||
# Register a stub package so the broken __init__ never executes (as eval_repro.py).
|
||||
import types # noqa: E402
|
||||
|
||||
_models_pkg = types.ModuleType("models")
|
||||
_models_pkg.__path__ = [os.path.join(UPSTREAM, "models")]
|
||||
sys.modules["models"] = _models_pkg
|
||||
|
||||
from models.pose_model import WiFlowPoseModel # noqa: E402
|
||||
|
||||
SEED = 42
|
||||
K = 17
|
||||
N_SUBC = 70
|
||||
TRUNK_IN = 540
|
||||
BATCH = 32 # <= 64 per protocol (GPU shared with the efficiency sweep)
|
||||
MAX_EPOCHS = 60
|
||||
PATIENCE = 8
|
||||
LR_ADAPTER = 1e-4
|
||||
LR_TRUNK_FT = 1e-5 # 10x lower for the pretrained trunk vs the fresh adapter
|
||||
L_SHOULDER, L_HIP = 5, 11
|
||||
THRESHOLDS = (0.1, 0.2, 0.3, 0.4, 0.5)
|
||||
|
||||
|
||||
def set_seed(seed=SEED):
|
||||
random.seed(seed)
|
||||
np.random.seed(seed)
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed_all(seed)
|
||||
torch.backends.cudnn.deterministic = True
|
||||
torch.backends.cudnn.benchmark = False
|
||||
|
||||
|
||||
def resample_subcarriers(frame_major, n_out=N_SUBC):
|
||||
"""(nFrames, nSc) -> (nFrames, n_out) by per-frame linear interpolation.
|
||||
|
||||
Identity for nSc == n_out. Normalized index domain [0, 1] on both sides.
|
||||
"""
|
||||
nf, nsc = frame_major.shape
|
||||
if nsc == n_out:
|
||||
return frame_major
|
||||
xi = np.linspace(0.0, 1.0, nsc)
|
||||
xo = np.linspace(0.0, 1.0, n_out)
|
||||
return np.stack([np.interp(xo, xi, frame_major[f]) for f in range(nf)]).astype(np.float32)
|
||||
|
||||
|
||||
def load_dataset():
|
||||
csi, kps, confs, ts, native70 = [], [], [], [], []
|
||||
shape_counts = {}
|
||||
with open(DATA) as f:
|
||||
for line in f:
|
||||
r = json.loads(line)
|
||||
nsc, nf = r["csi_shape"]
|
||||
shape_counts[f"{nsc}x{nf}"] = shape_counts.get(f"{nsc}x{nf}", 0) + 1
|
||||
assert nf == 20, r["csi_shape"]
|
||||
# Aligner layout bug: data is frame-major despite the declared
|
||||
# [nSc, nFrames] shape -- reshape (nFrames, nSc), then resample the
|
||||
# subcarrier axis to 70 and transpose to (70 subcarriers, 20 frames).
|
||||
fm = np.asarray(r["csi"], dtype=np.float32).reshape(nf, nsc)
|
||||
csi.append(resample_subcarriers(fm).T)
|
||||
kp = np.asarray(r["kp"], dtype=np.float32)
|
||||
assert kp.shape == (K, 2), kp.shape
|
||||
kps.append(kp)
|
||||
confs.append(r["conf"])
|
||||
ts.append(r["ts_start"])
|
||||
native70.append(nsc == N_SUBC)
|
||||
assert all(ts[i] <= ts[i + 1] for i in range(len(ts) - 1)), "records not time-sorted"
|
||||
return (np.stack(csi), np.stack(kps), np.asarray(confs, dtype=np.float32),
|
||||
np.asarray(native70), shape_counts, ts[0], ts[-1])
|
||||
|
||||
|
||||
def temporal_split(n):
|
||||
n_train = int(round(n * 0.70))
|
||||
n_val = int(round(n * 0.15))
|
||||
return slice(0, n_train), slice(n_train, n_train + n_val), slice(n_train + n_val, n)
|
||||
|
||||
|
||||
class AdaptedWiFlow(nn.Module):
|
||||
"""1x1 Conv1d adapter 70->540 + upstream WiFlow-STD trunk with K=17 pool head."""
|
||||
|
||||
def __init__(self, k=K, dropout=0.5):
|
||||
super().__init__()
|
||||
self.adapter = nn.Conv1d(N_SUBC, TRUNK_IN, kernel_size=1)
|
||||
nn.init.kaiming_normal_(self.adapter.weight, mode="fan_out", nonlinearity="relu")
|
||||
nn.init.constant_(self.adapter.bias, 0)
|
||||
self.trunk = WiFlowPoseModel(dropout=dropout)
|
||||
# K=17 via the parameter-free adaptive pool: decoder emits [B, 2, 15, 20]
|
||||
# spatial maps; pooling H->17 instead of 15 yields [B, 17, 2] with no new
|
||||
# parameters, so the pretrained state_dict loads strict=True for any K.
|
||||
self.trunk.avg_pool = nn.AdaptiveAvgPool2d((k, 1))
|
||||
|
||||
def forward(self, x):
|
||||
return self.trunk(self.adapter(x))
|
||||
|
||||
|
||||
def load_pretrained_trunk(trunk, path):
|
||||
state = torch.load(path, map_location="cpu", weights_only=True)
|
||||
# Defensive remap as in eval_repro.py (no-op for the retrained checkpoint).
|
||||
renames = {"att.": "attention.", "final_conv.": "decoder."}
|
||||
state = {next((new + k[len(old):] for old, new in renames.items()
|
||||
if k.startswith(old)), k): v
|
||||
for k, v in state.items()}
|
||||
trunk.load_state_dict(state, strict=True)
|
||||
|
||||
|
||||
def pck_torso(pred, target, thresholds=THRESHOLDS):
|
||||
"""Upstream calculate_pck math, torso = l_shoulder(5)<->l_hip(11) for 17-kp COCO."""
|
||||
norm = torch.sqrt(((target[:, L_SHOULDER] - target[:, L_HIP]) ** 2).sum(dim=1))
|
||||
norm = torch.clamp(norm, min=0.01)
|
||||
dist = torch.sqrt(((pred - target) ** 2).sum(dim=2)) / norm.unsqueeze(1)
|
||||
return {f"pck@{int(t * 100)}": (dist <= t).float().mean().item() for t in thresholds}
|
||||
|
||||
|
||||
def mpjpe(pred, target):
|
||||
return torch.sqrt(((pred - target) ** 2).sum(dim=2)).mean().item()
|
||||
|
||||
|
||||
@torch.no_grad()
|
||||
def predict(model, x, batch=256):
|
||||
model.eval()
|
||||
return torch.cat([model(x[i:i + batch]) for i in range(0, len(x), batch)])
|
||||
|
||||
|
||||
def eval_preds(pred, target):
|
||||
out = pck_torso(pred, target)
|
||||
out["mpjpe"] = mpjpe(pred, target)
|
||||
# Constant-pose detector: std across test frames per coordinate, mean over
|
||||
# the 17x2 coordinates. 0.0 == degenerate constant predictor.
|
||||
out["pred_std"] = pred.std(dim=0).mean().item()
|
||||
return out
|
||||
|
||||
|
||||
def train_run(name, x_tr, y_tr, x_va, y_va, device, pretrained, freeze_trunk,
|
||||
lr_trunk):
|
||||
set_seed(SEED)
|
||||
model = AdaptedWiFlow().to(device)
|
||||
if pretrained:
|
||||
load_pretrained_trunk(model.trunk, CHECKPOINT)
|
||||
if freeze_trunk:
|
||||
for p in model.trunk.parameters():
|
||||
p.requires_grad = False
|
||||
groups = [{"params": model.adapter.parameters(), "lr": LR_ADAPTER}]
|
||||
else:
|
||||
groups = [{"params": model.adapter.parameters(), "lr": LR_ADAPTER},
|
||||
{"params": model.trunk.parameters(), "lr": lr_trunk}]
|
||||
opt = torch.optim.AdamW(groups)
|
||||
loss_fn = nn.MSELoss()
|
||||
|
||||
n = len(x_tr)
|
||||
best_val, best_state, best_epoch, bad = float("inf"), None, -1, 0
|
||||
history = []
|
||||
t0 = time.time()
|
||||
for epoch in range(MAX_EPOCHS):
|
||||
model.train()
|
||||
if freeze_trunk:
|
||||
model.trunk.eval() # keep BatchNorm running stats fixed: pure transfer
|
||||
perm = torch.randperm(n, device=device)
|
||||
ep_loss = 0.0
|
||||
for i in range(0, n, BATCH):
|
||||
idx = perm[i:i + BATCH]
|
||||
opt.zero_grad()
|
||||
loss = loss_fn(model(x_tr[idx]), y_tr[idx])
|
||||
loss.backward()
|
||||
opt.step()
|
||||
ep_loss += loss.item() * len(idx)
|
||||
val_mpjpe = mpjpe(predict(model, x_va), y_va)
|
||||
history.append({"epoch": epoch, "train_mse": ep_loss / n, "val_mpjpe": val_mpjpe})
|
||||
marker = ""
|
||||
if val_mpjpe < best_val:
|
||||
best_val, best_epoch, bad = val_mpjpe, epoch, 0
|
||||
best_state = {k: v.detach().cpu().clone() for k, v in model.state_dict().items()}
|
||||
marker = " *"
|
||||
else:
|
||||
bad += 1
|
||||
print(f"[{name}] epoch {epoch:02d} train_mse {ep_loss / n:.6f} "
|
||||
f"val_mpjpe {val_mpjpe:.5f}{marker}", flush=True)
|
||||
if bad >= PATIENCE:
|
||||
print(f"[{name}] early stop at epoch {epoch} (best {best_epoch})", flush=True)
|
||||
break
|
||||
model.load_state_dict(best_state)
|
||||
torch.save(best_state, os.path.join(MEASB, f"{name}_best.pth"))
|
||||
return model, {"best_epoch": best_epoch, "best_val_mpjpe": best_val,
|
||||
"epochs_run": len(history), "wall_seconds": round(time.time() - t0, 1),
|
||||
"history": history}
|
||||
|
||||
|
||||
def run_suite(tag, csi, kps, device):
|
||||
"""Temporal 70/15/15 split, mean-pose baseline, three training runs."""
|
||||
n = len(csi)
|
||||
tr, va, te = temporal_split(n)
|
||||
print(f"=== suite {tag}: n={n} train={tr.stop} val={va.stop - va.start} "
|
||||
f"test={te.stop - te.start} ===", flush=True)
|
||||
|
||||
# CSI normalization constant from TRAIN split only.
|
||||
train_p99 = float(np.percentile(csi[tr], 99))
|
||||
train_max = float(csi[tr].max())
|
||||
print(f"[{tag}] train p99={train_p99:.3f} max={train_max:.3f} -> /p99, clip [0,1]",
|
||||
flush=True)
|
||||
csi_n = np.clip(csi / train_p99, 0.0, 1.0).astype(np.float32)
|
||||
|
||||
x = torch.from_numpy(csi_n).to(device)
|
||||
y = torch.from_numpy(kps).to(device)
|
||||
x_tr, y_tr = x[tr], y[tr]
|
||||
x_va, y_va = x[va], y[va]
|
||||
x_te, y_te = x[te], y[te]
|
||||
|
||||
suite = {
|
||||
"n_windows": n,
|
||||
"split": {"n_train": int(tr.stop), "n_val": int(va.stop - va.start),
|
||||
"n_test": int(te.stop - te.start)},
|
||||
"csi_norm": {"method": "divide by train-split p99 amplitude, clip [0,1]",
|
||||
"train_p99": train_p99, "train_max": train_max},
|
||||
"runs": {},
|
||||
}
|
||||
|
||||
# Honesty bar: mean-pose predictor fit on TRAIN, evaluated on TEST.
|
||||
mean_pose = y_tr.mean(dim=0, keepdim=True).expand(len(y_te), -1, -1)
|
||||
suite["mean_pose_baseline"] = eval_preds(mean_pose, y_te)
|
||||
suite["mean_pose_baseline"]["note"] = "train-split mean pose; pred_std 0 by construction"
|
||||
print(f"[{tag}] mean-pose baseline:", json.dumps(suite["mean_pose_baseline"]),
|
||||
flush=True)
|
||||
|
||||
configs = [
|
||||
("pretrained", dict(pretrained=True, freeze_trunk=False, lr_trunk=LR_TRUNK_FT)),
|
||||
("scratch", dict(pretrained=False, freeze_trunk=False, lr_trunk=LR_ADAPTER)),
|
||||
("frozen_trunk", dict(pretrained=True, freeze_trunk=True, lr_trunk=0.0)),
|
||||
]
|
||||
for name, cfg in configs:
|
||||
print(f"=== run: {tag}/{name} {cfg} ===", flush=True)
|
||||
model, train_info = train_run(f"{tag}_{name}", x_tr, y_tr, x_va, y_va,
|
||||
device, **cfg)
|
||||
test_metrics = eval_preds(predict(model, x_te), y_te)
|
||||
n_trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
|
||||
suite["runs"][name] = {"config": cfg, "trainable_params": n_trainable,
|
||||
"train": {k: v for k, v in train_info.items()
|
||||
if k != "history"},
|
||||
"history": train_info["history"],
|
||||
"test": test_metrics}
|
||||
print(f"[{tag}/{name}] TEST:", json.dumps(test_metrics), flush=True)
|
||||
return suite
|
||||
|
||||
|
||||
def main():
|
||||
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||
print(f"device {device}, torch {torch.__version__}", flush=True)
|
||||
set_seed(SEED)
|
||||
|
||||
csi, kps, confs, native70, shape_counts, ts_first, ts_last = load_dataset()
|
||||
print(f"shape distribution: {shape_counts}", flush=True)
|
||||
|
||||
results = {
|
||||
"protocol": {
|
||||
"dataset": DATA, "n_windows": len(csi),
|
||||
"ts_first": ts_first, "ts_last": ts_last,
|
||||
"conf_mean": float(confs.mean()), "conf_min": float(confs.min()),
|
||||
"csi_shape_distribution": shape_counts,
|
||||
"csi_layout_note": "aligner stores frame-major data under a transposed "
|
||||
"[nSc, nFrames] shape label; corrected on load",
|
||||
"csi_resample": "per-frame linear interp of subcarrier axis to 70 bins "
|
||||
"(identity for native-70 frames); native-70 windows still "
|
||||
"contain ~20.4% internally zero-padded short frames",
|
||||
"split": "temporal 70/15/15 (no shuffle across time)",
|
||||
"model": "1x1 Conv1d 70->540 adapter + WiFlowPoseModel trunk, "
|
||||
"AdaptiveAvgPool2d((17,1)) head (parameter-free K=17)",
|
||||
"checkpoint": CHECKPOINT,
|
||||
"checkpoint_note": "measurement-(a) retrained checkpoint (~96% PCK@20 on "
|
||||
"WiFlow data); att./final_conv. remap applied "
|
||||
"defensively (no-op, already new-style keys)",
|
||||
"optimizer": f"AdamW, adapter lr {LR_ADAPTER}, fine-tuned trunk lr "
|
||||
f"{LR_TRUNK_FT} (10x lower), scratch all {LR_ADAPTER}",
|
||||
"batch": BATCH, "max_epochs": MAX_EPOCHS, "patience": PATIENCE,
|
||||
"precision": "fp32", "seed": SEED,
|
||||
"pck": "torso-normalized, torso = ||l_shoulder(5) - l_hip(11)||, "
|
||||
"clamp min 0.01, mean over keypoints x frames "
|
||||
"(upstream math; upstream 2/12 indices are a 15-kp convention)",
|
||||
},
|
||||
# Primary: all 2,046 windows (pre-registered n), subcarrier axis resampled.
|
||||
"all2046": None,
|
||||
# Secondary robustness check: the 1,347 native [70,20] windows only.
|
||||
"native70": None,
|
||||
}
|
||||
|
||||
results["all2046"] = run_suite("all2046", csi, kps, device)
|
||||
results["native70"] = run_suite("native70", csi[native70], kps[native70], device)
|
||||
|
||||
out = os.path.join(MEASB, "measurement_b.json")
|
||||
with open(out, "w") as f:
|
||||
json.dump(results, f, indent=2)
|
||||
print(f"wrote {out}", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,33 @@
|
||||
#!/bin/bash
|
||||
set -ex
|
||||
cd ~/wiflow-std-bench
|
||||
|
||||
# 1. clone upstream at the pinned commit
|
||||
if [ ! -d upstream ]; then
|
||||
git clone https://github.com/DY2434/WiFlow-WiFi-Pose-Estimation-with-Spatio-Temporal-Decoupling upstream
|
||||
fi
|
||||
cd upstream && git checkout 06899d294a0f44709d601a53e91dbf24759daefb && cd ..
|
||||
|
||||
# 2. documented deviation: fix upstream import bug (TemporalConvNet does not exist)
|
||||
sed -i 's/from .tcn import TemporalConvNet/from .tcn import TemporalBlock/; s/'"'"'TemporalConvNet'"'"'/'"'"'TemporalBlock'"'"'/' upstream/models/__init__.py
|
||||
|
||||
# 3. venv: torch cu128 (RTX 5080 = sm_120 needs >=2.7; their pin 2.3.1 predates Blackwell)
|
||||
if [ ! -d venv ]; then
|
||||
python3 -m venv venv
|
||||
./venv/bin/pip install -q --upgrade pip
|
||||
./venv/bin/pip install -q torch --index-url https://download.pytorch.org/whl/cu128
|
||||
./venv/bin/pip install -q numpy pandas matplotlib seaborn scikit-learn opencv-python-headless scipy tqdm psutil kagglehub
|
||||
fi
|
||||
./venv/bin/python -c "import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.get_device_name(0))"
|
||||
|
||||
# 4. dataset via kagglehub (anonymous, public dataset)
|
||||
DS=$(./venv/bin/python -c "import kagglehub; print(kagglehub.dataset_download('kaka2434/wiflow-dataset'))")
|
||||
echo "dataset at: $DS"
|
||||
|
||||
# 5. run.py hardcodes ../preprocessed_csi_data relative to upstream/
|
||||
ln -sfn "$DS/preprocessed_csi_data" ~/wiflow-std-bench/preprocessed_csi_data
|
||||
|
||||
# 6. train with upstream defaults (seed 42 set inside run.py)
|
||||
../venv/bin/python ../clean_nan.py 2>/dev/null || venv/bin/python clean_nan.py
|
||||
cd upstream
|
||||
../venv/bin/python run.py --gpu 0 --batch_size 64 --epochs 50 --output_dir ../train_output
|
||||
@@ -0,0 +1,332 @@
|
||||
"""Configurable compact variants of the WiFlow-STD pose model (ADR-152 efficiency sweep).
|
||||
|
||||
This is a parameterized copy of upstream models/{pose_model,tcn,convnet,attention}.py
|
||||
(DY2434/WiFlow @ 06899d29, Apache-2.0). upstream/ is NOT modified. Deviations from
|
||||
upstream, all forced by shrinking channels and documented per variant in run_sweep.py:
|
||||
|
||||
1. TCN grouped-conv groups: upstream hardcodes groups=20, which does not divide
|
||||
the compact channel counts (e.g. 270, 135, 85). Rule here:
|
||||
- groups_mode='gcd20': per-conv groups = gcd(channels, 20) (== 20 wherever
|
||||
upstream's choice is valid, incl. the 540-ch input conv; falls back to the
|
||||
largest common divisor with 20 otherwise).
|
||||
- groups_mode='depthwise': groups = channels (tiny variant only).
|
||||
2. Conv2d downsampling strides: upstream uses 4 stride-(1,2) blocks because
|
||||
240/2^4 = 15 == n_keypoints. With smaller TCN output widths that would leave
|
||||
<15 rows and AdaptiveAvgPool2d((15,1)) would duplicate rows across keypoints.
|
||||
Rule: halve the width only while the result stays >= 15 (stride-2 blocks
|
||||
first, stride-1 after). Full model: 240 -> 4 halvings = upstream exactly.
|
||||
3. input_pw_groups (tiny only): the dense 540->c pointwise + residual downsample
|
||||
in TCN block 1 cost 2*540*c params (a ~117k floor that alone exceeds the
|
||||
tiny <100k budget). tiny groups these two convs (groups=4; 4 | gcd(540, 68)).
|
||||
4. Decoder mid-channels: upstream 64->32; here c_last -> max(c_last // 2, 4).
|
||||
"""
|
||||
import math
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
|
||||
|
||||
def tcn_groups(channels: int, mode: str) -> int:
|
||||
if mode == 'depthwise':
|
||||
return channels
|
||||
if mode == 'gcd20':
|
||||
return math.gcd(channels, 20)
|
||||
raise ValueError(mode)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- TCN (copy of tcn.py)
|
||||
class Chomp1d(nn.Module):
|
||||
def __init__(self, chomp_size):
|
||||
super().__init__()
|
||||
self.chomp_size = chomp_size
|
||||
|
||||
def forward(self, x):
|
||||
return x[:, :, :-self.chomp_size].contiguous()
|
||||
|
||||
|
||||
class CompactGroupedTemporalBlock(nn.Module):
|
||||
"""Upstream InnerGroupedTemporalBlock with parameterized groups."""
|
||||
|
||||
def __init__(self, n_inputs, n_outputs, kernel_size, stride, dilation, padding,
|
||||
dropout=0.2, groups_mode='gcd20', pw_groups=1):
|
||||
super().__init__()
|
||||
g_in = tcn_groups(n_inputs, groups_mode)
|
||||
g_out = tcn_groups(n_outputs, groups_mode)
|
||||
self.groups = (g_in, g_out)
|
||||
self.pw_groups = pw_groups
|
||||
|
||||
self.conv1_group = nn.Conv1d(n_inputs, n_inputs, kernel_size, stride=stride,
|
||||
padding=padding, dilation=dilation,
|
||||
groups=g_in, bias=False)
|
||||
self.chomp1 = Chomp1d(padding) if padding > 0 else nn.Identity()
|
||||
self.bn1_group = nn.BatchNorm1d(n_inputs)
|
||||
self.relu1_group = nn.SiLU(inplace=True)
|
||||
|
||||
self.conv1_pw = nn.Conv1d(n_inputs, n_outputs, 1, groups=pw_groups, bias=False)
|
||||
self.bn1_pw = nn.BatchNorm1d(n_outputs)
|
||||
self.relu1_pw = nn.SiLU(inplace=True)
|
||||
self.dropout1 = nn.Dropout(dropout)
|
||||
|
||||
self.conv2_group = nn.Conv1d(n_outputs, n_outputs, kernel_size, stride=1,
|
||||
padding=padding, dilation=dilation,
|
||||
groups=g_out, bias=False)
|
||||
self.chomp2 = Chomp1d(padding) if padding > 0 else nn.Identity()
|
||||
self.bn2_group = nn.BatchNorm1d(n_outputs)
|
||||
self.relu2_group = nn.SiLU(inplace=True)
|
||||
|
||||
self.conv2_pw = nn.Conv1d(n_outputs, n_outputs, 1, bias=False)
|
||||
self.bn2_pw = nn.BatchNorm1d(n_outputs)
|
||||
self.relu2_pw = nn.SiLU(inplace=True)
|
||||
self.dropout2 = nn.Dropout(dropout)
|
||||
|
||||
self.downsample = nn.Sequential(
|
||||
nn.Conv1d(n_inputs, n_outputs, 1, groups=pw_groups, bias=False),
|
||||
nn.BatchNorm1d(n_outputs)
|
||||
) if n_inputs != n_outputs else nn.Identity()
|
||||
|
||||
def forward(self, x):
|
||||
res = self.downsample(x)
|
||||
out = self.conv1_group(x)
|
||||
out = self.chomp1(out)
|
||||
out = self.bn1_group(out)
|
||||
out = self.relu1_group(out)
|
||||
out = self.conv1_pw(out)
|
||||
out = self.bn1_pw(out)
|
||||
out = self.relu1_pw(out)
|
||||
out = self.dropout1(out)
|
||||
out = self.conv2_group(out)
|
||||
out = self.chomp2(out)
|
||||
out = self.bn2_group(out)
|
||||
out = self.relu2_group(out)
|
||||
out = self.conv2_pw(out)
|
||||
out = self.bn2_pw(out)
|
||||
out = self.relu2_pw(out)
|
||||
out = self.dropout2(out)
|
||||
return F.silu(out + res)
|
||||
|
||||
|
||||
class CompactTemporalBlock(nn.Module):
|
||||
def __init__(self, num_inputs, num_channels, kernel_size=3, dropout=0.2,
|
||||
groups_mode='gcd20', input_pw_groups=1):
|
||||
super().__init__()
|
||||
layers = []
|
||||
for i, out_channels in enumerate(num_channels):
|
||||
dilation_size = 2 ** i
|
||||
in_channels = num_inputs if i == 0 else num_channels[i - 1]
|
||||
layers.append(CompactGroupedTemporalBlock(
|
||||
in_channels, out_channels, kernel_size, stride=1,
|
||||
dilation=dilation_size, padding=(kernel_size - 1) * dilation_size,
|
||||
dropout=dropout, groups_mode=groups_mode,
|
||||
pw_groups=input_pw_groups if i == 0 else 1))
|
||||
self.network = nn.Sequential(*layers)
|
||||
|
||||
def forward(self, x):
|
||||
return self.network(x)
|
||||
|
||||
|
||||
# ------------------------------------------------------- Conv2d path (copy of convnet.py)
|
||||
class AsymmetricConvBlock(nn.Module):
|
||||
"""Upstream block with parameterized width stride (upstream: always (1,2))."""
|
||||
|
||||
def __init__(self, in_channels, out_channels, dropout=0.3, stride_w=2):
|
||||
super().__init__()
|
||||
self.block = nn.Sequential(
|
||||
nn.Conv2d(in_channels, out_channels, kernel_size=(1, 3),
|
||||
stride=(1, stride_w), padding=(0, 1)),
|
||||
nn.BatchNorm2d(out_channels),
|
||||
nn.SiLU(inplace=True),
|
||||
nn.Dropout2d(dropout),
|
||||
nn.Conv2d(out_channels, out_channels, kernel_size=(1, 3), padding=(0, 1)),
|
||||
nn.BatchNorm2d(out_channels),
|
||||
nn.SiLU(inplace=True),
|
||||
nn.Dropout2d(dropout),
|
||||
nn.Conv2d(out_channels, out_channels, kernel_size=(1, 3), padding=(0, 1)),
|
||||
nn.BatchNorm2d(out_channels)
|
||||
)
|
||||
self.downsample = nn.Sequential(
|
||||
nn.Conv2d(in_channels, out_channels, kernel_size=1,
|
||||
stride=(1, stride_w), bias=False),
|
||||
nn.BatchNorm2d(out_channels)
|
||||
)
|
||||
self.activation = nn.SiLU(inplace=True)
|
||||
|
||||
def forward(self, x):
|
||||
return self.activation(self.block(x) + self.downsample(x))
|
||||
|
||||
|
||||
class ConvBlock1(nn.Module):
|
||||
def __init__(self, in_channels, out_channels, dropout=0.3):
|
||||
super().__init__()
|
||||
self.block = nn.Sequential(
|
||||
nn.Conv2d(in_channels, out_channels, kernel_size=(1, 3), padding=(0, 1)),
|
||||
nn.BatchNorm2d(out_channels),
|
||||
nn.SiLU(inplace=True),
|
||||
nn.Dropout2d(dropout),
|
||||
nn.Conv2d(out_channels, out_channels, kernel_size=(1, 3), padding=(0, 1)),
|
||||
nn.BatchNorm2d(out_channels),
|
||||
nn.SiLU(inplace=True),
|
||||
nn.Dropout2d(dropout),
|
||||
nn.Conv2d(out_channels, out_channels, kernel_size=(1, 3), padding=(0, 1)),
|
||||
nn.BatchNorm2d(out_channels)
|
||||
)
|
||||
self.downsample = nn.Sequential(
|
||||
nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=1, bias=False),
|
||||
nn.BatchNorm2d(out_channels)
|
||||
)
|
||||
self.activation = nn.SiLU(inplace=True)
|
||||
|
||||
def forward(self, x):
|
||||
return self.activation(self.block(x) + self.downsample(x))
|
||||
|
||||
|
||||
# ----------------------------------------------------- attention (verbatim attention.py)
|
||||
class AxialAttention(nn.Module):
|
||||
def __init__(self, in_planes, out_planes, groups=8, stride=1, bias=False, width=False):
|
||||
assert (in_planes % groups == 0) and (out_planes % groups == 0)
|
||||
super().__init__()
|
||||
self.in_planes = in_planes
|
||||
self.out_planes = out_planes
|
||||
self.groups = groups
|
||||
self.group_planes = out_planes // groups
|
||||
self.stride = stride
|
||||
self.bias = bias
|
||||
self.width = width
|
||||
self.qkv_transform = nn.Conv1d(in_planes, out_planes * 3, kernel_size=1,
|
||||
stride=1, padding=0, bias=False)
|
||||
self.bn_qkv = nn.BatchNorm1d(out_planes * 3)
|
||||
self.bn_similarity = nn.BatchNorm2d(groups)
|
||||
self.bn_output = nn.BatchNorm1d(out_planes)
|
||||
if stride > 1:
|
||||
self.pooling = nn.AvgPool2d(stride, stride=stride)
|
||||
nn.init.normal_(self.qkv_transform.weight.data, 0, math.sqrt(1. / self.in_planes))
|
||||
|
||||
def forward(self, x):
|
||||
if self.width:
|
||||
x = x.permute(0, 2, 1, 3)
|
||||
else:
|
||||
x = x.permute(0, 3, 1, 2)
|
||||
N, W, C, H = x.shape
|
||||
x = x.contiguous().view(N * W, C, H)
|
||||
qkv = self.bn_qkv(self.qkv_transform(x))
|
||||
qkv = qkv.reshape(N * W, 3, self.out_planes, H).permute(1, 0, 2, 3)
|
||||
q, k, v = qkv[0], qkv[1], qkv[2]
|
||||
q = q.reshape(N * W, self.groups, self.group_planes, H)
|
||||
k = k.reshape(N * W, self.groups, self.group_planes, H)
|
||||
v = v.reshape(N * W, self.groups, self.group_planes, H)
|
||||
qk = torch.einsum('bgci, bgcj->bgij', q, k)
|
||||
qk = self.bn_similarity(qk)
|
||||
similarity = F.softmax(qk, dim=-1)
|
||||
sv = torch.einsum('bgij,bgcj->bgci', similarity, v)
|
||||
sv = sv.reshape(N * W, self.out_planes, H)
|
||||
out = self.bn_output(sv)
|
||||
out = out.view(N, W, self.out_planes, H)
|
||||
if self.width:
|
||||
out = out.permute(0, 2, 1, 3)
|
||||
else:
|
||||
out = out.permute(0, 2, 3, 1)
|
||||
if self.stride > 1:
|
||||
out = self.pooling(out)
|
||||
return out
|
||||
|
||||
|
||||
class DualAxialAttention(nn.Module):
|
||||
def __init__(self, in_planes, out_planes, groups=8, stride=1, bias=False):
|
||||
super().__init__()
|
||||
self.width_axis = AxialAttention(in_planes, out_planes, groups, stride, bias, width=True)
|
||||
self.height_axis = AxialAttention(out_planes, out_planes, groups, stride, bias, width=False)
|
||||
|
||||
def forward(self, x):
|
||||
return self.height_axis(self.width_axis(x))
|
||||
|
||||
|
||||
# --------------------------------------------------------------- full model
|
||||
def compute_strides(width: int, n_blocks: int, target: int = 15):
|
||||
"""Halve width while result stays >= target (upstream: 240 -> 4 halvings -> 15)."""
|
||||
strides = []
|
||||
for _ in range(n_blocks):
|
||||
nxt = (width + 1) // 2 # conv k=3 s=2 p=1: out = ceil(in/2)
|
||||
if nxt >= target:
|
||||
strides.append(2)
|
||||
width = nxt
|
||||
else:
|
||||
strides.append(1)
|
||||
return strides, width
|
||||
|
||||
|
||||
class CompactWiFlowPoseModel(nn.Module):
|
||||
"""Parameterized upstream WiFlowPoseModel.
|
||||
|
||||
Upstream config == tcn_channels=[540,440,340,240], conv_channels=[8,16,32,64],
|
||||
attn_groups=8, groups_mode='gcd20' (gcd(c,20)==20 for all upstream channels),
|
||||
input_pw_groups=1 -> identical architecture, 2,225,042 params.
|
||||
"""
|
||||
|
||||
def __init__(self, tcn_channels, conv_channels, attn_groups,
|
||||
groups_mode='gcd20', input_pw_groups=1, dropout=0.3,
|
||||
num_subcarriers=540, num_keypoints=15):
|
||||
super().__init__()
|
||||
self.tcn = CompactTemporalBlock(
|
||||
num_inputs=num_subcarriers, num_channels=tcn_channels, kernel_size=3,
|
||||
dropout=dropout, groups_mode=groups_mode, input_pw_groups=input_pw_groups)
|
||||
|
||||
self.up = ConvBlock1(1, conv_channels[0])
|
||||
|
||||
strides, self.final_width = compute_strides(
|
||||
tcn_channels[-1], len(conv_channels), target=num_keypoints)
|
||||
self.conv_strides = strides
|
||||
self.residual_blocks = nn.ModuleList()
|
||||
in_channels = conv_channels[0]
|
||||
for out_channels, s in zip(conv_channels, strides):
|
||||
self.residual_blocks.append(
|
||||
AsymmetricConvBlock(in_channels, out_channels, stride_w=s))
|
||||
in_channels = out_channels
|
||||
|
||||
c_last = conv_channels[-1]
|
||||
self.attention = DualAxialAttention(c_last, c_last, groups=attn_groups)
|
||||
|
||||
c_mid = max(c_last // 2, 4)
|
||||
self.decoder = nn.Sequential(
|
||||
nn.Conv2d(c_last, c_mid, kernel_size=3, padding=1),
|
||||
nn.BatchNorm2d(c_mid),
|
||||
nn.SiLU(inplace=True),
|
||||
nn.Conv2d(c_mid, 2, kernel_size=1),
|
||||
nn.BatchNorm2d(2),
|
||||
nn.SiLU(inplace=True)
|
||||
)
|
||||
self.avg_pool = nn.AdaptiveAvgPool2d((num_keypoints, 1))
|
||||
self._initialize_weights()
|
||||
|
||||
def _initialize_weights(self):
|
||||
for m in self.modules():
|
||||
if isinstance(m, nn.Conv1d):
|
||||
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
|
||||
if m.bias is not None:
|
||||
nn.init.constant_(m.bias, 0)
|
||||
elif isinstance(m, (nn.BatchNorm1d, nn.LayerNorm)):
|
||||
nn.init.constant_(m.weight, 1)
|
||||
nn.init.constant_(m.bias, 0)
|
||||
elif isinstance(m, nn.Linear):
|
||||
nn.init.xavier_normal_(m.weight)
|
||||
if m.bias is not None:
|
||||
nn.init.constant_(m.bias, 0)
|
||||
|
||||
def forward(self, x):
|
||||
# [B, 540, 20]
|
||||
x = self.tcn(x) # [B, C_tcn, 20]
|
||||
x = x.transpose(1, 2).unsqueeze(1) # [B, 1, 20, C_tcn]
|
||||
x = self.up(x)
|
||||
for block in self.residual_blocks:
|
||||
x = block(x) # [B, C_conv, 20, W']
|
||||
x = x.permute(0, 1, 3, 2) # [B, C_conv, W', 20]
|
||||
x = self.attention(x)
|
||||
x = self.decoder(x) # [B, 2, W', 20]
|
||||
x = self.avg_pool(x).squeeze(-1) # [B, 2, 15]
|
||||
return x.transpose(1, 2) # [B, 15, 2]
|
||||
|
||||
|
||||
def describe(model: 'CompactWiFlowPoseModel'):
|
||||
params = sum(p.numel() for p in model.parameters())
|
||||
tcn_g = [blk.groups for blk in model.tcn.network]
|
||||
return {'params': params, 'tcn_groups_per_block': tcn_g,
|
||||
'conv_strides': model.conv_strides, 'final_width': model.final_width}
|
||||
@@ -0,0 +1,278 @@
|
||||
"""WiFlow-STD compact-variant efficiency sweep (ADR-152) — sequential overnight runner.
|
||||
|
||||
Trains compact variants of the upstream WiFlow-STD architecture on the same
|
||||
data/split as the full-size reference retraining (seed 42, file-level 70/15/15,
|
||||
upstream dataset.py) and evaluates PCK@10..50 + MPJPE on the full test split and
|
||||
the corruption-free test subset (file indices < 487).
|
||||
|
||||
Training mirrors upstream run.py/train.py defaults except:
|
||||
- fp32 only (no fp16 autocast / GradScaler — avoids the BN-poisoning trap
|
||||
documented in RESULTS.md defect 5; data on disk is already cleaned).
|
||||
- batch 64 (kept modest: another GPU job may share the 16 GB card tonight).
|
||||
- scheduler + early stopping keyed on val MPJPE (upstream early-stops on val MPE
|
||||
with patience 5; same here).
|
||||
|
||||
Usage:
|
||||
venv/bin/python sweep/run_sweep.py --dry-run # param counts only
|
||||
nohup venv/bin/python sweep/run_sweep.py > sweep/sweep.log 2>&1 &
|
||||
|
||||
Idempotent: variants already present in sweep/results.jsonl are skipped.
|
||||
|
||||
NOTE: deployed to ruvultra (~/wiflow-std-bench/sweep) as a standalone file, so
|
||||
it deliberately inlines its helpers. The reference implementations (upstream
|
||||
import shim, >1GB np.load mmap patch, key-remap loader, canonical evaluate
|
||||
loop) live in benchmarks/wiflow-std/_bench_common.py — keep copies in sync.
|
||||
"""
|
||||
import argparse
|
||||
import copy
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from torch.utils.data import DataLoader, Subset
|
||||
|
||||
# csi_windows.npy is ~13 GB; mmap large arrays instead of eagerly loading
|
||||
# ~15 GB into RAM (same patch as _bench_common._np_load_mmap).
|
||||
_np_load = np.load
|
||||
|
||||
|
||||
def _np_load_mmap(path, *a, **kw):
|
||||
if (isinstance(path, str) and path.endswith('.npy')
|
||||
and os.path.getsize(path) > 1 << 30 and 'mmap_mode' not in kw):
|
||||
kw['mmap_mode'] = 'r'
|
||||
return _np_load(path, *a, **kw)
|
||||
|
||||
|
||||
np.load = _np_load_mmap
|
||||
|
||||
BENCH = os.path.expanduser('~/wiflow-std-bench')
|
||||
SWEEP = os.path.join(BENCH, 'sweep')
|
||||
sys.path.insert(0, os.path.join(BENCH, 'upstream'))
|
||||
sys.path.insert(0, SWEEP)
|
||||
|
||||
from dataset import PreprocessedCSIKeypointsDataset, create_preprocessed_train_val_test_loaders # noqa: E402
|
||||
from losses.pose_loss import PoseLoss # noqa: E402
|
||||
from utils.metrics import calculate_pck, calculate_mpjpe # noqa: E402
|
||||
from model_compact import CompactWiFlowPoseModel, describe # noqa: E402
|
||||
|
||||
VARIANTS = [
|
||||
# name, tcn_channels, conv_channels, attn_groups, groups_mode, input_pw_groups
|
||||
dict(name='half', tcn=[270, 220, 170, 120], conv=[4, 8, 16, 32], attn_groups=4,
|
||||
groups_mode='gcd20', input_pw_groups=1),
|
||||
dict(name='quarter', tcn=[135, 110, 85, 60], conv=[2, 4, 8, 16], attn_groups=2,
|
||||
groups_mode='gcd20', input_pw_groups=1),
|
||||
dict(name='tiny', tcn=[68, 56, 44, 32], conv=[2, 4, 8, 16], attn_groups=2,
|
||||
groups_mode='depthwise', input_pw_groups=4),
|
||||
]
|
||||
|
||||
BATCH = 64
|
||||
EPOCHS = 50
|
||||
PATIENCE = 5
|
||||
LR = 1e-4
|
||||
WEIGHT_DECAY = 5e-5
|
||||
SEED = 42
|
||||
CORRUPT_FILE_START = 487 # files 487-499 were zero-filled by clean_nan.py
|
||||
|
||||
|
||||
def set_seed(seed=SEED):
|
||||
random.seed(seed)
|
||||
np.random.seed(seed)
|
||||
torch.manual_seed(seed)
|
||||
torch.cuda.manual_seed_all(seed)
|
||||
torch.backends.cudnn.deterministic = True
|
||||
torch.backends.cudnn.benchmark = False
|
||||
|
||||
|
||||
def build_model(v, dropout=0.5):
|
||||
return CompactWiFlowPoseModel(
|
||||
tcn_channels=v['tcn'], conv_channels=v['conv'], attn_groups=v['attn_groups'],
|
||||
groups_mode=v['groups_mode'], input_pw_groups=v['input_pw_groups'],
|
||||
dropout=dropout)
|
||||
|
||||
|
||||
@torch.no_grad()
|
||||
def evaluate(model, loader, device):
|
||||
model.eval()
|
||||
totals = {t: 0.0 for t in (0.1, 0.2, 0.3, 0.4, 0.5)}
|
||||
total_mpe, n = 0.0, 0
|
||||
for bx, by in loader:
|
||||
bx, by = bx.to(device), by.to(device)
|
||||
out = model(bx)
|
||||
bs = by.size(0)
|
||||
total_mpe += calculate_mpjpe(out, by) * bs
|
||||
pck = calculate_pck(out, by, thresholds=list(totals))
|
||||
for t in totals:
|
||||
totals[t] += pck[t] * bs
|
||||
n += bs
|
||||
return {'samples': n, 'mpjpe': total_mpe / n,
|
||||
**{f'pck@{int(t * 100)}': totals[t] / n for t in totals}}
|
||||
|
||||
|
||||
def train_variant(v, dataset, device):
|
||||
set_seed(SEED)
|
||||
train_loader, val_loader, test_loader = create_preprocessed_train_val_test_loaders(
|
||||
dataset=dataset, batch_size=BATCH, num_workers=2, random_seed=SEED)
|
||||
|
||||
set_seed(SEED) # re-seed after split so init is split-independent
|
||||
model = build_model(v).to(device)
|
||||
info = describe(model)
|
||||
print(f"[{v['name']}] params={info['params']:,} tcn_groups={info['tcn_groups_per_block']} "
|
||||
f"conv_strides={info['conv_strides']} final_width={info['final_width']}", flush=True)
|
||||
|
||||
criterion = PoseLoss(position_weight=1.0, bone_weight=0.2, loss_type='smooth_l1')
|
||||
optimizer = torch.optim.AdamW(model.parameters(), lr=LR, weight_decay=WEIGHT_DECAY,
|
||||
betas=(0.9, 0.999))
|
||||
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
|
||||
optimizer, mode='min', factor=0.5, patience=3, min_lr=LR / 1000,
|
||||
cooldown=1, threshold=1e-4)
|
||||
|
||||
best_val_mpe = float('inf')
|
||||
best_val_pck20 = 0.0
|
||||
best_epoch = 0
|
||||
best_state = None
|
||||
patience_counter = 0
|
||||
t0 = time.time()
|
||||
error = None
|
||||
epochs_run = 0
|
||||
|
||||
for epoch in range(1, EPOCHS + 1):
|
||||
model.train()
|
||||
ep_loss, nb = 0.0, 0
|
||||
te = time.time()
|
||||
for i, (bx, by) in enumerate(train_loader):
|
||||
bx = bx.to(device, non_blocking=True)
|
||||
by = by.to(device, non_blocking=True)
|
||||
optimizer.zero_grad(set_to_none=True)
|
||||
out = model(bx)
|
||||
loss, _parts = criterion(out, by)
|
||||
if not torch.isfinite(loss):
|
||||
error = f'non-finite loss at epoch {epoch} step {i}'
|
||||
break
|
||||
loss.backward()
|
||||
optimizer.step()
|
||||
ep_loss += loss.item()
|
||||
nb += 1
|
||||
if epoch == 1 and i % 500 == 0:
|
||||
print(f"[{v['name']}] e1 step {i}/{len(train_loader)} loss={loss.item():.5f}",
|
||||
flush=True)
|
||||
if error:
|
||||
break
|
||||
epochs_run = epoch
|
||||
|
||||
val = evaluate(model, val_loader, device)
|
||||
scheduler.step(val['mpjpe'])
|
||||
lr_now = optimizer.param_groups[0]['lr']
|
||||
print(f"[{v['name']}] epoch {epoch}/{EPOCHS} train_loss={ep_loss / max(nb, 1):.5f} "
|
||||
f"val_mpjpe={val['mpjpe']:.5f} val_pck20={val['pck@20'] * 100:.2f}% "
|
||||
f"lr={lr_now:.2e} ({time.time() - te:.0f}s)", flush=True)
|
||||
|
||||
if val['mpjpe'] < best_val_mpe:
|
||||
best_val_mpe = val['mpjpe']
|
||||
best_val_pck20 = val['pck@20']
|
||||
best_epoch = epoch
|
||||
best_state = copy.deepcopy(model.state_dict())
|
||||
patience_counter = 0
|
||||
else:
|
||||
patience_counter += 1
|
||||
if patience_counter >= PATIENCE:
|
||||
print(f"[{v['name']}] early stop at epoch {epoch} (best {best_epoch})", flush=True)
|
||||
break
|
||||
|
||||
train_seconds = time.time() - t0
|
||||
result = {
|
||||
'variant': v['name'], 'params': info['params'],
|
||||
'tcn_channels': v['tcn'], 'conv_channels': v['conv'],
|
||||
'attn_groups': v['attn_groups'], 'groups_mode': v['groups_mode'],
|
||||
'input_pw_groups': v['input_pw_groups'],
|
||||
'tcn_groups_per_block': info['tcn_groups_per_block'],
|
||||
'conv_strides': info['conv_strides'], 'final_width': info['final_width'],
|
||||
'batch_size': BATCH, 'max_epochs': EPOCHS, 'patience': PATIENCE,
|
||||
'lr': LR, 'weight_decay': WEIGHT_DECAY, 'seed': SEED, 'precision': 'fp32',
|
||||
'epochs_run': epochs_run, 'best_epoch': best_epoch,
|
||||
'best_val_mpjpe': best_val_mpe if best_state else None,
|
||||
'best_val_pck20': best_val_pck20 if best_state else None,
|
||||
'train_seconds': round(train_seconds, 1),
|
||||
'torch': torch.__version__, 'error': error,
|
||||
'finished_utc': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
|
||||
}
|
||||
|
||||
if best_state is not None:
|
||||
ckpt = os.path.join(SWEEP, f"{v['name']}_best.pth")
|
||||
torch.save(best_state, ckpt)
|
||||
result['checkpoint'] = ckpt
|
||||
model.load_state_dict(best_state)
|
||||
|
||||
eval_loader = DataLoader(test_loader.dataset, batch_size=256, shuffle=False,
|
||||
num_workers=2)
|
||||
result['test_full'] = evaluate(model, eval_loader, device)
|
||||
|
||||
w2f = dataset.window_to_file
|
||||
clean_idx = [i for i in test_loader.dataset.indices if w2f[i] < CORRUPT_FILE_START]
|
||||
clean_loader = DataLoader(Subset(dataset, clean_idx), batch_size=256,
|
||||
shuffle=False, num_workers=2)
|
||||
result['test_clean'] = evaluate(model, clean_loader, device)
|
||||
print(f"[{v['name']}] TEST clean: pck20={result['test_clean']['pck@20'] * 100:.2f}% "
|
||||
f"mpjpe={result['test_clean']['mpjpe']:.5f} | full: "
|
||||
f"pck20={result['test_full']['pck@20'] * 100:.2f}%", flush=True)
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--dry-run', action='store_true', help='print param counts and exit')
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.dry_run:
|
||||
for v in VARIANTS:
|
||||
m = build_model(v)
|
||||
info = describe(m)
|
||||
x = torch.randn(2, 540, 20)
|
||||
m.eval()
|
||||
y = m(x)
|
||||
print(f"{v['name']:8s} params={info['params']:>9,} "
|
||||
f"tcn={v['tcn']} conv={v['conv']} attn_g={v['attn_groups']} "
|
||||
f"mode={v['groups_mode']} pw_g={v['input_pw_groups']} "
|
||||
f"tcn_groups={info['tcn_groups_per_block']} strides={info['conv_strides']} "
|
||||
f"W'={info['final_width']} out={tuple(y.shape)}")
|
||||
return
|
||||
|
||||
results_path = os.path.join(SWEEP, 'results.jsonl')
|
||||
done = set()
|
||||
if os.path.exists(results_path):
|
||||
with open(results_path) as f:
|
||||
for line in f:
|
||||
try:
|
||||
done.add(json.loads(line)['variant'])
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
device = torch.device('cuda')
|
||||
print(f"torch {torch.__version__} on {torch.cuda.get_device_name(0)}", flush=True)
|
||||
data_dir = os.path.join(BENCH, 'preprocessed_csi_data')
|
||||
dataset = PreprocessedCSIKeypointsDataset(data_dir=data_dir, keypoint_scale=1000.0,
|
||||
enable_temporal_clean=True)
|
||||
|
||||
for v in VARIANTS:
|
||||
if v['name'] in done:
|
||||
print(f"[{v['name']}] already in results.jsonl — skipping", flush=True)
|
||||
continue
|
||||
print(f"\n===== variant: {v['name']} =====", flush=True)
|
||||
try:
|
||||
result = train_variant(v, dataset, device)
|
||||
except Exception as e: # record and move on to next variant
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
result = {'variant': v['name'], 'error': repr(e),
|
||||
'finished_utc': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())}
|
||||
with open(results_path, 'a') as f:
|
||||
f.write(json.dumps(result) + '\n')
|
||||
f.flush()
|
||||
print('\nSWEEP COMPLETE', flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Binary file not shown.
@@ -0,0 +1,772 @@
|
||||
{
|
||||
"torch": {
|
||||
"env": {
|
||||
"torch": "2.12.0+cpu",
|
||||
"platform": "Windows-11-10.0.26200-SP0",
|
||||
"processor": "Intel64 Family 6 Model 197 Stepping 2, GenuineIntel",
|
||||
"num_threads": 16,
|
||||
"checkpoint": "results\\retrained_best_pose_model.pth",
|
||||
"params": 2225042
|
||||
},
|
||||
"variants": {
|
||||
"fp32": {
|
||||
"file": "retrained_fp32_resaved.pth",
|
||||
"size_bytes": 9068948,
|
||||
"size_mb": 9.068948,
|
||||
"latency_batch1": {
|
||||
"batch_size": 1,
|
||||
"runs": 100,
|
||||
"median_ms_per_batch": 24.903650000851485,
|
||||
"median_ms_per_window": 24.903650000851485,
|
||||
"windows_per_second": 40.15475642991324
|
||||
},
|
||||
"latency_batch64": {
|
||||
"batch_size": 64,
|
||||
"runs": 30,
|
||||
"median_ms_per_batch": 184.02919999789447,
|
||||
"median_ms_per_window": 2.875456249967101,
|
||||
"windows_per_second": 347.77089723115813
|
||||
},
|
||||
"accuracy": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.9668200004577636,
|
||||
"pck@50": 0.9915333324432373,
|
||||
"mpjpe": 0.00936222033649683,
|
||||
"wall_seconds": 37.85407733917236
|
||||
}
|
||||
},
|
||||
"fp16": {
|
||||
"file": "retrained_fp16.pth",
|
||||
"size_bytes": 4580332,
|
||||
"size_mb": 4.580332,
|
||||
"latency_batch1": {
|
||||
"batch_size": 1,
|
||||
"runs": 100,
|
||||
"median_ms_per_batch": 23.936699999467237,
|
||||
"median_ms_per_window": 23.936699999467237,
|
||||
"windows_per_second": 41.776853117691964
|
||||
},
|
||||
"latency_batch64": {
|
||||
"batch_size": 64,
|
||||
"runs": 30,
|
||||
"median_ms_per_batch": 102.32584999903338,
|
||||
"median_ms_per_window": 1.5988414062348966,
|
||||
"windows_per_second": 625.4529036465817
|
||||
},
|
||||
"accuracy": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.966773332977295,
|
||||
"pck@50": 0.9915066654205322,
|
||||
"mpjpe": 0.009460017587244511,
|
||||
"wall_seconds": 21.632277250289917
|
||||
}
|
||||
},
|
||||
"int8_dynamic": {
|
||||
"file": "retrained_int8_dynamic.pth",
|
||||
"size_bytes": 9068948,
|
||||
"size_mb": 9.068948,
|
||||
"latency_batch1": {
|
||||
"batch_size": 1,
|
||||
"runs": 100,
|
||||
"median_ms_per_batch": 18.105350000041653,
|
||||
"median_ms_per_window": 18.105350000041653,
|
||||
"windows_per_second": 55.23229321707117
|
||||
},
|
||||
"latency_batch64": {
|
||||
"batch_size": 64,
|
||||
"runs": 30,
|
||||
"median_ms_per_batch": 168.77549999844632,
|
||||
"median_ms_per_window": 2.6371171874757238,
|
||||
"windows_per_second": 379.20195763359703
|
||||
},
|
||||
"accuracy": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.9668200004577636,
|
||||
"pck@50": 0.9915333324432373,
|
||||
"mpjpe": 0.00936222033649683,
|
||||
"wall_seconds": 45.35376596450806
|
||||
}
|
||||
}
|
||||
},
|
||||
"int8_dynamic_quant_report": {
|
||||
"eligible_module_counts": {
|
||||
"nn.Linear": 0,
|
||||
"nn.Conv1d": 21,
|
||||
"nn.Conv2d": 22
|
||||
},
|
||||
"modules_actually_quantized": [],
|
||||
"n_modules_quantized": 0,
|
||||
"params_total": 2225042,
|
||||
"params_quantized": 0,
|
||||
"params_quantized_fraction": 0.0
|
||||
},
|
||||
"accuracy_subset": {
|
||||
"description": "seed-42 file-level 70/15/15 test split, corrupted windows (files 487-499) excluded, seed-42 random subset",
|
||||
"subset_size": 10000,
|
||||
"clean_test_total": 10000
|
||||
}
|
||||
},
|
||||
"onnx": {
|
||||
"env": {
|
||||
"torch": "2.12.0+cpu",
|
||||
"onnxruntime": "1.26.0",
|
||||
"platform": "Windows-11-10.0.26200-SP0"
|
||||
},
|
||||
"export": {
|
||||
"mode": "dynamic-batch",
|
||||
"exporter": "torchscript",
|
||||
"file": "retrained_fp32_dynamic.onnx",
|
||||
"size_mb": 8.971781
|
||||
},
|
||||
"parity": {
|
||||
"fixture": "results/parity_fixture.npz (batch 2, seed 42)",
|
||||
"max_abs_diff_vs_stored_fixture": 2.384185791015625e-07,
|
||||
"max_abs_diff_vs_torch_now": 2.384185791015625e-07,
|
||||
"pass_lt_1e-4": true
|
||||
},
|
||||
"latency": {
|
||||
"batch1": {
|
||||
"batch_size": 1,
|
||||
"runs": 100,
|
||||
"median_ms_per_batch": 2.5410999987798277,
|
||||
"median_ms_per_window": 2.5410999987798277,
|
||||
"windows_per_second": 393.5303610563043
|
||||
},
|
||||
"batch64": {
|
||||
"batch_size": 64,
|
||||
"runs": 30,
|
||||
"median_ms_per_batch": 181.95204999938142,
|
||||
"median_ms_per_window": 2.8430007812403346,
|
||||
"windows_per_second": 351.7410218803118
|
||||
}
|
||||
},
|
||||
"ort_int8_dynamic_supplementary": {
|
||||
"file": "retrained_int8_ort_dynamic.onnx",
|
||||
"size_mb": 2.438794,
|
||||
"runs": true,
|
||||
"max_abs_diff_vs_fp32_fixture": 0.00827130675315857
|
||||
}
|
||||
},
|
||||
"onnx_accuracy": {
|
||||
"onnx_fp32": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.9668200004577636,
|
||||
"pck@50": 0.9915333324432373,
|
||||
"mpjpe": 0.00936222568154335,
|
||||
"wall_seconds": 22.34790802001953
|
||||
},
|
||||
"onnx_int8_ort_dynamic": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.965240001964569,
|
||||
"pck@50": 0.9915466655731201,
|
||||
"mpjpe": 0.01108054072111845,
|
||||
"wall_seconds": 55.742953062057495
|
||||
}
|
||||
},
|
||||
"latency_controlled_rerun": {
|
||||
"note": "3 interleaved repetitions per variant, median ms/window; quiet box",
|
||||
"fp32": {
|
||||
"batch1_ms_per_window_median": 10.969150001983508,
|
||||
"batch1_reps": [
|
||||
10.969150001983508,
|
||||
12.646450000829645,
|
||||
10.49820000116597
|
||||
],
|
||||
"batch64_ms_per_window_median": 2.2734187500077496,
|
||||
"batch64_reps": [
|
||||
2.377234374989712,
|
||||
2.124126562478068,
|
||||
2.2734187500077496
|
||||
]
|
||||
},
|
||||
"fp16": {
|
||||
"batch1_ms_per_window_median": 24.313550000442774,
|
||||
"batch1_reps": [
|
||||
25.1078499986761,
|
||||
21.856999999727122,
|
||||
24.313550000442774
|
||||
],
|
||||
"batch64_ms_per_window_median": 2.414695312495496,
|
||||
"batch64_reps": [
|
||||
2.5705156249955508,
|
||||
1.7137437499741281,
|
||||
2.414695312495496
|
||||
]
|
||||
},
|
||||
"int8_dynamic": {
|
||||
"batch1_ms_per_window_median": 15.627150000000256,
|
||||
"batch1_reps": [
|
||||
17.67525000104797,
|
||||
14.627999998992891,
|
||||
15.627150000000256
|
||||
],
|
||||
"batch64_ms_per_window_median": 2.0546906250160646,
|
||||
"batch64_reps": [
|
||||
2.0546906250160646,
|
||||
2.03407343752815,
|
||||
2.9325796875241394
|
||||
]
|
||||
},
|
||||
"onnx_fp32": {
|
||||
"batch1_ms_per_window_median": 3.186650001225644,
|
||||
"batch1_reps": [
|
||||
2.7332500012562377,
|
||||
3.1995500012271805,
|
||||
3.186650001225644
|
||||
],
|
||||
"batch64_ms_per_window_median": 1.9893374999924163,
|
||||
"batch64_reps": [
|
||||
1.5590843750032946,
|
||||
1.9893374999924163,
|
||||
2.2144343749914697
|
||||
]
|
||||
},
|
||||
"onnx_int8_ort_dynamic": {
|
||||
"batch1_ms_per_window_median": 6.50984999811044,
|
||||
"batch1_reps": [
|
||||
6.50984999811044,
|
||||
6.455249998907675,
|
||||
6.789299999581999
|
||||
],
|
||||
"batch64_ms_per_window_median": 5.770093750015803,
|
||||
"batch64_reps": [
|
||||
5.770093750015803,
|
||||
3.912374999970325,
|
||||
7.8067296875019565
|
||||
]
|
||||
}
|
||||
},
|
||||
"onnx_static_ptq": {
|
||||
"env": {
|
||||
"onnxruntime": "1.26.0",
|
||||
"torch": "2.12.0+cpu",
|
||||
"platform": "Windows-11-10.0.26200-SP0",
|
||||
"source_model": "retrained_fp32_dynamic.onnx",
|
||||
"preprocessed_model": {
|
||||
"file": "retrained_fp32_preproc.onnx",
|
||||
"size_mb": 8.981529
|
||||
}
|
||||
},
|
||||
"variants": {
|
||||
"minmax_all": {
|
||||
"file": "retrained_int8_static_minmax_all.onnx",
|
||||
"size_bytes": 2604286,
|
||||
"size_mb": 2.604286,
|
||||
"calibration": {
|
||||
"method": "minmax",
|
||||
"windows": 1000,
|
||||
"percentile": null,
|
||||
"seconds": 5.052440166473389
|
||||
},
|
||||
"scope": "all",
|
||||
"per_channel": true,
|
||||
"activation_type": "QInt8",
|
||||
"weight_type": "QInt8",
|
||||
"node_counts": {
|
||||
"Add": 9,
|
||||
"AveragePool": 1,
|
||||
"BatchNormalization": 12,
|
||||
"Concat": 10,
|
||||
"Conv": 43,
|
||||
"DequantizeLinear": 283,
|
||||
"Einsum": 4,
|
||||
"Gather": 16,
|
||||
"Mul": 39,
|
||||
"QuantizeLinear": 181,
|
||||
"Reshape": 14,
|
||||
"Shape": 2,
|
||||
"Sigmoid": 37,
|
||||
"Slice": 8,
|
||||
"Softmax": 2,
|
||||
"Squeeze": 1,
|
||||
"Transpose": 7,
|
||||
"Unsqueeze": 11
|
||||
},
|
||||
"max_abs_diff_vs_fp32_fixture": 0.015945255756378174,
|
||||
"accuracy": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.9545266661643982,
|
||||
"pck@50": 0.9913666645050049,
|
||||
"mpjpe": 0.014860070134699345,
|
||||
"wall_seconds": 43.455235958099365
|
||||
}
|
||||
},
|
||||
"minmax_conv": {
|
||||
"file": "retrained_int8_static_minmax_conv.onnx",
|
||||
"size_bytes": 2527421,
|
||||
"size_mb": 2.527421,
|
||||
"calibration": {
|
||||
"method": "minmax",
|
||||
"windows": 1000,
|
||||
"percentile": null,
|
||||
"seconds": 4.380746126174927
|
||||
},
|
||||
"scope": "conv",
|
||||
"per_channel": true,
|
||||
"activation_type": "QInt8",
|
||||
"weight_type": "QInt8",
|
||||
"node_counts": {
|
||||
"Add": 9,
|
||||
"AveragePool": 1,
|
||||
"BatchNormalization": 12,
|
||||
"Concat": 10,
|
||||
"Conv": 43,
|
||||
"DequantizeLinear": 156,
|
||||
"Einsum": 4,
|
||||
"Gather": 16,
|
||||
"Mul": 39,
|
||||
"QuantizeLinear": 78,
|
||||
"Reshape": 14,
|
||||
"Shape": 2,
|
||||
"Sigmoid": 37,
|
||||
"Slice": 8,
|
||||
"Softmax": 2,
|
||||
"Squeeze": 1,
|
||||
"Transpose": 7,
|
||||
"Unsqueeze": 11
|
||||
},
|
||||
"max_abs_diff_vs_fp32_fixture": 0.010693132877349854,
|
||||
"accuracy": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.9663399996757507,
|
||||
"pck@50": 0.9918666641235352,
|
||||
"mpjpe": 0.01084446222037077,
|
||||
"wall_seconds": 35.937947034835815
|
||||
}
|
||||
},
|
||||
"entropy_all": {
|
||||
"file": "retrained_int8_static_entropy_all.onnx",
|
||||
"size_bytes": 2604268,
|
||||
"size_mb": 2.604268,
|
||||
"calibration": {
|
||||
"method": "entropy",
|
||||
"windows": 512,
|
||||
"percentile": null,
|
||||
"seconds": 23.835066318511963
|
||||
},
|
||||
"scope": "all",
|
||||
"per_channel": true,
|
||||
"activation_type": "QInt8",
|
||||
"weight_type": "QInt8",
|
||||
"node_counts": {
|
||||
"Add": 9,
|
||||
"AveragePool": 1,
|
||||
"BatchNormalization": 12,
|
||||
"Concat": 10,
|
||||
"Conv": 43,
|
||||
"DequantizeLinear": 283,
|
||||
"Einsum": 4,
|
||||
"Gather": 16,
|
||||
"Mul": 39,
|
||||
"QuantizeLinear": 181,
|
||||
"Reshape": 14,
|
||||
"Shape": 2,
|
||||
"Sigmoid": 37,
|
||||
"Slice": 8,
|
||||
"Softmax": 2,
|
||||
"Squeeze": 1,
|
||||
"Transpose": 7,
|
||||
"Unsqueeze": 11
|
||||
},
|
||||
"max_abs_diff_vs_fp32_fixture": 0.015280365943908691,
|
||||
"accuracy": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.9530466662406921,
|
||||
"pck@50": 0.9912600006103516,
|
||||
"mpjpe": 0.015098519864678382,
|
||||
"wall_seconds": 51.514281034469604
|
||||
}
|
||||
},
|
||||
"entropy_conv": {
|
||||
"file": "retrained_int8_static_entropy_conv.onnx",
|
||||
"size_bytes": 2527403,
|
||||
"size_mb": 2.527403,
|
||||
"calibration": {
|
||||
"method": "entropy",
|
||||
"windows": 512,
|
||||
"percentile": null,
|
||||
"seconds": 9.634419918060303
|
||||
},
|
||||
"scope": "conv",
|
||||
"per_channel": true,
|
||||
"activation_type": "QInt8",
|
||||
"weight_type": "QInt8",
|
||||
"node_counts": {
|
||||
"Add": 9,
|
||||
"AveragePool": 1,
|
||||
"BatchNormalization": 12,
|
||||
"Concat": 10,
|
||||
"Conv": 43,
|
||||
"DequantizeLinear": 156,
|
||||
"Einsum": 4,
|
||||
"Gather": 16,
|
||||
"Mul": 39,
|
||||
"QuantizeLinear": 78,
|
||||
"Reshape": 14,
|
||||
"Shape": 2,
|
||||
"Sigmoid": 37,
|
||||
"Slice": 8,
|
||||
"Softmax": 2,
|
||||
"Squeeze": 1,
|
||||
"Transpose": 7,
|
||||
"Unsqueeze": 11
|
||||
},
|
||||
"max_abs_diff_vs_fp32_fixture": 0.012535125017166138,
|
||||
"accuracy": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.9659599989891052,
|
||||
"pck@50": 0.9918666648864746,
|
||||
"mpjpe": 0.010778637571632861,
|
||||
"wall_seconds": 41.01180171966553
|
||||
}
|
||||
},
|
||||
"percentile_all": {
|
||||
"file": "retrained_int8_static_percentile_all.onnx",
|
||||
"size_bytes": 2604052,
|
||||
"size_mb": 2.604052,
|
||||
"calibration": {
|
||||
"method": "percentile",
|
||||
"windows": 512,
|
||||
"percentile": 99.99,
|
||||
"seconds": 20.221954584121704
|
||||
},
|
||||
"scope": "all",
|
||||
"per_channel": true,
|
||||
"activation_type": "QInt8",
|
||||
"weight_type": "QInt8",
|
||||
"node_counts": {
|
||||
"Add": 9,
|
||||
"AveragePool": 1,
|
||||
"BatchNormalization": 12,
|
||||
"Concat": 10,
|
||||
"Conv": 43,
|
||||
"DequantizeLinear": 283,
|
||||
"Einsum": 4,
|
||||
"Gather": 16,
|
||||
"Mul": 39,
|
||||
"QuantizeLinear": 181,
|
||||
"Reshape": 14,
|
||||
"Shape": 2,
|
||||
"Sigmoid": 37,
|
||||
"Slice": 8,
|
||||
"Softmax": 2,
|
||||
"Squeeze": 1,
|
||||
"Transpose": 7,
|
||||
"Unsqueeze": 11
|
||||
},
|
||||
"max_abs_diff_vs_fp32_fixture": 0.017689883708953857,
|
||||
"accuracy": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.9639333323478698,
|
||||
"pck@50": 0.9916799991607667,
|
||||
"mpjpe": 0.012176512064039708,
|
||||
"wall_seconds": 49.365190744400024
|
||||
}
|
||||
},
|
||||
"percentile_conv": {
|
||||
"file": "retrained_int8_static_percentile_conv.onnx",
|
||||
"size_bytes": 2527241,
|
||||
"size_mb": 2.527241,
|
||||
"calibration": {
|
||||
"method": "percentile",
|
||||
"windows": 512,
|
||||
"percentile": 99.99,
|
||||
"seconds": 8.223475694656372
|
||||
},
|
||||
"scope": "conv",
|
||||
"per_channel": true,
|
||||
"activation_type": "QInt8",
|
||||
"weight_type": "QInt8",
|
||||
"node_counts": {
|
||||
"Add": 9,
|
||||
"AveragePool": 1,
|
||||
"BatchNormalization": 12,
|
||||
"Concat": 10,
|
||||
"Conv": 43,
|
||||
"DequantizeLinear": 156,
|
||||
"Einsum": 4,
|
||||
"Gather": 16,
|
||||
"Mul": 39,
|
||||
"QuantizeLinear": 78,
|
||||
"Reshape": 14,
|
||||
"Shape": 2,
|
||||
"Sigmoid": 37,
|
||||
"Slice": 8,
|
||||
"Softmax": 2,
|
||||
"Squeeze": 1,
|
||||
"Transpose": 7,
|
||||
"Unsqueeze": 11
|
||||
},
|
||||
"max_abs_diff_vs_fp32_fixture": 0.014725983142852783,
|
||||
"accuracy": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.9660599988937378,
|
||||
"pck@50": 0.9916066654205322,
|
||||
"mpjpe": 0.010310938355326652,
|
||||
"wall_seconds": 36.89548587799072
|
||||
}
|
||||
}
|
||||
},
|
||||
"latency": {
|
||||
"note": "3 interleaved repetitions per variant, median ms/window; onnx_fp32 / onnx_int8_ort_dynamic are same-session references",
|
||||
"onnx_fp32": {
|
||||
"batch1_reps": [
|
||||
4.5327999996516155,
|
||||
2.535649999117595,
|
||||
2.167549997466267
|
||||
],
|
||||
"batch64_reps": [
|
||||
1.9354515624740998,
|
||||
2.4948054687854437,
|
||||
1.9334703125082342
|
||||
],
|
||||
"batch1_ms_per_window_median": 2.535649999117595,
|
||||
"batch64_ms_per_window_median": 1.9354515624740998
|
||||
},
|
||||
"onnx_int8_ort_dynamic": {
|
||||
"batch1_reps": [
|
||||
5.698599999959697,
|
||||
5.721350000385428,
|
||||
4.805099997611251
|
||||
],
|
||||
"batch64_reps": [
|
||||
4.096601562508795,
|
||||
4.857628124995017,
|
||||
4.583800000006022
|
||||
],
|
||||
"batch1_ms_per_window_median": 5.698599999959697,
|
||||
"batch64_ms_per_window_median": 4.583800000006022
|
||||
},
|
||||
"entropy_all": {
|
||||
"batch1_reps": [
|
||||
6.444149999879301,
|
||||
5.038299999796436,
|
||||
5.713200000172947
|
||||
],
|
||||
"batch64_reps": [
|
||||
4.149468750028973,
|
||||
3.437125000004926,
|
||||
4.410960937491382
|
||||
],
|
||||
"batch1_ms_per_window_median": 5.713200000172947,
|
||||
"batch64_ms_per_window_median": 4.149468750028973
|
||||
},
|
||||
"entropy_conv": {
|
||||
"batch1_reps": [
|
||||
4.874750000453787,
|
||||
5.169099998965976,
|
||||
5.236699998931726
|
||||
],
|
||||
"batch64_reps": [
|
||||
3.010160156236452,
|
||||
3.1175546875203963,
|
||||
3.516850781238645
|
||||
],
|
||||
"batch1_ms_per_window_median": 5.169099998965976,
|
||||
"batch64_ms_per_window_median": 3.1175546875203963
|
||||
},
|
||||
"percentile_all": {
|
||||
"batch1_reps": [
|
||||
5.184749999898486,
|
||||
5.2898499998264015,
|
||||
5.916899999647285
|
||||
],
|
||||
"batch64_reps": [
|
||||
4.305105468745296,
|
||||
4.460741406262514,
|
||||
4.184502343747454
|
||||
],
|
||||
"batch1_ms_per_window_median": 5.2898499998264015,
|
||||
"batch64_ms_per_window_median": 4.305105468745296
|
||||
},
|
||||
"percentile_conv": {
|
||||
"batch1_reps": [
|
||||
4.916449999655015,
|
||||
7.150899999032845,
|
||||
5.284949998895172
|
||||
],
|
||||
"batch64_reps": [
|
||||
3.855813281262499,
|
||||
4.688969531230214,
|
||||
5.220103124997877
|
||||
],
|
||||
"batch1_ms_per_window_median": 5.284949998895172,
|
||||
"batch64_ms_per_window_median": 4.688969531230214
|
||||
},
|
||||
"minmax_all": {
|
||||
"batch1_reps": [
|
||||
6.463300000177696,
|
||||
7.149449998905766,
|
||||
5.3209000016067876
|
||||
],
|
||||
"batch64_reps": [
|
||||
3.9251343750095202,
|
||||
4.033442187505898,
|
||||
3.428199218745931
|
||||
],
|
||||
"batch1_ms_per_window_median": 6.463300000177696,
|
||||
"batch64_ms_per_window_median": 3.9251343750095202
|
||||
},
|
||||
"minmax_conv": {
|
||||
"batch1_reps": [
|
||||
5.9961499991914025,
|
||||
5.236549999608542,
|
||||
4.854399998293957
|
||||
],
|
||||
"batch64_reps": [
|
||||
4.368359375007458,
|
||||
3.249617187492504,
|
||||
3.0238906249735464
|
||||
],
|
||||
"batch1_ms_per_window_median": 5.236549999608542,
|
||||
"batch64_ms_per_window_median": 3.249617187492504
|
||||
}
|
||||
},
|
||||
"accuracy_subset": {
|
||||
"description": "seed-42 file-level 70/15/15 test split, corrupted windows excluded, seed-42 random subset (same as quantize_bench/eval_ort_accuracy)",
|
||||
"subset_size": 10000
|
||||
}
|
||||
},
|
||||
"tiny_variant": {
|
||||
"env": {
|
||||
"torch": "2.12.0+cpu",
|
||||
"onnxruntime": "1.26.0",
|
||||
"platform": "Windows-11-10.0.26200-SP0",
|
||||
"num_threads": 16,
|
||||
"checkpoint": "results\\tiny_best.pth",
|
||||
"checkpoint_size_bytes": 340555,
|
||||
"params": 56290,
|
||||
"variant_config": {
|
||||
"tcn": [
|
||||
68,
|
||||
56,
|
||||
44,
|
||||
32
|
||||
],
|
||||
"conv": [
|
||||
2,
|
||||
4,
|
||||
8,
|
||||
16
|
||||
],
|
||||
"attn_groups": 2,
|
||||
"groups_mode": "depthwise",
|
||||
"input_pw_groups": 4
|
||||
}
|
||||
},
|
||||
"export": {
|
||||
"mode": "dynamic-batch",
|
||||
"exporter": "torchscript",
|
||||
"opset": 17,
|
||||
"file": "tiny_fp32_dynamic.onnx",
|
||||
"size_bytes": 295279,
|
||||
"size_mb": 0.295279,
|
||||
"verified_batches": [
|
||||
1,
|
||||
2,
|
||||
64
|
||||
],
|
||||
"note": "AdaptiveAvgPool2d((15,1)) replaced at export by an exact mean(-1) + constant averaging matmul (final_width 16 is not a multiple of 15, which the TorchScript exporter rejects); exactness proven by the parity check vs the original torch model"
|
||||
},
|
||||
"parity": {
|
||||
"fixture": "results/parity_fixture.npz input (batch 2, seed 42); reference output recomputed with the tiny torch model",
|
||||
"max_abs_diff_vs_torch": 1.4901161193847656e-07,
|
||||
"pass_lt_1e-4": true
|
||||
},
|
||||
"int8_static_percentile_conv": {
|
||||
"file": "tiny_int8_static_percentile_conv.onnx",
|
||||
"size_bytes": 248278,
|
||||
"size_mb": 0.248278,
|
||||
"calibration": {
|
||||
"method": "percentile",
|
||||
"percentile": 99.99,
|
||||
"windows": 512,
|
||||
"scope": "conv-only TRAIN-split corruption-free",
|
||||
"seconds": 1.5347836017608643
|
||||
},
|
||||
"per_channel": true,
|
||||
"activation_type": "QInt8",
|
||||
"weight_type": "QInt8",
|
||||
"max_abs_diff_vs_fp32_fixture": 0.018491357564926147
|
||||
},
|
||||
"latency": {
|
||||
"note": "3 interleaved repetitions per variant, median ms/window; full-model sessions are same-session references",
|
||||
"tiny_onnx_fp32": {
|
||||
"batch1_reps": [
|
||||
0.6312500008789357,
|
||||
0.6834500018157996,
|
||||
0.6595999984710943
|
||||
],
|
||||
"batch64_reps": [
|
||||
0.37747578119251557,
|
||||
0.24196640623586063,
|
||||
0.2314671875183194
|
||||
],
|
||||
"batch1_ms_per_window_median": 0.6595999984710943,
|
||||
"batch64_ms_per_window_median": 0.24196640623586063
|
||||
},
|
||||
"tiny_onnx_int8_static_percentile_conv": {
|
||||
"batch1_reps": [
|
||||
0.7988500001374632,
|
||||
0.9382499993080273,
|
||||
0.8451000030618161
|
||||
],
|
||||
"batch64_reps": [
|
||||
0.9211476562995813,
|
||||
1.3045390625165965,
|
||||
1.026230468767153
|
||||
],
|
||||
"batch1_ms_per_window_median": 0.8451000030618161,
|
||||
"batch64_ms_per_window_median": 1.026230468767153
|
||||
},
|
||||
"full_onnx_fp32_reference": {
|
||||
"batch1_reps": [
|
||||
2.267249998112675,
|
||||
2.80170000041835,
|
||||
2.132149998942623
|
||||
],
|
||||
"batch64_reps": [
|
||||
1.3050578124875756,
|
||||
1.4244992187855132,
|
||||
1.8014164062947202
|
||||
],
|
||||
"batch1_ms_per_window_median": 2.267249998112675,
|
||||
"batch64_ms_per_window_median": 1.4244992187855132
|
||||
},
|
||||
"full_onnx_int8_static_percentile_conv_reference": {
|
||||
"batch1_reps": [
|
||||
5.529599999135826,
|
||||
4.768399998283712,
|
||||
6.215800000063609
|
||||
],
|
||||
"batch64_reps": [
|
||||
3.815724218725336,
|
||||
3.1025562500417436,
|
||||
4.333318749957016
|
||||
],
|
||||
"batch1_ms_per_window_median": 5.529599999135826,
|
||||
"batch64_ms_per_window_median": 3.815724218725336
|
||||
}
|
||||
},
|
||||
"accuracy_subset": {
|
||||
"description": "seed-42 file-level 70/15/15 test split, corrupted windows excluded, seed-42 random subset (same as quantize_bench/eval_ort_accuracy/static_ptq_bench)",
|
||||
"subset_size": 10000
|
||||
},
|
||||
"accuracy": {
|
||||
"tiny_onnx_fp32": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.941106667804718,
|
||||
"pck@50": 0.99369333152771,
|
||||
"mpjpe": 0.012527281279861927,
|
||||
"wall_seconds": 10.927234888076782
|
||||
},
|
||||
"tiny_onnx_int8_static_percentile_conv": {
|
||||
"samples": 10000,
|
||||
"pck@20": 0.9268133331298828,
|
||||
"pck@50": 0.9932933319091797,
|
||||
"mpjpe": 0.014906252065300942,
|
||||
"wall_seconds": 12.320892333984375
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
{"variant": "half", "params": 843834, "tcn_channels": [270, 220, 170, 120], "conv_channels": [4, 8, 16, 32], "attn_groups": 4, "groups_mode": "gcd20", "input_pw_groups": 1, "tcn_groups_per_block": [[20, 10], [10, 20], [20, 10], [10, 20]], "conv_strides": [2, 2, 2, 1], "final_width": 15, "batch_size": 64, "max_epochs": 50, "patience": 5, "lr": 0.0001, "weight_decay": 5e-05, "seed": 42, "precision": "fp32", "epochs_run": 28, "best_epoch": 23, "best_val_mpjpe": 0.008576328293592842, "best_val_pck20": 0.9690593021534107, "train_seconds": 1346.4, "torch": "2.11.0+cu128", "error": null, "finished_utc": "2026-06-11T03:09:47Z", "checkpoint": "/home/ruvultra/wiflow-std-bench/sweep/half_best.pth", "test_full": {"samples": 54000, "mpjpe": 0.009419974447676428, "pck@10": 0.8740543655289544, "pck@20": 0.9610469643628156, "pck@30": 0.9813556064146537, "pck@40": 0.9896086878246731, "pck@50": 0.9934827546013726}, "test_clean": {"samples": 52560, "mpjpe": 0.008980081718602137, "pck@10": 0.8840944136840205, "pck@20": 0.9662253179869514, "pck@30": 0.9847971080282144, "pck@40": 0.9917795997050618, "pck@50": 0.9946956242600532}}
|
||||
{"variant": "quarter", "params": 338600, "tcn_channels": [135, 110, 85, 60], "conv_channels": [2, 4, 8, 16], "attn_groups": 2, "groups_mode": "gcd20", "input_pw_groups": 1, "tcn_groups_per_block": [[20, 5], [5, 10], [10, 5], [5, 20]], "conv_strides": [2, 2, 1, 1], "final_width": 15, "batch_size": 64, "max_epochs": 50, "patience": 5, "lr": 0.0001, "weight_decay": 5e-05, "seed": 42, "precision": "fp32", "epochs_run": 50, "best_epoch": 50, "best_val_mpjpe": 0.008780752391864856, "best_val_pck20": 0.9672531302240159, "train_seconds": 1754.4, "torch": "2.11.0+cu128", "error": null, "finished_utc": "2026-06-11T03:39:06Z", "checkpoint": "/home/ruvultra/wiflow-std-bench/sweep/quarter_best.pth", "test_full": {"samples": 54000, "mpjpe": 0.009705399298005634, "pck@10": 0.8646123917014511, "pck@20": 0.9553815319449813, "pck@30": 0.979827209190086, "pck@40": 0.9887037501511751, "pck@50": 0.9931309027671814}, "test_clean": {"samples": 52560, "mpjpe": 0.009279253277105465, "pck@10": 0.8742288637923323, "pck@20": 0.9605315079427745, "pck@30": 0.9833016723076865, "pck@40": 0.9908206971631566, "pck@50": 0.9942719799017071}}
|
||||
{"variant": "tiny", "params": 56290, "tcn_channels": [68, 56, 44, 32], "conv_channels": [2, 4, 8, 16], "attn_groups": 2, "groups_mode": "depthwise", "input_pw_groups": 4, "tcn_groups_per_block": [[540, 68], [68, 56], [56, 44], [44, 32]], "conv_strides": [2, 1, 1, 1], "final_width": 16, "batch_size": 64, "max_epochs": 50, "patience": 5, "lr": 0.0001, "weight_decay": 5e-05, "seed": 42, "precision": "fp32", "epochs_run": 50, "best_epoch": 47, "best_val_mpjpe": 0.012602971208592256, "best_val_pck20": 0.9397210340146666, "train_seconds": 1540.1, "torch": "2.11.0+cu128", "error": null, "finished_utc": "2026-06-11T04:04:50Z", "checkpoint": "/home/ruvultra/wiflow-std-bench/sweep/tiny_best.pth", "test_full": {"samples": 54000, "mpjpe": 0.012859782406853305, "pck@10": 0.7640358444319831, "pck@20": 0.9364815320968628, "pck@30": 0.9731568422317505, "pck@40": 0.9866444962642811, "pck@50": 0.992488939108672}, "test_clean": {"samples": 52560, "mpjpe": 0.012502924276904246, "pck@10": 0.770895526488985, "pck@20": 0.9411073559313967, "pck@30": 0.9764840687790962, "pck@40": 0.9886695077067278, "pck@50": 0.9936238432039409}}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"checkpoint": "/home/ruvultra/wiflow-std-bench/upstream/test/best_pose_model.pth",
|
||||
"test_full": {
|
||||
"samples": 54000,
|
||||
"mpjpe": 0.009834060806367133,
|
||||
"pck@10": 0.8686346120127925,
|
||||
"pck@20": 0.9608815324571398,
|
||||
"pck@30": 0.9789111610695168,
|
||||
"pck@40": 0.9857975759682832,
|
||||
"pck@50": 0.9898827553325229
|
||||
},
|
||||
"test_clean": {
|
||||
"samples": 52560,
|
||||
"mpjpe": 0.009432755044379373,
|
||||
"pck@10": 0.876996495807189,
|
||||
"pck@20": 0.9661454100405608,
|
||||
"pck@30": 0.9823453060205306,
|
||||
"pck@40": 0.987909734176537,
|
||||
"pck@50": 0.9911238361167036
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"published": {
|
||||
"pck@20": 0.9725,
|
||||
"pck@30": 0.9863,
|
||||
"pck@40": 0.9916,
|
||||
"pck@50": 0.9948,
|
||||
"mpjpe": 0.007
|
||||
},
|
||||
"params_millions": 2.225042,
|
||||
"data_dir": "C:\\Users\\ruv\\.cache\\kagglehub\\datasets\\kaka2434\\wiflow-dataset\\versions\\1\\preprocessed_csi_data",
|
||||
"device": "cpu",
|
||||
"test_full": {
|
||||
"samples": 54000,
|
||||
"mpjpe": NaN,
|
||||
"pck@10": 5.6790124349020145e-05,
|
||||
"pck@20": 0.0007876543271596785,
|
||||
"pck@30": 0.007780246982971827,
|
||||
"pck@40": 0.05529259262923841,
|
||||
"pck@50": 0.1542370371548114,
|
||||
"wall_seconds": 118.03756999969482
|
||||
},
|
||||
"test_drop_last": {
|
||||
"samples": 53952,
|
||||
"mpjpe": NaN,
|
||||
"pck@10": 5.6840649370682976e-05,
|
||||
"pck@20": 0.0007883550872372227,
|
||||
"pck@30": 0.007787168910892621,
|
||||
"pck@40": 0.055318307667895535,
|
||||
"pck@50": 0.15425316342412276,
|
||||
"wall_seconds": 120.87458372116089
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,333 @@
|
||||
"""ADR-152 edge optimization follow-up: ONNX Runtime STATIC post-training
|
||||
quantization (calibration-based QDQ) of the retrained WiFlow-STD model, to
|
||||
improve on the dynamic-int8 result (2.44 MB, PCK@20 96.52%, 6.5 ms/win b1).
|
||||
|
||||
Static PTQ pre-computes activation ranges from calibration data, so inference
|
||||
uses QLinearConv/QDQ kernels instead of dynamic ConvInteger -- typically both
|
||||
faster and (with good calibration) closer to fp32 accuracy.
|
||||
|
||||
Method:
|
||||
- Calibration set: corruption-free windows drawn ONLY from the seed-42
|
||||
file-level TRAINING split (same split as eval_repro.py; corrupted windows
|
||||
excluded via results/nan_windows_mask.npy | big_windows_mask.npy), chosen
|
||||
with np.random.default_rng(42). Never test windows.
|
||||
- quantize_static, QuantFormat.QDQ, per-channel int8 weights, int8
|
||||
activations; calibration methods MinMax / Entropy / Percentile(99.99);
|
||||
scopes "all" (ORT default op set) vs "conv" (op_types_to_quantize=
|
||||
["Conv"] -- leaves the attention path, which exports as Einsum/Softmax
|
||||
and elementwise ops, in fp32).
|
||||
- Model is pre-processed first (quant_pre_process: symbolic shape
|
||||
inference + ORT graph optimization, folds BatchNormalization into Conv).
|
||||
- Accuracy: identical protocol to eval_ort_accuracy.py -- the 10,000-window
|
||||
seed-42 subset of the corruption-free test split (PCK@20/50, MPJPE).
|
||||
- Latency: median ms/window at batch 1 (100 runs) and batch 64 (30 runs),
|
||||
3 interleaved repetitions across all variants (fp32 and dynamic-int8
|
||||
sessions included as same-session reference points).
|
||||
|
||||
Usage:
|
||||
PYTHONUTF8=1 .venv/Scripts/python.exe static_ptq_bench.py \
|
||||
[--data-dir <preprocessed_csi_data>] [--subset 10000]
|
||||
[--calib-minmax 1000] [--calib-hist 512] [--skip-accuracy]
|
||||
|
||||
Writes/merges into results/edge_optimization.json under key "onnx_static_ptq".
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import collections
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
|
||||
from _bench_common import RESULTS # noqa: E402
|
||||
# quantize_bench sets up upstream imports + the np.load mmap patch
|
||||
# (both via _bench_common.import_upstream)
|
||||
from quantize_bench import build_test_subset # noqa: E402
|
||||
import quantize_bench as qb # noqa: E402
|
||||
from eval_ort_accuracy import evaluate_ort # noqa: E402
|
||||
|
||||
FP32_ONNX = os.path.join(RESULTS, "retrained_fp32_dynamic.onnx")
|
||||
DYN_INT8_ONNX = os.path.join(RESULTS, "retrained_int8_ort_dynamic.onnx")
|
||||
PREPROC_ONNX = os.path.join(RESULTS, "retrained_fp32_preproc.onnx")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# calibration data: corruption-free TRAINING-split windows only
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def build_calibration_windows(data_dir, n_windows):
|
||||
"""Seed-42 file-level 70/15/15 TRAIN split (exactly as eval_repro.py),
|
||||
minus corrupted windows, then a seed-42 random draw of n_windows."""
|
||||
dataset = qb.PreprocessedCSIKeypointsDataset(
|
||||
data_dir=data_dir, keypoint_scale=1000.0, enable_temporal_clean=True)
|
||||
train_loader, _va, _te = qb.create_preprocessed_train_val_test_loaders(
|
||||
dataset=dataset, batch_size=64, num_workers=0, random_seed=42)
|
||||
train_indices = np.asarray(train_loader.dataset.indices)
|
||||
|
||||
corrupted = (np.load(os.path.join(RESULTS, "nan_windows_mask.npy"))
|
||||
| np.load(os.path.join(RESULTS, "big_windows_mask.npy")))
|
||||
clean = train_indices[~corrupted[train_indices]]
|
||||
print(f"train split: {len(train_indices)} windows, "
|
||||
f"{len(train_indices) - len(clean)} corrupted excluded, "
|
||||
f"{len(clean)} clean")
|
||||
|
||||
rng = np.random.default_rng(42)
|
||||
sel = np.sort(rng.choice(clean, size=n_windows, replace=False))
|
||||
xs = np.stack([dataset[int(i)][0].numpy() for i in sel]).astype(np.float32)
|
||||
print(f"calibration tensor: {xs.shape} from {n_windows} clean TRAIN windows")
|
||||
return xs
|
||||
|
||||
|
||||
def make_reader(windows, batch_size=64):
|
||||
from onnxruntime.quantization import CalibrationDataReader
|
||||
|
||||
class WindowReader(CalibrationDataReader):
|
||||
def __init__(self):
|
||||
self._batches = [windows[i:i + batch_size]
|
||||
for i in range(0, len(windows), batch_size)]
|
||||
self._it = iter(self._batches)
|
||||
|
||||
def get_next(self):
|
||||
b = next(self._it, None)
|
||||
return None if b is None else {"input": b}
|
||||
|
||||
def rewind(self):
|
||||
self._it = iter(self._batches)
|
||||
|
||||
def __len__(self):
|
||||
return len(self._batches)
|
||||
|
||||
return WindowReader()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# quantization variants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def preprocess_model():
|
||||
from onnxruntime.quantization.shape_inference import quant_pre_process
|
||||
quant_pre_process(FP32_ONNX, PREPROC_ONNX)
|
||||
return PREPROC_ONNX
|
||||
|
||||
|
||||
def quantize_variant(src, dst, method, scope, calib_windows):
|
||||
from onnxruntime.quantization import (CalibrationMethod, QuantFormat,
|
||||
QuantType, quantize_static)
|
||||
methods = {
|
||||
"minmax": CalibrationMethod.MinMax,
|
||||
"entropy": CalibrationMethod.Entropy,
|
||||
"percentile": CalibrationMethod.Percentile,
|
||||
}
|
||||
# NB: do NOT pass CalibMaxIntermediateOutputs -- in ORT 1.26 the MinMax
|
||||
# calibrater clears its buffer every N batches and then raises
|
||||
# "No data is collected" if the batch count is divisible by N.
|
||||
extra = {}
|
||||
if method == "percentile":
|
||||
extra["CalibPercentile"] = 99.99
|
||||
op_types = ["Conv"] if scope == "conv" else None
|
||||
|
||||
t0 = time.time()
|
||||
quantize_static(
|
||||
src, dst, make_reader(calib_windows),
|
||||
quant_format=QuantFormat.QDQ,
|
||||
op_types_to_quantize=op_types,
|
||||
per_channel=True,
|
||||
activation_type=QuantType.QInt8,
|
||||
weight_type=QuantType.QInt8,
|
||||
calibrate_method=methods[method],
|
||||
extra_options=extra,
|
||||
)
|
||||
secs = time.time() - t0
|
||||
|
||||
import onnx
|
||||
ops = collections.Counter(n.op_type for n in onnx.load(dst).graph.node)
|
||||
return {
|
||||
"file": os.path.basename(dst),
|
||||
"size_bytes": os.path.getsize(dst),
|
||||
"size_mb": os.path.getsize(dst) / 1e6,
|
||||
"calibration": {"method": method,
|
||||
"windows": int(len(calib_windows)),
|
||||
"percentile": extra.get("CalibPercentile"),
|
||||
"seconds": secs},
|
||||
"scope": scope,
|
||||
"per_channel": True,
|
||||
"activation_type": "QInt8",
|
||||
"weight_type": "QInt8",
|
||||
"node_counts": {k: v for k, v in sorted(ops.items())},
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# latency (3 interleaved reps, like the latency_controlled_rerun)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def ort_session(path):
|
||||
import onnxruntime as ort
|
||||
return ort.InferenceSession(path, providers=["CPUExecutionProvider"])
|
||||
|
||||
|
||||
def bench_ort(sess, batch, n_runs):
|
||||
rng = np.random.default_rng(123)
|
||||
x = rng.random((batch, 540, 20), dtype=np.float32)
|
||||
inp = sess.get_inputs()[0].name
|
||||
for _ in range(max(5, n_runs // 10)):
|
||||
sess.run(None, {inp: x})
|
||||
times = []
|
||||
for _ in range(n_runs):
|
||||
t0 = time.perf_counter()
|
||||
sess.run(None, {inp: x})
|
||||
times.append(time.perf_counter() - t0)
|
||||
return statistics.median(times) * 1e3 / batch # ms/window
|
||||
|
||||
|
||||
def interleaved_latency(sessions, reps=3, runs_b1=100, runs_b64=30):
|
||||
lat = {name: {"batch1_reps": [], "batch64_reps": []} for name in sessions}
|
||||
for rep in range(reps):
|
||||
for name, sess in sessions.items():
|
||||
lat[name]["batch1_reps"].append(bench_ort(sess, 1, runs_b1))
|
||||
lat[name]["batch64_reps"].append(bench_ort(sess, 64, runs_b64))
|
||||
print(f" rep {rep + 1}/{reps} {name}: "
|
||||
f"b1={lat[name]['batch1_reps'][-1]:.2f} "
|
||||
f"b64={lat[name]['batch64_reps'][-1]:.3f} ms/win", flush=True)
|
||||
for name in lat:
|
||||
lat[name]["batch1_ms_per_window_median"] = statistics.median(
|
||||
lat[name]["batch1_reps"])
|
||||
lat[name]["batch64_ms_per_window_median"] = statistics.median(
|
||||
lat[name]["batch64_reps"])
|
||||
return lat
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main():
|
||||
import onnxruntime
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--data-dir", default=os.path.join(
|
||||
os.path.expanduser("~"), ".cache", "kagglehub", "datasets", "kaka2434",
|
||||
"wiflow-dataset", "versions", "1", "preprocessed_csi_data"))
|
||||
parser.add_argument("--subset", type=int, default=10000)
|
||||
parser.add_argument("--calib-minmax", type=int, default=1000)
|
||||
parser.add_argument("--calib-hist", type=int, default=512,
|
||||
help="calibration windows for Entropy/Percentile "
|
||||
"(histogram calibraters hold all intermediate "
|
||||
"activations in RAM)")
|
||||
parser.add_argument("--skip-accuracy", action="store_true")
|
||||
parser.add_argument("--methods", default="minmax,entropy,percentile",
|
||||
help="comma list of calibration methods to (re)run; "
|
||||
"results merge into existing onnx_static_ptq")
|
||||
parser.add_argument("--out", default=os.path.join(RESULTS, "edge_optimization.json"))
|
||||
args = parser.parse_args()
|
||||
|
||||
results = {
|
||||
"env": {
|
||||
"onnxruntime": onnxruntime.__version__,
|
||||
"torch": torch.__version__,
|
||||
"platform": platform.platform(),
|
||||
"source_model": os.path.basename(FP32_ONNX),
|
||||
},
|
||||
"variants": {},
|
||||
}
|
||||
|
||||
# ---- calibration data (TRAIN split only) -------------------------------
|
||||
calib_mm = build_calibration_windows(args.data_dir, args.calib_minmax)
|
||||
calib_hist = calib_mm[:args.calib_hist]
|
||||
|
||||
# ---- preprocess + quantize ---------------------------------------------
|
||||
print("\n=== quant_pre_process (shape inference + graph optimization) ===")
|
||||
src = preprocess_model()
|
||||
results["env"]["preprocessed_model"] = {
|
||||
"file": os.path.basename(src),
|
||||
"size_mb": os.path.getsize(src) / 1e6,
|
||||
}
|
||||
|
||||
matrix = [(m, s) for m in args.methods.split(",")
|
||||
for s in ("all", "conv")]
|
||||
for method, scope in matrix:
|
||||
name = f"{method}_{scope}"
|
||||
dst = os.path.join(RESULTS, f"retrained_int8_static_{name}.onnx")
|
||||
calib = calib_mm if method == "minmax" else calib_hist
|
||||
print(f"\n=== quantize_static: {name} "
|
||||
f"({len(calib)} calib windows) ===", flush=True)
|
||||
try:
|
||||
results["variants"][name] = quantize_variant(
|
||||
src, dst, method, scope, calib)
|
||||
print(f" {results['variants'][name]['size_mb']:.3f} MB")
|
||||
except Exception as e: # noqa: BLE001
|
||||
results["variants"][name] = {"error": f"{type(e).__name__}: {e}"}
|
||||
print(f" FAILED: {e}")
|
||||
|
||||
# ---- fixture parity (sanity, batch 2) ----------------------------------
|
||||
fixture = np.load(os.path.join(RESULTS, "parity_fixture.npz"))
|
||||
fx, fy = fixture["input"], fixture["output"]
|
||||
sessions = {}
|
||||
for name, info in results["variants"].items():
|
||||
if "error" in info:
|
||||
continue
|
||||
path = os.path.join(RESULTS, info["file"])
|
||||
try:
|
||||
sess = ort_session(path)
|
||||
yq = sess.run(None, {sess.get_inputs()[0].name: fx})[0]
|
||||
info["max_abs_diff_vs_fp32_fixture"] = float(np.abs(yq - fy).max())
|
||||
sessions[name] = sess
|
||||
except Exception as e: # noqa: BLE001
|
||||
info["run_error"] = f"{type(e).__name__}: {e}"
|
||||
print("\nfixture max-abs-diff vs fp32:",
|
||||
{n: round(results["variants"][n].get("max_abs_diff_vs_fp32_fixture",
|
||||
float("nan")), 5)
|
||||
for n in results["variants"]})
|
||||
|
||||
# ---- latency: 3 interleaved reps incl. fp32 + dynamic-int8 reference ----
|
||||
print("\n=== latency (3 interleaved reps) ===")
|
||||
lat_sessions = {"onnx_fp32": ort_session(FP32_ONNX),
|
||||
"onnx_int8_ort_dynamic": ort_session(DYN_INT8_ONNX)}
|
||||
lat_sessions.update(sessions)
|
||||
results["latency"] = {
|
||||
"note": "3 interleaved repetitions per variant, median ms/window; "
|
||||
"onnx_fp32 / onnx_int8_ort_dynamic are same-session references",
|
||||
**interleaved_latency(lat_sessions),
|
||||
}
|
||||
|
||||
# ---- accuracy on the standard 10k corruption-free test subset ----------
|
||||
if not args.skip_accuracy:
|
||||
loader, n_clean = build_test_subset(args.data_dir, args.subset)
|
||||
results["accuracy_subset"] = {
|
||||
"description": "seed-42 file-level 70/15/15 test split, corrupted "
|
||||
"windows excluded, seed-42 random subset (same as "
|
||||
"quantize_bench/eval_ort_accuracy)",
|
||||
"subset_size": min(args.subset, n_clean) if args.subset else n_clean,
|
||||
}
|
||||
for name, sess in sessions.items():
|
||||
print(f"\n=== accuracy: {name} ===")
|
||||
results["variants"][name]["accuracy"] = evaluate_ort(
|
||||
sess, loader, name)
|
||||
print(json.dumps(results["variants"][name]["accuracy"], indent=2))
|
||||
|
||||
# ---- merge into edge_optimization.json ----------------------------------
|
||||
merged = {}
|
||||
if os.path.exists(args.out):
|
||||
with open(args.out) as f:
|
||||
merged = json.load(f)
|
||||
prev = merged.get("onnx_static_ptq")
|
||||
if prev: # nested merge so partial --methods reruns don't clobber
|
||||
prev["env"] = results["env"]
|
||||
prev["variants"].update(results["variants"])
|
||||
prev.setdefault("latency", {}).update(results["latency"])
|
||||
if "accuracy_subset" in results:
|
||||
prev["accuracy_subset"] = results["accuracy_subset"]
|
||||
else:
|
||||
merged["onnx_static_ptq"] = results
|
||||
with open(args.out, "w") as f:
|
||||
json.dump(merged, f, indent=2)
|
||||
print(f"\nwrote {args.out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,313 @@
|
||||
"""ADR-152 efficiency-sweep follow-up: edge pipeline for the TINY compact
|
||||
WiFlow-STD variant (56,290 params, results/tiny_best.pth, trained overnight
|
||||
2026-06-10/11 -- see RESULTS.md "Efficiency sweep").
|
||||
|
||||
Headline question: what does the smallest deployable WiFlow-class model look
|
||||
like (KB + ms + PCK)? Reuses the onnx_bench.py / static_ptq_bench.py
|
||||
machinery on the tiny checkpoint:
|
||||
|
||||
1. Load tiny_best.pth with remote/sweep/model_compact.py
|
||||
(depthwise TCN groups, input_pw_groups=4, conv [2,4,8,16], attn groups 2).
|
||||
2. Export ONNX: dynamic batch, opset 17, TorchScript exporter (dynamo=False)
|
||||
-- same recipe that worked for the full model; verified at batch 1/2/64.
|
||||
One forced deviation: tiny's stride schedule [2,1,1,1] leaves final_width
|
||||
16, and the TorchScript exporter cannot export AdaptiveAvgPool2d((15,1))
|
||||
when 15 is not a factor of the input height (the full model never hit
|
||||
this -- its width was exactly 15). The adaptive pool over a fixed-size
|
||||
feature map is a fixed linear map, so the export wrapper replaces it with
|
||||
an exact matmul equivalent (PyTorch adaptive-pool bin semantics:
|
||||
bin i averages rows floor(i*H/K)..ceil((i+1)*H/K)); the W axis (20->1,
|
||||
a factor) becomes mean(-1). Exactness is proven by the parity check
|
||||
below, which compares against the ORIGINAL torch model with the real
|
||||
AdaptiveAvgPool2d.
|
||||
3. Torch-vs-ORT parity on the stored fixture input
|
||||
(results/parity_fixture.npz, batch 2, seed 42 -- same 540x20 input layout;
|
||||
reference output recomputed with the tiny torch model). PASS < 1e-4.
|
||||
4. Static QDQ conv-only int8 (quant_pre_process + quantize_static,
|
||||
per-channel QInt8 weights+activations, Percentile(99.99) calibration on
|
||||
512 corruption-free TRAIN-split windows -- the winning recipe and
|
||||
calibration count from static_ptq_bench.py. 512, not "about 500":
|
||||
ORT 1.26's histogram collector np.asarray()'s the per-batch maxima, so
|
||||
the calibration count must be a multiple of the batch size 64 or the
|
||||
ragged last batch crashes it).
|
||||
5. Disk size + CPU latency b1/b64 (3 interleaved reps, median ms/window)
|
||||
for tiny fp32 + tiny int8, with the full-model ONNX fp32 + static-int8
|
||||
sessions interleaved as same-session references.
|
||||
6. Accuracy (PCK@20/50 + MPJPE) on the identical 10k-window seed-42
|
||||
corruption-free test subset for tiny fp32 + tiny int8.
|
||||
|
||||
Usage:
|
||||
PYTHONUTF8=1 .venv/Scripts/python.exe tiny_edge_bench.py \
|
||||
[--data-dir <preprocessed_csi_data>] [--subset 10000] [--calib 512]
|
||||
(--calib must be a multiple of 64; see step 4 above)
|
||||
|
||||
Writes/merges into results/edge_optimization.json under key "tiny_variant".
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import sys
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
RESULTS = os.path.join(HERE, "results")
|
||||
sys.path.insert(0, HERE)
|
||||
sys.path.insert(0, os.path.join(HERE, "remote", "sweep"))
|
||||
|
||||
# quantize_bench sets up upstream imports + the np.load mmap patch
|
||||
from quantize_bench import build_test_subset # noqa: E402
|
||||
from eval_ort_accuracy import evaluate_ort # noqa: E402
|
||||
from static_ptq_bench import ( # noqa: E402
|
||||
build_calibration_windows,
|
||||
interleaved_latency,
|
||||
make_reader,
|
||||
ort_session,
|
||||
)
|
||||
from model_compact import CompactWiFlowPoseModel, describe # noqa: E402
|
||||
|
||||
TINY_CKPT = os.path.join(RESULTS, "tiny_best.pth")
|
||||
TINY_FP32_ONNX = os.path.join(RESULTS, "tiny_fp32_dynamic.onnx")
|
||||
TINY_PREPROC_ONNX = os.path.join(RESULTS, "tiny_fp32_preproc.onnx")
|
||||
TINY_INT8_ONNX = os.path.join(RESULTS, "tiny_int8_static_percentile_conv.onnx")
|
||||
FULL_FP32_ONNX = os.path.join(RESULTS, "retrained_fp32_dynamic.onnx")
|
||||
FULL_INT8_ONNX = os.path.join(RESULTS, "retrained_int8_static_percentile_conv.onnx")
|
||||
|
||||
# Exact tiny config from remote/sweep/run_sweep.py VARIANTS (measured 56,290
|
||||
# params, clean-test PCK@20 94.11% -- results/efficiency_sweep.jsonl).
|
||||
TINY = dict(tcn=[68, 56, 44, 32], conv=[2, 4, 8, 16], attn_groups=2,
|
||||
groups_mode="depthwise", input_pw_groups=4)
|
||||
|
||||
|
||||
def load_tiny_model():
|
||||
model = CompactWiFlowPoseModel(
|
||||
tcn_channels=TINY["tcn"], conv_channels=TINY["conv"],
|
||||
attn_groups=TINY["attn_groups"], groups_mode=TINY["groups_mode"],
|
||||
input_pw_groups=TINY["input_pw_groups"], dropout=0.5)
|
||||
state = torch.load(TINY_CKPT, map_location="cpu", weights_only=True)
|
||||
model.load_state_dict(state, strict=True)
|
||||
model.eval()
|
||||
return model
|
||||
|
||||
|
||||
def adaptive_pool_matrix(h_in, h_out):
|
||||
"""Exact AdaptiveAvgPool1d as a (h_out, h_in) averaging matrix, using
|
||||
PyTorch's bin rule: bin i covers rows floor(i*h_in/h_out) ..
|
||||
ceil((i+1)*h_in/h_out)."""
|
||||
w = torch.zeros(h_out, h_in)
|
||||
for i in range(h_out):
|
||||
s = (i * h_in) // h_out
|
||||
e = -((-(i + 1) * h_in) // h_out) # ceil division
|
||||
w[i, s:e] = 1.0 / (e - s)
|
||||
return w
|
||||
|
||||
|
||||
class ExportWrapper(torch.nn.Module):
|
||||
"""CompactWiFlowPoseModel forward with the AdaptiveAvgPool2d((K,1))
|
||||
replaced by an exact fixed linear map (mean over the factor W axis, then
|
||||
a constant averaging matmul over the non-factor H axis) so the
|
||||
TorchScript ONNX exporter accepts it. Bit-equivalent up to float
|
||||
round-off; proven by the parity check against the original model."""
|
||||
|
||||
def __init__(self, m, num_keypoints=15):
|
||||
super().__init__()
|
||||
self.m = m
|
||||
self.register_buffer(
|
||||
"pool_w_t", adaptive_pool_matrix(m.final_width, num_keypoints).t())
|
||||
|
||||
def forward(self, x):
|
||||
m = self.m
|
||||
x = m.tcn(x)
|
||||
x = x.transpose(1, 2).unsqueeze(1)
|
||||
x = m.up(x)
|
||||
for block in m.residual_blocks:
|
||||
x = block(x)
|
||||
x = x.permute(0, 1, 3, 2)
|
||||
x = m.attention(x)
|
||||
x = m.decoder(x) # [B, 2, H=final_width, T=20]
|
||||
x = x.mean(-1) # W-axis pool (20 -> 1, a factor)
|
||||
x = x.matmul(self.pool_w_t) # exact adaptive H pool: [B, 2, K]
|
||||
return x.transpose(1, 2) # [B, K, 2]
|
||||
|
||||
|
||||
def export_onnx(model):
|
||||
"""Dynamic-batch TorchScript export (the recipe that worked for the full
|
||||
model in onnx_bench.py), verified at batch 1/2/64. Uses ExportWrapper
|
||||
(see docstring) because final_width 16 is not a multiple of 15."""
|
||||
wrapper = ExportWrapper(model).eval()
|
||||
x = torch.rand(2, 540, 20)
|
||||
with torch.no_grad():
|
||||
torch.onnx.export(
|
||||
wrapper, (x,), TINY_FP32_ONNX, opset_version=17,
|
||||
input_names=["input"], output_names=["output"], dynamo=False,
|
||||
dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}})
|
||||
sess = ort_session(TINY_FP32_ONNX)
|
||||
inp = sess.get_inputs()[0].name
|
||||
for b in (1, 2, 64):
|
||||
y = sess.run(None, {inp: np.zeros((b, 540, 20), dtype=np.float32)})[0]
|
||||
assert y.shape == (b, 15, 2), y.shape
|
||||
return {
|
||||
"mode": "dynamic-batch", "exporter": "torchscript", "opset": 17,
|
||||
"file": os.path.basename(TINY_FP32_ONNX),
|
||||
"size_bytes": os.path.getsize(TINY_FP32_ONNX),
|
||||
"size_mb": os.path.getsize(TINY_FP32_ONNX) / 1e6,
|
||||
"verified_batches": [1, 2, 64],
|
||||
"note": "AdaptiveAvgPool2d((15,1)) replaced at export by an exact "
|
||||
"mean(-1) + constant averaging matmul (final_width 16 is not "
|
||||
"a multiple of 15, which the TorchScript exporter rejects); "
|
||||
"exactness proven by the parity check vs the original torch "
|
||||
"model",
|
||||
}
|
||||
|
||||
|
||||
def quantize_tiny(calib_windows):
|
||||
"""quant_pre_process + static QDQ conv-only Percentile(99.99) int8 --
|
||||
the winning recipe from static_ptq_bench.py."""
|
||||
from onnxruntime.quantization import (CalibrationMethod, QuantFormat,
|
||||
QuantType, quantize_static)
|
||||
from onnxruntime.quantization.shape_inference import quant_pre_process
|
||||
|
||||
quant_pre_process(TINY_FP32_ONNX, TINY_PREPROC_ONNX)
|
||||
t0 = time.time()
|
||||
quantize_static(
|
||||
TINY_PREPROC_ONNX, TINY_INT8_ONNX, make_reader(calib_windows),
|
||||
quant_format=QuantFormat.QDQ,
|
||||
op_types_to_quantize=["Conv"],
|
||||
per_channel=True,
|
||||
activation_type=QuantType.QInt8,
|
||||
weight_type=QuantType.QInt8,
|
||||
calibrate_method=CalibrationMethod.Percentile,
|
||||
extra_options={"CalibPercentile": 99.99},
|
||||
)
|
||||
return {
|
||||
"file": os.path.basename(TINY_INT8_ONNX),
|
||||
"size_bytes": os.path.getsize(TINY_INT8_ONNX),
|
||||
"size_mb": os.path.getsize(TINY_INT8_ONNX) / 1e6,
|
||||
"calibration": {"method": "percentile", "percentile": 99.99,
|
||||
"windows": int(len(calib_windows)),
|
||||
"scope": "conv-only TRAIN-split corruption-free",
|
||||
"seconds": time.time() - t0},
|
||||
"per_channel": True,
|
||||
"activation_type": "QInt8",
|
||||
"weight_type": "QInt8",
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
import onnxruntime
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--data-dir", default=os.path.join(
|
||||
os.path.expanduser("~"), ".cache", "kagglehub", "datasets", "kaka2434",
|
||||
"wiflow-dataset", "versions", "1", "preprocessed_csi_data"))
|
||||
parser.add_argument("--subset", type=int, default=10000)
|
||||
parser.add_argument("--calib", type=int, default=512,
|
||||
help="calibration windows; must be a multiple of the "
|
||||
"64-window calibration batch (ORT histogram "
|
||||
"collector rejects ragged batches)")
|
||||
parser.add_argument("--skip-accuracy", action="store_true")
|
||||
parser.add_argument("--out", default=os.path.join(RESULTS, "edge_optimization.json"))
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.calib % 64 != 0:
|
||||
parser.error(
|
||||
f"--calib must be a multiple of 64 (got {args.calib}): ORT 1.26's "
|
||||
f"histogram calibration collector np.asarray()'s the per-batch "
|
||||
f"maxima and crashes on a ragged final batch (calibration batch "
|
||||
f"size is 64)")
|
||||
|
||||
model = load_tiny_model()
|
||||
info = describe(model)
|
||||
print(f"tiny model: {info['params']:,} params, tcn_groups={info['tcn_groups_per_block']}, "
|
||||
f"strides={info['conv_strides']}, final_width={info['final_width']}")
|
||||
assert info["params"] == 56290, info["params"]
|
||||
|
||||
results = {
|
||||
"env": {
|
||||
"torch": torch.__version__,
|
||||
"onnxruntime": onnxruntime.__version__,
|
||||
"platform": platform.platform(),
|
||||
"num_threads": torch.get_num_threads(),
|
||||
"checkpoint": os.path.relpath(TINY_CKPT, HERE),
|
||||
"checkpoint_size_bytes": os.path.getsize(TINY_CKPT),
|
||||
"params": info["params"],
|
||||
"variant_config": TINY,
|
||||
},
|
||||
}
|
||||
|
||||
# ---- export + parity ----------------------------------------------------
|
||||
print("\n=== ONNX export (dynamic batch, opset 17, torchscript) ===")
|
||||
results["export"] = export_onnx(model)
|
||||
print(f" {results['export']['size_mb']:.3f} MB, batches {results['export']['verified_batches']} OK")
|
||||
|
||||
fixture = np.load(os.path.join(RESULTS, "parity_fixture.npz"))
|
||||
fx = fixture["input"] # (2, 540, 20), seed 42 -- same input layout as full model
|
||||
sess_fp32 = ort_session(TINY_FP32_ONNX)
|
||||
y_ort = sess_fp32.run(None, {sess_fp32.get_inputs()[0].name: fx})[0]
|
||||
with torch.no_grad():
|
||||
y_torch = model(torch.from_numpy(fx)).numpy()
|
||||
results["parity"] = {
|
||||
"fixture": "results/parity_fixture.npz input (batch 2, seed 42); "
|
||||
"reference output recomputed with the tiny torch model",
|
||||
"max_abs_diff_vs_torch": float(np.abs(y_ort - y_torch).max()),
|
||||
"pass_lt_1e-4": bool(np.abs(y_ort - y_torch).max() < 1e-4),
|
||||
}
|
||||
print("parity:", json.dumps(results["parity"], indent=2))
|
||||
assert results["parity"]["pass_lt_1e-4"], "torch-vs-ORT parity FAILED"
|
||||
|
||||
# ---- static PTQ int8 ------------------------------------------------------
|
||||
print(f"\n=== static QDQ int8 (Percentile conv-only, {args.calib} calib windows) ===")
|
||||
calib = build_calibration_windows(args.data_dir, args.calib)
|
||||
results["int8_static_percentile_conv"] = quantize_tiny(calib)
|
||||
print(f" {results['int8_static_percentile_conv']['size_mb']:.3f} MB")
|
||||
sess_int8 = ort_session(TINY_INT8_ONNX)
|
||||
yq = sess_int8.run(None, {sess_int8.get_inputs()[0].name: fx})[0]
|
||||
results["int8_static_percentile_conv"]["max_abs_diff_vs_fp32_fixture"] = float(
|
||||
np.abs(yq - y_torch).max())
|
||||
|
||||
# ---- latency (3 interleaved reps, full-model sessions as references) -----
|
||||
print("\n=== latency (3 interleaved reps) ===")
|
||||
lat_sessions = {
|
||||
"tiny_onnx_fp32": sess_fp32,
|
||||
"tiny_onnx_int8_static_percentile_conv": sess_int8,
|
||||
"full_onnx_fp32_reference": ort_session(FULL_FP32_ONNX),
|
||||
"full_onnx_int8_static_percentile_conv_reference": ort_session(FULL_INT8_ONNX),
|
||||
}
|
||||
results["latency"] = {
|
||||
"note": "3 interleaved repetitions per variant, median ms/window; "
|
||||
"full-model sessions are same-session references",
|
||||
**interleaved_latency(lat_sessions),
|
||||
}
|
||||
|
||||
# ---- accuracy on the standard 10k corruption-free test subset ------------
|
||||
if not args.skip_accuracy:
|
||||
loader, n_clean = build_test_subset(args.data_dir, args.subset)
|
||||
results["accuracy_subset"] = {
|
||||
"description": "seed-42 file-level 70/15/15 test split, corrupted "
|
||||
"windows excluded, seed-42 random subset (same as "
|
||||
"quantize_bench/eval_ort_accuracy/static_ptq_bench)",
|
||||
"subset_size": min(args.subset, n_clean) if args.subset else n_clean,
|
||||
}
|
||||
results["accuracy"] = {}
|
||||
for name, sess in (("tiny_onnx_fp32", sess_fp32),
|
||||
("tiny_onnx_int8_static_percentile_conv", sess_int8)):
|
||||
print(f"\n=== accuracy: {name} ===")
|
||||
results["accuracy"][name] = evaluate_ort(sess, loader, name)
|
||||
print(json.dumps(results["accuracy"][name], indent=2))
|
||||
|
||||
# ---- merge into edge_optimization.json -----------------------------------
|
||||
merged = {}
|
||||
if os.path.exists(args.out):
|
||||
with open(args.out) as f:
|
||||
merged = json.load(f)
|
||||
merged["tiny_variant"] = results
|
||||
with open(args.out, "w") as f:
|
||||
json.dump(merged, f, indent=2)
|
||||
print(f"\nwrote {args.out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user