diff --git a/benchmarks/wiflow-std/RESULTS.md b/benchmarks/wiflow-std/RESULTS.md index a1906c53..673df59e 100644 --- a/benchmarks/wiflow-std/RESULTS.md +++ b/benchmarks/wiflow-std/RESULTS.md @@ -264,9 +264,117 @@ same split. WiFlow-STD assets stand ready on ruvultra (`~/wiflow-std-bench/`). Also worth investigating: ADR-079's protocol predicts ~9k windows per 30 min; the May session under-delivered ~8× (aligner drop rate?). +## Measurement (b) (MEASURED 2026-06-10/11) + +The data baseline unblocked: the 2026-06-10 22:10–22:40 collection session produced +**2,046 paired windows** (`ruvultra:~/wiflow-std-bench/paired-20260610.jsonl`; ONE +subject, ONE room, ONE ESP32 node, varied poses: walk/raise/squat/kick/wave/turn/ +jump/sit; aligner `scripts/align-ground-truth.js`, non-overlapping 20-frame windows +~0.42 s; 17 COCO keypoints in normalized [0,1] camera coords; MediaPipe confidence +mean 0.802, min 0.692 — all windows pass the conf>0.5 filter). The −4 h timestamp +bug and the empty-frame confidence-dilution aligner findings are recorded +separately; results only here. Trained on ruvultra (RTX 5080, torch 2.11+cu128, +fp32, batch 32, GPU shared with the efficiency sweep). Scripts mirrored in +`remote/measb/`; raw metrics + full training curves in `results/measurement_b.json`. + +### Two new aligner/dataset findings (forced deviations, MEASURED) + +1. **`csi_shape` is heterogeneous, not [70, 20]**: 1,347× [70,20], 284× [134,20], + 243× [26,20], 130× [12,20], 42× [20,20]. The ESP32 stream emits mixed frame + types and `extractCsiMatrix` stamps each window's subcarrier count from + `window[0].subcarriers`, zero-padding/truncating the other frames — even + native-70 windows contain ~20.4% internally zero-padded short frames + (subcarriers 40–69 all-zero). Handling: the primary suite ("all 2,046") + linearly resamples every frame's subcarrier axis to 70 bins (identity for + native-70 frames) so the pre-registered n and split sizes hold; a secondary + suite restricts to the 1,347 native [70,20] windows as a homogeneity check. +2. **Aligner layout bug**: `extractCsiMatrix` fills `matrix[f * nSc + s]` + (frame-major) but declares `shape: [nSc, nFrames]` — the stored shape label is + transposed relative to the data. Confirmed by coherent per-frame zero-tails; + corrected on load (`reshape(nFrames, nSc).T`). + +### Protocol (pre-registered, followed) + +Temporal split, no shuffling across time: first 70% train (1,432), next 15% val +(307), last 15% test (307); seed 42 elsewhere. Model: learned 1×1 Conv1d 70→540 +adapter prepended to the upstream WiFlow-STD trunk; K=17 via the parameter-free +adaptive pool (`AdaptiveAvgPool2d((17,1))` — pretrained weights load strict for +any K). CSI normalized by the TRAIN-split p99 amplitude (129.7 all / 130.9 +native-70), clipped to [0,1]. Three runs, ≤60 epochs, early-stop patience 8 on +val MPJPE, AdamW (adapter lr 1e-4; pretrained trunk lr 1e-5, 10× lower; scratch +all 1e-4), fp32. Pretrained init = the measurement-(a) **retrained** checkpoint +(`upstream/test/best_pose_model.pth`, ~96% PCK@20 on WiFlow data; the +`att.`/`final_conv.` key remap from `eval_repro.py` applied defensively — a no-op, +that checkpoint already uses post-rename keys). Frozen-trunk run: trunk +`requires_grad=False` **and** held in `.eval()` so BatchNorm running stats cannot +drift — a pure transfer probe; only the 70→540 adapter (38,340 params) trains. + +PCK is torso-normalized with **torso = ‖l_shoulder(5) − l_hip(11)‖** (upstream +`calculate_pck` math — per-frame norm clamped at 0.01, mean over keypoints × +frames — but upstream's `NECK_IDX/PELVIS_IDX = 2, 12` is a 15-keypoint +convention; on 17-kp COCO those indices are right_eye/right_hip, so the indices +were replaced, not the math). MPJPE is in normalized image units (not meters). + +### Results — primary suite, all 2,046 windows (test = last 307) + +| Run | PCK@10 | PCK@20 | PCK@30 | PCK@40 | PCK@50 | MPJPE | pred std | best ep | +|---|---|---|---|---|---|---|---|---| +| **mean-pose baseline** (honesty bar) | **73.1%** | **95.9%** | **98.7%** | 99.3% | 99.3% | **0.0148** | 0 (by constr.) | — | +| (i) pretrained-init, full fine-tune | 26.0% | 65.0% | 88.0% | 96.4% | 98.9% | 0.0313 | 0.0113 | 58/60 | +| (ii) scratch | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.2554 | 0.0002 | 4 (stop @13) | +| (iii) frozen-trunk (adapter only) | 0.0% | 0.0% | 0.2% | 3.2% | 14.4% | 0.1260 | 0.0073 | 59/60 | + +Secondary suite (native [70,20] windows only, n=1,347, test=202) reproduces the +same ordering: mean-baseline 96.0% / pretrained 67.1% / scratch 0.0% / +frozen-trunk 0.0% PCK@20 (MPJPE 0.0153 / 0.0318 / 0.2236 / 0.1343) — the +subcarrier-resampling choice does not change any conclusion. + +### Interpretation + +- **Did pretraining-transfer happen? Partially — as optimization transfer, not + feature transfer, and not past the honesty bar.** + - *Pretrained vs scratch*: dramatic (65.0% vs 0.0% PCK@20). The pretrained init + is the only configuration that trains at all under the pre-registered budget. + - *Frozen-trunk*: near-zero (0.0% PCK@20, 14.4% @50). WiFlow-STD's frozen + features do **not** transfer to our ESP32 domain through a linear subcarrier + adapter — the pretrained benefit is a well-conditioned initialization (incl. + calibrated BN/output scales), not reusable CSI→pose features. + - *Everything vs mean-pose baseline*: **no run beats it.** A constant + train-mean pose scores 95.9% torso-PCK@20 / 0.0148 MPJPE on this test split, + because a single subject in one camera frame barely moves in normalized + coordinates. The fine-tuned model is a real, non-constant model + (pred std 0.0113 > 0 — passes the constant-pose detector that retracted the + old 92.9% figure) but its deviations from the mean hurt: it fits train-period + temporal dynamics that do not generalize across the temporal split. +- **Verdict for ADR-152 §2.2(b): fine-tuning WiFlow-STD on this dataset does not + demonstrate CSI→pose signal beyond the mean pose.** Until a model beats the + mean-pose baseline on a temporal split, no PCK number from this line may be + cited as pose-estimation capability. + +### Caveats (honest, pre-registered) + +- Single subject, single room, single session (30 min), single ESP32 node — + in-domain temporal split only; nothing here speaks to cross-room or + cross-subject generalization. +- 2k windows vs the 360k-window WiFlow-STD corpus — **NOT comparable** to the + ~96% in-domain measurement-(a) number, and the published 97.25% even less so. +- The scratch run's total collapse (it cannot even reach the mean pose; its + output BatchNorm/SiLU head must learn output scale from random init at lr 1e-4) + is an optimization outcome under the fixed budget, not proof the architecture + cannot learn from scratch — the pretrained-vs-scratch gap partially reflects + this conditioning advantage. +- Mixed-subcarrier frames (finding 1) mean even the "clean" windows carry ~20% + zero-padded frames; collection-side frame-type filtering should precede the + next session. +- Mean-baseline PCK is inflated by low pose variance relative to torso size + (~0.2–0.3 image units); PCK@10 (73.1%) shows the same ceiling effect at a + stricter threshold — the bar is the bar, but a livelier dataset would lower it. + ## Pending -- (b) fine-tune on our ESP32 17-keypoint eval set — **BLOCKED-ON-DATA**, see above. +- (b) fine-tune on our ESP32 17-keypoint eval set — **MEASURED 2026-06-10/11**, + see above: no run beats the mean-pose baseline; pretraining transfers as + optimization aid only. - (c) our internal WiFlow on their dataset (15-keypoint subset mapping) — also affected: there is currently no validated internal pose model to compare (the 92.9% artifact is retracted; the MM-Fi SOTA models in ADR-150 §3 are a diff --git a/benchmarks/wiflow-std/remote/measb/train_measb.py b/benchmarks/wiflow-std/remote/measb/train_measb.py new file mode 100644 index 00000000..5789836f --- /dev/null +++ b/benchmarks/wiflow-std/remote/measb/train_measb.py @@ -0,0 +1,369 @@ +"""ADR-152 SS2.2 measurement (b): WiFlow-STD fine-tuned on our fresh ESP32 paired dataset. + +Dataset: ~/wiflow-std-bench/paired-20260610.jsonl -- 2,046 paired windows collected +2026-06-10 22:10-22:40 (ONE subject, ONE room, ONE ESP32 node, varied poses). +Per record: csi = flat float32 list, csi_shape, kp = 17 COCO [x, y] normalized [0,1] +camera coords, conf (MediaPipe mean confidence, all > 0.5 in this set), ts_start/ts_end. +Aligner: scripts/align-ground-truth.js, non-overlapping 20-frame windows (~0.42 s each). + +Dataset findings (MEASURED on this file, 2026-06-10): + - csi_shape is HETEROGENEOUS, not uniformly [70, 20]: 1,347x [70,20], 284x [134,20], + 243x [26,20], 130x [12,20], 42x [20,20]. The ESP32 stream emits mixed frame types + and the aligner stamps each window's subcarrier count from frame[0] + (extractCsiMatrix: nSc = window[0].subcarriers), zero-padding/truncating the rest. + Even native-70 windows contain ~20.4% internally zero-padded short frames + (subcarriers 40..69 all-zero for those frames). + - LAYOUT BUG: the aligner fills matrix[f * nSc + s] (frame-major) but declares + shape [nSc, nFrames]. The true layout is (frame, subcarrier); we reshape + (nFrames, nSc) and transpose. Confirmed by coherent per-frame zero-tails. + - Handling here (primary suite, "all2046"): every frame's subcarrier axis is + linearly resampled to 70 bins (np.interp over a normalized index domain; + identity for native-70 frames) so the pre-registered n=2,046 and split sizes + hold. Secondary suite ("native70") restricts to the 1,347 native [70,20] + windows (temporal 70/15/15 of those) as a homogeneity robustness check. + +Pre-registered protocol (followed exactly): + 1. TEMPORAL split (records are time-sorted; asserted): first 70% train (1,432), + next 15% val (307), last 15% test (307). No shuffling across time. Seed 42 + for everything else. + 2. Model: upstream WiFlow-STD trunk (WiFlowPoseModel) with a learned 1x1 Conv1d + projection 70->540 prepended, and K=17 via the parameter-free adaptive pool + (AdaptiveAvgPool2d((17, 1)) instead of (15, 1)) -- pretrained weights load + for any K. CSI normalization: divide by the TRAIN-split 99th-percentile + amplitude, clip to [0, 1] (documented in output JSON). + 3. Three runs, <=60 epochs, early-stop patience 8 on val MPJPE, batch 32, + AdamW, fp32 (no autocast): + (i) pretrained-init: trunk init from upstream/test/best_pose_model.pth + (the measurement-(a) retrained checkpoint, ~96% PCK@20 on WiFlow data; + key remap att.->attention. / final_conv.->decoder. applied defensively + as in eval_repro.py -- a no-op for this checkpoint, which already uses + the new names). Discriminative lr: adapter 1e-4, trunk 1e-5. + (ii) scratch: same architecture, random init, all params lr 1e-4. + (iii) frozen-trunk: pretrained trunk frozen (requires_grad=False AND held in + .eval() so BatchNorm running stats cannot drift -- pure transfer probe); + only the 70->540 adapter trains, lr 1e-4. + 4. Metrics on the temporal TEST split: torso-normalized PCK@10/20/30/40/50 and + MPJPE. Upstream utils/metrics.py calculate_pck(use_torso_norm=True) hardcodes + NECK_IDX/PELVIS_IDX = 2, 12 -- a 15-keypoint convention that is WRONG for our + 17 COCO keypoints (2 = right_eye, 12 = right_hip). We therefore reimplement the + identical math (per-frame norm distance, clamp min 0.01, mean over all + keypoints x frames) with torso = ||l_shoulder(5) - l_hip(11)||. + Also reported: prediction std across test frames (constant-pose detector; + must be > 0) and the mean-pose-predictor baseline (train-split mean pose + evaluated on test -- the honesty bar). + +Usage (on ruvultra): + nice -n 10 nohup ~/wiflow-std-bench/venv/bin/python train_measb.py > train_measb.log 2>&1 & +""" + +import json +import os +import random +import sys +import time + +import numpy as np +import torch +import torch.nn as nn + +BENCH = os.path.expanduser("~/wiflow-std-bench") +UPSTREAM = os.path.join(BENCH, "upstream") +MEASB = os.path.join(BENCH, "measb") +DATA = os.path.join(BENCH, "paired-20260610.jsonl") +CHECKPOINT = os.path.join(UPSTREAM, "test", "best_pose_model.pth") + +sys.path.insert(0, UPSTREAM) + +# Upstream defect (1): models/__init__.py imports a name tcn.py does not define. +# Register a stub package so the broken __init__ never executes (as eval_repro.py). +import types # noqa: E402 + +_models_pkg = types.ModuleType("models") +_models_pkg.__path__ = [os.path.join(UPSTREAM, "models")] +sys.modules["models"] = _models_pkg + +from models.pose_model import WiFlowPoseModel # noqa: E402 + +SEED = 42 +K = 17 +N_SUBC = 70 +TRUNK_IN = 540 +BATCH = 32 # <= 64 per protocol (GPU shared with the efficiency sweep) +MAX_EPOCHS = 60 +PATIENCE = 8 +LR_ADAPTER = 1e-4 +LR_TRUNK_FT = 1e-5 # 10x lower for the pretrained trunk vs the fresh adapter +L_SHOULDER, L_HIP = 5, 11 +THRESHOLDS = (0.1, 0.2, 0.3, 0.4, 0.5) + + +def set_seed(seed=SEED): + random.seed(seed) + np.random.seed(seed) + torch.manual_seed(seed) + if torch.cuda.is_available(): + torch.cuda.manual_seed_all(seed) + torch.backends.cudnn.deterministic = True + torch.backends.cudnn.benchmark = False + + +def resample_subcarriers(frame_major, n_out=N_SUBC): + """(nFrames, nSc) -> (nFrames, n_out) by per-frame linear interpolation. + + Identity for nSc == n_out. Normalized index domain [0, 1] on both sides. + """ + nf, nsc = frame_major.shape + if nsc == n_out: + return frame_major + xi = np.linspace(0.0, 1.0, nsc) + xo = np.linspace(0.0, 1.0, n_out) + return np.stack([np.interp(xo, xi, frame_major[f]) for f in range(nf)]).astype(np.float32) + + +def load_dataset(): + csi, kps, confs, ts, native70 = [], [], [], [], [] + shape_counts = {} + with open(DATA) as f: + for line in f: + r = json.loads(line) + nsc, nf = r["csi_shape"] + shape_counts[f"{nsc}x{nf}"] = shape_counts.get(f"{nsc}x{nf}", 0) + 1 + assert nf == 20, r["csi_shape"] + # Aligner layout bug: data is frame-major despite the declared + # [nSc, nFrames] shape -- reshape (nFrames, nSc), then resample the + # subcarrier axis to 70 and transpose to (70 subcarriers, 20 frames). + fm = np.asarray(r["csi"], dtype=np.float32).reshape(nf, nsc) + csi.append(resample_subcarriers(fm).T) + kp = np.asarray(r["kp"], dtype=np.float32) + assert kp.shape == (K, 2), kp.shape + kps.append(kp) + confs.append(r["conf"]) + ts.append(r["ts_start"]) + native70.append(nsc == N_SUBC) + assert all(ts[i] <= ts[i + 1] for i in range(len(ts) - 1)), "records not time-sorted" + return (np.stack(csi), np.stack(kps), np.asarray(confs, dtype=np.float32), + np.asarray(native70), shape_counts, ts[0], ts[-1]) + + +def temporal_split(n): + n_train = int(round(n * 0.70)) + n_val = int(round(n * 0.15)) + return slice(0, n_train), slice(n_train, n_train + n_val), slice(n_train + n_val, n) + + +class AdaptedWiFlow(nn.Module): + """1x1 Conv1d adapter 70->540 + upstream WiFlow-STD trunk with K=17 pool head.""" + + def __init__(self, k=K, dropout=0.5): + super().__init__() + self.adapter = nn.Conv1d(N_SUBC, TRUNK_IN, kernel_size=1) + nn.init.kaiming_normal_(self.adapter.weight, mode="fan_out", nonlinearity="relu") + nn.init.constant_(self.adapter.bias, 0) + self.trunk = WiFlowPoseModel(dropout=dropout) + # K=17 via the parameter-free adaptive pool: decoder emits [B, 2, 15, 20] + # spatial maps; pooling H->17 instead of 15 yields [B, 17, 2] with no new + # parameters, so the pretrained state_dict loads strict=True for any K. + self.trunk.avg_pool = nn.AdaptiveAvgPool2d((k, 1)) + + def forward(self, x): + return self.trunk(self.adapter(x)) + + +def load_pretrained_trunk(trunk, path): + state = torch.load(path, map_location="cpu", weights_only=True) + # Defensive remap as in eval_repro.py (no-op for the retrained checkpoint). + renames = {"att.": "attention.", "final_conv.": "decoder."} + state = {next((new + k[len(old):] for old, new in renames.items() + if k.startswith(old)), k): v + for k, v in state.items()} + trunk.load_state_dict(state, strict=True) + + +def pck_torso(pred, target, thresholds=THRESHOLDS): + """Upstream calculate_pck math, torso = l_shoulder(5)<->l_hip(11) for 17-kp COCO.""" + norm = torch.sqrt(((target[:, L_SHOULDER] - target[:, L_HIP]) ** 2).sum(dim=1)) + norm = torch.clamp(norm, min=0.01) + dist = torch.sqrt(((pred - target) ** 2).sum(dim=2)) / norm.unsqueeze(1) + return {f"pck@{int(t * 100)}": (dist <= t).float().mean().item() for t in thresholds} + + +def mpjpe(pred, target): + return torch.sqrt(((pred - target) ** 2).sum(dim=2)).mean().item() + + +@torch.no_grad() +def predict(model, x, batch=256): + model.eval() + return torch.cat([model(x[i:i + batch]) for i in range(0, len(x), batch)]) + + +def eval_preds(pred, target): + out = pck_torso(pred, target) + out["mpjpe"] = mpjpe(pred, target) + # Constant-pose detector: std across test frames per coordinate, mean over + # the 17x2 coordinates. 0.0 == degenerate constant predictor. + out["pred_std"] = pred.std(dim=0).mean().item() + return out + + +def train_run(name, x_tr, y_tr, x_va, y_va, device, pretrained, freeze_trunk, + lr_trunk): + set_seed(SEED) + model = AdaptedWiFlow().to(device) + if pretrained: + load_pretrained_trunk(model.trunk, CHECKPOINT) + if freeze_trunk: + for p in model.trunk.parameters(): + p.requires_grad = False + groups = [{"params": model.adapter.parameters(), "lr": LR_ADAPTER}] + else: + groups = [{"params": model.adapter.parameters(), "lr": LR_ADAPTER}, + {"params": model.trunk.parameters(), "lr": lr_trunk}] + opt = torch.optim.AdamW(groups) + loss_fn = nn.MSELoss() + + n = len(x_tr) + best_val, best_state, best_epoch, bad = float("inf"), None, -1, 0 + history = [] + t0 = time.time() + for epoch in range(MAX_EPOCHS): + model.train() + if freeze_trunk: + model.trunk.eval() # keep BatchNorm running stats fixed: pure transfer + perm = torch.randperm(n, device=device) + ep_loss = 0.0 + for i in range(0, n, BATCH): + idx = perm[i:i + BATCH] + opt.zero_grad() + loss = loss_fn(model(x_tr[idx]), y_tr[idx]) + loss.backward() + opt.step() + ep_loss += loss.item() * len(idx) + val_mpjpe = mpjpe(predict(model, x_va), y_va) + history.append({"epoch": epoch, "train_mse": ep_loss / n, "val_mpjpe": val_mpjpe}) + marker = "" + if val_mpjpe < best_val: + best_val, best_epoch, bad = val_mpjpe, epoch, 0 + best_state = {k: v.detach().cpu().clone() for k, v in model.state_dict().items()} + marker = " *" + else: + bad += 1 + print(f"[{name}] epoch {epoch:02d} train_mse {ep_loss / n:.6f} " + f"val_mpjpe {val_mpjpe:.5f}{marker}", flush=True) + if bad >= PATIENCE: + print(f"[{name}] early stop at epoch {epoch} (best {best_epoch})", flush=True) + break + model.load_state_dict(best_state) + torch.save(best_state, os.path.join(MEASB, f"{name}_best.pth")) + return model, {"best_epoch": best_epoch, "best_val_mpjpe": best_val, + "epochs_run": len(history), "wall_seconds": round(time.time() - t0, 1), + "history": history} + + +def run_suite(tag, csi, kps, device): + """Temporal 70/15/15 split, mean-pose baseline, three training runs.""" + n = len(csi) + tr, va, te = temporal_split(n) + print(f"=== suite {tag}: n={n} train={tr.stop} val={va.stop - va.start} " + f"test={te.stop - te.start} ===", flush=True) + + # CSI normalization constant from TRAIN split only. + train_p99 = float(np.percentile(csi[tr], 99)) + train_max = float(csi[tr].max()) + print(f"[{tag}] train p99={train_p99:.3f} max={train_max:.3f} -> /p99, clip [0,1]", + flush=True) + csi_n = np.clip(csi / train_p99, 0.0, 1.0).astype(np.float32) + + x = torch.from_numpy(csi_n).to(device) + y = torch.from_numpy(kps).to(device) + x_tr, y_tr = x[tr], y[tr] + x_va, y_va = x[va], y[va] + x_te, y_te = x[te], y[te] + + suite = { + "n_windows": n, + "split": {"n_train": int(tr.stop), "n_val": int(va.stop - va.start), + "n_test": int(te.stop - te.start)}, + "csi_norm": {"method": "divide by train-split p99 amplitude, clip [0,1]", + "train_p99": train_p99, "train_max": train_max}, + "runs": {}, + } + + # Honesty bar: mean-pose predictor fit on TRAIN, evaluated on TEST. + mean_pose = y_tr.mean(dim=0, keepdim=True).expand(len(y_te), -1, -1) + suite["mean_pose_baseline"] = eval_preds(mean_pose, y_te) + suite["mean_pose_baseline"]["note"] = "train-split mean pose; pred_std 0 by construction" + print(f"[{tag}] mean-pose baseline:", json.dumps(suite["mean_pose_baseline"]), + flush=True) + + configs = [ + ("pretrained", dict(pretrained=True, freeze_trunk=False, lr_trunk=LR_TRUNK_FT)), + ("scratch", dict(pretrained=False, freeze_trunk=False, lr_trunk=LR_ADAPTER)), + ("frozen_trunk", dict(pretrained=True, freeze_trunk=True, lr_trunk=0.0)), + ] + for name, cfg in configs: + print(f"=== run: {tag}/{name} {cfg} ===", flush=True) + model, train_info = train_run(f"{tag}_{name}", x_tr, y_tr, x_va, y_va, + device, **cfg) + test_metrics = eval_preds(predict(model, x_te), y_te) + n_trainable = sum(p.numel() for p in model.parameters() if p.requires_grad) + suite["runs"][name] = {"config": cfg, "trainable_params": n_trainable, + "train": {k: v for k, v in train_info.items() + if k != "history"}, + "history": train_info["history"], + "test": test_metrics} + print(f"[{tag}/{name}] TEST:", json.dumps(test_metrics), flush=True) + return suite + + +def main(): + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + print(f"device {device}, torch {torch.__version__}", flush=True) + set_seed(SEED) + + csi, kps, confs, native70, shape_counts, ts_first, ts_last = load_dataset() + print(f"shape distribution: {shape_counts}", flush=True) + + results = { + "protocol": { + "dataset": DATA, "n_windows": len(csi), + "ts_first": ts_first, "ts_last": ts_last, + "conf_mean": float(confs.mean()), "conf_min": float(confs.min()), + "csi_shape_distribution": shape_counts, + "csi_layout_note": "aligner stores frame-major data under a transposed " + "[nSc, nFrames] shape label; corrected on load", + "csi_resample": "per-frame linear interp of subcarrier axis to 70 bins " + "(identity for native-70 frames); native-70 windows still " + "contain ~20.4% internally zero-padded short frames", + "split": "temporal 70/15/15 (no shuffle across time)", + "model": "1x1 Conv1d 70->540 adapter + WiFlowPoseModel trunk, " + "AdaptiveAvgPool2d((17,1)) head (parameter-free K=17)", + "checkpoint": CHECKPOINT, + "checkpoint_note": "measurement-(a) retrained checkpoint (~96% PCK@20 on " + "WiFlow data); att./final_conv. remap applied " + "defensively (no-op, already new-style keys)", + "optimizer": f"AdamW, adapter lr {LR_ADAPTER}, fine-tuned trunk lr " + f"{LR_TRUNK_FT} (10x lower), scratch all {LR_ADAPTER}", + "batch": BATCH, "max_epochs": MAX_EPOCHS, "patience": PATIENCE, + "precision": "fp32", "seed": SEED, + "pck": "torso-normalized, torso = ||l_shoulder(5) - l_hip(11)||, " + "clamp min 0.01, mean over keypoints x frames " + "(upstream math; upstream 2/12 indices are a 15-kp convention)", + }, + # Primary: all 2,046 windows (pre-registered n), subcarrier axis resampled. + "all2046": None, + # Secondary robustness check: the 1,347 native [70,20] windows only. + "native70": None, + } + + results["all2046"] = run_suite("all2046", csi, kps, device) + results["native70"] = run_suite("native70", csi[native70], kps[native70], device) + + out = os.path.join(MEASB, "measurement_b.json") + with open(out, "w") as f: + json.dump(results, f, indent=2) + print(f"wrote {out}", flush=True) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/wiflow-std/results/measurement_b.json b/benchmarks/wiflow-std/results/measurement_b.json new file mode 100644 index 00000000..329c67ad --- /dev/null +++ b/benchmarks/wiflow-std/results/measurement_b.json @@ -0,0 +1,1540 @@ +{ + "protocol": { + "dataset": "/home/ruvultra/wiflow-std-bench/paired-20260610.jsonl", + "n_windows": 2046, + "ts_first": "2026-06-10T22:10:14.105Z", + "ts_last": "2026-06-10T22:39:37.445Z", + "conf_mean": 0.8024770021438599, + "conf_min": 0.6919999718666077, + "csi_shape_distribution": { + "70x20": 1347, + "12x20": 130, + "26x20": 243, + "134x20": 284, + "20x20": 42 + }, + "csi_layout_note": "aligner stores frame-major data under a transposed [nSc, nFrames] shape label; corrected on load", + "csi_resample": "per-frame linear interp of subcarrier axis to 70 bins (identity for native-70 frames); native-70 windows still contain ~20.4% internally zero-padded short frames", + "split": "temporal 70/15/15 (no shuffle across time)", + "model": "1x1 Conv1d 70->540 adapter + WiFlowPoseModel trunk, AdaptiveAvgPool2d((17,1)) head (parameter-free K=17)", + "checkpoint": "/home/ruvultra/wiflow-std-bench/upstream/test/best_pose_model.pth", + "checkpoint_note": "measurement-(a) retrained checkpoint (~96% PCK@20 on WiFlow data); att./final_conv. remap applied defensively (no-op, already new-style keys)", + "optimizer": "AdamW, adapter lr 0.0001, fine-tuned trunk lr 1e-05 (10x lower), scratch all 0.0001", + "batch": 32, + "max_epochs": 60, + "patience": 8, + "precision": "fp32", + "seed": 42, + "pck": "torso-normalized, torso = ||l_shoulder(5) - l_hip(11)||, clamp min 0.01, mean over keypoints x frames (upstream math; upstream 2/12 indices are a 15-kp convention)" + }, + "all2046": { + "n_windows": 2046, + "split": { + "n_train": 1432, + "n_val": 307, + "n_test": 307 + }, + "csi_norm": { + "method": "divide by train-split p99 amplitude, clip [0,1]", + "train_p99": 129.69725036621094, + "train_max": 181.0193328857422 + }, + "runs": { + "pretrained": { + "config": { + "pretrained": true, + "freeze_trunk": false, + "lr_trunk": 1e-05 + }, + "trainable_params": 2263382, + "train": { + "best_epoch": 58, + "best_val_mpjpe": 0.029583850875496864, + "epochs_run": 60, + "wall_seconds": 52.5 + }, + "history": [ + { + "epoch": 0, + "train_mse": 0.02910329114291921, + "val_mpjpe": 0.22232387959957123 + }, + { + "epoch": 1, + "train_mse": 0.024953276938732775, + "val_mpjpe": 0.20638489723205566 + }, + { + "epoch": 2, + "train_mse": 0.02218410537146323, + "val_mpjpe": 0.19414909183979034 + }, + { + "epoch": 3, + "train_mse": 0.019919338766173276, + "val_mpjpe": 0.18246465921401978 + }, + { + "epoch": 4, + "train_mse": 0.01802002595914476, + "val_mpjpe": 0.17038212716579437 + }, + { + "epoch": 5, + "train_mse": 0.01622012431353497, + "val_mpjpe": 0.1601421982049942 + }, + { + "epoch": 6, + "train_mse": 0.014466259217503684, + "val_mpjpe": 0.14984126389026642 + }, + { + "epoch": 7, + "train_mse": 0.012797557150518429, + "val_mpjpe": 0.1391102522611618 + }, + { + "epoch": 8, + "train_mse": 0.011128359266124956, + "val_mpjpe": 0.12883472442626953 + }, + { + "epoch": 9, + "train_mse": 0.009428741285504576, + "val_mpjpe": 0.11573321372270584 + }, + { + "epoch": 10, + "train_mse": 0.007814127998686703, + "val_mpjpe": 0.10186194628477097 + }, + { + "epoch": 11, + "train_mse": 0.006260629412952249, + "val_mpjpe": 0.08770060539245605 + }, + { + "epoch": 12, + "train_mse": 0.004784003726087053, + "val_mpjpe": 0.07141995429992676 + }, + { + "epoch": 13, + "train_mse": 0.0037322006651392863, + "val_mpjpe": 0.06400816887617111 + }, + { + "epoch": 14, + "train_mse": 0.003136915010374102, + "val_mpjpe": 0.054206982254981995 + }, + { + "epoch": 15, + "train_mse": 0.002630213634553235, + "val_mpjpe": 0.053094107657670975 + }, + { + "epoch": 16, + "train_mse": 0.002348081870837002, + "val_mpjpe": 0.04918904975056648 + }, + { + "epoch": 17, + "train_mse": 0.0021652982065547776, + "val_mpjpe": 0.047478385269641876 + }, + { + "epoch": 18, + "train_mse": 0.002081252001069861, + "val_mpjpe": 0.04686350375413895 + }, + { + "epoch": 19, + "train_mse": 0.001956615429215508, + "val_mpjpe": 0.04568779841065407 + }, + { + "epoch": 20, + "train_mse": 0.0018100869313777326, + "val_mpjpe": 0.04311814159154892 + }, + { + "epoch": 21, + "train_mse": 0.001841872943534032, + "val_mpjpe": 0.04282193258404732 + }, + { + "epoch": 22, + "train_mse": 0.0016938508156551745, + "val_mpjpe": 0.04222533479332924 + }, + { + "epoch": 23, + "train_mse": 0.001639947665896912, + "val_mpjpe": 0.04105218127369881 + }, + { + "epoch": 24, + "train_mse": 0.0016050036819703752, + "val_mpjpe": 0.041164666414260864 + }, + { + "epoch": 25, + "train_mse": 0.0016553318232048156, + "val_mpjpe": 0.04081456735730171 + }, + { + "epoch": 26, + "train_mse": 0.0015356652061428936, + "val_mpjpe": 0.039222899824380875 + }, + { + "epoch": 27, + "train_mse": 0.00142027766251281, + "val_mpjpe": 0.039004016667604446 + }, + { + "epoch": 28, + "train_mse": 0.00136006194981096, + "val_mpjpe": 0.037944238632917404 + }, + { + "epoch": 29, + "train_mse": 0.0013483136894920185, + "val_mpjpe": 0.03772488608956337 + }, + { + "epoch": 30, + "train_mse": 0.0013302022606842012, + "val_mpjpe": 0.037234146147966385 + }, + { + "epoch": 31, + "train_mse": 0.0012543508802493775, + "val_mpjpe": 0.036311447620391846 + }, + { + "epoch": 32, + "train_mse": 0.0012697824719231888, + "val_mpjpe": 0.03540641441941261 + }, + { + "epoch": 33, + "train_mse": 0.001192611671448295, + "val_mpjpe": 0.03567550331354141 + }, + { + "epoch": 34, + "train_mse": 0.0012180106831433553, + "val_mpjpe": 0.03450343385338783 + }, + { + "epoch": 35, + "train_mse": 0.0011910481499403966, + "val_mpjpe": 0.03567817807197571 + }, + { + "epoch": 36, + "train_mse": 0.0011471521077474694, + "val_mpjpe": 0.03440209850668907 + }, + { + "epoch": 37, + "train_mse": 0.0011404801567225565, + "val_mpjpe": 0.03589555248618126 + }, + { + "epoch": 38, + "train_mse": 0.0010755654849117237, + "val_mpjpe": 0.03390049189329147 + }, + { + "epoch": 39, + "train_mse": 0.0010479743632763707, + "val_mpjpe": 0.033616892993450165 + }, + { + "epoch": 40, + "train_mse": 0.0010437304157290253, + "val_mpjpe": 0.03435032442212105 + }, + { + "epoch": 41, + "train_mse": 0.0010178630141406085, + "val_mpjpe": 0.03435724228620529 + }, + { + "epoch": 42, + "train_mse": 0.00098468104179678, + "val_mpjpe": 0.03354224935173988 + }, + { + "epoch": 43, + "train_mse": 0.0009745247844169593, + "val_mpjpe": 0.03312605619430542 + }, + { + "epoch": 44, + "train_mse": 0.0009810815873561043, + "val_mpjpe": 0.032423462718725204 + }, + { + "epoch": 45, + "train_mse": 0.0009392871002422115, + "val_mpjpe": 0.031215427443385124 + }, + { + "epoch": 46, + "train_mse": 0.0009089983260264235, + "val_mpjpe": 0.03185856714844704 + }, + { + "epoch": 47, + "train_mse": 0.0009086706155947508, + "val_mpjpe": 0.031408071517944336 + }, + { + "epoch": 48, + "train_mse": 0.000895850013423836, + "val_mpjpe": 0.03207845985889435 + }, + { + "epoch": 49, + "train_mse": 0.0008529244051262862, + "val_mpjpe": 0.03133287653326988 + }, + { + "epoch": 50, + "train_mse": 0.0008321587580344614, + "val_mpjpe": 0.030738580971956253 + }, + { + "epoch": 51, + "train_mse": 0.0008275811449889947, + "val_mpjpe": 0.03121490404009819 + }, + { + "epoch": 52, + "train_mse": 0.0008260782953747158, + "val_mpjpe": 0.03016432374715805 + }, + { + "epoch": 53, + "train_mse": 0.0008019006703121785, + "val_mpjpe": 0.02996252477169037 + }, + { + "epoch": 54, + "train_mse": 0.0007858164738551532, + "val_mpjpe": 0.03053499199450016 + }, + { + "epoch": 55, + "train_mse": 0.0007676650293941438, + "val_mpjpe": 0.03045601397752762 + }, + { + "epoch": 56, + "train_mse": 0.000756923873796955, + "val_mpjpe": 0.029843738302588463 + }, + { + "epoch": 57, + "train_mse": 0.0007355798776625713, + "val_mpjpe": 0.02989450842142105 + }, + { + "epoch": 58, + "train_mse": 0.000737365201901249, + "val_mpjpe": 0.029583850875496864 + }, + { + "epoch": 59, + "train_mse": 0.0007215407794060487, + "val_mpjpe": 0.029640641063451767 + } + ], + "test": { + "pck@10": 0.260203093290329, + "pck@20": 0.6497412919998169, + "pck@30": 0.8794788122177124, + "pck@40": 0.9637861251831055, + "pck@50": 0.9885035157203674, + "mpjpe": 0.0312512144446373, + "pred_std": 0.011318008415400982 + } + }, + "scratch": { + "config": { + "pretrained": false, + "freeze_trunk": false, + "lr_trunk": 0.0001 + }, + "trainable_params": 2263382, + "train": { + "best_epoch": 4, + "best_val_mpjpe": 0.25484445691108704, + "epochs_run": 13, + "wall_seconds": 8.1 + }, + "history": [ + { + "epoch": 0, + "train_mse": 0.10147931118750705, + "val_mpjpe": 0.2967173159122467 + }, + { + "epoch": 1, + "train_mse": 0.08677088540026595, + "val_mpjpe": 0.2839488983154297 + }, + { + "epoch": 2, + "train_mse": 0.08117229266706126, + "val_mpjpe": 0.2935337424278259 + }, + { + "epoch": 3, + "train_mse": 0.07839870677646978, + "val_mpjpe": 0.2990759015083313 + }, + { + "epoch": 4, + "train_mse": 0.07353206353480589, + "val_mpjpe": 0.25484445691108704 + }, + { + "epoch": 5, + "train_mse": 0.07079217333200924, + "val_mpjpe": 0.27269694209098816 + }, + { + "epoch": 6, + "train_mse": 0.07097245003591036, + "val_mpjpe": 0.2684358060359955 + }, + { + "epoch": 7, + "train_mse": 0.06630552453951463, + "val_mpjpe": 0.2869749665260315 + }, + { + "epoch": 8, + "train_mse": 0.0623299669495175, + "val_mpjpe": 0.27154049277305603 + }, + { + "epoch": 9, + "train_mse": 0.05751657300737983, + "val_mpjpe": 0.2639182209968567 + }, + { + "epoch": 10, + "train_mse": 0.05713276381932157, + "val_mpjpe": 0.2644321620464325 + }, + { + "epoch": 11, + "train_mse": 0.05235974107707679, + "val_mpjpe": 0.27891167998313904 + }, + { + "epoch": 12, + "train_mse": 0.049985795130942784, + "val_mpjpe": 0.27126413583755493 + } + ], + "test": { + "pck@10": 0.0, + "pck@20": 0.0, + "pck@30": 0.0, + "pck@40": 0.0, + "pck@50": 0.0, + "mpjpe": 0.25539571046829224, + "pred_std": 0.00020971425692550838 + } + }, + "frozen_trunk": { + "config": { + "pretrained": true, + "freeze_trunk": true, + "lr_trunk": 0.0 + }, + "trainable_params": 38340, + "train": { + "best_epoch": 59, + "best_val_mpjpe": 0.12482486665248871, + "epochs_run": 60, + "wall_seconds": 29.5 + }, + "history": [ + { + "epoch": 0, + "train_mse": 0.029202866321169463, + "val_mpjpe": 0.21773657202720642 + }, + { + "epoch": 1, + "train_mse": 0.0224896340667369, + "val_mpjpe": 0.19948658347129822 + }, + { + "epoch": 2, + "train_mse": 0.020371714398764364, + "val_mpjpe": 0.19316966831684113 + }, + { + "epoch": 3, + "train_mse": 0.01890122553323234, + "val_mpjpe": 0.185868039727211 + }, + { + "epoch": 4, + "train_mse": 0.017790827702460342, + "val_mpjpe": 0.1816880851984024 + }, + { + "epoch": 5, + "train_mse": 0.0170674164410077, + "val_mpjpe": 0.17853409051895142 + }, + { + "epoch": 6, + "train_mse": 0.016503224898566746, + "val_mpjpe": 0.17588891088962555 + }, + { + "epoch": 7, + "train_mse": 0.016050895155724866, + "val_mpjpe": 0.1737031638622284 + }, + { + "epoch": 8, + "train_mse": 0.015696923451383685, + "val_mpjpe": 0.17200572788715363 + }, + { + "epoch": 9, + "train_mse": 0.015415839257769744, + "val_mpjpe": 0.17056800425052643 + }, + { + "epoch": 10, + "train_mse": 0.015174580862818483, + "val_mpjpe": 0.16929033398628235 + }, + { + "epoch": 11, + "train_mse": 0.014953630175129328, + "val_mpjpe": 0.16805949807167053 + }, + { + "epoch": 12, + "train_mse": 0.014754313642585743, + "val_mpjpe": 0.16697341203689575 + }, + { + "epoch": 13, + "train_mse": 0.014575892117644488, + "val_mpjpe": 0.16597621142864227 + }, + { + "epoch": 14, + "train_mse": 0.014411240686043348, + "val_mpjpe": 0.16503198444843292 + }, + { + "epoch": 15, + "train_mse": 0.014253408819770014, + "val_mpjpe": 0.1640642285346985 + }, + { + "epoch": 16, + "train_mse": 0.014096723281513046, + "val_mpjpe": 0.16311785578727722 + }, + { + "epoch": 17, + "train_mse": 0.013935973050915661, + "val_mpjpe": 0.1620997190475464 + }, + { + "epoch": 18, + "train_mse": 0.013768192911364512, + "val_mpjpe": 0.1610598862171173 + }, + { + "epoch": 19, + "train_mse": 0.01359110634107996, + "val_mpjpe": 0.15999113023281097 + }, + { + "epoch": 20, + "train_mse": 0.013404639427388846, + "val_mpjpe": 0.15887050330638885 + }, + { + "epoch": 21, + "train_mse": 0.013210025600381403, + "val_mpjpe": 0.1577041745185852 + }, + { + "epoch": 22, + "train_mse": 0.013011203418022761, + "val_mpjpe": 0.1564933955669403 + }, + { + "epoch": 23, + "train_mse": 0.01280682523573577, + "val_mpjpe": 0.1552220582962036 + }, + { + "epoch": 24, + "train_mse": 0.012595021171110303, + "val_mpjpe": 0.15387073159217834 + }, + { + "epoch": 25, + "train_mse": 0.012377229519188404, + "val_mpjpe": 0.15247991681098938 + }, + { + "epoch": 26, + "train_mse": 0.012163139260556111, + "val_mpjpe": 0.1511073112487793 + }, + { + "epoch": 27, + "train_mse": 0.011953740006094206, + "val_mpjpe": 0.14971870183944702 + }, + { + "epoch": 28, + "train_mse": 0.011747080442500847, + "val_mpjpe": 0.14838044345378876 + }, + { + "epoch": 29, + "train_mse": 0.0115443037344174, + "val_mpjpe": 0.14704924821853638 + }, + { + "epoch": 30, + "train_mse": 0.011346364679990867, + "val_mpjpe": 0.14575056731700897 + }, + { + "epoch": 31, + "train_mse": 0.011153917161321174, + "val_mpjpe": 0.14449527859687805 + }, + { + "epoch": 32, + "train_mse": 0.010972691205585135, + "val_mpjpe": 0.14327144622802734 + }, + { + "epoch": 33, + "train_mse": 0.010805318735581536, + "val_mpjpe": 0.1422281265258789 + }, + { + "epoch": 34, + "train_mse": 0.010652749653510209, + "val_mpjpe": 0.14111895859241486 + }, + { + "epoch": 35, + "train_mse": 0.010515908767890663, + "val_mpjpe": 0.1401691883802414 + }, + { + "epoch": 36, + "train_mse": 0.010389003710165703, + "val_mpjpe": 0.1393871158361435 + }, + { + "epoch": 37, + "train_mse": 0.010272538305036516, + "val_mpjpe": 0.1385311633348465 + }, + { + "epoch": 38, + "train_mse": 0.010164186171146745, + "val_mpjpe": 0.1378026008605957 + }, + { + "epoch": 39, + "train_mse": 0.010061318601101803, + "val_mpjpe": 0.13706167042255402 + }, + { + "epoch": 40, + "train_mse": 0.009963676453094575, + "val_mpjpe": 0.13636554777622223 + }, + { + "epoch": 41, + "train_mse": 0.009868574287097214, + "val_mpjpe": 0.13567958772182465 + }, + { + "epoch": 42, + "train_mse": 0.009777017921084465, + "val_mpjpe": 0.13502979278564453 + }, + { + "epoch": 43, + "train_mse": 0.009687256239687598, + "val_mpjpe": 0.13433508574962616 + }, + { + "epoch": 44, + "train_mse": 0.009599470465792624, + "val_mpjpe": 0.1336650252342224 + }, + { + "epoch": 45, + "train_mse": 0.009513282612507237, + "val_mpjpe": 0.13304953277111053 + }, + { + "epoch": 46, + "train_mse": 0.009427554078150395, + "val_mpjpe": 0.1323200762271881 + }, + { + "epoch": 47, + "train_mse": 0.00934415341838778, + "val_mpjpe": 0.13170869648456573 + }, + { + "epoch": 48, + "train_mse": 0.009261736115596815, + "val_mpjpe": 0.13112133741378784 + }, + { + "epoch": 49, + "train_mse": 0.00918193509562888, + "val_mpjpe": 0.13044753670692444 + }, + { + "epoch": 50, + "train_mse": 0.009104542855652018, + "val_mpjpe": 0.12986503541469574 + }, + { + "epoch": 51, + "train_mse": 0.009028687011762704, + "val_mpjpe": 0.12926670908927917 + }, + { + "epoch": 52, + "train_mse": 0.008955816504176102, + "val_mpjpe": 0.12868435680866241 + }, + { + "epoch": 53, + "train_mse": 0.00888419084188492, + "val_mpjpe": 0.12810716032981873 + }, + { + "epoch": 54, + "train_mse": 0.008814793345440367, + "val_mpjpe": 0.12754392623901367 + }, + { + "epoch": 55, + "train_mse": 0.00874635915391605, + "val_mpjpe": 0.12702825665473938 + }, + { + "epoch": 56, + "train_mse": 0.008679542893429376, + "val_mpjpe": 0.12642037868499756 + }, + { + "epoch": 57, + "train_mse": 0.008613041066174401, + "val_mpjpe": 0.12594333291053772 + }, + { + "epoch": 58, + "train_mse": 0.008546595810262184, + "val_mpjpe": 0.12541112303733826 + }, + { + "epoch": 59, + "train_mse": 0.008480435232085555, + "val_mpjpe": 0.12482486665248871 + } + ], + "test": { + "pck@10": 0.0, + "pck@20": 0.0003832151705864817, + "pck@30": 0.002299291081726551, + "pck@40": 0.032381683588027954, + "pck@50": 0.14428050816059113, + "mpjpe": 0.12595407664775848, + "pred_std": 0.007278506178408861 + } + } + }, + "mean_pose_baseline": { + "pck@10": 0.7305997014045715, + "pck@20": 0.9586127400398254, + "pck@30": 0.9871622920036316, + "pck@40": 0.9925273060798645, + "pck@50": 0.9932937026023865, + "mpjpe": 0.01475962158292532, + "pred_std": 0.0, + "note": "train-split mean pose; pred_std 0 by construction" + } + }, + "native70": { + "n_windows": 1347, + "split": { + "n_train": 943, + "n_val": 202, + "n_test": 202 + }, + "csi_norm": { + "method": "divide by train-split p99 amplitude, clip [0,1]", + "train_p99": 130.91983032226562, + "train_max": 180.31361389160156 + }, + "runs": { + "pretrained": { + "config": { + "pretrained": true, + "freeze_trunk": false, + "lr_trunk": 1e-05 + }, + "trainable_params": 2263382, + "train": { + "best_epoch": 59, + "best_val_mpjpe": 0.028786057606339455, + "epochs_run": 60, + "wall_seconds": 34.7 + }, + "history": [ + { + "epoch": 0, + "train_mse": 0.03143434640719365, + "val_mpjpe": 0.24106381833553314 + }, + { + "epoch": 1, + "train_mse": 0.02710387347343112, + "val_mpjpe": 0.22779834270477295 + }, + { + "epoch": 2, + "train_mse": 0.02450904537936473, + "val_mpjpe": 0.21331433951854706 + }, + { + "epoch": 3, + "train_mse": 0.02252072293196794, + "val_mpjpe": 0.2012370377779007 + }, + { + "epoch": 4, + "train_mse": 0.02079117415455088, + "val_mpjpe": 0.192302405834198 + }, + { + "epoch": 5, + "train_mse": 0.019259598619706556, + "val_mpjpe": 0.18497928977012634 + }, + { + "epoch": 6, + "train_mse": 0.01786766991943658, + "val_mpjpe": 0.17789718508720398 + }, + { + "epoch": 7, + "train_mse": 0.016434193224362705, + "val_mpjpe": 0.16694092750549316 + }, + { + "epoch": 8, + "train_mse": 0.015003233426077685, + "val_mpjpe": 0.15971370041370392 + }, + { + "epoch": 9, + "train_mse": 0.013477064646417049, + "val_mpjpe": 0.14975418150424957 + }, + { + "epoch": 10, + "train_mse": 0.011819301292592559, + "val_mpjpe": 0.1359071433544159 + }, + { + "epoch": 11, + "train_mse": 0.010180270561033886, + "val_mpjpe": 0.12011609226465225 + }, + { + "epoch": 12, + "train_mse": 0.008623048163405278, + "val_mpjpe": 0.10962960869073868 + }, + { + "epoch": 13, + "train_mse": 0.007252912756553352, + "val_mpjpe": 0.10335882008075714 + }, + { + "epoch": 14, + "train_mse": 0.006106390551617517, + "val_mpjpe": 0.09540214389562607 + }, + { + "epoch": 15, + "train_mse": 0.005359676358915178, + "val_mpjpe": 0.09057636559009552 + }, + { + "epoch": 16, + "train_mse": 0.004789241548125905, + "val_mpjpe": 0.08553972095251083 + }, + { + "epoch": 17, + "train_mse": 0.004348343574518016, + "val_mpjpe": 0.07920289039611816 + }, + { + "epoch": 18, + "train_mse": 0.0040117833513493946, + "val_mpjpe": 0.07419152557849884 + }, + { + "epoch": 19, + "train_mse": 0.003737294021796352, + "val_mpjpe": 0.07071558386087418 + }, + { + "epoch": 20, + "train_mse": 0.0034692754511240465, + "val_mpjpe": 0.06795826554298401 + }, + { + "epoch": 21, + "train_mse": 0.003230852333676714, + "val_mpjpe": 0.06458016484975815 + }, + { + "epoch": 22, + "train_mse": 0.0030218517133217475, + "val_mpjpe": 0.062487438321113586 + }, + { + "epoch": 23, + "train_mse": 0.0028510230088711164, + "val_mpjpe": 0.05932319909334183 + }, + { + "epoch": 24, + "train_mse": 0.0026618994672191138, + "val_mpjpe": 0.05705098807811737 + }, + { + "epoch": 25, + "train_mse": 0.0025389514019572563, + "val_mpjpe": 0.05447569862008095 + }, + { + "epoch": 26, + "train_mse": 0.002362748779097724, + "val_mpjpe": 0.052511945366859436 + }, + { + "epoch": 27, + "train_mse": 0.0022396585923320322, + "val_mpjpe": 0.05106504261493683 + }, + { + "epoch": 28, + "train_mse": 0.002125039399420785, + "val_mpjpe": 0.04928405210375786 + }, + { + "epoch": 29, + "train_mse": 0.00200098952770328, + "val_mpjpe": 0.047419942915439606 + }, + { + "epoch": 30, + "train_mse": 0.0018960386049249314, + "val_mpjpe": 0.04589642211794853 + }, + { + "epoch": 31, + "train_mse": 0.001827567362094205, + "val_mpjpe": 0.04474344849586487 + }, + { + "epoch": 32, + "train_mse": 0.0017325681195251668, + "val_mpjpe": 0.04346025735139847 + }, + { + "epoch": 33, + "train_mse": 0.0016443500521692545, + "val_mpjpe": 0.04234813526272774 + }, + { + "epoch": 34, + "train_mse": 0.0015841188279107955, + "val_mpjpe": 0.04019446671009064 + }, + { + "epoch": 35, + "train_mse": 0.0015091487361574476, + "val_mpjpe": 0.03928297013044357 + }, + { + "epoch": 36, + "train_mse": 0.0014544202454046427, + "val_mpjpe": 0.03781558573246002 + }, + { + "epoch": 37, + "train_mse": 0.001395829413150371, + "val_mpjpe": 0.03714558482170105 + }, + { + "epoch": 38, + "train_mse": 0.001346534527176527, + "val_mpjpe": 0.03650708124041557 + }, + { + "epoch": 39, + "train_mse": 0.0013110679464438419, + "val_mpjpe": 0.03521648421883583 + }, + { + "epoch": 40, + "train_mse": 0.0012685508586511743, + "val_mpjpe": 0.034147679805755615 + }, + { + "epoch": 41, + "train_mse": 0.0012353137454012072, + "val_mpjpe": 0.03370988741517067 + }, + { + "epoch": 42, + "train_mse": 0.0011922467519970628, + "val_mpjpe": 0.03350502997636795 + }, + { + "epoch": 43, + "train_mse": 0.0011716993518957862, + "val_mpjpe": 0.033042099326848984 + }, + { + "epoch": 44, + "train_mse": 0.0011406343915200498, + "val_mpjpe": 0.03243575617671013 + }, + { + "epoch": 45, + "train_mse": 0.001107781613393749, + "val_mpjpe": 0.032004695385694504 + }, + { + "epoch": 46, + "train_mse": 0.0010832754548437584, + "val_mpjpe": 0.03172597289085388 + }, + { + "epoch": 47, + "train_mse": 0.0010682939976939688, + "val_mpjpe": 0.03182212635874748 + }, + { + "epoch": 48, + "train_mse": 0.00104656588691825, + "val_mpjpe": 0.030928654596209526 + }, + { + "epoch": 49, + "train_mse": 0.0010045616469501043, + "val_mpjpe": 0.030643220990896225 + }, + { + "epoch": 50, + "train_mse": 0.0009883961464749492, + "val_mpjpe": 0.030708612874150276 + }, + { + "epoch": 51, + "train_mse": 0.0009810250272978321, + "val_mpjpe": 0.03019033558666706 + }, + { + "epoch": 52, + "train_mse": 0.0009541419821851595, + "val_mpjpe": 0.03077096864581108 + }, + { + "epoch": 53, + "train_mse": 0.0009370211081990176, + "val_mpjpe": 0.030223999172449112 + }, + { + "epoch": 54, + "train_mse": 0.0009154953339190468, + "val_mpjpe": 0.029796740040183067 + }, + { + "epoch": 55, + "train_mse": 0.0009150283336086339, + "val_mpjpe": 0.029676740989089012 + }, + { + "epoch": 56, + "train_mse": 0.0008937600550828412, + "val_mpjpe": 0.02944893389940262 + }, + { + "epoch": 57, + "train_mse": 0.0008770538711143127, + "val_mpjpe": 0.029552502557635307 + }, + { + "epoch": 58, + "train_mse": 0.0008620407871436822, + "val_mpjpe": 0.028965305536985397 + }, + { + "epoch": 59, + "train_mse": 0.0008538496438097821, + "val_mpjpe": 0.028786057606339455 + } + ], + "test": { + "pck@10": 0.23704135417938232, + "pck@20": 0.6712288856506348, + "pck@30": 0.8878858685493469, + "pck@40": 0.9685497879981995, + "pck@50": 0.9866045713424683, + "mpjpe": 0.0317576602101326, + "pred_std": 0.008104924112558365 + } + }, + "scratch": { + "config": { + "pretrained": false, + "freeze_trunk": false, + "lr_trunk": 0.0001 + }, + "trainable_params": 2263382, + "train": { + "best_epoch": 0, + "best_val_mpjpe": 0.2236219048500061, + "epochs_run": 9, + "wall_seconds": 5.3 + }, + "history": [ + { + "epoch": 0, + "train_mse": 0.11516269241891257, + "val_mpjpe": 0.2236219048500061 + }, + { + "epoch": 1, + "train_mse": 0.09848621406237874, + "val_mpjpe": 0.26384586095809937 + }, + { + "epoch": 2, + "train_mse": 0.09299878020744182, + "val_mpjpe": 0.2880570888519287 + }, + { + "epoch": 3, + "train_mse": 0.08682021771262331, + "val_mpjpe": 0.3054330348968506 + }, + { + "epoch": 4, + "train_mse": 0.08520131534477055, + "val_mpjpe": 0.2791931629180908 + }, + { + "epoch": 5, + "train_mse": 0.08170030018747296, + "val_mpjpe": 0.30333569645881653 + }, + { + "epoch": 6, + "train_mse": 0.0791756407706381, + "val_mpjpe": 0.32705754041671753 + }, + { + "epoch": 7, + "train_mse": 0.07421789592485539, + "val_mpjpe": 0.2974965274333954 + }, + { + "epoch": 8, + "train_mse": 0.07339104860705005, + "val_mpjpe": 0.26831597089767456 + } + ], + "test": { + "pck@10": 0.0, + "pck@20": 0.00029120559338480234, + "pck@30": 0.000873616780154407, + "pck@40": 0.000873616780154407, + "pck@50": 0.0011648223735392094, + "mpjpe": 0.22364823520183563, + "pred_std": 1.8200555132352747e-05 + } + }, + "frozen_trunk": { + "config": { + "pretrained": true, + "freeze_trunk": true, + "lr_trunk": 0.0 + }, + "trainable_params": 38340, + "train": { + "best_epoch": 59, + "best_val_mpjpe": 0.13141994178295135, + "epochs_run": 60, + "wall_seconds": 20.4 + }, + "history": [ + { + "epoch": 0, + "train_mse": 0.02987076517901643, + "val_mpjpe": 0.22486324608325958 + }, + { + "epoch": 1, + "train_mse": 0.024687130979932883, + "val_mpjpe": 0.20780175924301147 + }, + { + "epoch": 2, + "train_mse": 0.02164670548669417, + "val_mpjpe": 0.1994500756263733 + }, + { + "epoch": 3, + "train_mse": 0.020454474889761192, + "val_mpjpe": 0.19498063623905182 + }, + { + "epoch": 4, + "train_mse": 0.019394560161018044, + "val_mpjpe": 0.1884152740240097 + }, + { + "epoch": 5, + "train_mse": 0.018231849114026397, + "val_mpjpe": 0.18403740227222443 + }, + { + "epoch": 6, + "train_mse": 0.017505407153627778, + "val_mpjpe": 0.18113455176353455 + }, + { + "epoch": 7, + "train_mse": 0.01698620681979813, + "val_mpjpe": 0.17879396677017212 + }, + { + "epoch": 8, + "train_mse": 0.01656942880363821, + "val_mpjpe": 0.17679837346076965 + }, + { + "epoch": 9, + "train_mse": 0.0162232256062874, + "val_mpjpe": 0.17509010434150696 + }, + { + "epoch": 10, + "train_mse": 0.015934029979234283, + "val_mpjpe": 0.17365199327468872 + }, + { + "epoch": 11, + "train_mse": 0.015692025662053175, + "val_mpjpe": 0.17239253222942352 + }, + { + "epoch": 12, + "train_mse": 0.015484217627414464, + "val_mpjpe": 0.17132554948329926 + }, + { + "epoch": 13, + "train_mse": 0.015298402398105707, + "val_mpjpe": 0.17031940817832947 + }, + { + "epoch": 14, + "train_mse": 0.015126366212703328, + "val_mpjpe": 0.16935476660728455 + }, + { + "epoch": 15, + "train_mse": 0.014965638048227873, + "val_mpjpe": 0.16846652328968048 + }, + { + "epoch": 16, + "train_mse": 0.014815389178693295, + "val_mpjpe": 0.16761361062526703 + }, + { + "epoch": 17, + "train_mse": 0.014672840479267554, + "val_mpjpe": 0.16681094467639923 + }, + { + "epoch": 18, + "train_mse": 0.014535051236207639, + "val_mpjpe": 0.1660136580467224 + }, + { + "epoch": 19, + "train_mse": 0.014397492208582283, + "val_mpjpe": 0.16519594192504883 + }, + { + "epoch": 20, + "train_mse": 0.014256202853813151, + "val_mpjpe": 0.1643320620059967 + }, + { + "epoch": 21, + "train_mse": 0.014104339031429521, + "val_mpjpe": 0.16340284049510956 + }, + { + "epoch": 22, + "train_mse": 0.013928960720403071, + "val_mpjpe": 0.1623145341873169 + }, + { + "epoch": 23, + "train_mse": 0.013706462573327227, + "val_mpjpe": 0.16097597777843475 + }, + { + "epoch": 24, + "train_mse": 0.013429858762284985, + "val_mpjpe": 0.1594797819852829 + }, + { + "epoch": 25, + "train_mse": 0.013145321320390778, + "val_mpjpe": 0.15782670676708221 + }, + { + "epoch": 26, + "train_mse": 0.01287676781625314, + "val_mpjpe": 0.15616852045059204 + }, + { + "epoch": 27, + "train_mse": 0.012623840798786817, + "val_mpjpe": 0.1545725166797638 + }, + { + "epoch": 28, + "train_mse": 0.012390714562433838, + "val_mpjpe": 0.15311352908611298 + }, + { + "epoch": 29, + "train_mse": 0.012183478241341934, + "val_mpjpe": 0.15184184908866882 + }, + { + "epoch": 30, + "train_mse": 0.01199968609848274, + "val_mpjpe": 0.1506134420633316 + }, + { + "epoch": 31, + "train_mse": 0.011834799889810897, + "val_mpjpe": 0.14960341155529022 + }, + { + "epoch": 32, + "train_mse": 0.011684763023070794, + "val_mpjpe": 0.14860056340694427 + }, + { + "epoch": 33, + "train_mse": 0.011545304603052987, + "val_mpjpe": 0.14774590730667114 + }, + { + "epoch": 34, + "train_mse": 0.01141634290842369, + "val_mpjpe": 0.1467970609664917 + }, + { + "epoch": 35, + "train_mse": 0.011290905805559451, + "val_mpjpe": 0.14608632028102875 + }, + { + "epoch": 36, + "train_mse": 0.011167934634510431, + "val_mpjpe": 0.14519096910953522 + }, + { + "epoch": 37, + "train_mse": 0.011049817476228022, + "val_mpjpe": 0.14451351761817932 + }, + { + "epoch": 38, + "train_mse": 0.010937488122328658, + "val_mpjpe": 0.14371512830257416 + }, + { + "epoch": 39, + "train_mse": 0.01083158944019394, + "val_mpjpe": 0.14298222959041595 + }, + { + "epoch": 40, + "train_mse": 0.010734344577794047, + "val_mpjpe": 0.1423673927783966 + }, + { + "epoch": 41, + "train_mse": 0.010643748225780523, + "val_mpjpe": 0.14172928035259247 + }, + { + "epoch": 42, + "train_mse": 0.010559881792412646, + "val_mpjpe": 0.1411534994840622 + }, + { + "epoch": 43, + "train_mse": 0.010476248792134627, + "val_mpjpe": 0.1405744105577469 + }, + { + "epoch": 44, + "train_mse": 0.010396061145156338, + "val_mpjpe": 0.13997428119182587 + }, + { + "epoch": 45, + "train_mse": 0.010316670441670167, + "val_mpjpe": 0.13942934572696686 + }, + { + "epoch": 46, + "train_mse": 0.010237011800728509, + "val_mpjpe": 0.13873447477817535 + }, + { + "epoch": 47, + "train_mse": 0.0101589905710403, + "val_mpjpe": 0.1381523609161377 + }, + { + "epoch": 48, + "train_mse": 0.010078080192066073, + "val_mpjpe": 0.13758216798305511 + }, + { + "epoch": 49, + "train_mse": 0.009999273604617581, + "val_mpjpe": 0.1370079517364502 + }, + { + "epoch": 50, + "train_mse": 0.009922577063271565, + "val_mpjpe": 0.13653233647346497 + }, + { + "epoch": 51, + "train_mse": 0.009846728385948553, + "val_mpjpe": 0.13581179082393646 + }, + { + "epoch": 52, + "train_mse": 0.009771280800518782, + "val_mpjpe": 0.13525959849357605 + }, + { + "epoch": 53, + "train_mse": 0.00969787394037334, + "val_mpjpe": 0.13470490276813507 + }, + { + "epoch": 54, + "train_mse": 0.009626131754641125, + "val_mpjpe": 0.13417582213878632 + }, + { + "epoch": 55, + "train_mse": 0.009555609366428878, + "val_mpjpe": 0.13361123204231262 + }, + { + "epoch": 56, + "train_mse": 0.009484434124054674, + "val_mpjpe": 0.13312137126922607 + }, + { + "epoch": 57, + "train_mse": 0.009413423253695381, + "val_mpjpe": 0.13248831033706665 + }, + { + "epoch": 58, + "train_mse": 0.009343502207531782, + "val_mpjpe": 0.1319369524717331 + }, + { + "epoch": 59, + "train_mse": 0.009271281455324475, + "val_mpjpe": 0.13141994178295135 + } + ], + "test": { + "pck@10": 0.0, + "pck@20": 0.0, + "pck@30": 0.0, + "pck@40": 0.009318578988313675, + "pck@50": 0.08619685471057892, + "mpjpe": 0.13426505029201508, + "pred_std": 0.005987017415463924 + } + } + }, + "mean_pose_baseline": { + "pck@10": 0.7387886047363281, + "pck@20": 0.9603960514068604, + "pck@30": 0.985148549079895, + "pck@40": 0.9892253875732422, + "pck@50": 0.9900990128517151, + "mpjpe": 0.015258259139955044, + "pred_std": 0.0, + "note": "train-split mean pose; pred_std 0 by construction" + } + } +} \ No newline at end of file diff --git a/docs/adr/ADR-152-wifi-pose-sota-2026-intake.md b/docs/adr/ADR-152-wifi-pose-sota-2026-intake.md index 77fe9e35..a22eefee 100644 --- a/docs/adr/ADR-152-wifi-pose-sota-2026-intake.md +++ b/docs/adr/ADR-152-wifi-pose-sota-2026-intake.md @@ -108,6 +108,7 @@ Pull the Apache-2.0 weights + 360k-sample dataset; run three measurements: (a) t ## 4. Open questions (carried from the research run) 1. Does WiFlow-STD retain accuracy when fine-tuned on ESP32-S3/C6 CSI (fewer subcarriers, lower SNR), scored on our 17-keypoint set? (§2.2 answers this.) + > **Partial answer (MEASURED 2026-06-11, measurement (b) on 2,046 single-room windows — `benchmarks/wiflow-std/RESULTS.md`):** pretrained init shows strong *optimization* transfer (65% PCK@20 vs scratch's 0% collapse under the same budget) but **no feature transfer** (frozen-trunk + linear adapter ≈ 0%). And no run beat the mean-pose baseline (95.9% PCK@20 — single subject, near-static normalized coords), so no CSI→pose capability is citable from this data. A definitive answer needs multi-subject/multi-position data where the mean pose is weak. 2. Is the PerceptAlign dataset downloadable under a usable license, and does the two-checkerboard procedure work with ESP32 transceiver geometry? (§2.1.4 gate.) 3. Will esp_wifi_sensing evolve toward 802.11bf compliance, replacing opportunistic CSI extraction?