Merge pull request #1489 from ruvnet/feat/sar-metaharness

feat(harness): scaffold wifi-densepose-sar-harness with darwin, router, flywheel
This commit is contained in:
rUv
2026-07-31 10:33:01 -04:00
committed by GitHub
29 changed files with 965 additions and 1 deletions
@@ -0,0 +1,45 @@
# ADR-286: `wifi-densepose-sar-harness` — a MetaHarness minted via `vendor/metaharness`
| Field | Value |
|-------|-------|
| **Status** | Accepted — implemented, **published** |
| **Date** | 2026-07-30 |
| **Parent** | ADR-287 (`wifi-densepose-sar`, the crate this harness assists development on) |
| **Relates to** | ADR-182 (`harness/ruview/`, the first MetaHarness-minted harness in this repo), ADR-285 (`harness/homecore/`, the WASM-first pattern this harness's `@metaharness/kernel` dependency follows) |
| **Published** | [`wifi-densepose-sar-harness` v0.1.0](https://www.npmjs.com/package/wifi-densepose-sar-harness) on npm (2026-07-31) |
## 0. PROOF discipline
Every claim below about what's "real" versus "illustrative"/"SYNTHETIC" is checked by a passing test in this harness's own suite (14 tests: 5 router + 5 flywheel + 4 install-smoke). Nothing here is asserted without a corresponding `__tests__/*.test.ts` file exercising it.
## 1. Context
`wifi-densepose-sar` (ADR-287) is a new, narrowly-scoped research crate. Rather than hand-roll a bespoke development-assistance setup for it, `vendor/metaharness` (the `ruvnet/metaharness` generator, vendored as a git submodule alongside this repo's other `vendor/*` submodules) was used to scaffold one directly: `npx metaharness analyze v2/crates/wifi-densepose-sar --scaffold wifi-densepose-sar-harness --host claude-code` recommended and generated `template: vertical:coding` with four agents (architect/implementer/reviewer/test-writer) and `doctor`/`review-diff` commands — the same generator that produced `harness/ruview/` (ADR-182) and `harness/homecore/` (ADR-285).
The user's ask that shaped this ADR's scope was specific: wire in **darwin, router, and flywheel** — three complementary `@metaharness/*` packages the base scaffold doesn't include by default (only Darwin Mode ships built-in).
## 2. Decision
Land the scaffold at `harness/wifi-densepose-sar/`, and add real wiring for the three requested pieces, each as an actual npm dependency (not a stub, not a `try/catch` optional import):
1. **`@metaharness/darwin`** (devDependency) — wired by the scaffold itself. `npm run evolve` (real sandbox) / `evolve:dry` (mock sandbox) mutates the harness's own operating config and keeps only measurably-improving changes.
2. **`@metaharness/router`** — `src/router.ts` wires a real `Router` (k-NN over labelled examples, cost-optimal selection against a quality bar) with two example model tiers (`cheap-tier` $1/MTok, `frontier-tier` $15/MTok). Exposed as a CLI command (`route <e0> <e1> <e2> <e3>`) with a matching `.claude/commands/route.md` guidance file.
3. **`@metaharness/flywheel`** — `src/flywheel.ts` wires the real `runFlywheelGenerations` promotion loop (propose → evaluate → gate → promote, Ed25519-signed, independently replayable via `verifyReplayBundle`) with a SYNTHETIC proposer/evaluator (`dataSource: 'SYNTHETIC'`, no live model call). Exposed as `flywheel [generations]` with a matching `.claude/commands/flywheel.md` guidance file.
Every new CLI subcommand gets a `.claude/commands/<name>.md` file, matching the pattern the base scaffold's `doctor`/`review-diff` already establish — the MCP tool listing (`mcp__wifi-densepose-sar-harness__*`) is derived from these, so a command without one isn't fully wired into the harness's own guidance surface even if the CLI itself works.
## 3. What this explicitly is NOT
- **Not evolving the crate.** Darwin/Flywheel mutate the harness's own operating policy (agent prompts, review checklist depth) — not `wifi-densepose-sar`'s Rust code or its runtime performance. Actually optimizing the crate (the incremental-phasor-rotation work, ADR-287 §7) was done directly, not through this harness's self-improvement loop.
- **Not a live routing/promotion system.** The router's labelled examples are illustrative seed data, not measured eval-log observations. The flywheel's proposer/evaluator are deterministic stand-ins, not a real model call or a real coding-task benchmark suite. Both are honestly labeled as such in their own source files and in this harness's `CLAUDE.md`.
- **Not manifest-verified.** `.harness/manifest.json`/`manifest.sha256` reflect the initial scaffold output and were not regenerated after adding `router.ts`/`flywheel.ts` — this scaffold has no `manifest:update` script (unlike `harness/homecore/`). Documented as a known gap in the harness's own README.
## 4. A real bug the flywheel wiring found
The first version of the SYNTHETIC evaluator returned a constant `noopRate`. `@metaharness/flywheel`'s default promotion gate requires `noopRate` to *strictly improve* generation over generation (one of its five conjunctive clauses) — a constant value, however good, fails that clause forever, so nothing could ever be promoted. Fixed by making `noopRate` actually respond to the (synthetic) policy content; every generation promotes now. Kept as a cautionary note in `src/flywheel.ts`'s comments: a flywheel evaluator with a frozen metric is silently broken, not silently fine.
## 5. Consequences
- 14 tests (5 router + 5 flywheel + 4 install-smoke), 0 failed; `npm run build` clean under strict TypeScript.
- Published to npm as `wifi-densepose-sar-harness` v0.1.0 — `npx wifi-densepose-sar-harness init` works from a cold install.
- No risk to any other harness or crate in this repo — this harness only reads/assists on `wifi-densepose-sar`, and its MCP server, memory namespace, and Claude Code plugin are scoped to its own name.
+3 -1
View File
@@ -9,7 +9,7 @@ Latest proposed decisions:
- [ADR-264: Versioned wire protocol for RTL8720F CFR and Range-FFT reports](ADR-264-rtl8720f-radar-wire-protocol.md)
- [ADR-263: Adopt RTL8720F 2.4 GHz FMCW radar as an optional RuView sensing platform](ADR-263-rtl8720f-2-4ghz-fmcw-radar-platform.md)
This folder contains 208 Architecture Decision Records (ADRs) that document every significant technical choice in the RuView / WiFi-DensePose project. (The index tables below list a curated subset per domain; see the directory listing for the full set.)
This folder contains 210 Architecture Decision Records (ADRs) that document every significant technical choice in the RuView / WiFi-DensePose project. (The index tables below list a curated subset per domain; see the directory listing for the full set.)
## Why ADRs?
@@ -142,7 +142,9 @@ Statuses: **Proposed** (under discussion), **Accepted** (approved and/or impleme
| [ADR-280](ADR-280-active-sensing-programmable-perception.md) | Active sensing & programmable perception control plane | Accepted (implemented) |
| [ADR-281](ADR-281-ble-cs-delay-doppler-pose-factorization.md) | BLE Channel Sounding, delay-Doppler tensors, P3162 import, factorized pose | Accepted (implemented) |
| [ADR-282](ADR-282-ruview-ecosystem-positioning.md) | Ecosystem positioning + mandatory L0L5 evidence ladder | Accepted |
| [ADR-287](ADR-287-coherent-wideband-rf-tomography-crate.md) | `wifi-densepose-sar` — coherent wideband RF tomography research crate | Accepted (implemented, published) |
| [ADR-285](ADR-285-homecore-wasm-first-metaharness.md) | WASM-first Homecore developer metaharness via `npx homecore` | Accepted (implemented and validated) |
| [ADR-286](ADR-286-wifi-densepose-sar-harness-via-metaharness.md) | `wifi-densepose-sar-harness` — MetaHarness with darwin/router/flywheel | Accepted (implemented, published) |
---
@@ -0,0 +1,23 @@
{
"name": "wifi-densepose-sar-harness",
"version": "0.1.0",
"description": "Harness for wifi-densepose-sar",
"author": {
"displayName": "Generated by metaharness",
"url": "https://www.npmjs.com/package/metaharness"
},
"license": "MIT",
"categories": [
"agent-harness",
"metaharness-scaffold",
"Engineering",
"software-engineering"
],
"tags": [
"metaharness",
"agent-harness",
"vertical:coding",
"software-engineering"
],
"homepage": "https://github.com/ruvnet/agent-harness-generator"
}
@@ -0,0 +1,12 @@
---
description: "Health-check the harness: kernel load, MCP wiring, memory backend, host adapter."
---
Run a full health check and print a PASS/FAIL table.
1. Kernel loads and `kernelInfo().version` matches package.json.
2. The MCP server starts and lists its tools.
3. The memory backend is reachable.
4. The configured host adapter is present.
Exit non-zero if any check fails.
@@ -0,0 +1,10 @@
---
description: "Run the SYNTHETIC @metaharness/flywheel self-improvement demo and report the lift curve."
---
Run the propose → evaluate → gate → promote loop and report what it found.
1. Run `npm run build` first if `dist/flywheel.js` doesn't exist yet.
2. Run `wifi-densepose-sar-harness flywheel [generations]` (default 3 if omitted).
3. Report each generation's `primary` score and `delta`, the total promotions, and whether `verifyReplayBundle` passed.
4. State plainly: this run's `dataSource` is `SYNTHETIC` — the proposer and evaluator are deterministic stand-ins (see `src/flywheel.ts`'s honesty note), not a real model call or a real coding-task benchmark. Do not report its numbers as if they reflect real harness improvement.
@@ -0,0 +1,10 @@
---
description: "Review the current working diff for correctness, security, and reuse."
---
Review the current git diff.
1. `git diff` to read the change.
2. Report only high-confidence findings as `file:line — issue — fix`.
3. Separate bugs from nits.
4. End with APPROVE or REQUEST-CHANGES and a one-line reason.
@@ -0,0 +1,10 @@
---
description: "Route a 4-axis task embedding to the cost-optimal model tier via @metaharness/router."
---
Route one query to the cheapest model tier predicted to clear the quality bar.
1. Run `npm run build` first if `dist/router.js` doesn't exist yet.
2. Run `wifi-densepose-sar-harness route <physicsExplanation> <codeReview> <numericalDebugging> <docWriting>` — four 0..1 numbers scoring how much the query looks like each of those four shapes (see `src/router.ts` for the axis definitions).
3. Report the picked tier (`cheap-tier` or `frontier-tier`), its predicted quality, and whether it cleared the 0.8 quality bar.
4. Remind the user: the labelled examples behind this decision are illustrative seed data (see `src/router.ts`'s honesty note), not measured eval logs — the routing mechanism is real, the specific pick isn't backed by production data yet.
@@ -0,0 +1,40 @@
{
"permissions": {
"allow": [
"Bash(npx wifi-densepose-sar-harness*)",
"mcp__wifi-densepose-sar-harness__*",
"mcp__code_index__*",
"Bash(npm test*)",
"Bash(npm run*)",
"Bash(git diff*)",
"Bash(git status*)",
"Bash(git log*)"
],
"deny": [
"Read(./.env)",
"Read(./.env.*)",
"Bash(git push*)",
"Bash(rm -rf*)"
]
},
"mcpServers": {
"wifi-densepose-sar-harness": {
"command": "npx",
"args": [
"-y",
"wifi-densepose-sar-harness@latest",
"mcp",
"start"
]
},
"code_index": {
"command": "npx",
"args": [
"-y",
"wifi-densepose-sar-harness@latest",
"mcp",
"index"
]
}
}
}
@@ -0,0 +1,53 @@
---
name: evolve
description: "Evolve this harness with Darwin Mode — frozen model, evolving harness (real, sandboxed, safety-gated)."
---
# evolve — Darwin Mode self-improvement
`wifi-densepose-sar-harness` ships with **Darwin Mode** (`@metaharness/darwin`, ADR-070…146): the model
is frozen; the *harness* evolves. Each generation mutates ONE of the 7 surface files
(planner, contextBuilder, reviewer, retry/tool/memory/score policy), sandboxes each
child, scores it, and keeps only variants that *measurably* improve — building an
archive of successful descendants.
## Run it
```bash
npm run evolve # real substrate: runs your test command per variant (deterministic mutator — no API key, no network)
npm run evolve:dry # mock substrate: fast, fully offline, no test execution
```
Or directly:
```bash
npx metaharness-darwin evolve . --sandbox real --generations 3 --children 4
```
## Safety (secure by default)
- **Deterministic mutator** is the default — **no network, no API key, air-gapped**.
- Every mutation passes the `validateGeneratedCode` gate: no new imports, network,
filesystem, shell, env access, or dependencies — pure refactor/tuning only.
- Mutations run in a **sandbox**; only variants that pass your tests are archived.
- Nothing is promoted without measured improvement (guard against Goodharting).
See `@metaharness/darwin` for selection strategies (`--selection`, `--crossover`,
`--curriculum`), statistical gates (`--fdr`, `--bench`), and the real-LLM mutator (library API).
## What the benchmarks taught us (measured, full SWE-bench Lite 300)
Defaults worth carrying into how you evolve and run this harness (full evidence + CIs in
`@metaharness/darwin`'s `LEARNINGS.md` / `bench/results/RESULTS.md`):
1. **Closed-loop repair is the #1 lever (~2×).** Feeding test/compiler failure back and retrying took
resolve-rate 7.7% → 15.3% on the *same cheap model*. Iterate against ground truth, don't single-shot.
2. **Cheap-first + cost-aware routing.** Track **$/resolve**, not just resolve-rate; a cheap model
resolved 31× cheaper per fix than a frontier one. Reserve frontier for *measured* capability gaps.
3. **Tier the models (Barbarian & Scholar).** Cheap sweep + frontier on *only the residual* = 33.3%
at ~6× lower cost than running frontier everywhere.
4. **Put the output-format contract in a system message + example**, and size prompts to the model's
real context window — this alone took a weak local model from 0% to ~50% valid output.
5. **Only trust batch evaluation of the final artifact** — in-loop counters drift 1.55×.
6. **The harness multiplies the model; it can't rescue one below the task's reasoning floor.** Pick
the smallest model *above* the floor, then let evolution do the rest.
@@ -0,0 +1,15 @@
---
name: plan-change
description: "Turn a feature request into a minimal, file-level implementation plan before any code."
---
# plan-change
Produce an implementation plan for a requested change.
1. Restate the goal in one sentence.
2. List the files to touch and why.
3. Name the smallest interface that satisfies it.
4. Flag anything that ripples beyond three files or widens a permission.
Hand the plan to the implementer; do not write code in this step.
@@ -0,0 +1,39 @@
{
"schema": 1,
"generator": "0.1.0",
"template": "vertical:coding",
"template_version": "0.0.0",
"vars": {
"name": "wifi-densepose-sar-harness",
"description": "Harness for wifi-densepose-sar",
"host": "claude-code"
},
"hosts": [
"claude-code"
],
"files": {
".claude/commands/doctor.md": "2f1475fa0ed34729cb2ed9d6cecf29e99781da0a56afef133361e3ff0a17bb52",
".claude/commands/review-diff.md": "2ab52f01487bfe67335f4de5193d7a6fb0f72f646411114613d8fa5da071ef2b",
".claude/settings.json": "ea983de8f425d313ee42848a7da58ce98e05713a06dc2f45a5119f421a311e07",
".claude/skills/plan-change/SKILL.md": "84e1c44ca264b999ebf80cd8fa0e5ebf554275bbd8023b1530ad053a44482e30",
".claude-plugin/plugin.json": "f716a379f3e077b47dae884b5ca0d4a077e9a1ab1f091afa8f0f031ead1008e5",
"bin/cli.js": "78bd27074b3ce22bff63729995032c7b262a414ebfc8d75345fc82b40a4a84ee",
"CLAUDE.md": "4e9e11558e124605fd1a5de4be116c2b8d9dc15a3bc7c8f6789a7eec23dac19d",
"package.json": "a9f103fc452b5497a41333d2e80c834de07972e951be1701f25f98a5acaaea9d",
"README.md": "0efc949a11c20a8179261887d44afac60fb6e28af1ee0cf13451bd665b1cfbd4",
"src/agents/architect.ts": "b146bb8ad7729d7738f6c07d7770fecb6a13cf7683019c8b8afaec7c13806515",
"src/agents/implementer.ts": "55898d303ae87574348821ef2c270eceeec733278bb94041d767308b3a509291",
"src/agents/reviewer.ts": "ef5ab428ea799be7caeb05e761723122c52ab35d9ef4a18b618647ff72549ad8",
"src/agents/test-writer.ts": "f966a4d3a97a02b98606aac104f6fcbcf294c0186c15d7a809aa7555b20c0b5f",
"src/init.ts": "e19b1dbe6e4c3b7282a102bf068c20630e8cb0323d8abefa477d0f2dbbc68ce2",
"tsconfig.json": "8b4e730a1aa39162ac574455d7a98e1881f5313ca80ffe503b9652dcf0c76b9d",
"vitest.config.ts": "e9e94875611ab1cd602c1f6923902c2dab7953a75ac022962973639d1937587a",
"__tests__/smoke.test.ts": "e38dfc1389b8b419841f2c6513854dbb291e957be0c1c325c507dc58ef3d8c5e",
"LICENSE": "0580604803a2c38a1e4fe2c86d5e2f6d213cc3317cd5bc050691e484c523aedc",
".claude/skills/evolve/SKILL.md": "05965b34edd9bbea83391ae20c5c22bf2653396e7b54557cfcf395fd7d11294a"
},
"generated_at": "2026-07-30T22:39:05.579Z",
"meta": {
"surface": "cli"
}
}
@@ -0,0 +1 @@
b5ac57b4ae092fda710fef597519f3f623ebd9bb40bee9c7e010d0ad42200241
+45
View File
@@ -0,0 +1,45 @@
# wifi-densepose-sar-harness
Harness for [wifi-densepose-sar](https://crates.io/crates/wifi-densepose-sar) (ADR-287) — the coherent wideband RF tomography research crate this harness assists development on. Both are published: the crate on crates.io, this harness itself as [`wifi-densepose-sar-harness`](https://www.npmjs.com/package/wifi-densepose-sar-harness) on npm.
> Advanced Coding harness · domain: `software-engineering`. Generated with [create-agent-harness](https://github.com/ruvnet/agent-harness-generator).
## Behavioral rules
- Use the harness's MCP tools (`mcp__wifi-densepose-sar-harness__*`) for orchestration
- Memory and routing are handled by the kernel — you don't need to learn them
- Defer destructive operations to the user
## Agents
| Agent | Tier | Role |
|---|---|---|
| `architect` | opus | Designs the change before code is written. |
| `implementer` | sonnet | Writes code that matches the surrounding style. |
| `reviewer` | opus | Hunts correctness bugs in the diff. |
| `test-writer` | sonnet | Adds the missing tests for the change. |
## Skills
- `/plan-change` — Turn a feature request into a minimal, file-level implementation plan before any code.
- `/evolve` — Run Darwin Mode (`npm run evolve` / `evolve:dry`) to self-mutate the harness's own operating policy and keep only measurable improvements.
## Commands
Each command below has a matching `.claude/commands/<name>.md` guidance file — the MCP tool listing (`mcp__wifi-densepose-sar-harness__*`) is derived from these, so a new CLI subcommand isn't fully wired up until it has one too.
- `doctor` — Health-check the harness: kernel load, MCP wiring, memory backend, host adapter.
- `review-diff` — Review the current working diff for correctness, security, and reuse.
- `route <e0> <e1> <e2> <e3>` — cost-optimal model routing via `@metaharness/router` (needs `npm run build` first).
- `flywheel [generations]` — run the SYNTHETIC self-improvement demo via `@metaharness/flywheel` (needs `npm run build` first).
## Architecture
This harness uses [@metaharness/kernel](https://www.npmjs.com/package/@metaharness/kernel) — a Rust-compiled WASM module with a NAPI-RS native fallback — so the same code runs identically on every platform.
### Darwin, router, flywheel
Three complementary self-improvement/cost pieces, all real npm dependencies (not aspirational):
- **Darwin Mode** (`@metaharness/darwin`, devDependency) — `npm run evolve` (real sandbox) / `npm run evolve:dry` (mock sandbox) mutates the harness's own config and keeps only changes that measurably improve it. Wired by the scaffold itself.
- **Router** (`@metaharness/router`) — `src/router.ts` wires a real `Router` with a `qualityBar: 0.8` cost-optimal policy over two example model tiers. Its labelled examples are illustrative/seed data (see the file's honesty note), not measured eval-log observations — the routing *mechanism* is real and tested (`__tests__/router.test.ts`), the specific decisions it makes today are not yet backed by real data.
- **Flywheel** (`@metaharness/flywheel`) — `src/flywheel.ts` wires the real `runFlywheelGenerations` promotion loop (propose → evaluate → gate → promote, Ed25519-signed, independently replayable) with a SYNTHETIC proposer/evaluator (`dataSource: 'SYNTHETIC'`, no model call). It proves the wiring end-to-end (`__tests__/flywheel.test.ts` checks a real lift curve and a passing `verifyReplayBundle`); a LIVE run needs a real Proposer (model call) and Evaluator (real coding-task holdout/anchor suites) supplied by the operator — see the file's comments for exactly what those seams are.
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 wifi-densepose-sar-harness authors
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+42
View File
@@ -0,0 +1,42 @@
# wifi-densepose-sar-harness
Harness for wifi-densepose-sar
> **Advanced Coding** — Architect → implement → review → test, with a code-index MCP and push-guarded git perms.
>
> Generated with [`create-agent-harness`](https://github.com/ruvnet/agent-harness-generator). Multi-host scaffolding with a kernel that resolves native → wasm → js (js backend in the published beta; see `harness doctor`).
## Install
```bash
npm install -g wifi-densepose-sar-harness
wifi-densepose-sar-harness init
wifi-densepose-sar-harness doctor
```
## Agents
| Agent | Role |
|---|---|
| `architect` | Designs the change before code is written. |
| `implementer` | Writes code that matches the surrounding style. |
| `reviewer` | Hunts correctness bugs in the diff. |
| `test-writer` | Adds the missing tests for the change. |
This harness ships with the **claude-code** adapter.
## Darwin, router, flywheel
- `npm run evolve` / `evolve:dry` — Darwin Mode self-mutation of the harness's own config (`@metaharness/darwin`).
- `npm run route -- <e0> <e1> <e2> <e3>` (after `npm run build`) — cost-optimal model routing via `@metaharness/router`.
- `npm run flywheel:dry` — the SYNTHETIC `@metaharness/flywheel` self-improvement demo (propose → evaluate → gate → promote, signed + independently replayable).
See `CLAUDE.md`'s "Darwin, router, flywheel" section and the comments at the top of `src/router.ts` / `src/flywheel.ts` for what's real wiring vs. illustrative/synthetic data.
## Known gaps
- `.harness/manifest.json` / `manifest.sha256` reflect the initial `metaharness analyze --scaffold` output and were not regenerated after adding `src/router.ts`, `src/flywheel.ts`, and their tests — this scaffold has no `manifest:update` script (unlike `harness/homecore/`). Treat the manifest as historical provenance for the scaffold step, not a current file-integrity check.
## License
MIT
@@ -0,0 +1,37 @@
// SPDX-License-Identifier: MIT
import { describe, it, expect } from 'vitest';
import { runSarFlywheelDemo, verifySarFlywheelDemo, SAR_ROOT_POLICY } from '../src/flywheel.js';
describe('runSarFlywheelDemo — @metaharness/flywheel wiring (SYNTHETIC data)', () => {
it('runs the configured number of generations and stamps SYNTHETIC provenance', async () => {
const result = await runSarFlywheelDemo(3);
expect(result.generationsRun).toBeGreaterThan(0);
expect(result.generationsRun).toBeLessThanOrEqual(3);
expect(result.replayBundle.data_source).toBe('SYNTHETIC');
});
it('produces a non-empty lift curve rooted at the initial policy', async () => {
const result = await runSarFlywheelDemo(3);
expect(result.liftCurve.length).toBeGreaterThan(0);
expect(result.liftCurve[0].generation).toBe(0);
});
it('the final policy still carries every root lever', async () => {
const result = await runSarFlywheelDemo(2);
for (const key of Object.keys(SAR_ROOT_POLICY)) {
expect(result.finalPolicy).toHaveProperty(key);
}
});
it('the replay bundle independently verifies (no trust in the producer)', async () => {
const result = await runSarFlywheelDemo(3);
const verdict = verifySarFlywheelDemo(result);
expect(verdict.pass).toBe(true);
});
it('is deterministic in structure across repeated runs (same generations requested)', async () => {
const a = await runSarFlywheelDemo(2);
const b = await runSarFlywheelDemo(2);
expect(a.generationsRun).toBe(b.generationsRun);
});
});
@@ -0,0 +1,36 @@
// SPDX-License-Identifier: MIT
import { describe, it, expect } from 'vitest';
import { sarTaskRouter, routeSarQuery, SAR_ROUTER_CANDIDATES } from '../src/router.js';
describe('sarTaskRouter — @metaharness/router wiring', () => {
it('has both a cheap and a frontier candidate', () => {
const ids = SAR_ROUTER_CANDIDATES.map((c) => c.id);
expect(ids).toContain('cheap-tier');
expect(ids).toContain('frontier-tier');
});
it('routes a physics-explanation-shaped query to the cheap tier', () => {
const pick = routeSarQuery([1, 0, 0, 0]);
expect(pick.id).toBe('cheap-tier');
expect(pick.metBar).toBe(true);
});
it('routes a code-review-shaped query to the frontier tier (cheap tier misses the quality bar)', () => {
const pick = routeSarQuery([0, 1, 0, 0]);
expect(pick.id).toBe('frontier-tier');
});
it('always returns a candidate with a nonnegative predicted quality and a positive cost', () => {
for (const q of [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]] as const) {
const pick = routeSarQuery(q);
expect(pick.predictedQuality).toBeGreaterThanOrEqual(0);
expect(pick.costPerMTok).toBeGreaterThan(0);
}
});
it('sarTaskRouter.route and routeSarQuery agree (same underlying router)', () => {
const a = sarTaskRouter.route([1, 0, 0, 0]);
const b = routeSarQuery([1, 0, 0, 0]);
expect(a).toEqual(b);
});
});
@@ -0,0 +1,37 @@
// SPDX-License-Identifier: MIT
// Generated by metaharness — a real smoke test for wifi-densepose-sar-harness.
//
// This is NOT a placeholder: it boots the actual kernel + host adapter the
// harness depends on, so `npm test` fails loudly if @metaharness/kernel or
// @metaharness/host-claude-code is missing, broken, or version-skewed. It is the
// fastest signal that `npm install` produced a runnable harness.
import { describe, it, expect } from 'vitest';
import { loadKernel } from '@metaharness/kernel';
import adapter from '@metaharness/host-claude-code';
import { run } from '../bin/cli.js';
describe('wifi-densepose-sar-harness — install smoke test', () => {
it('loads the kernel and reports a version + a known backend', async () => {
const kernel = await loadKernel();
const info = kernel.kernelInfo();
expect(typeof info.version).toBe('string');
expect(info.version.length).toBeGreaterThan(0);
expect(['native', 'wasm', 'js']).toContain(kernel.backend);
});
it('resolves the host adapter with a name', () => {
expect(typeof adapter.name).toBe('string');
expect(adapter.name.length).toBeGreaterThan(0);
});
it('the CLI doctor command succeeds (exit 0)', async () => {
const code = await run(['doctor']);
expect(code).toBe(0);
});
it('an unknown CLI command exits non-zero', async () => {
const code = await run(['definitely-not-a-command']);
expect(code).not.toBe(0);
});
});
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env node
// SPDX-License-Identifier: MIT
// Generated by metaharness — the `wifi-densepose-sar-harness` CLI entry point.
//
// This is plain ESM JavaScript on purpose: it runs as-is via `npx wifi-densepose-sar-harness`
// with NO build step. `npm run build` (tsc) is only needed if you extend the
// TypeScript in src/. The published package ships this file directly (see the
// "bin" + "files" fields in package.json), so `npx wifi-densepose-sar-harness` works the moment
// `npm install` has resolved @metaharness/kernel + @metaharness/host-claude-code.
import { loadKernel } from '@metaharness/kernel';
import adapter from '@metaharness/host-claude-code';
const HARNESS_NAME = 'wifi-densepose-sar-harness';
/** `wifi-densepose-sar-harness init` — boot the kernel + host adapter and report status. */
async function init() {
const kernel = await loadKernel();
const info = kernel.kernelInfo();
console.log(`${HARNESS_NAME} — kernel ${info.version} (${kernel.backend})`);
console.log(`Host adapter: ${adapter.name}`);
console.log(`Run \`${HARNESS_NAME} doctor\` to verify the install.`);
return 0;
}
/**
* `wifi-densepose-sar-harness route <e0> <e1> <e2> <e3>` — route a 4-axis
* task embedding to the cost-optimal model tier via @metaharness/router.
* Needs `npm run build` first (src/router.ts is TypeScript; this command
* imports the compiled dist/ output so the base CLI stays build-free).
*/
async function route(args) {
const embedding = args.map(Number);
if (embedding.length !== 4 || embedding.some((n) => Number.isNaN(n))) {
console.error('Usage: wifi-densepose-sar-harness route <physicsExplanation> <codeReview> <numericalDebugging> <docWriting> (four 0..1 numbers)');
return 2;
}
let routeSarQuery;
try {
({ routeSarQuery } = await import('../dist/router.js'));
} catch (err) {
console.error(`route: dist/router.js not found — run \`npm run build\` first. (${err.message})`);
return 1;
}
const pick = routeSarQuery(embedding);
console.log(`route -> ${pick.id} (predicted quality ${pick.predictedQuality.toFixed(3)}, $${pick.costPerMTok}/MTok, met bar: ${pick.metBar})`);
return 0;
}
/**
* `wifi-densepose-sar-harness flywheel [generations]` — run the SYNTHETIC
* @metaharness/flywheel demo (see src/flywheel.ts) and print the lift curve
* + an independent replay-bundle verification. Needs `npm run build` first.
*/
async function flywheel(args) {
const generations = args[0] ? Number(args[0]) : 3;
if (Number.isNaN(generations) || generations < 1) {
console.error('Usage: wifi-densepose-sar-harness flywheel [generations>=1]');
return 2;
}
let runSarFlywheelDemo, verifySarFlywheelDemo;
try {
({ runSarFlywheelDemo, verifySarFlywheelDemo } = await import('../dist/flywheel.js'));
} catch (err) {
console.error(`flywheel: dist/flywheel.js not found — run \`npm run build\` first. (${err.message})`);
return 1;
}
console.log(`Running ${generations}-generation flywheel demo (dataSource: SYNTHETIC — see src/flywheel.ts)...`);
const result = await runSarFlywheelDemo(generations);
for (const point of result.liftCurve) {
console.log(` gen ${point.generation}: primary=${point.primary.toFixed(3)} delta=${point.delta.toFixed(3)} anchor=${point.anchor ?? 'n/a'}`);
}
const verdict = verifySarFlywheelDemo(result);
console.log(`generations run: ${result.generationsRun} · promotions: ${result.promotions.length} · replay verified: ${verdict.pass}`);
return verdict.pass ? 0 : 1;
}
/** `wifi-densepose-sar-harness doctor` — verify the install end-to-end (kernel + host resolve). */
async function doctor() {
const kernel = await loadKernel();
const info = kernel.kernelInfo();
const checks = [
['kernel loads', !!kernel],
['kernel reports a version', typeof info.version === 'string' && info.version.length > 0],
['kernel backend is native|wasm|js', ['native', 'wasm', 'js'].includes(kernel.backend)],
['host adapter has a name', typeof adapter?.name === 'string' && adapter.name.length > 0],
];
let ok = true;
for (const [label, pass] of checks) {
console.log(`${pass ? 'PASS' : 'FAIL'} ${label}`);
if (!pass) ok = false;
}
console.log(
ok
? `\n${HARNESS_NAME}: all checks passed (kernel ${info.version}, ${kernel.backend} backend, host ${adapter.name})`
: `\n${HARNESS_NAME}: doctor found problems`,
);
return ok ? 0 : 1;
}
/**
* Dispatch one CLI invocation. Exported (not just run on import) so a test can
* drive it without spawning a subprocess. Returns the intended exit code.
*/
export async function run(argv) {
const cmd = argv[0] ?? 'init';
switch (cmd) {
case 'init':
return init();
case 'doctor':
return doctor();
case 'route':
return route(argv.slice(1));
case 'flywheel':
return flywheel(argv.slice(1));
case '--version':
case '-v': {
const kernel = await loadKernel();
console.log(kernel.version());
return 0;
}
case '--help':
case '-h':
console.log(`Usage: ${HARNESS_NAME} <command>\n\n init boot the kernel + host adapter (default)\n doctor verify the install end-to-end\n route route a 4-axis task embedding to a cost-optimal model tier (needs \`npm run build\`)\n flywheel run the SYNTHETIC self-improvement demo loop (needs \`npm run build\`)\n --version print the kernel version`);
return 0;
default:
console.error(`Unknown command: ${cmd}. Try \`${HARNESS_NAME} --help\`.`);
return 2;
}
}
// CLI guard: execute only when invoked directly (not when imported by a test).
// npm's bin shims pass a NON-normalized argv[1] (e.g. ".../.bin/../<pkg>/bin/cli.js"
// on Windows) and may differ in case, so realpath BOTH sides before comparing —
// a naive string === misses the npx/shim path and the CLI silently no-ops.
import { fileURLToPath } from 'node:url';
import { realpathSync } from 'node:fs';
import { argv } from 'node:process';
const invokedDirectly = (() => {
if (!argv[1]) return false;
try {
const a = realpathSync(argv[1]);
const b = realpathSync(fileURLToPath(import.meta.url));
return process.platform === 'win32' ? a.toLowerCase() === b.toLowerCase() : a === b;
} catch {
return false;
}
})();
if (invokedDirectly) {
run(argv.slice(2))
.then((code) => process.exit(code))
.catch((err) => {
console.error(err);
process.exit(1);
});
}
+48
View File
@@ -0,0 +1,48 @@
{
"name": "wifi-densepose-sar-harness",
"version": "0.1.0",
"description": "Harness for wifi-densepose-sar",
"license": "MIT",
"type": "module",
"bin": {
"wifi-densepose-sar-harness": "bin/cli.js"
},
"files": [
"bin/**",
"dist/**",
"src/**",
"tsconfig.json",
".claude/**",
"CLAUDE.md",
"README.md",
"LICENSE"
],
"scripts": {
"build": "tsc",
"test": "vitest run",
"init": "node ./bin/cli.js init",
"doctor": "node ./bin/cli.js doctor",
"evolve": "metaharness-darwin evolve . --sandbox real --generations 3 --children 4",
"evolve:dry": "metaharness-darwin evolve . --sandbox mock --generations 2 --children 3",
"route": "node ./bin/cli.js route",
"flywheel:dry": "npm run build && node ./bin/cli.js flywheel 3"
},
"dependencies": {
"@metaharness/kernel": "^0.1.0",
"@metaharness/host-claude-code": "^0.1.0",
"@metaharness/router": "^0.3.2",
"@metaharness/flywheel": "^0.1.7"
},
"devDependencies": {
"@types/node": "^20.0.0",
"typescript": "^5.4.0",
"vitest": "^3.0.0",
"@metaharness/darwin": "^0.2.2"
},
"engines": {
"node": ">=20.0.0"
},
"publishConfig": {
"access": "public"
}
}
@@ -0,0 +1,7 @@
// SPDX-License-Identifier: MIT
// Architect agent — Designs the change before code is written.
export const SYSTEM_PROMPT = `You are the architect. Before any code is written you produce the smallest design that satisfies the request: the files to touch, the interfaces to add, and the trade-offs. You never write the implementation — you hand a crisp plan to the implementer. Prefer reuse over new abstractions; call out any change that ripples beyond three files. You operate inside the wifi-densepose-sar-harness harness; defer destructive actions to the user.`;
export const NAME = 'architect';
export const TIER = 'opus' as const;
@@ -0,0 +1,7 @@
// SPDX-License-Identifier: MIT
// Implementer agent — Writes code that matches the surrounding style.
export const SYSTEM_PROMPT = `You implement the architect's plan. Match the existing code's naming, comment density, and idioms — your diff should read like the person who wrote the file kept writing. Make the minimal change; do not refactor unrelated code. Leave the tests to the test-writer unless asked. You operate inside the wifi-densepose-sar-harness harness; defer destructive actions to the user.`;
export const NAME = 'implementer';
export const TIER = 'sonnet' as const;
@@ -0,0 +1,7 @@
// SPDX-License-Identifier: MIT
// Reviewer agent — Hunts correctness bugs in the diff.
export const SYSTEM_PROMPT = `You review diffs for correctness, security, and reuse. Report only high-confidence findings, each with a file:line and a concrete fix. Distinguish a bug (will break) from a nit (style). Never approve a change that widens a permission, swallows an error, or ships a secret. You operate inside the wifi-densepose-sar-harness harness; defer destructive actions to the user.`;
export const NAME = 'reviewer';
export const TIER = 'opus' as const;
@@ -0,0 +1,7 @@
// SPDX-License-Identifier: MIT
// Test Writer agent — Adds the missing tests for the change.
export const SYSTEM_PROMPT = `You write the tests the change needs: the happy path, the boundary, and the one failure mode most likely to regress. Mirror the project's existing test style and runner. A test that cannot fail is worse than no test — assert behaviour, not implementation. You operate inside the wifi-densepose-sar-harness harness; defer destructive actions to the user.`;
export const NAME = 'test-writer';
export const TIER = 'sonnet' as const;
+114
View File
@@ -0,0 +1,114 @@
// SPDX-License-Identifier: MIT
//
// The wifi-densepose-sar harness's self-improvement loop, via
// @metaharness/flywheel: run -> measure -> mutate -> verify -> promote,
// with a frozen, conjunctive promotion gate and a signed, replayable lineage
// (see @metaharness/darwin's `npm run evolve` for the harness-wide mutation
// entry point this formalizes the promotion loop for).
//
// HONESTY NOTE (load-bearing): `runSarFlywheelDemo()` below wires the real
// @metaharness/flywheel API end-to-end, but its Proposer and Evaluator are
// SYNTHETIC stand-ins — a deterministic string mutation and a deterministic
// scoring function over that string, with NO model call and NO real
// benchmark suite. It exists to prove the wiring works (see
// __tests__/flywheel.test.ts: a non-empty lift curve, a verifiable replay
// bundle) and to give a `dataSource: 'SYNTHETIC'`-stamped demo, exactly as
// the flywheel's own design requires callers to be honest about data
// provenance. A LIVE run needs the operator to supply:
// - a real Proposer: an actual model call that improves one policy lever
// (e.g. the review-diff checklist, the architect's planning prompt);
// - a real Evaluator: scores that policy against real coding-task
// holdout/anchor suites (e.g. "did review-diff catch the seeded bug").
// Neither exists in this repo — wiring them is a live-API-key decision for
// whoever operates this harness, not something to fake here.
import {
runFlywheelGenerations,
meetsPromotionRule,
makeSigner,
verifyReplayBundle,
type Policy,
type PolicyGenome,
type Proposer,
type Evaluator,
type Suite,
type FlywheelResult,
} from '@metaharness/flywheel';
/** The gen-0 operating policy for the SAR harness's review agents. Opaque
* string levers — the flywheel never interprets their meaning, only the
* Evaluator does. */
export const SAR_ROOT_POLICY: Policy = {
reviewDepth: 'standard-checklist',
architectPlanning: 'single-pass',
};
/**
* SYNTHETIC proposer: deterministically lengthens/varies the target lever's
* value rather than calling a model. Stands in for a real model call that
* would draft an improved lever value.
*/
const syntheticProposer: Proposer = async (base: PolicyGenome, target: string) => {
const current = base.policy[target] ?? '';
return `${current}+g${base.generation + 1}`;
};
/**
* SYNTHETIC evaluator: scores a policy purely as a function of its own
* string content (longer, more "refined"-looking levers score marginally
* higher on quality and lower on no-op rate, both with a floor/ceiling) —
* a deterministic stand-in for actually running the harness's agents
* against a real coding-task suite. `noopRate` must move (not stay
* constant) for anything to ever promote: the default gate's clause 2
* requires it to strictly improve generation over generation (ADR-226's
* "the executor policy is the part that mattered" finding, encoded as a
* hard requirement) — a constant noopRate, even a "good" one, gates every
* candidate out forever.
*/
const syntheticEvaluator: Evaluator = async (policy: Policy, _suite: Suite) => {
const totalLength = Object.values(policy).reduce((s, v) => s + v.length, 0);
const primary = Math.min(0.5 + totalLength / 200, 0.98);
const noopRate = Math.max(0.3 - totalLength / 300, 0.02);
return {
primary,
noopRate,
costPerWin: 1 / primary,
regressed: false,
};
};
const SAR_HOLDOUT: Suite = {
id: 'sar-harness-holdout-synthetic',
items: ['seeded-review-task-1', 'seeded-review-task-2', 'seeded-review-task-3'],
};
const SAR_ANCHOR: Suite = {
id: 'sar-harness-anchor-synthetic',
items: ['frozen-regression-task-1'],
};
/**
* Run a small, fully SYNTHETIC flywheel demo end-to-end: propose, evaluate,
* gate, and (when it clears the gate) promote a few generations of mutated
* policy, returning the real @metaharness/flywheel result — a genuine lift
* curve and a signed, independently replayable bundle, just built from
* synthetic (not live) evidence.
*/
export async function runSarFlywheelDemo(maxGenerations = 3): Promise<FlywheelResult> {
return runFlywheelGenerations({
rootPolicy: SAR_ROOT_POLICY,
proposer: syntheticProposer,
evaluator: syntheticEvaluator,
promotionRule: meetsPromotionRule,
holdout: SAR_HOLDOUT,
anchor: SAR_ANCHOR,
maxGenerations,
signer: makeSigner(),
dataSource: 'SYNTHETIC',
});
}
/** Independently verify a flywheel demo's replay bundle (no trust in the producer). */
export function verifySarFlywheelDemo(result: FlywheelResult) {
return verifyReplayBundle(result.replayBundle);
}
+21
View File
@@ -0,0 +1,21 @@
// SPDX-License-Identifier: MIT
// Generated by create-agent-harness — your harness's `wifi-densepose-sar-harness init` entry.
import { loadKernel } from '@metaharness/kernel';
import adapter from '@metaharness/host-claude-code';
const HARNESS_NAME = 'wifi-densepose-sar-harness';
async function main(): Promise<number> {
const kernel = await loadKernel();
const info = kernel.kernelInfo();
console.log(`${HARNESS_NAME} — kernel ${info.version} (${kernel.backend})`);
console.log(`Host adapter: ${adapter.name}`);
console.log(`Run \`${HARNESS_NAME} doctor\` to verify the install.`);
return 0;
}
main().then(c => process.exit(c)).catch(err => {
console.error(err);
process.exit(1);
});
+77
View File
@@ -0,0 +1,77 @@
// SPDX-License-Identifier: MIT
//
// Cost-optimal task routing for the wifi-densepose-sar harness, via
// @metaharness/router (ADR-040's DRACO Phase-2 finding, productized): route
// each agent query to the cheapest model predicted to clear a quality bar,
// instead of sending every query to the frontier tier by default.
//
// HONESTY NOTE: the candidate `examples` below are SEED/ILLUSTRATIVE data —
// four hand-picked (embedding, quality) points per candidate, not measured
// eval-log observations. They exist so `sarTaskRouter` is a real, runnable
// k-NN router out of the box (see __tests__/router.test.ts), not so its
// routing decisions should be trusted for production cost savings. Replace
// `SAR_ROUTER_CANDIDATES[*].examples` with real (query embedding → quality
// achieved) rows from your own eval logs before relying on this for
// production routing — see @metaharness/router's README ("the more
// examples, the closer it gets to the per-query oracle").
import { Router, type RouterCandidate } from '@metaharness/router';
/**
* A 4-axis feature embedding for a harness query, used only to pick a
* nearby labelled example — NOT a real text embedding. Axes (each 0..1):
* [0] physicsExplanation — "explain the range-resolution formula"-shaped
* [1] codeReview — "review this diff for correctness"-shaped
* [2] numericalDebugging — "why did this reconstruction test fail"-shaped
* [3] docWriting — "write/update the tutorial"-shaped
* A caller with a real embedding model should project onto whatever
* dimensionality that model produces instead — the router only needs
* consistent vectors, not these specific four axes.
*/
export type SarTaskEmbedding = readonly [number, number, number, number];
export const SAR_ROUTER_CANDIDATES: RouterCandidate[] = [
{
id: 'cheap-tier',
costPerMTok: 1,
examples: [
{ embedding: [1, 0, 0, 0], quality: 0.9 }, // physics explanations: cheap tier does fine
{ embedding: [0, 0, 0, 1], quality: 0.85 }, // doc writing: cheap tier does fine
{ embedding: [0, 1, 0, 0], quality: 0.55 }, // code review: cheap tier is weak
{ embedding: [0, 0, 1, 0], quality: 0.5 }, // numerical debugging: cheap tier is weak
],
},
{
id: 'frontier-tier',
costPerMTok: 15,
examples: [
{ embedding: [1, 0, 0, 0], quality: 0.95 },
{ embedding: [0, 0, 0, 1], quality: 0.93 },
{ embedding: [0, 1, 0, 0], quality: 0.92 }, // code review: frontier tier needed
{ embedding: [0, 0, 1, 0], quality: 0.9 }, // numerical debugging: frontier tier needed
],
},
];
/**
* Cost-optimal router for the harness's four query shapes above. `qualityBar`
* of 0.8 matches the router README's worked example: return the cheapest
* candidate predicted to clear 80% quality, or the best-predicted candidate
* if none do.
*/
export const sarTaskRouter = new Router({
qualityBar: 0.8,
candidates: SAR_ROUTER_CANDIDATES,
// k=1: each candidate has only 4 (orthogonal, one-hot) examples covering
// the 4 task axes. The router's default k=5 would average ALL of a
// candidate's examples regardless of query similarity once a candidate
// has <=5 examples, collapsing every query to the same prediction. k=1
// makes it pick the single nearest labelled task type, which is what
// this small illustrative dataset is shaped for.
k: 1,
});
/** Route one query embedding to the cost-optimal model tier. */
export function routeSarQuery(queryEmbedding: SarTaskEmbedding) {
return sarTaskRouter.route([...queryEmbedding]);
}
+19
View File
@@ -0,0 +1,19 @@
{
"compilerOptions": {
"target": "ES2022",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"lib": ["ES2022"],
"outDir": "./dist",
"rootDir": "./src",
"declaration": true,
"declarationMap": true,
"sourceMap": true,
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true
},
"include": ["src/**/*.ts"],
"exclude": ["node_modules", "dist", "__tests__"]
}
@@ -0,0 +1,23 @@
// SPDX-License-Identifier: MIT
// Generated by metaharness. Strips the `#!/usr/bin/env node` shebang from
// importable entrypoints (e.g. bin/cli.js) before Vite parses them — Vite/esbuild
// (used internally by Vitest) does NOT strip shebangs, so importing a shebanged
// module throws `SyntaxError: Invalid or unexpected token`. See issue #44.
// Has no effect on direct CLI execution or `npm run doctor` (those bypass Vite).
import { defineConfig } from 'vitest/config';
export default defineConfig({
plugins: [
{
name: 'strip-shebang',
enforce: 'pre',
transform(code: string) {
if (code.startsWith('#!')) {
// Replace with a blank line so source line numbers stay aligned.
return { code: code.replace(/^#![^\n]*/, ''), map: null };
}
return null;
},
},
],
});