Agent harness
A routing and cost-governance layer over a terminal coding agent: four seats, a priced ledger, a bounded tuner and a verifier.
Context
- Role
- Builder: routing, daemon, ledger, auto-tuner, verification
- Team
- Solo, personal infrastructure
- Duration
- 2026
- Stack
- Python, PowerShell, TOML, Grok Build, DeepSeek, Z.ai
- Status
- Personal tool; public repository
The harness is the layer I built around a terminal coding agent. It does not ship the agent or the models. It governs which seat runs a task, what the turn cost, and what has to be true before a change is accepted.
Four model seats. Three switchable profiles. A Python ledger that prices every inference. A tuner that adjusts one setting at a time, inside hard bounds, and refuses to write a config it cannot parse. A verifier that checks the whole arrangement before a session trusts it.
The expensive model was the default because nobody measured.
Escalating every task to the strongest model feels safe. It is a cost decision hidden as a quality decision, and without a per-session view there is no way to argue with it. The bill arrives at the end of the month with no attribution.
The context paid for a library it never used
Every prompt carried skills the task did not need. Ten always-on skills moved to on-demand routing cut roughly 1,000 prompt tokens per turn. That is a small number per turn and a large number across a working week.
Automation can corrupt the thing it manages
A tuner that writes a broken TOML file is worse than no tuner. A profile switch that overwrites the context window the tuner just raised is worse than no profile. The safety case for automation is the interesting engineering, not the automation itself.
Design arrived after the code
The interface was being built before a direction was agreed. The harness added a gate: directions on a canvas, reviewed, then one chosen. The build waits for the pick.
Automate the decision, keep the refusal.
Approach
Measure the escalation before trusting it
I ran a 33-task benchmark twice: once with the cheap model as the default and escalation only for planning, hard bugs and review, and once with blanket escalation. Blanket escalation solved one more task, a 3% gain, for 8% more cost. The cheap default stayed.
That result became policy. The routing skill names the seat for each kind of work; the subagent personas put planning and review on the judgment seat. Escalation is a decision with a reason, not a reflex.
| deepseek-flash | Volume workhorse for implementation, exploration and routine edits |
|---|---|
| deepseek-flash-high | The same model at high effort, with a larger output budget; the starter default |
| deepseek-pro | Judgment seat for planning, review and hard bugs; both personas run here |
| glm-flash | Long-context overflow seat on a second provider, for bulk work |
Make the spend visible
The ledger tails the agent's unified log and appends one row per inference. It records prompt tokens, cached tokens, cache misses, completion and reasoning tokens, and an estimated dollar cost. It aggregates session signals into a second file and refreshes a report every fifteen minutes.
| Input | ~/.grok/logs/unified.jsonl, one inference event at a time |
|---|---|
| Pricing | DeepSeek card: $0.0028 cache hit, $0.14 cache miss, $0.28 output per million tokens |
| Output | usage.jsonl, session_stats.jsonl and a generated report |
| Scope | Calls from other providers are recorded and kept out of the DeepSeek dollar total |
Make the automation bounded
The tuner changes one setting at a time. A context window can only rise, in 64K steps, between 128K and 512K, and at most once per seat per day. Reasoning effort can only fall, and only when reasoning tokens exceed 20% of output. A failing optional MCP server can be disabled. Nothing else is in scope.
| Parse first | The new TOML is parsed before it is written; an invalid config is refused |
|---|---|
| Dry run | --dry-run prints the intended change without touching the file |
| Backup | A timestamped copy is written before the first automatic edit |
| Audit | Every applied change is appended to a log with the key, old value, new value and reason |
| Pins | Values pinned in the overrides file are never overwritten |
| Carry | A profile switch re-applies the tuner state instead of wiping it |
Verify the whole machine
The verifier parses all three configs and requires the same seat set and the same default. It rejects retired seat names, checks that the old proxy ports are closed, and counts the skill junctions against the expected set. A failure is a failed check, not a warning in a log.
A separate integrity canary covers seventeen policy files: hooks, rules and gates. It stores a baseline and alerts on unexpected changes. The point is to notice drift in the guardrails, before the guardrails quietly stop guarding.
Close the loop on context and design
The same routing idea applies to skills and tools. Only the one to three skills a task needs are loaded. Pencil and Inspo stay off in normal sessions; the design lane starts a child session with them, produces canvas directions, and exits. The parent build session never carries the design tools.
Hooks were scoped to files changed in the current task, which cut the UI-quality check from 104 seconds to about 12. A gate that is too slow gets disabled, so the latency was part of the design.
Shipped
Routing policy and profiles
Four seats defined in three configs, with matching defaults and a documented rule for when to leave the cheap seat. Profile switches preserve the live tuning values, so a switch does not undo a day of measured adjustments.
The ledger and report
A background daemon prices each inference and aggregates sessions. The report shows where spend came from by model and by day. It runs on a schedule through a hidden launcher, so there is no console window to close.
The tuner
A second daemon proposes bounded changes every fifteen minutes. It writes only after a successful parse, keeps a backup, records the reason, and respects pins. The window is raise-only because lowering it does not shrink the prompt; it only triggers compaction sooner. That distinction came from watching the setting oscillate.
The verifier and canary
One command checks configs, seats, defaults, retired ports and skill junctions. A second check covers the policy files. Both are read-only, so they are safe to run before trusting a session or installing an update.
The design lane
A scripted child session that takes a brief and references, produces two or three canvas directions, exports previews, and exits. The parent session stays lean. The chosen direction becomes a design contract before implementation.
This is personal infrastructure, not a product. It runs on my machine, prices my traffic and governs my sessions. The repository is the part that is safe to share: policy, automation, configs and checks, without keys or session data.
Evidence
~1,000 fewer prompt tokens per turn after moving ten always-on skills to on-demand routing.
104 s → ~12 s for the UI-quality hook, an 8.7× reduction after scoping the check to changed files.
17 policy files under an integrity canary. The guardrails have their own guardrail.
Reflection
Cost is a design constraint. Once the ledger existed, the routing argument became empirical: here is the task count, here is the spend, here is the difference. The expensive model stopped being the safe default.
The tuner taught me to design the refusal path first. Parsing before writing, backing up before editing, pinning what must not move: those choices matter more than the adjustment logic. An automation that cannot refuse is a liability.
The next version would price every provider instead of only DeepSeek, attribute cost per task rather than per session, and run the benchmark on a larger set. Thirty-three tasks are enough to make a decision and not enough to call it a law. The ledger is also a file and a report; a small dashboard would make the trend visible without opening a terminal.
Measure the default, then defend it.