The Model Exam: Seven Passes, Two Zeroes, One Home

Episode 17 of One Man One Legion — third stop in the engine-room arc, after the mini-agent and the model stable. The stable ended on a promise: the stress test that would grade the newest horse. This is it.

Hours after the 27B came up, the fleet held its exam. The goal was not to make the model stronger but to test whether task-splitting, hard constraints, and a division of labor could match usage to the limits of a 4-bit thinker. Design and grading were the cloud model’s job; the subject was local.

The baseline: seven tasks, real asserts

Capability first: seven coding tasks of rising difficulty, from a discount parameter and an empty-list crash to a log filter, a two-file change, and a messy-style-preservation test. Every answer was executed against assertion scripts, not eyeballed. Result: 7/7 at 13-142 seconds per task and 194-1,834 output tokens, at temperature 0.2. The caveats sit next to the passes: one answer was lifted out by hand (no code block), one key-name mismatch scored as verifier strictness, one put an example block before the real code. Known flaw: on this stack, thinking leaks into the body. Verdict: qualified for daily code duty.

The A/B: what splitting actually buys

Then the experiment: one task — a column-dedup command for a small CSV tool, assembled versions 53 lines — two usage patterns, six logged requests. Group A got the whole job in one prompt; Group B got three constrained steps (touch only what’s necessary, keep style, no drive-by refactors, say when unsure): the function, the main() branch, a self-check.

Run	max_tokens	Time (s)	Output
A one-shot	3,000	223	0 chars — dead
A retry	8,000	256	1,465 chars
B1 function	3,000	125	298 chars
B2 main()	3,000	58	604 chars
B3 self-check	1,500	123	0 chars — dead
B3 retry	6,000	260	468 chars

The first killer of a 4-bit thinking model is its own budget: thinking burns 500-3,000+ tokens before the body starts, and two of six requests starved to zero characters — the 0-byte corpse file is still archived. Iron rule: never call it below 6,000 max_tokens.

Splitting bought containment, not quality — the successful one-shot passed all five verifier cases cleanly; the split path passed four, leaking raw rows on the missing-column fifth. The one-shot rewrote the entire file, so review meant reading everything; each split step returned one function, a diff you can hold in your head. The three good split steps took 443 seconds against 479 for the one-shot’s two attempts — no slower, both dead runs visible above.

The most valuable find was self-inflicted. That missing-column leak? The split prompt itself said to return the original list unchanged — the model obeyed, literally. A small model in split mode obeys every design decision in a template to the letter; the template must be reviewed before it becomes law. Given a real budget, the B3 self-check caught exactly that bug, plus two more.

The habitat race: same weights, four doors

The same weights answer to four doors: the agent framework’s alias, a wrapper that mounts Claude Code, a lightweight terminal harness on a handwritten route, and the bare localhost endpoint. The exam measured each door’s ticket out of the 38,912-token window. Claude Code costs 24,769 tokens of system prompt — 64% of the window, a ticket designed for a 200K-token model worn by a 38.9K one, leaving 6.1K of working space. The harness charges 7,710 — 31% of that — for some 23K of dialogue space, 3.8× the working room. The bare endpoint is free.

Two upgrades were refused with numbers: 48K with CPU offload collapsed throughput to 3.7 tokens/s against the ~14 baseline — an 8,000-token answer would take 36 minutes; and prefix caching wouldn’t start at that window, its ceiling of about 12 seconds a round not worth 5,600 tokens of context.

Six rules, 241 tokens

The winning harness can’t vary instructions per model, so six global rules were written to be harmless to the cloud model too: read before editing, minimal changes, stop after two failures, ask when unsure, verify after the change, split big jobs. Measured cost: the ticket rose from 7,710 to 7,951 — 241 tokens, 3.1%. With the rules as the only variable, correctness stayed 5/5, rules-on consumed 49,394 prompt tokens against 64,907 rules-off, and only rules-on asked about an ambiguity. Told to refactor boldly, the cloud model correctly overrode the minimal-change rule — the injection is protocol-level, never outranking a direct instruction.

Final proof: the log filter, hardest of the seven. Re-run that evening on the bare endpoint, it died the zero-character death; inside the harness it finished end-to-end, every requirement met, for 68,984 tokens — the harness absorbs output across rounds, dissolving the budget trap. Along the way the model caught a bug in the verifier itself — an expected count of 3 where the fixture holds 2 — and refused to fix either side, stopping to ask which was right. Rule four, executed as designed.

The verdict, still standing

At 21:05 the verdict landed: the harness is this model’s home; the bare endpoint serves pipelines, where agents enforce split discipline; Claude Code keeps the cloud model, the wrapper stays a spare; hard jobs get cloud review. The logic compresses to a line: big models are judged by intelligence, small models by how frugally you run them — one door saves space, one saves the ticket, the third saves neither. Nine days later, the serving process born that evening still holds the GPU.

Scope, as logged: one task family, one round per cell, 53-line-scale single-file edits; larger repos untested. An exam, not a benchmark.

Back to the fleet audit — next in the engine room: swapping the cloud brain without stopping the factories.