---
title: "The Model Exam: Seven Passes, Two Zeroes, One Home"
date: 2026-08-27
description: "E17: the fleet grades its local 27B — a 7/7 baseline, an A/B with two zero-output deaths, a four-habitat race, six rules for 241 tokens."
category: "AI & Compute"
tags:
  - "One Man One Legion"
  - "AI Agents"
  - "Local LLM"
  - "vLLM"
  - "Model Evaluation"
verifiedHardware: "Workstation ~/.openclaw/workspace/qwen38-eval/ (dir mtime 2026-08-18): Q38压测报告-20260818.md (v2.0 — A/B raw table, T1-T7 appendix, tuning log) + WORKLOG-20260818.md (four-habitat section, rules ON/OFF, verdict), run.py (A/B runner — prompts verbatim incl. the return-original-list wording; temperature 0.2) + run_tests.py (baseline runner), t1-t7 .md/.py pairs (per-task time and token counts in file headers), A_full.md (0 bytes) / A_full8k.md / B1_dedup.md / B2_main.md / B3_check.md (split-run outputs), toolA.py + toolB.py (53 lines each; toolB carries the missing-column print bug), verify.py (T1-T5 asserts + five dedup cases) + data.csv / empty.csv, cc-q38.sh + start_qwen38-final.sh; ~/.openclaw/workspace/memory/2026-08-18.md (ticket 24769 vs 7710 = 31%, rules 7710→7951, ON/OFF 49394/64907, log-filter revival 68984, verifier-bug stop-and-ask); ~/.dsh/AGENTS.md (six rules, on disk 2026-08-27); live state 2026-08-27: ps / ss / nvidia-smi — the 27B serving process up since Aug 18 · timestamps Asia/Shanghai"
keyTakeaways:
  - "Graded by execution, not vibes: seven coding tasks of rising difficulty all passed against assertion scripts — 7/7 at 13-142 seconds and 194-1,834 output tokens — with every caveat (hand-extracted code, verifier strictness, thinking leak) logged next to the passes."
  - "The A/B found what splitting actually buys: not quality (the one-shot passed all five verifier cases cleanly; the split path passed four and leaked raw rows on the fifth) but containment — one-function diffs instead of whole-file rewrites. Meanwhile max_tokens starvation killed two of six requests with zero characters of output, and the split mode's one bug was authored by the prompt itself."
  - "Four habitats, one winner: the 24,769-token Claude Code ticket eats 64% of the 38,912-token window, the terminal harness charges 7,710 for 3.8× the working room, and six global rules cost 241 tokens while consuming less (49,394 vs 64,907) — so the harness became the model's home, the bare endpoint serves pipelines, and the mount wrapper stays a spare."
measuredOn: 2026-08-27
faq:
  - q: "Doesn't a 7/7 pass mean the local model is as good as the cloud one?"
    a: "No — and the exam never claimed that. The baseline only qualified the 27B for daily single-file code duty. The series' own mini-agent test (episode 16) showed a smaller local model stalling where a cloud model passed with the same harness, and this exam's verdict explicitly routes hard jobs and design calls to cloud review. What the 7/7 buys is the right to be used, not the right to everything."
  - q: "Why keep using a model that died with zero output twice in six requests?"
    a: "Because the deaths were a usage bug, not a model bug — the thinking phase consumed the output budget before the body started. Raising max_tokens to 6,000+ fixed it, and the harness route dissolves the trap entirely by absorbing output across rounds. That was the exam's whole point: match the usage pattern to the model's limits instead of upgrading the model."
---
*[Episode 17 of One Man One Legion](/posts/2026-08-27-one-man-one-legion-fleet-audit/) — third stop in the engine-room arc, after [the mini-agent](/posts/2026-08-27-one-man-legion-ep16-mini-agent/) and [the model stable](/posts/2026-08-27-one-man-legion-ep14-model-stable/). The stable ended on a promise: the stress test that would grade the newest horse. This is it.*

Hours after the 27B came up, the fleet held its exam. The goal was not to make the model stronger but to test whether task-splitting, hard constraints, and a division of labor could match usage to the limits of a 4-bit thinker. Design and grading were the cloud model's job; the subject was local.

## The baseline: seven tasks, real asserts

Capability first: seven coding tasks of rising difficulty, from a discount parameter and an empty-list crash to a log filter, a two-file change, and a messy-style-preservation test. Every answer was executed against assertion scripts, not eyeballed. Result: 7/7 at 13-142 seconds per task and 194-1,834 output tokens, at temperature 0.2. The caveats sit next to the passes: one answer was lifted out by hand (no code block), one key-name mismatch scored as verifier strictness, one put an example block before the real code. Known flaw: on this stack, thinking leaks into the body. Verdict: qualified for daily code duty.

## The A/B: what splitting actually buys

Then the experiment: one task — a column-dedup command for a small CSV tool, assembled versions 53 lines — two usage patterns, six logged requests. Group A got the whole job in one prompt; Group B got three constrained steps (touch only what's necessary, keep style, no drive-by refactors, say when unsure): the function, the main() branch, a self-check.

| Run | max_tokens | Time (s) | Output |
|---|---|---|---|
| A one-shot | 3,000 | 223 | 0 chars — dead |
| A retry | 8,000 | 256 | 1,465 chars |
| B1 function | 3,000 | 125 | 298 chars |
| B2 main() | 3,000 | 58 | 604 chars |
| B3 self-check | 1,500 | 123 | 0 chars — dead |
| B3 retry | 6,000 | 260 | 468 chars |

The first killer of a 4-bit thinking model is its own budget: thinking burns 500-3,000+ tokens before the body starts, and two of six requests starved to zero characters — the 0-byte corpse file is still archived. Iron rule: never call it below 6,000 max_tokens.

Splitting bought containment, not quality — the successful one-shot passed all five verifier cases cleanly; the split path passed four, leaking raw rows on the missing-column fifth. The one-shot rewrote the entire file, so review meant reading everything; each split step returned one function, a diff you can hold in your head. The three good split steps took 443 seconds against 479 for the one-shot's two attempts — no slower, both dead runs visible above.

The most valuable find was self-inflicted. That missing-column leak? The split prompt itself said to return the original list unchanged — the model obeyed, literally. A small model in split mode obeys every design decision in a template to the letter; the template must be reviewed before it becomes law. Given a real budget, the B3 self-check caught exactly that bug, plus two more.

## The habitat race: same weights, four doors

The same weights answer to four doors: the agent framework's alias, a wrapper that mounts Claude Code, a lightweight terminal harness on a handwritten route, and the bare localhost endpoint. The exam measured each door's ticket out of the 38,912-token window. Claude Code costs 24,769 tokens of system prompt — 64% of the window, a ticket designed for a 200K-token model worn by a 38.9K one, leaving 6.1K of working space. The harness charges 7,710 — 31% of that — for some 23K of dialogue space, 3.8× the working room. The bare endpoint is free.

Two upgrades were refused with numbers: 48K with CPU offload collapsed throughput to 3.7 tokens/s against the ~14 baseline — an 8,000-token answer would take 36 minutes; and prefix caching wouldn't start at that window, its ceiling of about 12 seconds a round not worth 5,600 tokens of context.

## Six rules, 241 tokens

The winning harness can't vary instructions per model, so six global rules were written to be harmless to the cloud model too: read before editing, minimal changes, stop after two failures, ask when unsure, verify after the change, split big jobs. Measured cost: the ticket rose from 7,710 to 7,951 — 241 tokens, 3.1%. With the rules as the only variable, correctness stayed 5/5, rules-on consumed 49,394 prompt tokens against 64,907 rules-off, and only rules-on asked about an ambiguity. Told to refactor boldly, the cloud model correctly overrode the minimal-change rule — the injection is protocol-level, never outranking a direct instruction.

Final proof: the log filter, hardest of the seven. Re-run that evening on the bare endpoint, it died the zero-character death; inside the harness it finished end-to-end, every requirement met, for 68,984 tokens — the harness absorbs output across rounds, dissolving the budget trap. Along the way the model caught a bug in the verifier itself — an expected count of 3 where the fixture holds 2 — and refused to fix either side, stopping to ask which was right. Rule four, executed as designed.

## The verdict, still standing

At 21:05 the verdict landed: the harness is this model's home; the bare endpoint serves pipelines, where agents enforce split discipline; Claude Code keeps the cloud model, the wrapper stays a spare; hard jobs get cloud review. The logic compresses to a line: big models are judged by intelligence, small models by how frugally you run them — one door saves space, one saves the ticket, the third saves neither. Nine days later, the serving process born that evening still holds the GPU.

Scope, as logged: one task family, one round per cell, 53-line-scale single-file edits; larger repos untested. An exam, not a benchmark.

*[Back to the fleet audit](/posts/2026-08-27-one-man-one-legion-fleet-audit/) — next in the engine room: swapping the cloud brain without stopping the factories.*