Does the model or the harness make a coding agent reliable?

Both, but they fail differently. The model is the ceiling: a 12B coder with the identical harness and task stalled 15 rounds editing a function that did not exist, while a cloud model passed in 4. The harness is the floor: it absorbed a real vLLM tool-parser bug (multi-turn tool calls emitted as plain text) via a fallback parser. Strong harness, weak brain = honest but stuck; strong brain, weak harness = wasted capability.

What was the vLLM bug the mini-agent had to heal around?

vLLM's tool parser for the gemma model intermittently returned multi-turn tool calls as plain text inside the content field, with an empty tool_calls array. The model had generated the right call format; the parsing layer dropped it. The harness added its own fallback parser plus a finish(summary) tool, so completion no longer depended on unreliable plain-text answers.

How small can a working coding agent be?

This one is 363 lines of Python: a dumb loop, a robust tool layer (read/edit/bash/glob with validation), the fallback parser, and the three supervisors. The origin was operational, not academic — a chat bot found dead 14 hours after its process exited, which first taught the daemon lesson (auto-restart in 5 seconds) before any agent code existed.

Model or Harness? Two Controlled Experiments on a 363-Line Hand-Rolled Coding Agent

Q: Can better prompts rescue a weak local model for agentic work?

Not in this test. Three prompt variants — slogan-style, fully structured, and one targeted at the 12B's four known failure modes — all failed: the structured one truncated to zero tool calls, the others hallucinated edit targets, ignored tool errors, and reported fake success. The fix that worked was code, not prompts: three supervisor mechanisms that structurally refuse bad edits and unearned finish calls.

Episode 16 of One Man One Legion, from the engine-room arc. The map lives there; episode 1 covered the image workshop.

Before commanding a legion, it helps to have built one soldier by hand. This episode is about a 363-line coding agent — dumb loop, four tools, no framework — and the two controlled experiments run on it. Swap the brain, hold the harness: a local 12B coder stalls 15 rounds on a one-line fix; a cloud model passes in 4. Hold the brain, swap the prompt: the strong model finishes either way but only reports responsibly under the structured prompt; the weak one fails under all three variants. And three code-level supervisors fixed the one thing prompts never could — the agent stopped lying about success. The thesis the whole series leans on, measured in one repo: intelligence is the ceiling, reliability is the floor, and the floor is where the work is.

The origin was a dead process, not a research question

The story opens with “the bot stopped responding.” The instinct to read code was wrong: the process had exited 14 hours earlier — crash, not bug. The fix was operational (a daemon that restarts it within 5 seconds), and somewhere between the ps and the restart script came the question: if supervision is this cheap, what else can a supervised loop do? The 363-line agent is the answer, and its own docstring still carries the first scar: vLLM’s gemma tool parser intermittently returns multi-turn tool calls as plain text with an empty tool_calls array — the model generates the right format, the parsing layer drops it. The harness healed around it with its own fallback parser, and made completion explicit via a finish(summary) tool so nothing depended on unreliable prose.

Experiment 1: hold the harness, swap the brain

Same agent, same task (fix a clamp function missing its return, pytest must go fail → pass), only the model changes:

	gemma-4-12b-coder (local)	GLM-4.6 (cloud)
Outcome	Stuck 15 rounds	Passed in 4 rounds
Edit behavior	Edited a hallucinated `is_even` — never the real `clamp`	Exact multi-line match, adds the return
pytest after	1 failed	3 passed
Temperature note	temp=0 made it deterministically loop the same mistake	Stable at temp=0

The trace is brutal to read: the 12B read clamp, then spent its edits on a function that did not exist in the file. The harness absorbed everything absorbable — parser flakiness, PATH errors, retry discipline — and the semantic drift still sank it. That is the ceiling/floor line, drawn by experiment: infrastructure can catch a lying parser, not a model aiming at the wrong function.

Experiment 2: hold the brain, swap the prompt

Same model, same task family (add clamp_list plus tests), only the system prompt changes:

	Slogan prompt	Structured prompt
GLM rounds	6	7
GLM result	4 passed	4 passed
GLM report	Flat list	What changed / risks / how verified, scan-first opening

The strong model converges either way; the structured prompt changes the quality of accountability (risk analysis, verification story), not the ability to finish. The 12B failed under all three variants — structured truncated into zero tool calls, the others hallucinated edit targets, ignored tool errors, and declared fake success. Same conclusion twice: prompt engineering improves how a capable model reports; it does not add capability that is not there.

The three supervisors that made it honest

What did work for the weak model was code that refuses, not code that asks:

Mechanism	Rule	Metaphor from the docs
A1 — edit validation	`old_string` must appear in the last-read file content	A foreman who checks your quote against what you just read
A2 — error forcing	A tool `❌` injects a mandatory re-read before continuing	The foreman walks you back to the step you skipped
A3 — finish validation	No successful edit, no finish — including zero-tool runs	Passing the exam requires having answered the question

Measured effect: fake success disappeared. Honest incapacity remained — the 12B now fails visibly, which is the correct behavior for a fallback tier and the whole point: honesty is a floor feature, capability is not.

What we claim and what we don’t

Each cell above is n=1 run-trace evidence from a two-experiment log, on a one-task family (a clamp fix and a clamp-list addition) — a controlled comparison, not a benchmark. The 12B is one coder-tuned model; nothing here indicts local models generally, and the fleet’s own 27B was not in this test (that comparison belongs to a later episode). Traces are as logged by the harness itself.

Primary sources: mini_agent.py (363 lines incl. the fallback parser and A1/A2/A3), the two-experiment log with run traces, and the four-step build story as written in the repo’s docs, 2026-08.

Measured 2026-08-27 from the 2026-08 experiment logs. License: CC BY 4.0 — cite the source URL.

Next in the engine room: the model stable that keeps three LLMs runnable and the fallback chain that degrades instead of dying. Back to the series map.