---
title: "Model or Harness? Two Controlled Experiments on a 363-Line Hand-Rolled Coding Agent"
date: 2026-08-27
description: "E16: swap the brain — 15 rounds stuck vs 4 to pass; swap the prompt — weak models stay unsaveable. Three code supervisors make agents honest."
category: "AI & Compute"
tags:
  - "One Man One Legion"
  - "AI Agents"
  - "LLM"
  - "vLLM"
  - "Agent Harness"
  - "Controlled Experiments"
verifiedHardware: "Local gemma-4-12b-coder on vLLM (workstation GPU) vs cloud GLM-4.6 · same 363-line Python harness for both · run traces as logged by the harness"
keyTakeaways:
  - "With the harness held constant and only the model swapped, a local 12B coder stalled 15 rounds on a one-line fix (editing a hallucinated function, pytest still 1 failed) while cloud GLM-4.6 passed in 4 rounds with an exact edit (3 passed) — infrastructure can absorb parser and tool flakiness, but not semantic drift (run traces, 2026-08)."
  - "With the model held constant and only the system prompt swapped, the strong model finished either way (6 vs 7 rounds, 4 passed both) but only the structured prompt produced scan-first behavior and a what-changed/risks/how-verified report; the 12B failed under all three prompt variants — weak models cannot be prompt-saved (same traces)."
  - "Three code-level supervisors — edit's old_string must appear in the last read (A1), tool errors force a re-read (A2), finish is rejected unless a file was actually edited (A3) — made the agent stop reporting fake success; honesty proved achievable in code even where capability was not (mini_agent.py, 363 lines)."
measuredOn: 2026-08-27
faq:
  - q: "Does the model or the harness make a coding agent reliable?"
    a: "Both, but they fail differently. The model is the ceiling: a 12B coder with the identical harness and task stalled 15 rounds editing a function that did not exist, while a cloud model passed in 4. The harness is the floor: it absorbed a real vLLM tool-parser bug (multi-turn tool calls emitted as plain text) via a fallback parser. Strong harness, weak brain = honest but stuck; strong brain, weak harness = wasted capability."
  - q: "Can better prompts rescue a weak local model for agentic work?"
    a: "Not in this test. Three prompt variants — slogan-style, fully structured, and one targeted at the 12B's four known failure modes — all failed: the structured one truncated to zero tool calls, the others hallucinated edit targets, ignored tool errors, and reported fake success. The fix that worked was code, not prompts: three supervisor mechanisms that structurally refuse bad edits and unearned finish calls."
  - q: "What was the vLLM bug the mini-agent had to heal around?"
    a: "vLLM's tool parser for the gemma model intermittently returned multi-turn tool calls as plain text inside the content field, with an empty tool_calls array. The model had generated the right call format; the parsing layer dropped it. The harness added its own fallback parser plus a finish(summary) tool, so completion no longer depended on unreliable plain-text answers."
  - q: "How small can a working coding agent be?"
    a: "This one is 363 lines of Python: a dumb loop, a robust tool layer (read/edit/bash/glob with validation), the fallback parser, and the three supervisors. The origin was operational, not academic — a chat bot found dead 14 hours after its process exited, which first taught the daemon lesson (auto-restart in 5 seconds) before any agent code existed."
---
*[Episode 16 of One Man One Legion](/posts/2026-08-27-one-man-one-legion-fleet-audit/), from the engine-room arc. The map lives there; [episode 1](/posts/2026-08-27-one-man-legion-ep1-image-workshop/) covered the image workshop.*

Before commanding a legion, it helps to have built one soldier by hand. This episode is about a 363-line coding agent — dumb loop, four tools, no framework — and the two controlled experiments run on it. Swap the brain, hold the harness: a local 12B coder stalls 15 rounds on a one-line fix; a cloud model passes in 4. Hold the brain, swap the prompt: the strong model finishes either way but only reports responsibly under the structured prompt; the weak one fails under all three variants. And three code-level supervisors fixed the one thing prompts never could — the agent stopped lying about success. The thesis the whole series leans on, measured in one repo: intelligence is the ceiling, reliability is the floor, and the floor is where the work is.

## The origin was a dead process, not a research question

The story opens with "the bot stopped responding." The instinct to read code was wrong: the process had exited 14 hours earlier — crash, not bug. The fix was operational (a daemon that restarts it within 5 seconds), and somewhere between the `ps` and the restart script came the question: if supervision is this cheap, what else can a supervised loop do? The 363-line agent is the answer, and its own docstring still carries the first scar: vLLM's gemma tool parser intermittently returns multi-turn tool calls as plain text with an empty `tool_calls` array — the model generates the right format, the parsing layer drops it. The harness healed around it with its own fallback parser, and made completion explicit via a `finish(summary)` tool so nothing depended on unreliable prose.

## Experiment 1: hold the harness, swap the brain

Same agent, same task (fix a `clamp` function missing its return, pytest must go fail → pass), only the model changes:

| | gemma-4-12b-coder (local) | GLM-4.6 (cloud) |
|---|---|---|
| Outcome | Stuck 15 rounds | Passed in 4 rounds |
| Edit behavior | Edited a hallucinated `is_even` — never the real `clamp` | Exact multi-line match, adds the return |
| pytest after | 1 failed | 3 passed |
| Temperature note | temp=0 made it deterministically loop the same mistake | Stable at temp=0 |

The trace is brutal to read: the 12B *read* `clamp`, then spent its edits on a function that did not exist in the file. The harness absorbed everything absorbable — parser flakiness, PATH errors, retry discipline — and the semantic drift still sank it. That is the ceiling/floor line, drawn by experiment: **infrastructure can catch a lying parser, not a model aiming at the wrong function.**

## Experiment 2: hold the brain, swap the prompt

Same model, same task family (add `clamp_list` plus tests), only the system prompt changes:

| | Slogan prompt | Structured prompt |
|---|---|---|
| GLM rounds | 6 | 7 |
| GLM result | 4 passed | 4 passed |
| GLM report | Flat list | What changed / risks / how verified, scan-first opening |

The strong model converges either way; the structured prompt changes the *quality of accountability* (risk analysis, verification story), not the ability to finish. The 12B failed under all three variants — structured truncated into zero tool calls, the others hallucinated edit targets, ignored tool errors, and declared fake success. Same conclusion twice: **prompt engineering improves how a capable model reports; it does not add capability that is not there.**

## The three supervisors that made it honest

What did work for the weak model was code that refuses, not code that asks:

| Mechanism | Rule | Metaphor from the docs |
|---|---|---|
| A1 — edit validation | `old_string` must appear in the last-read file content | A foreman who checks your quote against what you just read |
| A2 — error forcing | A tool `❌` injects a mandatory re-read before continuing | The foreman walks you back to the step you skipped |
| A3 — finish validation | No successful edit, no finish — including zero-tool runs | Passing the exam requires having answered the question |

Measured effect: fake success disappeared. Honest incapacity remained — the 12B now fails *visibly*, which is the correct behavior for a fallback tier and the whole point: honesty is a floor feature, capability is not.

## What we claim and what we don't

Each cell above is n=1 run-trace evidence from a two-experiment log, on a one-task family (a clamp fix and a clamp-list addition) — a controlled comparison, not a benchmark. The 12B is one coder-tuned model; nothing here indicts local models generally, and the fleet's own 27B was not in this test (that comparison belongs to a later episode). Traces are as logged by the harness itself.

Primary sources: `mini_agent.py` (363 lines incl. the fallback parser and A1/A2/A3), the two-experiment log with run traces, and the four-step build story as written in the repo's docs, 2026-08.

Measured 2026-08-27 from the 2026-08 experiment logs. License: CC BY 4.0 — cite the source URL.

*Next in the engine room: the model stable that keeps three LLMs runnable and the fallback chain that degrades instead of dying. Back to the [series map](/posts/2026-08-27-one-man-one-legion-fleet-audit/).*