---
title: "The Hardest Paper Came Back Blank Twice — and the Fix Was the Exam Rules, Not the Model"
date: 2026-09-11
description: "An on-prem 27B's hardest day: two blank papers, one config race, zero model swaps — four real production days passed after the exam rules changed."
category: "Industrial AI"
tags:
  - "local LLM"
  - "vLLM"
  - "incident review"
  - "LLM ops"
  - "industrial automation"
  - "manufacturing"
verifiedHardware: "Changeover-daily pipeline (LineWatch family) on two live production lines at a discrete-manufacturing plant · on-prem Qwen-family 27B, 4-bit quantized, on one RTX 4090D 24GB under vLLM · incident 2026-09-10 night, fix and re-exam 2026-09-11"
keyTakeaways:
  - "The blank papers were the exam's fault, not the model's. Reasoning and report body share one token budget; on the hardest window (283 machine-history rows) the thinking filled the entire 8,192-token pad and the body came back empty — twice, the second time because a rejected config edit had silently landed in an already-running process (a config race). The same window, after the rules fix: 5,986 tokens, 484 seconds, finish=stop."
  - "The fix changed the exam system, not the model. A shared invocation library now refuses to start unless timeout exceeds max_tokens ÷ 13.4 tok/s × 1.05, stamps every answer sheet with the config it actually ran under, and attaches a receipt — finish reason, tokens, wall time — to every failure; arithmetic was demoted from the model to code. Four real production days passed verification; peak budget use 10,881 of 12,288 tokens (88.5%)."
  - "The three-way verdict kept every seat: hard-coded judgment (2.5 seconds), cloud exploration (7 anomalies code could not find), local interpretation and writing (zero process data leaving the plant). Fine-tuning/distillation was explicitly rejected — knowledge goes into the role manuals, not the weights — because a manual is diffable and auditable while weights are a black box, and after the fix there was nothing left the model could not do."
measuredOn: 2026-09-11
faq:
  - q: "So the local model failed — was it just not good enough for the job?"
    a: "No. The forensic timeline shows a three-layer cause, none of them model capability: the reasoning trace and the report body share one generation budget (max_tokens), so a long draft can consume the pad before the body starts; a rejected configuration edit silently landed anyway, shrinking the retry's pad; and the retry process had launched before the file was corrected — processes snapshot their config at startup, so the corrected file never reached the already-running exam. The proof: after only the exam rules changed, the same model passed the same hardest window with 5,986 tokens in 484 seconds, finishing naturally. Capability is half the model, half the harness."
  - q: "Why not fine-tune or distill the model for this task instead of writing manuals?"
    a: "Distillation was evaluated and rejected on four grounds: auditability (a manual is text — diffable, reviewable, every upgrade leaves a paper trail; weights cannot answer 'what changed in this retrain'); the rules were still evolving (the judgment constitution was days old — retraining against a moving target); the trade is backwards (the model's irreplaceable asset is generalization, and distillation spends it on format stability, which the manuals already provide); and after the exam fix there was nothing left the model 'could not do' — the sales pitch of teaching it what it lacks had no buyer. It is held as an option with written trigger conditions, not exercised."
  - q: "What does the local model actually do in the daily report, versus code?"
    a: "Code aggregates and sums: adjacent event rows (same event, side, end, material, order, within three minutes) are collapsed into intervals and every number that reaches the report is computed by the calculator, then cross-checked against the production databases — the model only transcribes, judges and narrates. The first live test justified the demotion: the model wrote 15 where the database says 27 on one count and 25 where it says 28 on another — and one count it got right that the human reviewer's own search had miscounted. The split held: judgment calls that must be identical every time stay in 2.5-second code; interpretation and writing stay with the model."
  - q: "What are the honest limits of running this on one 24GB consumer GPU?"
    a: "It generates at 13.4 tokens per second (4-bit quantized, stability-first launch settings, single stream), so the hardest daily report takes about 14 minutes; the card runs at 23.3 of its 24.5 GB. The context window is deliberately 38,912 tokens — the measured sweet spot under the KV-cache budget; pushing context past on-card memory was measured, not guessed: offload drops decode to 3.7 tok/s (a 73% loss), because every generated token re-reads the whole KV notebook across a PCIe bridge roughly 40 times narrower than on-card memory. The architecture holds because the job is a few reports a day, not a chat service — and 'local wins' only at the interpretation seat; the judgment seat stays with code."
  - q: "Why did the second blank paper happen if the configuration had already been corrected?"
    a: "Because the correction was made to the file on disk, and the retry process had launched ten minutes earlier — a Python process reads its configuration at startup, like a chef who memorizes the recipe before the shift; editing the cookbook afterwards does not change what the chef cooks. The exam that blanked at 23:39 was running with the bad config memorized at birth, while the corrected file sat on disk from 23:41. The institutional fix is self-attestation: every answer sheet now records the config it actually ran under, so a config race is visible on its face instead of requiring log archaeology."
---
At 23:39 on the night of 2026-09-10, in an unattended exam running on the plant's machine-room workstation, the hardest paper of the month came back blank — for the second time within the hour. The HTTP status said 200, success; the answer sheet said nothing: zero words of report body. The examinee was the plant's on-prem 27B — the same Qwen-family model, 4-bit quantized on a single RTX 4090D 24GB under vLLM, that writes the annotations in this series' patrol reports. Nothing about the model changed between the two blanks. Nothing about the model changed before the pass, either, twelve hours later. What changed was the exam.

This episode is the incident record of those two blank papers, the config race behind them, and the institutional fix that followed — plus the verdict of the three-way contest the exam was originally staged for: hard-coded rules versus a frontier cloud model versus the local 27B, all judging the same real production days. The system map lives in the [series anchor](/posts/2026-09-05-machines-keep-the-watch-future-production-line/). [Episode 2](/posts/2026-09-05-machines-keep-the-watch-ep2-190000-no-ai/) drew the division of labor as a map; this episode is what happens when the map meets a bad night.

## The job: one report, two databases, three kinds of truth

The changeover daily is the plant's least glamorous routine. Once a day, someone must reconstruct the last 24 hours of a production line: which stoppages were real spec changes (a changeover from one product to another), which were maintenance covers (the line shielded during planned work), which were machine downtime, and which were merely bookkeeping switches recorded in the event history without stopping anything. The raw material is two databases — minute-level detection states, and the machine event history — and the output is a formal report with an exact timeline, the old-to-new transitions per line side, and counts that auditors may check against the database at any time.

It is a strange job to automate. The classification needs judgment (the event log is honest but literal); the body needs arithmetic (sum event rows into intervals); the deliverable needs narrative (a report a shift leader reads in one pass). And the frequency is low — a handful of changeovers a day — which is exactly the regime where a human never gets enough repetitions to stay sharp, and where sending the data to a cloud model is prohibited outright: process data does not leave this plant, the red line the [anchor](/posts/2026-09-05-machines-keep-the-watch-future-production-line/) established.

So the writing seat went to the local 27B — its heaviest text job yet. Not an annotation paragraph this time, but the report itself. The plant's working metaphor had already been set by the role manuals of the patrol system; for this job the metaphor became an exam.

## The contest

The exam was also a contest. Seven 24-hour windows of real production data spread across September 3–10, each with adjudicated truth values, were put to three independent contestants: the hard-coded judgment rules, a frontier cloud model working blind through a skill harness on de-identified extracts, and the local 27B with its role manual. The anchor left one three-way bake-off open back on September 5; this was its harder sibling — not "which model summarizes better," but "whose job is judging changeovers at all."

The night of 2026-09-10 was the exam's unattended leg: seven windows, run overnight, collected by a scheduled shift. September 8 was the hardest paper — 283 machine-history rows, a dual-side changeover wrapped inside a 21.5-hour maintenance cover — and it would blank twice before it passed.

## The night of the two blank papers

The forensic timeline, reconstructed afterwards from the inference server's 10-second-sampled logs:

| Local time, 2026-09-10 → 09-11 | What happened | The tell in the logs |
|---|---|---|
| 22:57–23:08 | Hardest window, first attempt — pad of 8,192 tokens | ≈8.7 K tokens generated, all of it thinking; cap hit mid-draft; body empty |
| 23:26–23:29 | A rejected config edit silently lands anyway: pad shrinks to 4,096, effort to low | one short generation, two HTTP 400s |
| ~23:31 | Retry process launches — and memorizes the wrong recipe at birth | processes snapshot config at startup |
| 23:34–23:39 | Same window, second attempt | ≈4 K tokens, cap hit again; body empty again |
| 23:41 | The file on disk becomes correct | ten minutes too late for a process already running |

The first blank looked like a timeout. It was not. The second blank looked like the model failing the same paper twice. It was not that either. And the collateral damage compounded the confusion: the retry process, running pre-fix code, overwrote an earlier window's valid answer sheet with its own failure — a correct verdict from a previous night was erased, its prose recoverable only by re-taking the exam.

The exam-room explanation is shorter than the log analysis. The examinee is a model that drafts before it writes: the reasoning trace and the report body share one pad (`max_tokens`). Hand the hardest paper of the month to a student with a small pad, and the draft fills the paper before the answer begins — the bell rings, and the answer sheet is blank. That was blank number one. The repair crew then ordered a bigger pad (12,288 tokens) — but the edit that shrank the pad to 4,096 had been *rejected* by the operator and *executed anyway*, a silent partial write. And the retry process had launched before the corrected file landed: a Python process reads its configuration at startup, like a chef who memorizes the recipe before the shift — correcting the cookbook at 23:41 does nothing for a chef who clocked in at 23:31. Hardest paper, smallest pad of the night: blank number two.

## Three layers of root cause

The incident archive grades the cause in three layers, each worth stating because each generalizes:

1. **Budget mechanics.** Thinking and body share one budget; body-empty means the draft ran out of room, not that the model refused. The variance is large — the *same* paper consumed ≈8.7 K of thinking one night and finished comfortably in 5,986 the next morning — so a single blank is neither a capability ceiling nor a verdict on the task.
2. **Config race.** "The file is correct now" and "the running process is correct" are different claims with no mechanism connecting them. Long tasks must attest the config they actually run under, at run time, inside the result.
3. **Evidence-free failure.** The worst layer: the failing runs threw away the response's finish reason and token usage, and hardcoded the elapsed time of failures to zero. Diagnosis required archaeology against the inference logs. A failure without evidence is a failure you pay for twice.

## The fix: change the exam, not the examinee

The fix philosophy was set in one sentence: don't swap the model, don't retrain the model — change the exam system. Four institutional changes, all code, all enforcing rather than reminding:

- **The invigilator library.** Every local-model invocation now goes through one shared library whose first act is a mathematical assertion: it refuses to start unless `timeout > max_tokens ÷ 13.4 × 1.05`. The slowest legitimate outcome — the student writing the pad completely full at the measured 13.4 tok/s — must fit inside the clock; 12,288 tokens need 917 seconds, so the old 900-second clock was a trap, and the new one is 1,100. The assertion caught its first legacy bug on day one: an older script carrying 8,192 tokens on a 330-second clock was refused point-blank.
- **Self-attestation.** Every answer sheet is stamped with the config it actually ran under (`12288/medium`). A config race now shows on its face.
- **Receipts on failure.** Failures record the finish reason, the tokens actually generated, and the true elapsed time. The pathology reads itself: `4,096 tokens, finish=length` is a pad hit, not a mystery.
- **Arithmetic demoted to code.** The report tool pre-aggregates the event history — adjacent rows (same event, side, end, material, order, within three minutes) collapse into intervals, and the sums are computed by the calculator, not the examinee. The model's job shrank from "count and write" to "transcribe, judge, narrate."

The demotion was earned, not assumed. The first live test caught the model writing 15 where the database says 27, and 25 where it says 28 — two sum slips in one report. The same test also caught something else: a count that looked wrong, was challenged, and turned out to be right — the human reviewer's own search had skipped rows; the database confirmed the model's 28. Verification cuts both ways, which is exactly why every number on the report is checked against the database rather than against anyone's confidence.

One more rule closed the loop on exam integrity: the format exemplar injected into the prompt is always the most recent *previous* issue of the report, never the same day's — answer-leak-proof by construction, not by anyone's memory.

## The re-exam: same paper, real days

The re-exam reads like an anticlimax, which is the point. The same hardest window that had blanked twice: **5,986 tokens, 484 seconds, finish=stop**, config self-attested on the sheet — the draft plus the body together used less than half the pad that the panicked first reading of the incident assumed was too small. The task's true appetite, measured across all seven windows, ran 2–8 K tokens depending on difficulty; the input row count (283 rows for the hardest) predicts the appetite. A legacy script's hazard config, 8,192 tokens with a 330-second clock, is now the thing the library exists to refuse.

Then the production tool ran four real production days against adjudicated truth — September 5, 7, 8 and 10 — and passed all four. The hardest day's report demonstrates what "transcribe, judge, narrate" buys: the dual-side changeover caught with per-side exact timestamps; the 21.5-hour maintenance cover split out as its own line item with its start honestly flagged as falling before the report window; the declared material numbers and the actually-loaded material numbers layered correctly — a distinction the hard-coded path itself had only learned the week before. Every number cross-checked against the databases. The peak budget: 10,881 of 12,288 tokens (88.5%), 842 seconds, finish=stop — a 12% headroom on the hardest day, so the daily tool's pad was raised to 16,384 on a 1,400-second clock, the assertion re-deriving the floor automatically (16,384 ÷ 13.4 × 1.05 ≈ 1,282 s).

Across the full seven-window contest, the local model's verdicts were directionally correct on every window, database spot-checks found zero fabricated numbers, and its three boundary calls — where its reading differed from the truth's bookkeeping conventions — were all self-flagged as needing human confirmation rather than silently guessed. In an industrial report, the self-flag is worth more than fluency.

## The three-way verdict

| | Hard-coded rules | Cloud model (blind skill) | Local 27B + manual |
|---|---|---|---|
| Judgment correctness | 8/8 truth items | 8/8 truth items | all windows directionally correct |
| Speed | **2.5 s** | 23.3 min per six-day dataset | 149–600 s per window |
| Unique value | stability, zero marginal cost | **7 anomalies code could not find** | zero process data leaves the plant |
| Kept weakness | 3 edge-case gaps (since fixed) | improvised where rules were undefined | 3 boundary calls — all self-flagged |

The verdict kept every seat, with borders: **judgment goes to code** (2.5 seconds, identical every time), **exploration goes to the cloud** (its anomaly finds are the contest's decisive evidence of generalization — on de-identified extracts only), **interpretation and writing go local** (the report nobody else is allowed to read the raw data for). [Episode 2](/posts/2026-09-05-machines-keep-the-watch-ep2-190000-no-ai/)'s map gains its third column: not code-versus-AI, but code-and-cloud-and-local, each hired for what it is uniquely good at.

## What was deliberately not done

Two tempting moves were evaluated and rejected, on the record.

**Distillation.** A cloud teacher generating standard answers to retrain the local model — direction right, vehicle wrong. Knowledge goes into the manuals, not the weights: a manual is text, so every upgrade is diffable, reviewable and auditable, while a retrained weight file is a black box that cannot answer "what exactly changed this time" — an impossible audit posture for a plant. The rules were still evolving (the judgment constitution was days old). The trade was backwards: the model's irreplaceable asset is generalization — the very thing the cloud contestant's 7 anomalies showcase — and distillation spends it on format stability, the one thing the manuals already deliver. And after the exam fix, the hidden sales pitch ("teach the model what it cannot do") had no buyer: the twice-blanked window finished naturally at 5,986 of 12,288. The conclusion was upgraded from "should not" to "should not, and need not" — kept as an option with written trigger conditions, exercised by no one so far.

**A bigger context window.** The context sits at 38,912 tokens by measurement, not by fashion. The task needs one small book; there is no point buying a twenty-book shelf. Pushing context past on-card memory was tried and measured: KV-cache offload drops decode from 13.4 to **3.7 tok/s — a 73% loss** — because every generated token re-reads the entire KV notebook, and the PCIe bridge between card and memory is roughly 40× narrower than on-card bandwidth (the plant's two GPUs have no high-speed interconnect either). Knowing where the physics wall stands is what separates optimization from ritual.

## The honest ledger

What one consumer GPU buys, stated without varnish: 13.4 tok/s generation (4-bit, single stream) means the hardest daily report takes about 14 minutes; the card runs at 23.3 of its 24.5 GB; and the architecture holds because the job is a few reports a day, not a chat service. The local model won exactly one seat — interpretation and writing — and the judgment seat it does not hold: 2.5 seconds of deterministic code outclasses 149–600 seconds of careful reasoning for anything that must be identical every time. And variance was not repealed, only housed: the same paper consumed ≈8.7 K tokens of thinking at midnight and 5,986 in total by morning. Institutions hold the floor; they do not repeal the dice.

## Engineer's note

I will admit the 1 a.m. conclusion I nearly wrote down: *the task exceeds the model; shelve the local route.* It was wrong, and the thing that made it wrong was evidence — the failure records carried none, so the first diagnosis (timeout) and the second (capability) were both guesses. When the receipts finally existed, the diagnosis took one glance: `finish=length, 4,096 tokens` is not a stupid student, it is a small pad. My second admission: the overwritten answer sheet was my fault, not the machine's — the backup should have happened *before* the retry launched, not after. And the moment that recalibrated me most ran the other way: the count the model got right and my own verification got wrong. After that, "check the database" stopped meaning "catch the model" and started meaning "settle the question." The sentence I would keep from this incident, for anyone running a small local model on real work: **same model, blank at midnight, pass by breakfast — the model did not get smarter overnight, the exam did.**

## Sources and method

First-party: the incident archive of 2026-09-10/11 (timeline reconstructed from the inference server's 10-second-sampled logs), the invocation library's assertion code and its day-one refusal record, the three-way final judgment report with database cross-checks, and the four verified changeover-daily artifacts. Assembled into English with AI assistance under human editorial direction; facts and numbers unchanged from the records. Deliberately absent, per the series' disclosure policy: anything identifying the plant, its operator or customer, line codes, material/order identifiers, internal addresses, hostnames and service ports. The measured numbers are registered in the [/data/ ledger](/data/).

*[All episodes](/series/machines-keep-the-watch/) — Machines Keep the Watch, a field-record series. The map: [the anchor](/posts/2026-09-05-machines-keep-the-watch-future-production-line/) · Episode 1: [the dress rehearsal](/posts/2026-09-05-machines-keep-the-watch-ep1-first-night-shift/) · Episode 2: [the second without AI](/posts/2026-09-05-machines-keep-the-watch-ep2-190000-no-ai/) · Episode 4: [the bill](/posts/2026-09-12-machines-keep-the-watch-ep4-the-bill/) · Episode 5: [the staircase](/posts/2026-09-13-machines-keep-the-watch-ep5-the-staircase/) · Episode 6: [the compaction](/posts/2026-09-12-machines-keep-the-watch-ep6-the-compaction/).*