--- title: "The Model Stable: Two Days in June, One Evening in August" date: 2026-08-27 description: "E14: the engine room's model stable — three local models totaling 70 GB, a two-day deployment battle, and the fallback horse that isn't saddled today." category: "AI & Compute" tags: - "One Man One Legion" - "AI Agents" - "vLLM" - "Local LLM" - "Open Models" verifiedHardware: "Workstation model directories (du -sh, counted 2026-08-27: gemma-4-12b-coder 23 GB, Qwen3.8-27B-AWQ 29 GB, Qwythos-9B-Claude-Mythos-5-1M 18 GB; dir mtimes 06-19 / 08-18 / 06-25), ~/本地大模型部署全记录.md (v1.0, 2026-06-26: two-day timeline 07:47 decision → 07:30 conclusions, seven pitfalls, seven iron rules, 4/3/3 room triage, 28673-token overflow error), ~/vLLM部署对比-Gemma vs Qwythos.md (2026-06-25), ~/gemma-coder-handbook.md (2026-06-19: 11648-line project vs 32K window, 190-line module 3 bugs missed + 1 false positive), ~/vllm-gemma4-tp2.log (2026-06-19 11:43 first serve attempt, 600s engine-core timeout), ~/models/download_qwen38.log (first line 2026-08-18 14:41:08, ModelScope), ~/models/download_qwen38_awq.sh (resume-safe mirror script), ~/models/start_qwen38.sh + vllm_qwen38.log (banner 08-18 19:58:41, vLLM 0.23.0, 38912-token window), ~/.openclaw/workspace/qwen38-eval/cc-q38.sh (mtime 08-18 20:02, effort-500 quirk comment, 3-4 min ready), live state 2026-08-27: ps (vLLM up since Aug 18), nvidia-smi (23314/24564 MiB on the 4090D, 267 MiB on the A4000), ss (no listener on the fallback endpoint), openclaw.json via jq (primary zai/glm-5.3, fallbacks [local-vllm/qwythos-9b], 27B under a separate provider) · timestamps Asia/Shanghai" keyTakeaways: - "The stable holds three locally owned models — a 23 GB coder, an 18 GB reasoner, a 29 GB workhorse, 70 GB in total — and each earned its stall for a different job: editor-side code commentary, disaster fallback, and the coding mount that carries Claude Code on local weights." - "The June deployment of the 9B fallback took two days and seven pitfalls; the August arrival of the 27B went from first download byte to a mounted coding agent in five hours and twenty-one minutes — because June's scars (mirror downloads, same-family parsers, the FlashInfer kill switch) had compiled into August's defaults." - "The stable's core discipline is knowing boundaries: the 12B was tested and assigned commentator-not-detective; the 9B was triaged across ten chat rooms as 4 fully capable, 3 degraded-acceptable, 3 forbidden — because a small model faking deep analysis is worse than silence. Today the named fallback endpoint is dark while the 27B holds the GPU: the stable is a promise kept by scripts, not a hot standby." measuredOn: 2026-08-27 faq: - q: "Why keep local models at all when the cloud model is better?" a: "Because same-provider backups die together. The deployment record's own argument: the cloud 5.2 and 4.7 share one API, so when that API blinks, both blink. Real failover has to cross providers — cloud primary, local 9B behind it — verified still in place in the config read on 2026-08-27." - q: "Is the disaster fallback actually running right now?" a: "No, and the series reports it as measured: the failover chain names the local 9B, but its endpoint has no listener today — its serving log ends with a clean shutdown — and the 27B holding the GPU is not in the chain at all. Re-lighting the fallback is one start script away, about the 3-4 minutes the 27B itself needs to come ready." --- *[Episode 14 of One Man One Legion](/posts/2026-08-27-one-man-one-legion-fleet-audit/) — second stop in the engine-room arc, after [the mini-agent experiment](/posts/2026-08-27-one-man-legion-ep16-mini-agent/). Upstairs, the workshops print, render, and speak. Down here, the fleet keeps its own brains.* On one workstation disk sit three directories: a 23 GB coder, an 18 GB reasoner, and a 29 GB workhorse — 70 GB of locally owned model weights, measured today. One GPU is awake and nearly full; the card beside it is essentially idle. The question this room answers: what does a one-man fleet actually do with its own models, when a better one is a single API call away? ## The first horse was a commentator June 19: the coder arrives, 23 GB. Its directory timestamp reads 11:27; sixteen minutes later a first serve attempt tries to span both GPUs and dies on a 600-second timeout waiting for engine cores. Single-GPU it is. The same day, a handbook is written with tested boundaries — not guesses: a real project of 11,648 lines is roughly 100K tokens against a 32K window, three to four times too big to load; hand the model a 190-line module to debug and it misses all 3 real bugs while flagging 1 false one. Verdict: a code commentator for the editor, not a project detective. The stable's first lesson was a boundary. ## Two days in June June 25, 07:47 — the decision to stand up a local fallback. The 18 GB download crawls at ~300 KB/s through the proxy, with a 17-hour estimate and frequent disconnects; switching to a domestic mirror at 13:10 lifts it to 2.3 MB/s, seven times faster, and the weights land by 15:12. Then comes the serving battle: seven pitfalls across two days, first light at 19:08, first chat room verified at 21:26, and a final conclusions note at 07:30 the next morning. Three pits deserve engraving. The tool-call parser took a fifth attempt to pair — the reasoning parser and the tool parser must come from the same qwen3 family, or they talk over each other. The 32K context window overflowed on the agent framework's own 28K system prompt (the error cites at least 28,673 input tokens), so the window went to 64K. And a tool named `web_search` went uncalled — the model confused it with a built-in from its training; renamed `internet_search`, it worked. Then the triage, honest as a chart: across ten chat rooms, the 9B is fully capable in 4 (single-tool jobs), degraded-acceptable in 3, and forbidden in 3 — deep trend analysis, stock data, long-form creation, where the article workflow alone needs 170K tokens of context, far past the local ceiling. The standing rule: publish nothing rather than let a 9B fake its way through analysis. And the reason the fallback exists at all: same-provider backups die together — the cloud 5.2 and 4.7 share one API — so real failover crosses providers, cloud first, local 9B behind it. Read fresh this morning, that chain is exactly two rungs. ## One evening in August August 18: the workhorse. First download byte logged 14:41 — domestic mirror, resume-safe script, which is June's download lesson promoted to infrastructure. The serving banner comes up at 19:58; by 20:02 a script mounts Claude Code on the local weights. 5 hours 21 minutes, first byte to coding agent. The August start script is June's scar tissue, worn as defaults: the same-family parser pair, the FlashInfer kill switch, a 38,912-token window, fp8 KV cache — under the same vLLM 0.23.0 banner as June's logs. The new pits were small and solved in comments: Claude Code sends an effort level this vLLM rejects with a 500, so the mount script now forces one explicitly — low, for a 27B. The 4090D's CUDA index is probed at runtime, because enumeration shifts between boots. The newest horse even runs inside a conda environment still named after the first. ## The honesty file Today is day 9 of the 27B's continuous run — 95% of the 4090D's 24 GB, the A4000 idle beside it. The failover chain names the local 9B as the fallback, but the endpoint it points at has no listener; that log ends with a clean shutdown. And the 27B holding the GPU is not in the chain at all — it is a work mount, not insurance. The stable is not a hot standby. It is a promise kept by scripts: one command re-lights the fallback, about the 3-4 minutes the 27B itself needs to come ready. What 70 GB of owned weights buys is not speed and not quality. It is a chain of custody — one link of the fleet's brain that no outage, quota, or pricing change can take away — plus a compounding record: June's two days bought August's one evening. *[Back to the fleet audit](/posts/2026-08-27-one-man-one-legion-fleet-audit/) — next in the engine-room arc: the stress test that graded the newest horse.*