Appendix BCase study: small models in a coding harness
The model is only one part
Can a small, cheap AI model do real work in place of a big one? This study gave 29 models from 9 vendors the same small coding job, inside the same harness. The clearest lesson was not about the models.
0 of 5
failures were wrong code
Every failure came from the parts around the model: it never used its tools, it saved the work in the wrong folder, or the server it ran on wasn't set up to offer tools.
0.3¢ to $0.25
for the same passing result
What one passing run cost at the prices the study recorded: a fraction of a cent up to about 25 cents, a spread of about 81 times.123
~9 min
for two self-hosted runs, vs 12–117 s in the cloud
Passing cloud runs took seconds. Two self-hosted models ran for about 9 minutes (550 and 529 seconds) and still failed.
Notes (3)
- 1Anthropic, “Pricing,” Claude Platform Docs, https://platform.claude.com/docs/en/about-claude/pricing (accessed October 26, 2026).
- 2OpenAI, “Pricing,” OpenAI (developer docs), https://developers.openai.com/api/docs/pricing (accessed October 26, 2026).
- 3Google, “Gemini API pricing,” Google AI for Developers, https://ai.google.dev/gemini-api/docs/pricing (accessed October 26, 2026).
Appendix B · page 1 of 7
Turn pages with the ← and → keys.