Appendix BCase study: small models in a coding harness
Cost and time
The same passing result cost from a fraction of a cent to about 25 cents, and bigger wasn't leaner.
Small cloud models
- grok-4-1-fast-non-reasoning$0.003
- gpt-6-luna$0.0039
- grok-4-1-fast-reasoning$0.004
- DeepSeek-V4-Flash$0.011
- gpt-5.6-luna$0.011
- gpt-5.4-nano$0.014
- gemini-3.5-flash-lite$0.02
- claude-haiku-4-5$0.03
- gemini-3.5-flash$0.059
- gpt-5.4-mini$0.06
- gemini-3.8-flash$0.077
Frontier models
- Mistral-Large-3$0.03
- gpt-6-sol$0.03
- DeepSeek-V4-Pro$0.036
- gpt-4.1$0.037
- kimi-k2.6$0.045
- grok-4.3$0.049
- claude-sonnet-5$0.063
- gpt-5$0.097
- claude-opus-4-6$0.15
- claude-sonnet-4-6$0.15
- grok-4-20-non-reasoning$0.21
- DeepSeek-V3.2$0.25
The one local pass isn't shown: local models have no per-token price.
Why can a job of under 100 lines cost 25 cents? A model doesn't remember between steps, so the harness re-sends the whole conversation every round.1 Runs read 16,000 to 578,000 tokens of input (a token is a word or part of a word). In a typical cloud run about 74% of that was served from the vendor's cache at a lower rate;2 the most expensive run, DeepSeek V3.2, got no cache discount at all. Bigger also wasn't leaner: 6 models finished in 3 tool calls, while grok-4-20-non-reasoning, a frontier model, made 36 for the same job.
Go deeper: how long each run took
Small cloud models
- Phi-4-mini-reasoning3 s · fail
- grok-4-1-fast-non-reasoning17 s
- claude-haiku-4-521 s
- gemini-3.5-flash22 s
- gemini-3.5-flash-lite23 s
- gemini-3.1-flash-lite27 s · fail
- gemini-3.8-flash31 s
- DeepSeek-V4-Flash36 s
- gpt-5.4-mini38 s
- gpt-6-luna41 s
- gpt-5.6-luna42 s
- gpt-5.4-nano53 s
- grok-4-1-fast-reasoning117 s
Local, downloadable models
- qwen3-coder:30b50 s
- qwen2.5-coder 1.5B197 s · fail
- gpt-oss:20b529 s · fail
- gemma3:27b550 s · fail
Frontier models
- gpt-4.112 s
- Mistral-Large-320 s
- kimi-k2.621 s
- claude-sonnet-521 s
- DeepSeek-V4-Pro22 s
- grok-4.330 s
- gpt-6-sol34 s
- claude-opus-4-641 s
- claude-sonnet-4-675 s
- DeepSeek-V3.282 s
- gpt-583 s
- grok-4-20-non-reasoning91 s
Notes (2)
- 1Anthropic, “Effective context engineering for AI agents,” Anthropic Engineering, September 29, 2025, https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents (accessed October 26, 2026).
- 2Anthropic, “Pricing,” Claude Platform Docs, https://platform.claude.com/docs/en/about-claude/pricing (accessed October 26, 2026).
Appendix B · page 6 of 7
Turn pages with the ← and → keys.