Chapter 7Small and free models
Where they stumble
Small models know less, and their own makers say so. Downloadable doesn't always mean small, either.
Their makers are candid about the limits. Microsoft says Phi-3-mini simply doesn't have the capacity to store much factual knowledge,1 and Google warns that its Gemma models may give incorrect or outdated facts.2 On a short factual-question test, smaller versions of the same model families answered far fewer questions correctly.3
"Open-weight" (downloadable) doesn't always mean small: OpenAI says its gpt-oss-120b runs on a single 80 GB data-center GPU,4 and Meta's Llama 4 Maverick has 400 billion parameters.5
Appendix B · Case study0 of 5 failures were wrong codeCase study: we gave 29 models the same small coding job. Every failure was about the parts around the model (tools, folders, setup), never wrong code. Local, downloadable models struggled most.
Read the case study →Notes (5)
- 1Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, et al., “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone” (preprint, arXiv, April 2024 (v4 2024-08-30)), https://doi.org/10.48550/arXiv.2404.14219.
- 2Google, “Gemma 4 model card,” Google AI for Developers, https://ai.google.dev/gemma/docs/core/model_card_4 (accessed October 26, 2026).
- 3Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus, “Measuring short-form factuality in large language models (SimpleQA)” (preprint, arXiv, November 7, 2024), https://doi.org/10.48550/arXiv.2411.04368.
- 4OpenAI, “Introducing gpt-oss,” August 5, 2025, https://openai.com/index/introducing-gpt-oss/ (accessed October 26, 2026).
- 5Meta, “The Llama 4 herd,” Meta AI, April 5, 2025, https://ai.meta.com/blog/llama-4-multimodal-intelligence/ (accessed October 26, 2026).
Chapter 7 · page 2 of 2
Turn pages with the ← and → keys.