Appendix BCase study: small models in a coding harness
What went wrong
Every failure came from the parts around the model: no tool calls, the wrong folder, or a server that offered no tools.
| Model | Group | What happened | Graded where it saved | Time |
|---|---|---|---|---|
| qwen2.5-coder 1.5B | Local | Never used a tool: described the tool calls in text instead of making them | No code to grade | 197 s |
| gemma3:27b | Local | Never used a tool, so no file was saved | No code to grade | 550 s |
| gpt-oss:20b | Local | Wrote working code, but into the wrong folder | Pass | 529 s |
| gemini-3.1-flash-lite | Small cloud | Wrote working code, but into the wrong folder | Pass | 27 s |
| Phi-4-mini-reasoning | Small cloud | The server it ran on was not set up to offer tools | No code to grade | 3 s |
- The setup mattered as much as the model. Two early failures turned out to be a bug in the harness and a sign-in problem. Both were fixed and the models re-run, so they aren't counted against the models.
- Labels can be wrong. Two models listed by the gateway's model catalogue as having no tool support used tools without trouble and passed.
- Tools reach further than you think. Two models that passed also ran tests in an unrelated project folder on the same computer before returning to their own. Nothing broke, but it's why agents belong in a sandbox.
Appendix B · page 4 of 7
Turn pages with the ← and → keys.