Appendix BCase study: small models in a coding harness
The task
Roman numerals in the Go programming language: turn 1987 into MCMLXXXVII and back again. Small and completely specified on purpose, so that writing the code wasn't the hard part.
The finished file runs about 60 to 90 lines. Every model got this brief, word for word:
In this folder, write roman.go in package roman with two functions: ToRoman(n int) string converts 1..3999 to a Roman numeral FromRoman(s string) (int, error) parses a Roman numeral back to a number Use the standard subtractive forms: IV, IX, XL, XC, CD, CM. FromRoman must return an error for an empty string and for a string containing a character that is not one of I, V, X, L, C, D, M. Write roman_test.go covering both functions, then run `go test ./...` and report the result.
How it was checked
After each run, a grader the model never saw was copied in: a test with 15 fixed numbers, four junk inputs that must be rejected, and a full round trip of every number from 1 to 3999. The model's own tests were deleted first, so it couldn't pass by writing an easy one. If the file was missing or broken, the harness said "carry on", up to three times. Code saved in the wrong folder was graded again where it was saved.
The harness is PounceCode 0.7.1 (from PounceApps, the site author's own app label). Models were reached through their vendors' own APIs, a hosted model gateway, or run on local hardware.
Go deeper: show the grader
package roman
// The held-out grader. The run never sees this file: it is copied in after
// the model has stopped, which is the whole point — a test the executor can
// read is a test the executor can edit.
import "testing"
var heldOutCases = []struct {
n int
s string
}{
{1, "I"}, {3, "III"}, {4, "IV"}, {9, "IX"}, {14, "XIV"}, {40, "XL"},
{90, "XC"}, {400, "CD"}, {900, "CM"}, {1066, "MLXVI"}, {1987, "MCMLXXXVII"},
{2024, "MMXXIV"}, {3999, "MMMCMXCIX"}, {58, "LVIII"}, {1994, "MCMXCIV"},
}
func TestHeldOutToRoman(t *testing.T) {
for _, c := range heldOutCases {
if got := ToRoman(c.n); got != c.s {
t.Errorf("ToRoman(%d) = %q, want %q", c.n, got, c.s)
}
}
}
func TestHeldOutFromRoman(t *testing.T) {
for _, c := range heldOutCases {
got, err := FromRoman(c.s)
if err != nil {
t.Errorf("FromRoman(%q) returned %v", c.s, err)
continue
}
if got != c.n {
t.Errorf("FromRoman(%q) = %d, want %d", c.s, got, c.n)
}
}
}
func TestHeldOutRejectsJunk(t *testing.T) {
for _, s := range []string{"", "ABC", "MMQ", "12"} {
if _, err := FromRoman(s); err == nil {
t.Errorf("FromRoman(%q) accepted junk", s)
}
}
}
func TestHeldOutRoundTrip(t *testing.T) {
for n := 1; n <= 3999; n++ {
s := ToRoman(n)
back, err := FromRoman(s)
if err != nil {
t.Fatalf("round trip %d -> %q failed: %v", n, s, err)
}
if back != n {
t.Fatalf("round trip %d -> %q -> %d", n, s, back)
}
}
}Appendix B · page 3 of 7
Turn pages with the ← and → keys.