Solve a multi-step arithmetic word problem
Turn a 3-5 step quantitative word problem into a single correct numeric answer.
Category: Math
Needs more data
Each model solved 29 GSM8K grade-school math word problems via Ollama (temperature 0, greedy, step-by-step prompt); each explicit final 'Answer: <number>' was compared exactly to the dataset's gold answer. Infra-error cases are excluded from pass rates.
Measured results
Each model actually ran; these are real pass counts against the verifier, best first.
| Model (Ollama tag) | Cases passed | Pass rate |
|---|---|---|
| gemma4:26b-mlx best c8656f50f0a6 | 29/29 | |
| qwen3-coder:30b 06c1097efce0 | 29/29 |
verifier A response is scoreable only when it ends with an explicit 'Answer: <number>' line and the provider does not report generation truncation; that number is compared exactly to the GSM8K gold answer.
Caveats
- Small-sample datapoint: 29 objective cases, a single greedy run per model. Best model 95% CI: 88%–100%.
- GSM8K is a widely published benchmark and is almost certainly in every model's training data; absolute pass rates flatter the models — relative rankings are the sturdier read.
- One deterministically selected source item (gsm8k-1161) is preserved in the evidence but excluded from scoring because its wording supports both 140 and the dataset gold 170.
- This is retrospective benchmark evidence, not a deployable runtime check, so it cannot produce a 'local with a check' verdict.
- Unlike regex generation, everyday arithmetic has no automatic runtime verifier: read this verdict as measured accuracy on labeled problems, not as a runtime safety net.
Preserved raw run (every prompt and output): eval/results/solve-multi-step-arithmetic-word-problem-r20260728-155709.json in the repo.
Built by Sam Carlton