Reformat records into a strict output template
Rewrite each input record into a fixed line template, same fields, same order, every time.
Category: Instruction following
Local with a check
Each model was asked to rewrite 10 small record sets (CSV, JSON, pipe-separated, key:value blocks) into a fully pinned line template via Ollama (temperature 0, greedy); output had to match the expected text exactly after trailing-whitespace normalization. Infra-error cases are excluded from pass rates.
Measured results
Each model actually ran; these are real pass counts against the verifier, best first.
| Model (Ollama tag) | Cases passed | Pass rate |
|---|---|---|
| gemma4:26b-mlx best 21c59a2eae30 | 10/10 | |
| devstral-small-2:latest 24277f07f62d | 10/10 | |
| qwen3.5:27b 7653528ba5cb | 10/10 | |
| qwen3-coder:30b 06c1097efce0 | 9/10 |
verifier The model's output is compared character-for-character to the expected reformatted text after stripping code fences, per-line trailing whitespace, and leading/trailing blank lines; any other deviation in field order, punctuation, casing, dates, padding, or line ordering fails the case. There is no partial credit for mostly-correct lines.
Caveats
- Small-sample datapoint: 10 objective cases, a single greedy run per model. Best model 95% CI: 72%–100%.
- Exact-match is all-or-nothing per case: one wrong character or one mis-sorted line fails the whole case, so scores understate outputs that are almost right.
- Every template is fully specified in the prompt with a worked example, so this measures instruction-following and mechanical transforms (casing, zero-padding, date rewriting, sorting), not knowledge.
- Record-reformatting tasks of this shape are common in instruction-tuning data, so absolute pass rates may flatter the models; relative rankings are the sturdier read.
Preserved raw run (every prompt and output): eval/results/reformat-records-into-strict-template-r20260702-101843.json in the repo.
Built by Sam Carlton