Produce JSON conforming to a given schema

← All tasks

Emit JSON that validates against a supplied JSON Schema with no extra or missing keys.

Category: Instruction following

🟡 Local with a check Evaluated July 2, 2026 · 10 cases · Ollama (local, Apple Silicon)

Each model was asked to emit a JSON value for 10 schema-constrained specs via Ollama (temperature 0, greedy); each output was parsed and validated against the case's JSON Schema. Infra-error cases are excluded from pass rates.

Measured results

Each model actually ran; these are real pass counts against the verifier, best first.

Model (Ollama tag)	Cases passed	Pass rate
gemma4:26b-mlx best 21c59a2eae30	10/10	100%
devstral-small-2:latest 24277f07f62d	10/10	100%
qwen3.5:27b 7653528ba5cb	10/10	100%
qwen3-coder:30b 06c1097efce0	10/10	100%

verifier The model's output is parsed as JSON and validated against the case's schema by a strict-subset validator (type, const/enum, required, additionalProperties, array bounds and uniqueness, numeric ranges and multipleOf, string length and pattern). It checks schema conformance only — it does not judge whether values match the instruction prose beyond what the schema pins down.

Caveats

• Small-sample datapoint: 10 objective cases, a single greedy run per model. Best model 95% CI: 72%–100%.
• Schema conformance is the only thing verified: an output can pass while getting details the schema does not pin (names, quantities, wording) wrong — semantic fidelity to the instruction prose is not scored.
• Schemas here lean on const, enum, and pattern to pin the values the instruction dictates, so an empty object or generic guess cannot pass; but any looseness a schema leaves open is invisible to the score.
• Emitting JSON to a schema is heavily represented in instruction-tuning data, so absolute pass rates likely flatter the models; relative rankings are the sturdier read.

Preserved raw run (every prompt and output): eval/results/emit-json-conforming-to-schema-r20260702-101519.json in the repo.