Generate a regex from a plain-English description

← All tasks

Turn a written matching rule into a working regular expression.

Category: Coding

🟡 Local with a check Evaluated July 2, 2026 · 10 cases · Ollama (local, Apple Silicon)

Each model was asked to produce a regex for 10 plain-English specs via Ollama (temperature 0, greedy); each regex was run against labeled match / no-match strings. Infra-error cases are excluded from pass rates.

Measured results

Each model actually ran; these are real pass counts against the verifier, best first.

Model (Ollama tag)	Cases passed	Pass rate
gemma4:26b-mlx best 21c59a2eae30	10/10	100%
qwen3.5:27b 7653528ba5cb	10/10	100%
qwen3-coder:30b 06c1097efce0	10/10	100%
devstral-small-2:latest 24277f07f62d	8/10	80%

verifier Each model's regex is run with .test() against labeled strings; it passes a case only if it matches every mustMatch string and rejects every mustNotMatch string (mustNotMatch includes near-misses that require correct anchoring).

Caveats

• Small-sample datapoint: 10 objective cases, a single greedy run per model. Best model 95% CI: 72%–100%.
• The verifier is the 'check': these results describe safety *with* an example-based verification step, which is how you would use a generated regex anyway.
• Regex-from-English is almost certainly well represented in training data, so absolute pass rates flatter the models; relative rankings are the sturdier read.

Preserved raw run (every prompt and output): eval/results/generate-regex-from-plain-english-r20260702-101100.json in the repo.