An AI IELTS-writing grader, benchmarked essay-by-essay against a teacher's hand-marked bands — measured honestly, and held conservative where the data said to.
The job that matters: catching what a real examiner catches. Recall is measured by automated span-overlap against every error the teacher marked.
| Essay | Task | Teacher errors | Caught | Recall |
|---|---|---|---|---|
| Xuân Mai | T2 | 32 | 27 | 84% |
| Trần Hải Long | T2 | 23 | 22 | 96% |
| Giang Trịnh | T2 | 45 | 43 | 96% |
| HG | T2 | 35 | 35 | 100% |
| Hương Trà | T2 | 19 | 19 | 100% |
| Task-2 margin total | 154 | 146 | 95% |
Two tracked-changes essays (50% / 57%) are excluded from the margin: ~85% of the teacher's marks there are the student deleting words for style, not grammar errors, so span-overlap under-counts them. Task-1 misses are chart-data errors a text grade can't see — now covered by live Task-1 data checks.
Predicted overall band vs. the teacher's official band, across 8 scored essays.
| Essay | Our band | Teacher | Δ |
|---|---|---|---|
| Giang Trịnh | 5.5 | 5.5 | 0 |
| HG | 5.0 | 5.0 | 0 |
| Thanh HW2 | 6.5 | 6.5 | 0 |
| Thanh HW3 | 7.0 | 7.0 | 0 |
| Xuân Mai | 5.5 | 6.0 | −0.5 |
| Trần Hải Long | 5.5 | 6.0 | −0.5 |
| Hương Trà | 6.0 | 6.5 | −0.5 |
| Hương Trà T1 | 5.5 | 6.5 | −1.0 |
Every band miss traces to one criterion — Lexical Resource — and nothing else.
| Criterion | Bias vs teacher | Verdict |
|---|---|---|
| Task Response | ~0 | On the money |
| Coherence & Cohesion | ~0 | On the money |
| Lexical Resource | −0.44 | The one systematic weakness — stingier on vocabulary |
| Grammatical Range | ~0 | On the money |
Three prompt formulations were A/B-tested against the teacher's real Lexical-Resource bands.
| Variant | Band error (MAE) | Bias | Result |
|---|---|---|---|
| Current (shipped) | 0.438 | −0.44 | Conservative — held |
| Descriptor anchor | 0.375 | +0.13 | Inflates weak essays (+1) |
| Hard floor | 0.625 | −0.63 | Crushes strong essays |
The lowest-error variant only wins by inflating weak essays a full band — the opposite of what a grader students trust should do. So Lexical Resource was held conservative rather than degraded: a measured decision not to ship a change, backed by the A/B data. Shipping the honest non-decision is the point.
On the job that matters — catching what a real examiner catches — it runs at ~95%, with band scores inside the human-examiner margin and a bias that never over-grades. The one weakness, I proved with data can't be safely fixed — so I held it.
So the numbers stay trustworthy: ground truth is one teacher, 9 essays — strong signal, small sample. Recall matching is automated span-overlap (±a few per count). The eval reproduces the v100 grader logic via DeepSeek; the live edge function adds caching + chart vision + band finalization on top.