AI-assisted claims triage: an insurance case study.

The problem behind the problem

The insurer's complaint data pointed at settlement speed. The operational data pointed somewhere else entirely: claims were sitting in an unsorted queue for about 2.9 days before a human opened them. Actual settlement, once a handler picked one up, was fast. The delay was almost entirely in the waiting.

That queue contained everything. A windscreen chip and a total loss with a potential injury arrived in the same inbox, in the same format, with no indication of which was which until somebody read fourteen pages of PDF. Handlers were spending the first hour of every day sorting rather than settling, and the most experienced adjusters were reading the simplest claims because nobody could tell them apart in advance.

Two previous attempts at fixing this had failed. A rules engine built on document filename patterns broke the first time a broker changed their template. A classification model trained on claim type codes worked well in testing and poorly in production, because the codes were entered after triage rather than before it — a leakage problem nobody caught until go-live.

What we measured before we built

Week one was spent instrumenting rather than designing. We could not have defended any improvement claim without knowing precisely what we started from, and the numbers that came back reframed the project for everyone involved.

~2.9^d

Time from submission to first human review, as reported by the project team

14^pp

Pages in a typical first-notification pack

0^%

Claims pre-sorted by severity before a handler opened them

We also sampled 400 historical claims and asked three senior adjusters to triage them independently. They agreed with each other 81% of the time — which set a realistic ceiling. A system that matched human triage perfectly would be matching a process that disagrees with itself one time in five. That number became the honest target, and it changed how the client's own leadership talked about the project.

The approach we chose

The tempting architecture was a single large model reading the whole pack and producing a routing decision. We rejected it. It would have been expensive per claim, slow on long packs, impossible to explain to the regulator, and it would have collapsed the two genuinely separate problems — what is in this document and what should happen next — into one opaque step.

Instead we split the work into a pipeline where every stage has its own confidence score and its own escape hatch:

1. Normalise. Every pack — email, broker portal upload, scanned post — lands in one queue with provenance attached and pages classified by type.
2. Extract. Layout-aware OCR plus schema-constrained LLM extraction pulls 34 fields, each with a per-field confidence score rather than one score for the document.
3. Verify. Extracted fields are cross-checked against the policy administration system: does this policy exist, was it in force on the date of loss, does the vehicle match?
4. Score. A gradient-boosted model scores severity and complexity from the verified fields — deliberately not from raw text, so it stays explainable and stable.
5. Route. An agent applies the routing policy, calls the claims system to assign the file, and writes a plain-English rationale into the claim notes.

Only the extraction and routing steps use a language model. The severity decision — the one the regulator cares most about — is made by a model whose inputs are structured, auditable fields that a human can inspect in a table.

What we built

A golden dataset before a single prompt

Before any pipeline code, we assembled 180 real claim packs with the correct extraction and correct routing agreed by the adjuster panel. Building that dataset took eleven days and was the single most useful thing we did. It defined “done”, it caught the leakage that had killed the previous attempt, and it meant every subsequent change could be measured rather than argued about.

Extraction tuned to the documents, not the demo

Real claim packs are photographed on phones at angles, faxed, and occasionally scanned upside down. We tuned chunking and page classification against that reality rather than against clean specimens. Hybrid retrieval over the policy corpus grounds the cover check, and every extracted field carries the page and bounding box it came from — so a reviewer sees the source, not just the answer.

A review queue designed as a product

Roughly three claims in ten still need a human, and that queue is where the system either earns trust or loses it. It shows the extracted fields beside the source page, highlights only what fell below threshold, and lets a reviewer correct a field in one keystroke. Those corrections flow back into the golden dataset weekly.

An agent with a budget and a stopping condition

The routing agent has a fixed tool surface — read policy, read claim, assign handler, write note — a token budget, and a hard rule that it stops and escalates rather than retrying indefinitely. It cannot settle, pay, or contact a customer. Narrow authority was the condition of getting approval to run it at all.

The guardrails that made it approvable

This is a regulated decision affecting customers at a difficult moment. The controls were not a compliance afterthought; they were the reason the project was allowed to proceed past the pilot.

• Permission-scoped retrieval. The agent retrieves only documents attached to the claim it is working and the policy it verified. It has no view across the book.
• Personal data redaction. Names, addresses, bank details and medical references are masked before any content reaches a hosted model, and restored only inside the insurer's boundary.
• Injection defence. Claim packs contain free text from third parties. Every document is screened for instruction-override patterns before extraction runs; 412 adversarial cases run in CI.
• No autonomous customer contact. The agent drafts; a human sends. Every outbound communication has a named approver.
• Seven-year decision log. Inputs, retrieved sources, model versions, scores, rationale and outcome retained and replayable for the full complaints window.
• Evaluation gates the release. Faithfulness, field accuracy, routing agreement and refusal correctness run on every change. A regression blocks the deploy the way a failing test would.

What was reported

The project team reported the time to a first triage decision moving from about 2.9 days before the build to about six hours at 90 days on live production traffic. The raw figures are shown below. Percentage changes are left out until the client has verified the measurement definition, window and population.

~2.9^d

Before: time from submission to first human review

~6^h

After: time to first triage decision, at 90 days

400^claims

Claims in the baseline sample used to set the target

What we would do differently

Three things, stated plainly, because a case study that reports only wins is marketing rather than engineering.

• We under-invested in the review queue for too long. For the first three weeks it was a thin admin screen, and reviewer trust dipped accordingly. The queue is a product surface, not an internal tool, and it should have been designed as one from the start.
• We started with a model that was larger than necessary. Extraction moved to a smaller, cheaper model in month two with no measurable quality loss. We should have tested downward earlier instead of assuming bigger was safer.
• We should have shipped the rationale text sooner. The plain-English explanation written into the claim notes did more for adjuster adoption than any accuracy improvement, and it was a two-day piece of work sitting in the backlog for a month.

Discuss a similar workflow