Benchmarking LLMs for Structured Data Extraction

A few fellow engineers and I were talking about field extraction the other day. They were both saying Gemini's been surprisingly good at pulling structured data off documents, which surprised me, since that's not really the model I'd have reached for first. So we ran an experiment using LangSmith.

I printed ten copies of a form with the usual set of fields, names, dates, IDs, a checkbox, a signature line, and asked my coworker to help me fill them out by hand. I ran OCR on them too, mostly out of curiosity. It was pretty bad, which wasn't unexpected, especially on checkboxes.

Testing it

I set up a LangGraph graph with one node that does the extraction, and passed in which provider to use as runtime context, so the eval could run the exact same graph once per model:
‍

def _extract(state: ExtractionState, runtime: Runtime[ExtractionContext]) -> dict:
    extractor = EXTRACTORS[runtime.context.provider]
    return {"result": extractor.extract(state["image_path"])}

I hand-labeled ground truth for every form and ran it all through LangSmith so I could score each model against the exact same data.

Scoring took a bit of tuning, mostly because a plain exact string match evaluator will ding a model for writing "Sr" instead of "Sr." or "management" instead of "Management," which isn't actually wrong, it's just formatting. So everything got normalized first (lowercased, trailing punctuation stripped, dates parsed into one format, phone numbers stripped down to digits) and scored three ways: fraction of fields correct for partial credit, whether the whole record was perfect for a stricter all-or-nothing exact match, and field accuracy per dollar.

Claude and Gemini ended up tied on average field accuracy, but Gemini still got a lot more whole forms completely right, which is a different thing to be measuring than it sounds like at first. The other thing that jumped out was the cheap tiers. Gemini's Flash Lite scored higher on field accuracy than OpenAI's flagship did, while Claude's Haiku and OpenAI's nano both fell off hard.

Picking a metric

No single model won across the board, and that's kind of the whole point. Field accuracy, exact match, and cost efficiency measure different things, so a model that dominates one of them can easily be the weakest on another. What's nice about having an eval running is that switching which one you care about is just a matter of changing what you sort by.

Optimizing for exact match

A form that's 11 out of 12 fields right still needs someone to go fix that twelfth field, and that costs about the same as fixing a form that's only 6 out of 12 right, since either way a person has to open it back up. That's the case for caring about exact match instead of average field accuracy.

Optimizing for field accuracy

Field accuracy gives partial credit for every field that's right, which matters more than exact match when a human is going to check every field anyway. Fewer wrong fields overall beats fewer broken records when the correction happens field by field instead of record by record.

‍Optimizing for cost

Cost efficiency matters most at volume, since a cheaper model that's a little less accurate can still get more forms extracted correctly per dollar than a pricier one that's only a little better.

So Chris and Mike were right, though the answer splits three ways. Gemini Pro won exact match and tied Claude Opus on field accuracy, and Gemini Flash Lite won on cost. Flash Lite is the one I didn't see coming, since it beat OpenAI's flagship on field accuracy too, at a tiny fraction of the price.

Technical References: 

Benchmarking LLMs for Structured Data Extraction GitHub Repo