How to catch silent data loss in an AI extraction pipeline

The short answer

Compare what landed against what the source contains, item by item, and treat any completion rate you haven't reconciled as unproven. My pipeline measured coverage per post. A list of 684 vehicles came back with 401 and still scored a perfect 1 of 1. That shape of error silently lost about 130,000 records.

Four gates now run over every processing wave before anything is written: coverage, grounding, no-shrink and shortfall.

What the metric hid

The pipeline turns Arabic recovery lists from Sudanese Facebook groups and Telegram channels into one record per vehicle, for a free public registry of cars looted in the war. Some posts describe one car. Some hold more than 1,000 chassis numbers.

One post had 1,402 chassis numbers in it. The model ran out of output tokens while writing and returned 16 vehicles. The post was marked processed. Re-extracted with the output limit fixed, it yielded 1,052 vehicles, each chassis checked against the source text.

Per-post coverage looked healthy the whole time. A post counts as done whether it gives back 16 rows or 1,052.

The summaries lied too. A reader that understood only one of two verdict file shapes rejected 195 entries and dropped 3,277 vehicles, behind a line that read ingested 10 new verdict file(s). The only check that has ever caught a silent loss on this project is counting database rows against what the model claimed.

The four gates

1. Coverage. Every post in a batch has to come back, or nothing from the batch is written. A model can stop early and still report success. In one benchmark Haiku analysed 1 of 21 posts and reported all 21 done.
2. Grounding. Every chassis number, plate, contact and price has to appear in the post text or in the OCR text of its photos. Anything else is nulled. This caught 106 invented identifiers, including six whole vehicle records and a real phone number copied onto 117 vehicles from a neighbouring post. The model that produced them flagged none of the 106.
3. No-shrink. A re-run can't replace a post's stored vehicles with fewer of them. A bad re-run gets refused and the good rows stay.
4. Shortfall. Count the chassis-shaped numbers in the source and compare them with the vehicles emitted. A gap gets flagged for a person to read.

Why shortfall flags and coverage blocks

Coverage and grounding are exact. A post is missing or present. A number is in the source or absent from it. Exact checks can act on their own: coverage blocks the batch, grounding nulls the value.

Shortfall is an estimate. Some posts are bare lists of tracking codes with no vehicle words, and they legitimately produce zero vehicles. A blocking shortfall gate would reject every one of them. A gate that cries wolf gets switched off, and a switched-off gate catches nothing. So it flags, and I read the flags.

The grounding fix

Grounding first checked identifiers against the post text alone. A chassis number stamped on a plate and photographed is absent from the text by definition, so the gate had to stay off for every post with a photo. Running it anyway once nulled 251 real identifiers.

Once the photos had been transcribed with Google Cloud Vision, I pointed the gate at the post text plus the OCR text. On one wave of 450 posts, 537 chassis numbers existed only in the OCR. A text-only check would have deleted all 537, and those stamped numbers are what a police impound yard checks at the gate.

Where this transfers

Any pipeline that turns documents into rows has this blind spot. You count the document and lose the line.

• Claims intake. Count the claim lines on each form and compare them with the rows written for that claim.
• KYC documents. Ground every extracted field, ID number and date of birth included, in the OCR of the document it came from. Null what you can't find.
• Invoices and receipts. Compare line items extracted with line items detected on the page, and flag the gap for a person.
• Reprocessing. Refuse any re-run that replaces records with fewer records.

Questions people ask

What is silent data loss?

Records that disappear while every job reports success. The only symptom is a count lower than the source, and you see it only if you count the source.

How do I stop an LLM from inventing data during extraction?

Check every extracted value against the source text mechanically and null what isn't there. On one wave of 4,703 vehicles the gate nulled 7 fields, 0.15%. Across the project the models self-reported 0 of 106 inventions.

Should a data quality check block or warn?

Let it act on its own when the check is exact: block a batch with a missing item, null a value that's absent from its source, refuse a re-run that shrinks. Warn when legitimate input can trip it, like a list that holds no vehicles.

The project behind it

The full build is in the vehicle recovery case study. How the registry is searched is in how to check if your looted car in Sudan has been found, and the cost side is in how I cut the AI cost about tenfold.

If you run claims intake or invoice processing and want these checks in your pipeline, this is how I work.