679,162 free public records from the chaos of Arabic social media
Families in Sudan were searching Facebook groups post by post for one chassis number. This pipeline ingests, transcribes and scores tens of thousands of Arabic posts into a registry anyone can query, and it had to be re-engineered until it could afford to run. The interesting part is what it cost to make it work, and what the metrics were hiding.
Personal project · Python, Claude and OpenAI APIs, Google Cloud Vision · 2026
Context
The project started for a private reason: a family vehicle was taken during the war in Sudan, and the only trace of vehicles like it is in Arabic-language Telegram channels and Facebook groups — recovery lists, auction posts, photographs of paperwork. Thousands of posts a week, no search, no structure, and a real answer hiding somewhere inside them.
It stopped being a private project quickly. The same posts that might contain one vehicle contain tens of thousands of others, each one somebody else's. So the system now maintains a searchable registry that other families can query — by partial chassis number, plate, Arabic or Latin digits, or free text.
What I did
- Built the collection layer: a live Telegram channel monitor with historical backfill, an intake bot so family and friends could forward anything suspicious, and a self-hosted Facebook harvester that searches inside groups rather than scrolling the feed — because search renders reliably where feed pagination gets throttled.
- Built the extraction layer: each recovery list, whether text or photographed, is parsed into individual searchable vehicle records rather than left as a wall of text. That is what turned a private search into a public registry.
- Rebuilt the architecture when the numbers said to. The original design sent every photograph to a vision model. Once I benchmarked it honestly, I moved to transcribing every image once through Google Cloud Vision at roughly $1.50 per thousand, after which the entire corpus is text — which is both cheaper and better. I benchmarked a competing model for the same backlog at around $145 and no quality gain, and did not use it.
- Cut spend in four layers that lose no information: an exact, deterministic scan for known identifiers before any model call; a relevance gate that drops posts with no vehicle vocabulary, no chassis-shaped number and no image, removing about 35% of the corpus; content-hash deduplication, because these lists are cross-posted between groups and one verdict can serve every copy; and model tiering, where a cheap screening pass handles the bulk and only borderline posts escalate. Large backlogs drain through the asynchronous batch API at half price.
- Added a provider fallback so a spend limit, rate limit or outage retries against a second vendor and tags the verdict with which model produced it, instead of silently dropping the post.
- Built four verification gates — coverage, grounding, no-shrink and shortfall — that run over every processing wave before anything is written to the database.
Collected, transcribed, scored and made searchable. Every image in the corpus has been transcribed exactly once.
The gate that earned its keep
Coverage was measured per post, which looked healthy and was hiding a disaster: a single list containing 684 vehicles could return 401 of them and still score a perfect pass, because the post itself had been processed. That shape of error had silently lost on the order of 130,000 records.
The fix was a shortfall gate that compares records emitted against chassis-shaped numbers present in the source, and flags the gap rather than blocking on it — because a list of bare tracking codes legitimately yields zero vehicles, and a gate that cries wolf gets switched off. A second fix mattered as much: the grounding check originally validated against post text alone, which would have discarded every identifier recovered from a photograph. On one wave that was 537 identifiers.
Where the corpus goes before it reaches a paid model (figures from the earlier processing run; the live collection has since grown to 106,595 posts)
The difference is not lost work. It is posts removed by the deterministic identifier scan, the relevance gate and content-hash deduplication — none of which cost an API call, and none of which discard information.
What I would tell you in an interview
Every expensive decision on this project was one I made before I measured. The vision-first architecture, the model choice, the coverage metric — each felt obviously right, and each was wrong in a way only a benchmark or a reconciliation exposed. The habit I took from it is the one I would bring to any operation: instrument the thing, compare what it claims against what actually landed, and treat a green metric you have not audited as an untested assumption.
Where this transfers
What it means for an employer: I turn chaotic multilingual data into a registry people can rely on, report honestly even when the dashboard says green, and manage model and cloud cost like a budget line. This is a document-processing pipeline with a cost ceiling and a correctness requirement, which describes a great deal of commercial work: claims intake, KYC and onboarding documents, invoice and receipt capture, compliance review. The pattern generalises directly — filter deterministically before you filter probabilistically, transcribe once, tier your models, and never trust a completion rate you have not reconciled against the source.