Entity Resolution: "Are These the Same Thing?" at a Cent a Thousand

Every CRM has three copies of the same customer. Every product catalog sells the same item under four titles. Every data warehouse contains "Acme Corp", "ACME Corporation", and "Acme (DO NOT USE)", and somewhere a dashboard is triple-counting all of them. Entity resolution, deciding whether two records refer to the same real-world thing, is the least glamorous problem in data and among the most expensive to ignore, because every duplicate silently corrupts every metric downstream.

The classical toolkit (exact keys, fuzzy string distance, trained matchers) fails in a characteristic way: it matches strings, and identity lives in meaning. "Bob's Burgers LLC, 42 Main St" and "Robert's Burger Restaurant, 42 Main Street" defeat edit distance and are obviously the same restaurant to anyone who reads them. Reading them is now a sub-cent operation with a decision model like Jev, which is the entire news.

The pattern: pairwise verdicts, properly asked

At its core, resolution is one judge question asked many times: given record A and record B, same entity, different, or unclear? The craft is the boundary legislation inside the question, because "same" is domain policy, not philosophy: same company (subsidiaries count? franchises?), same person (Jr. and Sr.?), same product (color variants? bundle vs unit?). Write the clauses in; the postgres build's fuzzy-join demo showed the mechanics, and this page is that section grown up.

Verdicts beat scores here for a workflow reason: a match probability still needs a threshold argument in a meeting; a same/different/unclear verdict with confidence routes itself, cascade-style: confident matches merge (or queue for merge), confident differents part ways, unclears go to the human lane, which is now a short list instead of the whole file.

The engineering reality: blocking, or the O(n²) tax

The naive version compares everything to everything, and a million records is half a trillion pairs; no per-verdict price survives that arithmetic. Every serious pipeline runs blocking first: cheap candidate generation (shared postcode, same first token, embedding nearest-neighbors, phonetic keys) that shrinks "all pairs" to "plausible pairs", and only plausible pairs meet the judge. The division of labor mirrors the classic-ML comparison: dumb-fast methods for recall of candidates, the language judge for precision on candidates, humans for the residue. Then materialize: resolved cluster IDs become columns per the database pattern, verdicts get cached (identity rarely changes daily), and incremental runs judge only new-or-changed records against existing clusters, which turns resolution from an annual cleanup project into a nightly job that costs less than the log storage.

Where it's already earning: CRM dedup before migrations, catalog dedup in ecommerce, supplier and invoice-entity matching in AP pipelines, security-alert entity linking, and research datasets where "same paper, same author?" decides whole analyses per the labeling discipline. Same machine everywhere; only the boundary clauses change.

Frequently asked questions

What is entity resolution?

Determining whether records across or within datasets refer to the same real-world entity (customer, company, product), then linking or merging them. Also known as record linkage, deduplication, or identity resolution.

How is the AI approach better than fuzzy matching?

Fuzzy matching compares characters; a language judge compares meaning, surviving abbreviations, reorderings, and descriptive variation that defeat edit distance. Keep fuzzy methods as the cheap blocking layer; promote the judge to the decision.

What does it cost at scale?

With blocking, judged pairs are a small multiple of record count, and at reported verdict prices million-record datasets resolve for single-digit dollars, versus the consulting-project pricing the problem historically commanded.

How do I handle the "unclear" pairs?

As the product working correctly: route them to human review with both records side by side, feed adjudications back as boundary clauses in the question, and watch the unclear rate fall with each iteration, per the standard calibration loop.