A Practical Guide to Entity Resolution

Entity resolution starts when a system has more records than real things.

A customer appears twice because one source has "Acme Inc." and another has "ACME Incorporated." A supplier changes address. A patient has two spellings of the same name. A product feed carries the same item with different titles, units, and manufacturer codes. A company database has branches, subsidiaries, former names, and legal entities mixed together.

The job is to decide which records refer to the same thing and to keep a usable representation of that thing without throwing away the evidence that produced it. That second part is where many projects become fragile. A merged row may be convenient for a dashboard, but the system still needs to know which source records were linked, which fields won, which model or rule made the decision, and which cases were uncertain.

This guide treats entity resolution as a general operating pattern. It can apply to customers, products, suppliers, companies, locations, patients, assets, files, tickets, or any other domain where duplicate or overlapping records accumulate.

The basic vocabulary

An entity is the real thing you care about. A source record is one system's version of that thing. A candidate pair is a possible match between two records. A cluster is a set of records that currently appear to describe the same entity. A canonical entity is the version of that entity that the rest of the business reads.

Those terms should stay separate. If the canonical table is treated as the only truth, it becomes hard to explain mistakes or rerun the system after rules change. If the evidence layer is preserved, the canonical table can be rebuilt when better matching logic, new data, or a human review changes the answer.

The reusable pipeline

Every serious entity resolution system has roughly the same shape, even when the domain changes.

The table is intentionally plain because this is the part that transfers across industries. The exact fields change. The sequence usually does not.

Start by defining the entity

Before choosing a model, decide what counts as one thing.

For people, the question might be whether two rows refer to the same individual, the same household, or the same customer account. For businesses, the question might be whether branches roll into a parent company, whether legal entities stay separate, or whether former names should merge. For products, the question might be whether pack size, region, or color variant creates a separate entity.

This is not a semantic detail. It changes the labels, the blockers, the evaluation set, the review policy, and the final reference table. A system that merges too broadly may destroy useful distinctions. A system that merges too narrowly may leave every downstream analysis full of duplicates.

Write the boundary down in ordinary language before building anything.

Keep source records immutable

The raw input should be append-only. If a CRM row, product feed, claim record, or supplier file arrives, store it with its source system, source primary key, raw payload, normalized payload, and ingest timestamp.

Do not overwrite the source record to make it look cleaner. Cleaned values belong beside the raw value, with the rule or parser version that produced them. That gives you a way to compare today's normalization with next month's normalization and to explain why a match changed.

In practice, this means the first table is not a golden record table. It is a source record table.

Normalize before matching

Normalization removes surface noise before the expensive part starts.

Names need case handling, punctuation cleanup, token ordering, nicknames, legal suffixes, transliteration, and sometimes language-specific handling. Addresses need postal normalization and geocoding when available. Products need unit conversion, brand cleanup, model-number parsing, pack-size handling, and category hints. Dates, phone numbers, emails, tax IDs, and URLs need their own validators.

This work is not glamorous, but it saves the matcher from wasting effort on differences that should never have reached the model. It also gives human reviewers a cleaner view of the evidence.

Keep failed normalization visible. A parser failure is often a signal that a source system changed its schema or that a new kind of record entered the pipeline.

Blocking is the part that makes the system possible

Without blocking, a million records create roughly five hundred billion possible pairs. No LLM, cross-encoder, or probabilistic matcher should be asked to work at that scale directly.

Blocking creates candidate pairs that are plausible enough to score. Good systems usually use several blockers and union the results.

The goal of blocking is high recall with manageable candidate volume. It is acceptable for blockers to generate false positives because the matcher can reject them later. It is more dangerous for blockers to miss true matches, because missed candidates never reach the scoring stage.

Track which blocker produced each candidate. That provenance is useful when a source starts generating bad candidates or when one blocker is carrying most of the recall.

Score cheaply before using expensive reasoning

After blocking, score candidate pairs with the cheapest reliable method first. Depending on the data, that might be a probabilistic linkage model, a gradient-boosted model, a small transformer, a cross-encoder, or a rule-based scorer with calibrated weights.

The scorer should produce more than a label. It should produce a score or confidence, the features that mattered, and enough metadata to reproduce the decision later. If two candidates are close, or if a pair sits near the threshold, route it to a stronger method or a reviewer.

This is where LLMs can help, but they should enter deliberately. The recent literature points in the same direction: LLMs are strongest when used for ambiguous cases, candidate selection, explanations, or cluster-level decisions. They are usually a poor first-pass blocker on large data.

Where LLMs fit

There are several useful LLM roles in entity resolution.

The pattern to avoid is sending every possible pair to a large model. It is expensive, difficult to audit, and often worse than a hybrid pipeline that spends model calls only where uncertainty is high.

LLM outputs should go into the evidence layer. Store the model name, prompt version, input fields, answer, confidence, explanation, latency, and token cost. That makes the LLM one source of evidence, not an invisible authority inside the golden record.

Resolve clusters, not just pairs

Pairwise scores are intermediate evidence. The final question is usually cluster membership: which records belong to the same entity?

This step depends on the problem.

Global consistency matters. If A matches B and B matches C, the system still needs to decide what to do when A and C look weak. Pairwise prompts alone do not solve that. The clustering layer should make those conflicts visible and version the outcome.

Publish a canonical view

The canonical table is the serving layer. It is what analytics, search, CRM workflows, support tools, procurement systems, or agent workflows read.

The canonical row should be derived from the cluster and its evidence. It should not erase the evidence.

For each attribute, choose a winning value with explicit survivorship logic. A trusted source may beat a newer source for tax ID. Recency may matter more for address. Completeness may matter more for product descriptions. A human override may win for legal names.

Attribute-level survivorship is usually better than row-level merging because different fields have different sources of authority.

This is the difference between a reference table and a pile of merged rows. The reference table can explain itself.

A minimum data model

You do not need a large platform to start, but the storage model should separate evidence from publication.

This model is deliberately boring. It gives you the thing that most entity resolution projects eventually need: a way to explain why the current entity exists, which records support it, and what would change if matching or survivorship rules changed.

How to evaluate it

Pairwise precision, recall, and F1 are still useful, but they are not enough.

Measure blocker recall so you know whether true matches are reaching the scorer. Measure candidate volume so you know whether the scorer is being flooded. Measure pairwise precision and recall for the scorer. Measure cluster quality when the output is deduplication. Measure review load, abstention rate, latency, and cost.

If LLMs are involved, also measure prompt sensitivity. A matching system that changes materially because the prompt was reworded is not stable enough to operate without guardrails. Token cost and runtime should be reported next to quality, because a small F1 gain may not justify a large operational cost.

The hardest errors are usually false positives that merge distinct entities. They pollute the canonical view and can be difficult to unwind. For many domains, the first production target should be high precision with a review queue for ambiguous cases, then gradual recall improvement.

A practical starting plan

That sequence works whether the records are customers, suppliers, products, patients, assets, companies, or locations. The domain changes the fields and rules. The architecture stays stable.

Common failure modes

Most of these failures come from compressing the whole problem into "are these two records the same?" That question is useful, but it is only one piece of the system.

Sources