Imagine merging three data sets: a company registry, a list of government contracts and a sanctions list. One record says “Huawei Technologies Co., Ltd.”, another says “HUAWEI TECHNOLOGIES CO LTD”, a third says “华为技术有限公司”. Are they the same company? What about “Huawei Device Co., Ltd.”?
Deciding when different records refer to the same real-world thing is called entity resolution (also record linkage, deduplication or entity matching). It is the step that most determines whether a knowledge graph is useful or misleading.
Why it is hard
- Variants: abbreviations, punctuation, word order, legal suffixes (Ltd, Limited, GmbH, 有限公司).
- Transliteration: names from Chinese, Arabic or Russian can be romanised in several ways. “Zhang Wei” is one of the most common names in the world.
- Errors: typos, outdated addresses, swapped fields.
- Change over time: companies merge, rename and move. People marry and change names.
- Ambiguity: many different people share a name, and parent companies and subsidiaries share brands.
- Deliberate obfuscation: in investigations, the people you are tracking may be trying not to be matched.
The two kinds of mistakes
- A false merge joins two different entities. In a sanctions or due-diligence context, this can wrongly accuse an innocent person or company.
- A false split fails to join records about the same entity. This hides connections you are looking for.
You cannot drive both to zero. Choose the balance according to the consequences. For screening, you want few missed matches, followed by human review. For published findings, you want very few false merges.
How entity resolution works
1. Standardise
Normalise case, punctuation and whitespace. Expand or strip legal forms. Parse addresses into components. Transliterate where appropriate. Format dates and identifiers consistently. This step alone resolves many easy cases.
2. Use identifiers where they exist
Identifiers beat names. Examples include company registration numbers, tax IDs, the Legal Entity Identifier (LEI) for companies in financial markets, ISINs, ORCID for researchers, and Wikidata IDs. If two records share a reliable identifier, they almost certainly match.
3. Block
Comparing every record with every other is impossible at scale: a million records would mean half a trillion pairs. Blocking compares only records that share something, such as the same postcode, the same first letters of a name or the same birth year.
4. Compare and score
For each candidate pair, compare fields with similarity measures: string distance for names, geographic distance for addresses, exact match for dates of birth. Then combine the comparisons into a score.
- Rules: “same registration number” or “name similarity > 0.9 and same city”. Transparent, but brittle.
- Probabilistic matching, based on the classic Fellegi–Sunter model (1969): each field agreement or disagreement contributes evidence for or against a match. Rare values count for more, so agreement on an unusual surname is stronger evidence than agreement on “Smith”. Open-source tools such as Splink implement this at scale.
- Machine learning and language models: classifiers trained on labelled pairs, or language models that judge whether two descriptions refer to the same entity. They are powerful for messy text, but they need careful evaluation and can be expensive at scale.
5. Cluster
Pairwise matches are linked into clusters, where each cluster is one entity. Watch for chaining: A matches B, B matches C, but A and C are clearly different. Good systems check cluster coherence.
6. Review and record
Send uncertain matches to human review. Record why records were merged and with what confidence. Your graph will need that provenance when someone challenges a link.
Measuring quality
Create a labelled sample of record pairs (match / non-match) and measure:
- Precision: of the pairs you merged, how many were true matches?
- Recall: of the true matches, how many did you find?
Measure on realistic, difficult examples, not just easy ones. Re-measure when data sources change.
Practical advice
- Start with identifiers and high-confidence rules, then add probabilistic matching for the rest.
- Keep the original records. Never overwrite them with the merged version, so you can undo a merge.
- Treat entity resolution as ongoing, not a one-time cleaning step. New data constantly arrives.
- Reuse reference data: open registries, Wikidata, and curated lists such as OpenSanctions already resolve many entities and provide identifiers to match against.
Most of the value of a knowledge graph comes from connecting information that was previously separate. Entity resolution is where those connections are made, or wrongly made, so it deserves more time and care than any visualisation layer.