Lucknow, India

Data Management

Deduplicate records without merging different people

Use matching rules, review queues and reversible decisions to manage duplicate data carefully.

IGS Editorial · 3 min read · Updated 17 November 2026

Deduplicate records without merging different people — editorial artwork
Illustrative data management image

Duplicate records can fragment history and create repeated work. Removing them is not simply a matter of finding similar names. Two people may share a name, one business may have several branches and an old contact detail may legitimately appear on more than one account.

Begin by defining what counts as the same entity for the intended use. The answer may differ between a mailing list, a billing account and a service history. A matching rule should reflect that purpose rather than assuming one universal customer record.

Profile before matching

Inspect the fields available and their reliability. Some identifiers may be unique by design but inconsistently populated. Addresses and phone numbers can change or be shared. Note missingness and source differences before selecting match criteria.

Preserve the original values. Normalised copies can help compare capitalisation, spacing or punctuation, but they should not overwrite source evidence prematurely. Keep a stable record reference so decisions can be traced back.

Use levels of confidence

Separate exact matches on reliable identifiers from possible matches on weaker combinations. A candidate list is a review aid, not an instruction to merge everything in it. Document why each rule suggests a match.

For example, an identical verified account reference may be strong evidence, while a similar name and shared office address may identify colleagues. Borderline cases need an authorised reviewer with enough context to make the decision.

Define surviving values

When records are confirmed duplicates, decide how the combined record is formed. The newest value is not always the most accurate. A verified source may take precedence over a later unverified import.

Keep conflicting values visible where needed and record the decision rule. Related transactions and communication preferences require particular care. A merge should not accidentally broaden contact permissions or lose restrictions attached to one source record.

Make the process recoverable

Test rules on a controlled sample before applying them widely. Record which records were merged, which values survived and who approved the action. Ensure there is an appropriate recovery method before performing irreversible changes.

Review false positives and missed matches separately. Tightening a rule may reduce incorrect merges while leaving more duplicates. The appropriate balance depends on the consequences of each error, not only on a headline match rate.

Prevent recurrence

Examine how duplicates enter the system: repeated imports, unclear search procedures or weak intake validation. Add checks at the point where they help without blocking legitimate records that happen to look similar.

Schedule proportionate monitoring and keep unresolved candidates in a review queue. The goal is trustworthy identity handling, not the largest possible reduction in row count. A smaller, well-supported set of merges is often more valuable than an aggressive cleanup that creates uncertainty about who owns each history.

Review a likely false match

Two records share a surname, phone number and address. They may be duplicates, or they may represent members of the same household. The decision depends on the entity definition and other evidence. A matching score should prompt review rather than create authority to combine their histories automatically.

Ask what consequence an incorrect merge would have. If private service details could be assigned to the wrong person, the review standard should reflect that risk. A mailing-list cleanup and an account-history merge may need very different controls.

Record the evidence used and keep the source records traceable. If a merge is approved, verify related transactions and preferences after the operation. If it is rejected, preserve that decision so the same pair is not repeatedly presented without new evidence. Review the rule that generated the candidate and assess whether similar false matches are likely. Good deduplication learns from rejected matches as well as accepted ones, improving confidence without treating a smaller row count as the only objective.