Data quality toolkit

Dataset deduplication

Find duplicate and near-duplicate records within a single dataset. MatchForge compares selected fields, groups similar records, and helps you create a cleaner dataset using findDuplicates().

From messy data to clean records

A quick look at how MatchForge detects and organizes duplicates.

INPUT DATA

Rahul Sharma

rahul@gmail.com

Rahul S.

rahul@gmail.com

Amit Shah

amit@gmail.com

Matching engine

Similarity analysis

OUTPUT
Duplicate group2 records

Rahul Sharma

Rahul S.

1 unique record

Example: 3 input records

1 duplicate group1 unique record

How dataset deduplication works

From selecting comparison fields to identifying duplicates and producing a clean dataset, here is how the process works.

01

Select fields

INPUT

Choose the fields used to identify duplicate records.

02

Compare records

COMPARISON

Calculate similarity scores using your chosen algorithm.

03

Group duplicates

GROUPING

Organize connected records into duplicate groups.

04

Get clean results

OUTPUT

Separate duplicate groups from unique records.

Basic usage

Pass an array of JavaScript objects to findDuplicates(). Each object represents one record.

typescript
import { findDuplicates } from "@matchforge/core";

const records = [
  { name: "Rahul Sharma", email: "rahul@gmail.com" },
  { name: "Rahul S.", email: "rahul@gmail.com" },
  { name: "Pavan Kapadiya", email: "pavan@gmail.com" },
  { name: "Amit Shah", email: "amit@gmail.com" }
];

const result = findDuplicates(records, {
  fields: [
    { key: "name", weight: 0.6 },
    { key: "email", weight: 0.4 }
  ],
  method: "jaro-winkler",
  threshold: 0.85
});

console.log(result.duplicateGroups);
console.log(result.uniqueRecords);

Configuration

Control which fields are compared and how similarity is calculated.

OptionDescription
fieldsFields to compare, with optional weights.
methodSimilarity algorithm used for comparisons.
thresholdMinimum similarity required to identify a match.
normalizationPreprocessing applied before comparison.
algorithmOptionsAdditional options for supported algorithms.
typescript
const result = findDuplicates(records, {
  fields: [
    { key: "name", weight: 0.6 },
    { key: "email", weight: 0.4 }
  ],
  method: "jaro-winkler",
  threshold: 0.85,
  normalization: {
    lowercase: true,
    trim: true,
    normalizeWhitespace: true,
    removeAccents: true
  }
});

Field weights let you assign different importance to fields. Normalization preprocesses values before comparison. See the preprocessing guide for more details.

Understanding results

The result contains duplicate groups, unique records, and summary statistics.

PropertyDescription
duplicateGroupsGroups containing detected duplicate records.
uniqueRecordsRecords that do not belong to a duplicate group.
totalRecordsTotal number of input records.
duplicateGroupsCountNumber of detected duplicate groups.
duplicateRecordsCountRecords contained in duplicate groups.
uniqueRecordsCountNumber of unique records.

Duplicate groups

Each group contains its records and the duplicate pairs connecting them. Pair information includes record indexes, overall similarity scores, and individual field scores. Records can be connected through transitive matches, even when every pair is not directly matched.

Export cleaned dataset

Use exportUniqueCSV() to export a cleaned CSV. Choose which records to retain from each duplicate group.

Keep most complete

Recommended

Retains the record with the most non-empty fields from each duplicate group.

keep: "most-complete"

Keep first record

Retains the first record from each duplicate group.

keep: "first"

Exclude duplicate groups

Removes all records belonging to duplicate groups from the exported dataset.

keep: "exclude"
typescript
import { exportUniqueCSV } from "@matchforge/core";

const csv = exportUniqueCSV(
  {
    ...result,
    records,
    headers: ["name", "email"],
    parseErrors: []
  },
  {
    keep: "most-complete"
  }
);

Unique records are retained in the exported dataset. The selected option determines how records belonging to duplicate groups are handled.

Review before exporting

Duplicate detection identifies potential duplicates based on similarity. Review the detected groups before choosing which records to retain.