Dataset deduplication
Find duplicate and near-duplicate records within a single dataset. MatchForge compares selected fields, groups similar records, and helps you create a cleaner dataset using findDuplicates().
From messy data to clean records
A quick look at how MatchForge detects and organizes duplicates.
Rahul Sharma
rahul@gmail.com
Rahul S.
rahul@gmail.com
Amit Shah
amit@gmail.com
Matching engine
Similarity analysis
Rahul Sharma
Rahul S.
Example: 3 input records
How dataset deduplication works
From selecting comparison fields to identifying duplicates and producing a clean dataset, here is how the process works.
Select fields
INPUTChoose the fields used to identify duplicate records.
Compare records
COMPARISONCalculate similarity scores using your chosen algorithm.
Group duplicates
GROUPINGOrganize connected records into duplicate groups.
Get clean results
OUTPUTSeparate duplicate groups from unique records.
Basic usage
Pass an array of JavaScript objects to findDuplicates(). Each object represents one record.
import { findDuplicates } from "@matchforge/core";
const records = [
{ name: "Rahul Sharma", email: "rahul@gmail.com" },
{ name: "Rahul S.", email: "rahul@gmail.com" },
{ name: "Pavan Kapadiya", email: "pavan@gmail.com" },
{ name: "Amit Shah", email: "amit@gmail.com" }
];
const result = findDuplicates(records, {
fields: [
{ key: "name", weight: 0.6 },
{ key: "email", weight: 0.4 }
],
method: "jaro-winkler",
threshold: 0.85
});
console.log(result.duplicateGroups);
console.log(result.uniqueRecords);Configuration
Control which fields are compared and how similarity is calculated.
| Option | Description |
|---|---|
fields | Fields to compare, with optional weights. |
method | Similarity algorithm used for comparisons. |
threshold | Minimum similarity required to identify a match. |
normalization | Preprocessing applied before comparison. |
algorithmOptions | Additional options for supported algorithms. |
const result = findDuplicates(records, {
fields: [
{ key: "name", weight: 0.6 },
{ key: "email", weight: 0.4 }
],
method: "jaro-winkler",
threshold: 0.85,
normalization: {
lowercase: true,
trim: true,
normalizeWhitespace: true,
removeAccents: true
}
});Field weights let you assign different importance to fields. Normalization preprocesses values before comparison. See the preprocessing guide for more details.
Understanding results
The result contains duplicate groups, unique records, and summary statistics.
| Property | Description |
|---|---|
duplicateGroups | Groups containing detected duplicate records. |
uniqueRecords | Records that do not belong to a duplicate group. |
totalRecords | Total number of input records. |
duplicateGroupsCount | Number of detected duplicate groups. |
duplicateRecordsCount | Records contained in duplicate groups. |
uniqueRecordsCount | Number of unique records. |
Duplicate groups
Each group contains its records and the duplicate pairs connecting them. Pair information includes record indexes, overall similarity scores, and individual field scores. Records can be connected through transitive matches, even when every pair is not directly matched.
Export cleaned dataset
Use exportUniqueCSV() to export a cleaned CSV. Choose which records to retain from each duplicate group.
Keep most complete
RecommendedRetains the record with the most non-empty fields from each duplicate group.
keep: "most-complete"Keep first record
Retains the first record from each duplicate group.
keep: "first"Exclude duplicate groups
Removes all records belonging to duplicate groups from the exported dataset.
keep: "exclude"import { exportUniqueCSV } from "@matchforge/core";
const csv = exportUniqueCSV(
{
...result,
records,
headers: ["name", "email"],
parseErrors: []
},
{
keep: "most-complete"
}
);Unique records are retained in the exported dataset. The selected option determines how records belonging to duplicate groups are handled.
Review before exporting
Duplicate detection identifies potential duplicates based on similarity. Review the detected groups before choosing which records to retain.