CSV dataset deduplication
Identify and remove duplicate records directly from CSV datasets. MatchForge parses your data, compares selected columns, groups similar records, and helps you export a cleaned dataset using findDuplicatesCSV().
How CSV deduplication works
A simple workflow takes your CSV data through duplicate detection and produces a cleaned dataset.
Provide CSV data
Upload or provide a CSV string containing the records you want to clean.
Configure comparison
Select columns, similarity methods, field weights, and a matching threshold.
Detect duplicates
MatchForge compares records and organizes detected duplicates into groups.
Export clean data
Choose a retention strategy and export the resulting dataset as CSV.
Basic usage
Pass CSV text and deduplication options to findDuplicatesCSV(). The function parses the CSV and runs duplicate detection on the resulting records.
import {
findDuplicatesCSV,
exportUniqueCSV,
} from "@matchforge/core";
const csv = `name,email,city
Rahul Sharma,rahul@gmail.com,Ahmedabad
Rahul S.,rahul@gmail.com,Ahmedabad
Pavan Kapadiya,pavan@gmail.com,Surat
Amit Shah,amit@gmail.com,Vadodara`;
const result = findDuplicatesCSV(csv, {
fields: [
{ key: "name", weight: 0.6 },
{ key: "email", weight: 0.4 },
],
method: "jaro-winkler",
threshold: 0.85,
});
console.log(result.duplicateGroups);
console.log(result.uniqueRecords);Unlike object-based deduplication, you do not need to manually parse the CSV or convert its rows into objects. The CSV function handles both operations internally.
Configure deduplication
The same comparison options available in object-based deduplication are supported for CSV datasets.
| Option | Description |
|---|---|
fields | Columns used to identify duplicate records, with optional weights. |
method | Similarity algorithm used for comparing field values. |
threshold | Minimum overall similarity required to identify duplicates. |
normalization | Preprocessing applied to values before comparison. |
algorithmOptions | Additional configuration for supported algorithms. |
const result = findDuplicatesCSV(csv, {
fields: [
{ key: "name", weight: 0.6 },
{ key: "email", weight: 0.4 },
],
method: "jaro-winkler",
threshold: 0.85,
normalization: {
lowercase: true,
trim: true,
normalizeWhitespace: true,
removeAccents: true,
},
});CSV parsing and validation
The function automatically parses the input CSV and includes the extracted headers, records, and parsing errors in its result.
const result = findDuplicatesCSV(csv, {
fields: [
{ key: "name" },
{ key: "email" },
],
});
console.log(result.headers);
console.log(result.records);
console.log(result.parseErrors);Check parseErrors to identify problems encountered while parsing the CSV. Parsing errors are returned as part of the result rather than being silently discarded.
Understanding results
findDuplicatesCSV() returns the duplicate detection results together with the original CSV records and parsing information.
| Property | Description |
|---|---|
headers | Column names extracted from the CSV. |
records | All parsed records represented as objects. |
duplicateGroups | Groups containing detected duplicate records. |
uniqueRecords | Records that do not belong to duplicate groups. |
parseErrors | CSV parsing errors returned by the parser. |
totalRecords | Total number of parsed input records. |
duplicateGroupsCount | Number of detected duplicate groups. |
duplicateRecordsCount | Number of records in duplicate groups. |
uniqueRecordsCount | Number of unique records. |
Export cleaned CSV
Use exportUniqueCSV() to generate a cleaned CSV from the deduplication result. Choose how records in duplicate groups should be handled.
Keep most complete
Retains the record with the highest number of non-empty fields from each duplicate group.
keep: "most-complete"Keep first record
Retains the first record encountered in each duplicate group.
keep: "first"Exclude duplicate groups
Removes all records belonging to duplicate groups from the exported dataset.
keep: "exclude"import { exportUniqueCSV } from "@matchforge/core";
const cleanedCSV = exportUniqueCSV(result, {
keep: "most-complete",
});Records not belonging to duplicate groups are retained regardless of the selected option.