CSV data tools

CSV dataset deduplication

Identify and remove duplicate records directly from CSV datasets. MatchForge parses your data, compares selected columns, groups similar records, and helps you export a cleaned dataset using findDuplicatesCSV().

How CSV deduplication works

A simple workflow takes your CSV data through duplicate detection and produces a cleaned dataset.

01

Provide CSV data

Upload or provide a CSV string containing the records you want to clean.

02

Configure comparison

Select columns, similarity methods, field weights, and a matching threshold.

03

Detect duplicates

MatchForge compares records and organizes detected duplicates into groups.

04

Export clean data

Choose a retention strategy and export the resulting dataset as CSV.

Basic usage

Pass CSV text and deduplication options to findDuplicatesCSV(). The function parses the CSV and runs duplicate detection on the resulting records.

typescript
import {
  findDuplicatesCSV,
  exportUniqueCSV,
} from "@matchforge/core";

const csv = `name,email,city
Rahul Sharma,rahul@gmail.com,Ahmedabad
Rahul S.,rahul@gmail.com,Ahmedabad
Pavan Kapadiya,pavan@gmail.com,Surat
Amit Shah,amit@gmail.com,Vadodara`;

const result = findDuplicatesCSV(csv, {
  fields: [
    { key: "name", weight: 0.6 },
    { key: "email", weight: 0.4 },
  ],
  method: "jaro-winkler",
  threshold: 0.85,
});

console.log(result.duplicateGroups);
console.log(result.uniqueRecords);

Unlike object-based deduplication, you do not need to manually parse the CSV or convert its rows into objects. The CSV function handles both operations internally.

Configure deduplication

The same comparison options available in object-based deduplication are supported for CSV datasets.

OptionDescription
fieldsColumns used to identify duplicate records, with optional weights.
methodSimilarity algorithm used for comparing field values.
thresholdMinimum overall similarity required to identify duplicates.
normalizationPreprocessing applied to values before comparison.
algorithmOptionsAdditional configuration for supported algorithms.
typescript
const result = findDuplicatesCSV(csv, {
  fields: [
    { key: "name", weight: 0.6 },
    { key: "email", weight: 0.4 },
  ],
  method: "jaro-winkler",
  threshold: 0.85,
  normalization: {
    lowercase: true,
    trim: true,
    normalizeWhitespace: true,
    removeAccents: true,
  },
});

CSV parsing and validation

The function automatically parses the input CSV and includes the extracted headers, records, and parsing errors in its result.

typescript
const result = findDuplicatesCSV(csv, {
  fields: [
    { key: "name" },
    { key: "email" },
  ],
});

console.log(result.headers);
console.log(result.records);
console.log(result.parseErrors);

Check parseErrors to identify problems encountered while parsing the CSV. Parsing errors are returned as part of the result rather than being silently discarded.

Understanding results

findDuplicatesCSV() returns the duplicate detection results together with the original CSV records and parsing information.

PropertyDescription
headersColumn names extracted from the CSV.
recordsAll parsed records represented as objects.
duplicateGroupsGroups containing detected duplicate records.
uniqueRecordsRecords that do not belong to duplicate groups.
parseErrorsCSV parsing errors returned by the parser.
totalRecordsTotal number of parsed input records.
duplicateGroupsCountNumber of detected duplicate groups.
duplicateRecordsCountNumber of records in duplicate groups.
uniqueRecordsCountNumber of unique records.

Export cleaned CSV

Use exportUniqueCSV() to generate a cleaned CSV from the deduplication result. Choose how records in duplicate groups should be handled.

Keep most complete

Retains the record with the highest number of non-empty fields from each duplicate group.

keep: "most-complete"

Keep first record

Retains the first record encountered in each duplicate group.

keep: "first"

Exclude duplicate groups

Removes all records belonging to duplicate groups from the exported dataset.

keep: "exclude"
typescript
import { exportUniqueCSV } from "@matchforge/core";

const cleanedCSV = exportUniqueCSV(result, {
  keep: "most-complete",
});

Records not belonging to duplicate groups are retained regardless of the selected option.