Blocking for Record-matching & Deduplication

Blocking & Candidate Generation

Blocking reduces the number of record comparisons by grouping records using selected fields. Instead of comparing every source record against every target record, MatchForge generates a smaller set of candidate pairs for further matching.

Why blocking?

Suppose the source dataset contains N records and the target dataset contains M records. Without blocking, exhaustive comparison requires N × M record comparisons. This becomes expensive as datasets grow.

Without blocking

N × M

Every source record may be compared with every target record.

With blocking

Fewer pairs

Only pairs sharing at least one blocking key are retained.

Blocking is not the final matching decision. It generates candidate pairs that can subsequently be evaluated using similarity algorithms and matching thresholds.

How candidate generation works

MatchForge generates a blocking key for each record using the selected field and method. Records with matching keys are included as candidate pairs.

SOURCE DATASET

Pavan Kapadiya

Delhi

Rahul Patel

Ahmedabad

TARGET DATASET

Pavan Kapadia

Delhi

Rahul Patel

Ahmedabad

Blocking key: cityCandidate pairs
(0, 0)Delhi ↔ Delhi
(1, 1)Ahmedabad ↔ Ahmedabad

Supported blocking methods

Each rule selects a source field and a target field, then applies one of the following blocking methods.

exact

Exact

Records are grouped when their normalized field values are identical.

Example: "Delhi" and "delhi" with lowercase normalization

prefix

Prefix

Records are grouped when the first configured number of characters match.

Example: "Pavan" and "Pavani" with length: 3

soundex

Soundex

Records are grouped when their Soundex phonetic codes match.

Example: "Smith" and "Smyth"

Basic usage

Import generateCandidates() and provide source records, target records, and at least one blocking rule.

typescript
import { generateCandidates } from "@matchforge/core";

const source = [
  { name: "Pavan Kapadiya", city: "Delhi" },
  { name: "Rahul Patel", city: "Ahmedabad" },
  { name: "Amit Shah", city: "Mumbai" },
];

const target = [
  { fullName: "Pavan Kapadia", location: "Delhi" },
  { fullName: "Rahul Patel", location: "Ahmedabad" },
  { fullName: "Neha Shah", location: "Mumbai" },
];

const result = generateCandidates(source, target, {
  rules: [
    {
      sourceKey: "city",
      targetKey: "location",
      method: "exact",
    },
  ],
});

console.log(result.pairs);
console.log(result.stats);

Every returned pair contains the original source and target array indexes. Use these indexes to retrieve the corresponding records from the input datasets.

Multiple blocking rules

You can configure multiple rules. MatchForge combines their candidate pairs using OR semantics: a pair is included if it matches at least one rule. Pairs generated by multiple rules are deduplicated.

typescript
const result = generateCandidates(source, target, {
  rules: [
    {
      sourceKey: "name",
      targetKey: "fullName",
      method: "prefix",
      length: 3,
    },
    {
      sourceKey: "city",
      targetKey: "location",
      method: "exact",
    },
  ],
  normalization: {
    lowercase: true,
    trim: true,
    normalizeWhitespace: true,
    removeAccents: true,
  },
});

Prefix blocking defaults to a length of 3 when length is not provided. The configured length must be a positive integer. Normalization is applied before generating blocking keys.

Configuration

rulesBlockingRule[]

Required. At least one rule.

sourceKeystring

Source record field used for the blocking key.

targetKeystring

Target record field used for the blocking key.

method"exact" | "prefix" | "soundex"

Blocking method.

lengthnumber

Optional prefix length. Defaults to 3.

normalizationNormalizeOptions

Optional normalization applied before key generation.

Available normalization options include lowercase, trim, removePunctuation, removeAccents, and normalizeWhitespace.

Candidate generation statistics

The result contains the generated pairs and statistics describing the reduction in the comparison search space.

typescript
console.log(result.stats);

// {
//   totalSourceRecords: 3,
//   totalTargetRecords: 3,
//   totalPossiblePairs: 9,
//   candidatePairs: 3,
//   reductionPercentage: 66.6666666667
// }

totalSourceRecords — Number of records in the source dataset.

totalTargetRecords — Number of records in the target dataset.

totalPossiblePairs — Total possible comparisons: source count × target count.

candidatePairs — Number of unique candidate pairs generated.

reductionPercentage — Percentage of possible comparisons eliminated by blocking.

Use blocking with record matching

Pass blocking configuration directly to matchRecords(). MatchForge generates candidate pairs first and performs matching only for those pairs.

typescript
import { matchRecords } from "@matchforge/core";

const result = matchRecords(source, target, {
  fields: [
    { sourceKey: "name", targetKey: "fullName" },
    { sourceKey: "city", targetKey: "location" },
  ],
  method: "jaro-winkler",
  threshold: 0.8,
  blocking: {
    rules: [
      {
        sourceKey: "city",
        targetKey: "location",
        method: "exact",
      },
    ],
  },
});

Use blocking with dataset deduplication

For deduplication, use the blocking option in findDuplicates(). Each duplicate blocking rule selects a single field from the dataset.

typescript
import { findDuplicates } from "@matchforge/core";

const result = findDuplicates(records, {
  fields: [
    { key: "name", weight: 0.7 },
    { key: "city", weight: 0.3 },
  ],
  method: "jaro-winkler",
  threshold: 0.8,
  blocking: {
    rules: [
      { key: "city", method: "exact" },
      { key: "name", method: "prefix", length: 3 },
    ],
  },
});

Important considerations

Blocking generates candidate pairs; it does not determine whether two records are a match.

Multiple rules use OR semantics. A pair matching any rule is included.

Duplicate candidate pairs are returned only once.

Missing, null, empty, and normalized-empty blocking values are skipped.

Present blocking field values must be strings. A non-string value causes an error.

A restrictive blocking strategy may exclude genuine matches. Evaluate candidate recall before relying on it.

Candidate indexes refer to the original source and target arrays.

Explore blocking interactively

Configure blocking methods, inspect candidate pairs, and see how blocking changes the number of comparisons.

Open blocking playground