Matching-Evaluation

Matching evaluation

Matching evaluation measures how accurately a record-matching system identifies matching and non-matching record pairs. MatchForge provides evaluateMatches() to compare matching predictions against labelled ground-truth pairs and calculate standard classification metrics.

How matching evaluation works

A matching system compares records and generates candidate matches based on similarity scores and matching configuration. Evaluation checks these predictions against known labels to determine how well the system performs.

01

Generate predictions

Use matchRecords() to generate predicted source-target matches.

02

Provide ground truth

Supply labelled pairs indicating actual matches and non-matches.

03

Calculate metrics

Evaluate predictions using precision, recall, F1-score, and accuracy.

Ground truth

Ground truth is a collection of labelled source-target pairs. Each pair identifies two records using their original dataset indexes and specifies whether they represent the same entity.

typescript
const groundTruth = [
  {
    sourceIndex: 0,
    targetIndex: 0,
    isMatch: true,
  },
  {
    sourceIndex: 0,
    targetIndex: 1,
    isMatch: false,
  },
  {
    sourceIndex: 1,
    targetIndex: 0,
    isMatch: false,
  },
];
PropertyDescription
sourceIndexZero-based index of the record in the source dataset.
targetIndexZero-based index of the record in the target dataset.
isMatchTrue if the pair represents the same entity; otherwise, false.

Ground-truth indexes refer to the original input dataset positions and start at zero. Each source-target pair must be labelled only once. Duplicate labelled pairs are rejected.

Confusion matrix

The confusion matrix summarizes the four possible outcomes when comparing predicted matches against ground truth.

Predicted outcome

Match

Predicted outcome

Non-match

Actually a match

True Positive (TP)

Actually a match

False Negative (FN)

Actually a non-match

False Positive (FP)

Actually a non-match

True Negative (TN)

True Positive (TP)

A genuine match correctly predicted as a match.

Predicted match · Actual match

False Positive (FP)

A non-match incorrectly predicted as a match.

Predicted match · Actual non-match

False Negative (FN)

A genuine match that was not predicted.

Predicted non-match · Actual match

True Negative (TN)

A non-match correctly predicted as a non-match.

Predicted non-match · Actual non-match

A pair is considered predicted as a match when it appears in the matching predictions. A labelled pair that does not appear in the predictions is treated as a predicted non-match.

Evaluation metrics

MatchForge calculates four metrics from the confusion matrix. Each metric is returned as a decimal value between 0 and 1.

Precision

TP / (TP + FP)

Of all pairs predicted as matches, how many are actually matches? Higher precision means fewer incorrect matches.

Recall

TP / (TP + FN)

Of all actual matches in the labelled data, how many did the matching system identify? Higher recall means fewer missed matches.

F1-score

2 × (Precision × Recall) / (Precision + Recall)

The harmonic mean of precision and recall. It provides a balance between finding genuine matches and avoiding incorrect matches.

Accuracy

(TP + TN) / (TP + FP + FN + TN)

The proportion of correctly predicted pairs among all labelled pairs evaluated.

When a metric's denominator is zero, MatchForge returns 0 for that metric. Accuracy alone can be misleading when the dataset contains many more non-matches than matches. Consider precision, recall, and F1-score together.

Basic usage

First, generate matching predictions using matchRecords(). Then prepare labelled ground-truth pairs and pass both to evaluateMatches().

typescript
import {
  matchRecords,
  evaluateMatches,
} from "@matchforge/core";

const source = [
  { name: "Rahul Sharma" },
  { name: "Amit Patel" },
];

const target = [
  { fullName: "Rahul Sharma" },
  { fullName: "Amit Shah" },
];

// Generate matching predictions.
const predictions = matchRecords(source, target, {
  sourceKey: "name",
  targetKey: "fullName",
  method: "jaro-winkler",
  threshold: 0.8,
  topK: 1,
});

// Labelled source-target pairs.
const groundTruth = [
  { sourceIndex: 0, targetIndex: 0, isMatch: true },
  { sourceIndex: 0, targetIndex: 1, isMatch: false },
  { sourceIndex: 1, targetIndex: 0, isMatch: false },
  { sourceIndex: 1, targetIndex: 1, isMatch: false },
];

// Evaluate the predictions.
const evaluation = evaluateMatches({
  predictions,
  groundTruth,
});

console.log(evaluation.confusionMatrix);
console.log(evaluation.metrics);

API reference

evaluateMatches()

Evaluates matching predictions against labelled ground-truth pairs and returns a confusion matrix and evaluation metrics.

evaluateMatches(options: EvaluationOptions)

EvaluationOptions

The evaluator accepts the predictions returned by matchRecords() and the labelled ground-truth pairs.

OptionTypeDescription
predictionsMatchingResultMatching predictions returned by matchRecords().
groundTruthGroundTruthPair[]Labelled source-target pairs with their actual match status.

EvaluationResult

The function returns the confusion matrix, calculated metrics, and the number of labelled pairs evaluated.

typescript
{
  confusionMatrix: {
    truePositives: 1,
    falsePositives: 1,
    falseNegatives: 1,
    trueNegatives: 1,
  },
  metrics: {
    precision: 0.5,
    recall: 0.5,
    f1Score: 0.5,
    accuracy: 0.5,
  },
  totalEvaluatedPairs: 4,
}
PropertyDescription
confusionMatrixCounts of true positives, false positives, false negatives, and true negatives.
metricsPrecision, recall, F1-score, and accuracy.
totalEvaluatedPairsNumber of explicitly labelled ground-truth pairs evaluated.

Important considerations

Unlabelled pairs are ignored

Only explicitly labelled ground-truth pairs contribute to the evaluation. An unlabelled pair is not automatically treated as a non-match or included in the confusion matrix.

Predictions depend on matching configuration

The similarity threshold and topK settings in matchRecords() affect which pairs appear in the predictions. Changing these settings can change the evaluation metrics.

Missing genuine matches count as false negatives

If a ground-truth pair is labelled as a match but does not appear in the matching predictions, it contributes to the false-negative count.

Labelled non-matches are important

Include representative non-matching pairs in the ground truth. Without labelled non-matches, false positives and true negatives cannot be evaluated reliably.

Use representative labelled data

Evaluation results are only as useful as the labelled pairs supplied. Include representative matches and non-matches that reflect the data your matching system will process.

NEXT STEP

Experiment with matching evaluation

Configure matching predictions, provide labelled pairs, and inspect the resulting confusion matrix and evaluation metrics.

Open matching evaluation playground