Matching evaluation
Matching evaluation measures how accurately a record-matching system identifies matching and non-matching record pairs. MatchForge provides evaluateMatches() to compare matching predictions against labelled ground-truth pairs and calculate standard classification metrics.
How matching evaluation works
A matching system compares records and generates candidate matches based on similarity scores and matching configuration. Evaluation checks these predictions against known labels to determine how well the system performs.
01
Generate predictions
Use matchRecords() to generate predicted source-target matches.
02
Provide ground truth
Supply labelled pairs indicating actual matches and non-matches.
03
Calculate metrics
Evaluate predictions using precision, recall, F1-score, and accuracy.
Ground truth
Ground truth is a collection of labelled source-target pairs. Each pair identifies two records using their original dataset indexes and specifies whether they represent the same entity.
const groundTruth = [
{
sourceIndex: 0,
targetIndex: 0,
isMatch: true,
},
{
sourceIndex: 0,
targetIndex: 1,
isMatch: false,
},
{
sourceIndex: 1,
targetIndex: 0,
isMatch: false,
},
];| Property | Description |
|---|---|
| sourceIndex | Zero-based index of the record in the source dataset. |
| targetIndex | Zero-based index of the record in the target dataset. |
| isMatch | True if the pair represents the same entity; otherwise, false. |
Ground-truth indexes refer to the original input dataset positions and start at zero. Each source-target pair must be labelled only once. Duplicate labelled pairs are rejected.
Confusion matrix
The confusion matrix summarizes the four possible outcomes when comparing predicted matches against ground truth.
Predicted outcome
Match
Predicted outcome
Non-match
Actually a match
True Positive (TP)
Actually a match
False Negative (FN)
Actually a non-match
False Positive (FP)
Actually a non-match
True Negative (TN)
True Positive (TP)
A genuine match correctly predicted as a match.
Predicted match · Actual match
False Positive (FP)
A non-match incorrectly predicted as a match.
Predicted match · Actual non-match
False Negative (FN)
A genuine match that was not predicted.
Predicted non-match · Actual match
True Negative (TN)
A non-match correctly predicted as a non-match.
Predicted non-match · Actual non-match
A pair is considered predicted as a match when it appears in the matching predictions. A labelled pair that does not appear in the predictions is treated as a predicted non-match.
Evaluation metrics
MatchForge calculates four metrics from the confusion matrix. Each metric is returned as a decimal value between 0 and 1.
Precision
TP / (TP + FP)Of all pairs predicted as matches, how many are actually matches? Higher precision means fewer incorrect matches.
Recall
TP / (TP + FN)Of all actual matches in the labelled data, how many did the matching system identify? Higher recall means fewer missed matches.
F1-score
2 × (Precision × Recall) / (Precision + Recall)The harmonic mean of precision and recall. It provides a balance between finding genuine matches and avoiding incorrect matches.
Accuracy
(TP + TN) / (TP + FP + FN + TN)The proportion of correctly predicted pairs among all labelled pairs evaluated.
When a metric's denominator is zero, MatchForge returns 0 for that metric. Accuracy alone can be misleading when the dataset contains many more non-matches than matches. Consider precision, recall, and F1-score together.
Basic usage
First, generate matching predictions using matchRecords(). Then prepare labelled ground-truth pairs and pass both to evaluateMatches().
import {
matchRecords,
evaluateMatches,
} from "@matchforge/core";
const source = [
{ name: "Rahul Sharma" },
{ name: "Amit Patel" },
];
const target = [
{ fullName: "Rahul Sharma" },
{ fullName: "Amit Shah" },
];
// Generate matching predictions.
const predictions = matchRecords(source, target, {
sourceKey: "name",
targetKey: "fullName",
method: "jaro-winkler",
threshold: 0.8,
topK: 1,
});
// Labelled source-target pairs.
const groundTruth = [
{ sourceIndex: 0, targetIndex: 0, isMatch: true },
{ sourceIndex: 0, targetIndex: 1, isMatch: false },
{ sourceIndex: 1, targetIndex: 0, isMatch: false },
{ sourceIndex: 1, targetIndex: 1, isMatch: false },
];
// Evaluate the predictions.
const evaluation = evaluateMatches({
predictions,
groundTruth,
});
console.log(evaluation.confusionMatrix);
console.log(evaluation.metrics);API reference
evaluateMatches()
Evaluates matching predictions against labelled ground-truth pairs and returns a confusion matrix and evaluation metrics.
evaluateMatches(options: EvaluationOptions)
EvaluationOptions
The evaluator accepts the predictions returned by matchRecords() and the labelled ground-truth pairs.
| Option | Type | Description |
|---|---|---|
| predictions | MatchingResult | Matching predictions returned by matchRecords(). |
| groundTruth | GroundTruthPair[] | Labelled source-target pairs with their actual match status. |
EvaluationResult
The function returns the confusion matrix, calculated metrics, and the number of labelled pairs evaluated.
{
confusionMatrix: {
truePositives: 1,
falsePositives: 1,
falseNegatives: 1,
trueNegatives: 1,
},
metrics: {
precision: 0.5,
recall: 0.5,
f1Score: 0.5,
accuracy: 0.5,
},
totalEvaluatedPairs: 4,
}| Property | Description |
|---|---|
| confusionMatrix | Counts of true positives, false positives, false negatives, and true negatives. |
| metrics | Precision, recall, F1-score, and accuracy. |
| totalEvaluatedPairs | Number of explicitly labelled ground-truth pairs evaluated. |
Important considerations
Unlabelled pairs are ignored
Only explicitly labelled ground-truth pairs contribute to the evaluation. An unlabelled pair is not automatically treated as a non-match or included in the confusion matrix.
Predictions depend on matching configuration
The similarity threshold and topK settings in matchRecords() affect which pairs appear in the predictions. Changing these settings can change the evaluation metrics.
Missing genuine matches count as false negatives
If a ground-truth pair is labelled as a match but does not appear in the matching predictions, it contributes to the false-negative count.
Labelled non-matches are important
Include representative non-matching pairs in the ground truth. Without labelled non-matches, false positives and true negatives cannot be evaluated reliably.
Use representative labelled data
Evaluation results are only as useful as the labelled pairs supplied. Include representative matches and non-matches that reflect the data your matching system will process.
NEXT STEP
Experiment with matching evaluation
Configure matching predictions, provide labelled pairs, and inspect the resulting confusion matrix and evaluation metrics.
Open matching evaluation playground