Independent project · Data science / Record linkage
Entity Resolution: Rules vs. Calibrated Classifier
Three methods, one labelled truth set, and an honest answer to whether the learned model was worth it.
As of v0.1.0
Every figure on this page resolves to a committed artifact at v0.1.0 (opens in a new tab).
The problem
Linking records that describe the same entity across systems that never agreed on an identifier is a pattern I work with professionally. In that setting I could measure coverage, how many records got a match, but never accuracy, because there was no ground truth. I wanted a public version where accuracy could be measured.
MusicBrainz and Discogs both catalogue albums independently, and MusicBrainz editors have linked 242,542 of its 2,324,821 album-type release groups to their Discogs masters. Those links are the labelled truth. Both datasets are CC0, so the whole thing is reproducible.
Why it is hard:
- Titles differ by punctuation, edition, and language.
- Artist credits vary, various-artists compilations especially.
- One-to-many links exist in both directions: rare, but real.
- Most MusicBrainz records have no Discogs link at all, and an unlinked record is not evidence of a non-match.
What I built
- Scope: album-type release groups only (482,514 sampled) against all Discogs masters; 241,752 truth pairs in scope. Singles, EPs, and a third source are on the roadmap.
- Blocking: five keys unioned, then a per-record cap of 200 candidates. Pair completeness 96.3% before the cap, 96.2% after; the 240 truth pairs lost to the cap are counted, not hidden.
- One normaliser shared by every method, so no matcher gets private preprocessing.
- Three methods: an exact-match rule, a weighted-score rules baseline (the shape of what I use at work), and one scikit-learn classifier with isotonic calibration on a separate fold.
- Tiered decisions: auto-accept, review, reject, with the review queue split into a floor part and an ambiguity part.
- Warehouse: a dbt bronze/silver/gold warehouse on DuckDB carrying the mapping table and its tests; only the test-fold mapping is committed.
- Number checker in CI: every cited figure in the docs must resolve to a committed artifact key, or the build fails.
CC0 dumps → deterministic sample + truth audit → normalise → block (cap 200) → pair features → fit / calibrate / test folds by MusicBrainz record → exact, rules, and learned tiers → artifacts → dbt bronze/silver/gold → findings and site
Three decisions that shaped it
Choose a population where coverage and accuracy can disagree
The first candidate for side A was Wikidata albums. Profiling showed its Discogs-linked population contained no unlinked records, so coverage would have equalled accuracy by construction and the coverage-versus-accuracy question could never be asked. MusicBrainz, sampled with a 0.50 unlinked share per side, was chosen instead (ADR 0001).
Evaluate at the grain where decisions are made
Every decision and metric is per MusicBrainz record, so cross-validation folds are split by MusicBrainz record, not by Discogs master (ADR 0003). Rules thresholds were searched on fit-fold decisions; the classifier was trained on labelled pairs only and calibrated on its own fold.
Commit only what is reproducible and licensed
First-release years come from MusicBrainz core tables because the convenient derived table ships under a non-commercial licence (ADR 0002). Only the test-fold mapping table is committed; the full mapping is a build output (ADR 0005). The full build’s ~13.8 GiB data peak does not fit a hosted runner, so CI verifies committed artifacts and the full build is owner-run.
What the data said
The rules baseline was the better single method
Found
- On the test fold at auto-accept, rules scored precision 0.9946, recall 0.931, F1 0.962.
- The calibrated classifier scored precision 0.9993, recall 0.850, F1 0.919.
- Review queues: 25,076 records for rules, 6,016 for the classifier.
Why it matters
A learned model is not automatically an upgrade. Here its measurable contribution was calibrated confidence and a review queue about a quarter the size, bought with lower recall, at different tier thresholds.
What I’d tell the decision-maker
- Keep the rules baseline as the reference method.
- Switch to the classifier only if reviewer hours are the binding cost and you accept the recall you give up.
- Do not compare the queues as if recall were equal.
Coverage is not accuracy
Found
The rules method auto-accepted 4,501 MusicBrainz records that have no Discogs link. Those are unverified, and they appear in coverage counts only, never in precision, recall, or F1.
Why it matters
In production matching, this is exactly the population a high match rate is built from. Without labels, a rising coverage number tells you nothing about false attribution.
What I’d tell the decision-maker
Report verified and unverified accepts as separate numbers, and treat the unverified share as review backlog, not as wins.
Where the data proved the plan wrong
- Various-artists credits: the first feature version treated “Various Artists” credits as ordinary artist names and mismatched compilations. Canonicalising them to one token (ADR 0004, feature version 0.2.0) fixed it, and the earlier stages were re-run with sample membership and counts unchanged.
- The candidate cap has a price: the per-record cap of 200 dropped 240 truth pairs. They are counted in the blocking report rather than absorbed into “recall”.
- Calibration was still overconfident: the classifier’s pair-level probabilities were calibrated, but decision-level confidence, the probability of the chosen candidate, remained overconfident because selecting the maximum inflates it. The findings name this rather than smoothing it over.
- The hosted runner couldn’t hold the data: the planned full-build CI workflow was replaced by a verify-only workflow after measuring the data peak.
How it was built
The project was AI-assisted, using Claude Code under a written brief with phase-gated review, which means I approved or amended every phase. The brief and its decision records (ADR 0001 to 0005) are in the repository.
Independent project on CC0 data; MIT licence; no employer code, data, or business rules.
Stack: Python, SQL, scikit-learn, dbt-core, DuckDB, uv, pytest, GitHub Actions, GitHub Pages.