The problem in one paragraph

You upload a PDF, a thesis, or a bibliography. You want a machine to tell you, for every entry, one of three things: this reference is real and its metadata is correct; this reference is real but something is wrong (a mangled author, a wrong year, a fabricated DOI); or this reference does not appear to exist at all. That sounds like a job for a large language model (LLM). It is almost the opposite. The reliable way to answer it is to take each citation string and look it up in an authoritative database of real publications, deciding — under uncertainty, with noisy input — whether a returned record is truly the same work. That decision problem has a name in computer science: entity resolution (also called record linkage or data matching). Everything in this article orbits that one idea.

Part 1 — The landscape: what really exists in 2026

Before the theory, a survey of the working products. They fall into four camps: public matching infrastructure (Crossref, OpenAlex), researcher-facing checkers (scite, Paperpal, Trinka), reference-manager metadata lookup (Zotero, Mendeley, EndNote), and publisher-side integrity screening (Cactus Paperpal Preflight, Clear Skies, publisher in-house tools).

Public matching infrastructure

Crossref Simple Text Query (STQ) is the granddaddy. You paste a reference list — up to 1,000 references per submission, ideally one per line in a consistent style — and it returns one best-match DOI per reference, or, if you tick “List All Possible DOIs,” several candidates for ambiguous cases. It requires no login or fee and is a public utility layered on the Crossref metadata index. Programmatically the same matching is reachable through the REST API at api.crossref.org/works?query.bibliographic=<string>, and results are sorted by a relevance score. Crossref explicitly warns that matching loosely formatted text is probabilistic: unmatched references are retained and periodically re-checked, because a DOI may not have existed yet when you first queried.

OpenAlex, run by the non-profit OurResearch (the team behind Unpaywall), is the largest fully open scholarly graph. As of its 2026 roadmap OpenAlex indexes 477 million works — described on the OpenAlex blog as “the largest connected repository of scholarship ever published” (roughly 271.3M core works plus a ~192M-record DataCite/repository expansion pack). Its REST API needs no key, and it aggregates Crossref, PubMed, ORCID and ROR daily. Semantic Scholar (Allen Institute for AI) covers “nearly 200 million papers covering all disciplines,” with some 2026 sources citing 214M+, and is especially strong in CS/AI. PubMed (E-utilities), DataCite (datasets/DOIs), and arXiv round out the free identifier-resolution services. DOI resolution itself — turning 10.xxxx/yyyy into a landing page via doi.org — is a separate, deterministic step and needs no matching at all.

Researcher-facing checkers

scite.ai Reference Check takes a PDF and produces a report: it uses machine learning to identify each reference and match citation statements (the sentence around an in-text citation) to their targets, then flags retractions and editorial concerns and shows how each cited work has itself been cited (supporting / contrasting / mentioning — scite’s “Smart Citations”). It only persists the report; the uploaded PDF sits in a restricted S3 bucket and is deleted after processing. It explicitly misses references without DOIs (news, policy documents). There is an upload API for programmatic runs.

Paperpal Reference Checker (by Cactus Communications) parses the document, structures each reference (authors, title, journal, year, volume, pages, DOI) and matches it to its in-text citation, then validates each entry against a database of “250M+ verified research articles” — per Paperpal’s own Reference Checker page, “to fill in missing details, correct inconsistencies, confirm validity, detect retractions.” It flags fake, broken, retracted, AI-generated, missing or unmatched references. It sits inside Paperpal Prime, priced at US$25/month (or $139/year, ~$11.58/month billed annually). Trinka takes a different angle: it validates references against Crossref and flags research-integrity risks — retracted papers, predatory journals, duplicate references, excessive self-citation — but does not itself fix them. Writefull Cite suggests citations from your text but is currently restricted to publishers and institutions. Tools such as ReciteWorks check that in-text (Author, Year) pairs match the reference list, a purely internal-consistency check that needs no external database.

Reference managers

Zotero, Mendeley and EndNote resolve metadata by identifier lookup: paste a DOI/PMID/ISBN/arXiv ID and they fetch a structured record from Crossref, PubMed, or a translation server. Zotero’s Translation Server (used by the open-source clibib tool) with Crossref as fallback is a good example of the “retrieve, don’t generate” pattern applied to bibliographies.

Publisher-side integrity screening

On the publisher side the driver is paper-mill fraud and a retraction surge. According to the arXiv study Mapping Academic Integrity: Global Retraction Trends (2025), “the year 2023 set a historical record, with over 14,000 retraction notices issued. This trend persisted through 2024 and 2025, which saw over 10,000 and 9,000 retractions, respectively.” Cactus’s Paperpal Preflight for Editorial Desk (PPE) screens submissions across ~25 elements including a citation-analysis module (excessive self-citation, citation clustering, unrelated or retracted references) and author/affiliation verification. Cactus partnered with the STM Integrity Hub, and Frontiers integrated PPE (alongside Clear Skies’ Papermill Alarm) into its AIRA screening system. Publishers Elsevier, Springer Nature and Wiley run in-house integrity checks too; STM’s Hylke Koers frames the role as surfacing “signals” for human editors to act on. Vendors deliberately keep the exact detection logic private so paper mills cannot game it — which means for these publisher-internal systems, the precise matching algorithms are not publicly documented, and we say so rather than speculate.

Part 2 — The algorithms and methods (the core of it)

A verification system is a pipeline. Each stage has decades of literature behind it. We’ll walk them in order, defining every term the first time it appears.

2.1 Parsing a messy reference string into fields

A raw reference — [5] E. Lundberg, "Humanism on Gallifrey," Science Fiction Studies, vol. 40, no. 2, p. 382, 2013. — is just a string. Reference parsing (or citation segmentation) is the task of labelling each span with a field: author, title, venue, volume, year, pages. The dominant classical method is a Conditional Random Field (CRF) — a sequence-labelling model that predicts a label for each token while accounting for the labels of neighbouring tokens (so it “knows” a year usually follows a volume, etc.).

Modern transformer-based parsers (and Neural ParsCit, which added a bidirectional LSTM) push accuracy higher, and in 2026 LLMs are increasingly used to extract fields directly — the VerusCite tool, for example, uses a served LLM to chunk a reference section and emit structured citations, reporting that Gemini 3.1 Flash Lite “correctly extracts over 99.8% of citations at under $0.03 per paper.” This is exactly the kind of task where LLMs shine: turning messy human text into structured fields. It is not where they should be asked to supply the DOI from memory.

2.2 Candidate generation: from millions to a shortlist

You cannot compare a query against every one of ~477M records. You first need blocking (also called candidate generation): a cheap filter that returns a few dozen plausible candidates, trading a tiny amount of recall for an enormous speed-up.

Techniques, cheapest first:

• Inverted index + BM25. An inverted index maps each term to the documents containing it (like a book index). BM25 is the classic ranking function that scores a document by how many query terms it contains, weighting rare terms more (via inverse document frequency) and dampening very frequent terms. It is the strong, cheap baseline — Crossref’s search-based matcher is essentially this.
• n-gram blocking. Break titles into overlapping character chunks (“citation” → “cita”, “itat”, “tati”…) so typos and truncations still share chunks.
• Sorted neighbourhood. Sort all records by a key (e.g. normalised title) and slide a window; near-duplicates end up adjacent.
• MinHash + Locality-Sensitive Hashing (LSH). MinHash produces a compact signature of a set (e.g. the set of a title’s word-shingles) such that the probability two signatures collide equals their Jaccard similarity (the size of the intersection over the union of two sets). LSH then hashes similar signatures into the same bucket, so you only compare items that land together. This turns an O(n²) all-pairs comparison into something near-linear. A common trick (documented even in entity-resolution patents) is to run MinHash several times and filter each bucket with a Jaccard threshold around 0.2 to kill false positives before expensive scoring.

2.3 String similarity and record-linkage theory

Once you have candidates, you score field-by-field. The workhorse similarity measures:

• Levenshtein (edit) distance — the minimum number of single-character insertions, deletions or substitutions to turn one string into another. Good for typos and OCR errors in titles.
• Jaro–Winkler — a similarity tuned for short strings that rewards matching prefixes; the standard choice for person names, where the beginning of a surname matters most.
• Jaccard — set overlap over token sets; robust to word reordering (“Smith and Jones” vs “Jones, Smith”).
• Cosine over TF-IDF character n-grams — represent each string as a vector of weighted character-chunk counts and take the cosine of the angle; strong for fuzzy, mixed-script, or partial matches.

How do you combine per-field scores into a single decision? The foundational answer is the Fellegi–Sunter model (1969), still the backbone of modern matchers. For each field comparison you estimate two probabilities: m = probability the fields agree given the pair is a true match, and u = probability they agree given a non-match. The match weight for a field is essentially log(m/u). The intuition is elegant: agreement is strong evidence only when it is rare among non-matches — two records agreeing on the surname “Xander” is far more informative than agreeing on “John.” Sum the weights across fields and apply two thresholds: above the upper one, “match”; below the lower one, “non-match”; in between, “possible match — send to review.” The open-source library Splink implements exactly this.

2.4 Embeddings and dense retrieval for scholarly text

A dense embedding maps a piece of text to a vector such that semantically similar texts land near each other; retrieval is then a nearest-neighbour search (via FAISS/HNSW indexes). For scholarly documents the specialised models are SciBERT (BERT pre-trained on scientific text), SPECTER and SPECTER2 (trained with a citation-aware objective — papers that cite each other are pulled together), and SciNCL (which samples training pairs from the citation-graph neighbourhood). General-purpose retrieval models — E5, GTE, BGE-M3, and the Qwen3-Embedding family — are also widely used.

2.5 Hybrid retrieval and reranking

Because lexical (BM25) and dense retrieval fail in different ways, the production default is hybrid: run both and fuse the ranked lists. The most robust fusion is Reciprocal Rank Fusion (RRF), which ignores raw scores entirely and combines items by 1/(k + rank) (typically k=60) — sidestepping the problem that BM25 scores and cosine similarities live on incompatible scales.

After fusion comes reranking: a more expensive model re-orders the top ~50–100 candidates. A cross-encoder feeds the query and a candidate together into a transformer and outputs a single relevance score — the most accurate option, but O(n) forward passes at query time, so it is slow (one tutorial measured a cross-encoder’s p99.9 latency at over 21 seconds at 40 QPS). monoT5 and BGE-reranker are common cross-encoders. ColBERT uses “late interaction” (the MaxSim operator over per-token vectors): it precomputes document token embeddings, so it delivers near-cross-encoder quality at ~23ms p50 — the practical middle ground.

2.6 Confidence calibration and abstention

This is the stage most amateur systems skip, and it is the difference between a trustworthy checker and a dangerous one. For citation verification the worst possible output is a confidently-wrong match — telling a user “yes, this resolves to DOI X” when it doesn’t. So the system must sometimes say “no confident match.” That is selective prediction (classification with a reject option). Chow (1970) established the optimal rule decades ago: abstain when the top confidence falls below a cost-derived threshold.

Two practical signals drive the decision: the absolute score of the top candidate, and the margin between the top-1 and top-2 candidates. A high top score with a razor-thin margin means “two papers look almost identical” — often a reprint, a preprint-vs-published pair, or two editions — and is a classic abstain case. To make scores mean what they claim, apply calibration: Platt scaling (fit a logistic function to raw scores) or isotonic regression (a non-parametric monotonic fit) so that “0.9 confidence” is right about 90% of the time. Conformal prediction goes further, producing prediction sets with a distribution-free, finite-sample coverage guarantee under exchangeability: abstain whenever the set is not a singleton. Cost-sensitive and conformal abstention frameworks (e.g. the 2025 CAP method pairing reinforcement learning with conformal thresholds) formalise the trade-off between coverage and error.

2.7 The metrics that matter

Evaluate with the standard information-retrieval toolkit, and add the one that is specific to this task:

• Precision — of the matches you returned, what fraction are correct? (For citation checking, keep this very high.)
• Recall — of the citations that do have a correct record, what fraction did you find?
• F1 — harmonic mean of precision and recall.
• Top-1 accuracy and Top-k recall — is the right record ranked first, or anywhere in the top k?
• MRR (Mean Reciprocal Rank) — averages 1/rank of the first correct hit.
• nDCG — rewards putting the most relevant items highest, discounted by position.
• Confidently-wrong rate — the fraction of high-confidence outputs that are actually wrong. This is the danger metric for a verifier and should be reported explicitly and driven near zero, even at the cost of recall.

Part 3 — The hallucination problem (why any of this is urgent)

The reason citation checking went from a niche librarian utility to a mainstream need is that LLMs fabricate references at alarming, measurable rates. The evidence is now substantial and consistent.

Why LLMs hallucinate citations

A base LLM is a next-token predictor: it generates the most statistically plausible continuation of text, one token at a time. A reference is, to the model, just a highly structured plausible-looking string — an author name, a title-shaped phrase, a journal-shaped phrase, and a DOI-shaped string of digits. The model has no grounding to a database and no notion that the string must resolve to a real object; it happily emits a DOI-shaped string that has never been registered. Notably, the title field is usually the most accurate and the DOI the least accurate — because a DOI is arbitrary and cannot be inferred from meaning, exactly the kind of identifier a model cannot reconstruct from memory.

RAG helps — but is not a cure

Retrieval-Augmented Generation (RAG) gives the model real retrieved documents to condition on, which is why tools like Elicit and SciSpace hallucinate far less than raw ChatGPT. But RAG is only a partial fix: if the retriever returns the wrong document, or the model still paraphrases an identifier from memory, or the retrieval simply misses, fabrication returns. A 2026 study of deep-research agents (“Cited but Not Verified”) found citation-hallucination rates of 11% to 57% across commercially deployed models even with web search, and noted that RAG “does not validate whether cited sources are accessible, topically relevant, or factually consistent with the claims they support.”

2025–2026 verifier research

A wave of purpose-built citation verifiers now grounds every reference against real sources:

• CiteCheck (arXiv, 2026) — retrieval-grounded, zero-shot; reports 88.7 macro-F1 and 88.9% accuracy on a physics benchmark of controlled corruptions, beating GPT/Claude/Gemini baselines by ~5.8 F1 points.
• CiteTracer (“Source or It Didn’t Happen,” multi-agent, 2026) — reaches 97.1% accuracy on a synthetic benchmark (class F1: REAL 97.0, POTENTIAL 95.8, HALLUCINATED 98.5) and detects 97.1% of fabrications on a real-world set drawn from ICLR 2026 and desk-rejected submissions.
• VerusCite — deliberately precision-first: a target false-positive rate below 1% with recall ~70–80%, and citation extraction “over 99.8%.” (These are the tool’s own V1 benchmark figures on a non-peer-reviewed write-up.)
• BibTeX-agent study (2026) — using the open-source clibib for deterministic BibTeX retrieval from Zotero’s translation server with Crossref fallback, a two-stage “search then revise” integration lifted accuracy +8.0pp to 91.5%, raised fully-correct entries from 50.9% to 78.3%, and kept regression at just 0.8% — demonstrating that architecture (separating retrieval from revision) matters independently of model quality.

These systems are converging on the same shape as Figure G: retrieve first, then have a structured verifier (increasingly an LLM used as a judge over retrieved evidence, not as a memory) decide EXACT / MINOR-error / MAJOR-hallucination. The problem is now operational at venue scale — NeurIPS 2025 and ICLR 2026 program chairs both assembled desk-reject queues for submissions with fabricated bibliographies, and ACM CCS 2026 published a transparency report enumerating AI-fabricated citations it flagged.

Part 4 — What 2026 AI genuinely changes

It is easy to over- or under-claim the role of AI here. The honest position, grounded in the evidence above:

Where LLMs genuinely help

• Parsing messy strings into fields — LLMs and modern transformer parsers turn a footnote-style, multi-language, OCR-mangled citation into clean fields better than brittle regex ever could.
• Query understanding — normalising “et al.”, expanding abbreviated venues, inferring that “PNAS” = “Proceedings of the National Academy of Sciences.”
• Reasoning-based reranking and match explanation — an LLM-as-judge can look at a candidate record next to the query and explain “this is the preprint version; the published DOI differs,” which is genuinely useful for the abstain decision.

Where LLMs must not be trusted

The enabling 2026 capabilities are long-context models (ingest a whole thesis bibliography at once), function calling / tool use (the model calls a Crossref/OpenAlex API instead of guessing), and agentic verification loops (search → compare → decide → optionally re-search). Used this way, the LLM is an orchestrator and explainer wrapped around a deterministic retrieval core.

The economics

Lexical retrieval (BM25 over an inverted index) is nearly free per query, needs no GPU, and — as Crossref’s own results show — is already a very strong matcher. Dense embeddings and cross-encoder rerankers add cost (GPU, latency, indexing) and add accuracy only sometimes (recall the +17pp reranking win on one benchmark but “marginal” on another, and BM25 beating SPECTER outside CS/BioMed). So the correct engineering stance is: BM25 + structured lookup is the baseline; add embeddings and a reranker only where a benchmark on your own data shows they pay for themselves.

A state-of-the-art 2026 architecture, end to end

Normalise and try exact-identifier lookup first → parse the string into fields with GROBID/AnyStyle or an LLM → generate candidates through multiple channels (exact keys, BM25, structured author+year, structured venue+year, optionally a dense channel) → fuse with RRF → score fields with Jaro-Winkler/edit-distance/Jaccard and optionally a cross-encoder or LLM-judge reranker → calibrate scores and apply top-1 threshold + top-1/top-2 margin + conformal abstention → return real / wrong-metadata / not-found with a confidence and an explanation. That is Figure A, filled in.

Part 5 — A practical blueprint you can build

Ordered by return on effort. You can ship value after step 1 and improve monotonically.

Step 1 — Exact identifier lookup with normalisation

Extract and normalise any DOI, PMID, ISBN or arXiv ID present. Lower-case DOIs, strip https://doi.org/ prefixes and trailing punctuation, canonicalise arXiv IDs (old vs new scheme). Resolve directly against Crossref/PubMed/DataCite/arXiv. This alone verifies a large share of well-formed references deterministically and cheaply.

# Pseudocode: exact-key channel
doi = normalize_doi(extract_doi(ref_string))
if doi:
    rec = crossref.get(doi)            # deterministic resolution
    if rec: return Verdict(REAL, rec, confidence=1.0)
    else:   return Verdict(NOT_FOUND, reason="DOI does not resolve")

Step 2 — Structured field extraction

For references without a usable identifier, parse into fields. Start with GROBID (F1 ~0.89 out of the box, ~0.92 retrained) or AnyStyle; for hard, mixed-format inputs an LLM extraction step is a strong option (VerusCite reports >99.8% extraction). Keep the raw string too — you will need it for the full-text BM25 channel.

Step 3 — Multi-channel candidate generation

Query the index through several doors and merge (Figure C):

• Exact keys (from step 1).
• BM25 full-text on the whole reference string / title (this is essentially what Crossref’s query.bibliographic does).
• Structured author+year — essential. When a citation is just “Smith et al., 2019,” there is no title to search; you must query author.family=Smith & publication_year=2019 and disambiguate on venue. A text-only or embeddings-only system has no answer here at all, because there is almost nothing to embed.
• Structured venue+year — journal + volume + first page is often enough to pin a specific article even with a garbled title.

Step 4 — Fuzzy field-level scoring

Score each candidate field-by-field: Jaro-Winkler on author surnames, edit distance / TF-IDF-cosine on titles, exact/near on year, page, volume. Combine with a Fellegi-Sunter-style weighted sum (or a learned classifier like Ditto if you have labels). The output is a single match score per candidate plus a per-field agreement breakdown you can later show the user.

Step 5 — Hybrid fusion and optional reranking

Fuse the channel rankings with RRF (k≈60). If a benchmark on your data justifies it, add a dense channel and/or rerank the top ~50 with a cross-encoder (BGE-reranker, monoT5) or ColBERT for lower latency. Measure the lift; drop the stage if it doesn’t pay.

Step 6 — Confidence calibration and abstention

Calibrate scores with Platt scaling or isotonic regression on a held-out set. Set two thresholds: an absolute minimum top-1 score, and a minimum top-1 minus top-2 margin. Below the absolute threshold → not found. Above it but below the margin → ambiguous / abstain (often duplicates). For a guaranteed error rate, layer conformal prediction and abstain on non-singleton sets. Optimise for a near-zero confidently-wrong rate, accepting lower recall.

Step 7 — Evaluation harness

Build a golden set of a few hundred references with known correct records, deliberately including hard cases: bare author+year, preprint/published duplicates, non-Latin names, retracted works, and a set of known fabrications so you can measure the confidently-wrong rate directly. Track precision, recall, F1, Top-1 accuracy, Top-k recall, MRR, nDCG — and the confidently-wrong rate as your primary safety KPI.

Summary blueprint checklist

Further reading — the key papers and tools

• Tkaczyk et al. — “Reference matching: for real this time” and the SBMV evaluation (Crossref blog / JCDL 2018): search-based vs parse-based reference matching, F1 96.3% vs 92.5%.
• Tkaczyk et al. — “Machine Learning vs. Rules… Bibliographic Reference and Citation Parsers” (JCDL 2018): GROBID/CERMINE/ParsCit/AnyStyle F1 comparison.
• Fellegi & Sunter — “A Theory for Record Linkage” (JASA 1969): the probabilistic-linkage foundation; see also the open-source Splink.
• Li et al. — “Deep Entity Matching with Pre-Trained Language Models” (Ditto, VLDB 2021): BERT-based matching, up to +29% F1.
• Cohan et al. / Singh et al. — SPECTER, SPECTER2 and SciRepEval (2020–2023): citation-aware scholarly embeddings and their benchmarks; SciNCL (Ostendorff et al., 2022).
• Cormack, Clarke & Büttcher — Reciprocal Rank Fusion (2009); Khattab & Zaharia — ColBERT late interaction (2020).
• Walters & Wilder (Scientific Reports 2023), Bhattacharyya et al. (Cureus 2023), Chelli et al. (JMIR 2024), Aljamaan et al. (JMIR Medical Informatics 2024): measured citation-fabrication rates.
• CiteCheck, CiteTracer (“Source or It Didn’t Happen”), and the clibib BibTeX-agent study (2026): retrieval-grounded verification with reported F1/accuracy.
• Chow (1970) and Angelopoulos & Bates (conformal prediction, 2021): selective prediction and abstention.
• Tools: Crossref STQ & REST API, OpenAlex, Semantic Scholar, GROBID, AnyStyle, Splink, FAISS/HNSW, FlagEmbedding (BGE), RAGatouille (ColBERT).