How do I build the best reference-matching model in 2026

Executive summary: Reference matching, wrong references and fabricated citations in 2026

Executive summary: how to match messy references correctly — and avoid falsely accusing real citations of being fabricated

A modern reference-verification system has to solve two different problems that are often accidentally treated as one. First, it must answer “which real scholarly record does this messy reference mean?” Only after that has been attempted thoroughly should it answer “does the cited work appear not to exist?” The literature reviewed here shows why this ordering matters: real citations routinely contain misspellings, missing fields, OCR damage, wrong years, incomplete author lists, incorrect identifiers and version differences, while bibliographic databases themselves can contain missing or conflicting metadata. A detector that equates “my matcher failed” with “the citation is fabricated” will therefore create false accusations. Crossref’s own real-reference benchmark is a particularly clear demonstration: its stronger search-plus-validation matcher reached an F1 of 0.9629 on 2,000 messy references, yet still produced both missed true matches and incorrect matches. [1]

The central conclusion for a 2026 system: build a high-recall candidate retriever + evidence-aware reranker + calibrated abstention layer, not a binary “found/not found” script. Search several bibliographic sources, preserve the raw reference, model each bibliographic field separately, understand publication versions and parent/child records, and reserve the word fabricated for cases that remain unsupported after escalation. This design direction is consistent with Crossref’s Search-Based Matching with Validation, EXmatcher’s combined raw-string/field approach, CiteTracer’s multi-source cascade, GhostCite’s database-plus-web verification, and RefChecker’s deliberate distinction between unverified and likely hallucinated. [2]

The datasets in your list are not interchangeable. Crossref’s 2,000-reference set and EXmatcher are direct record-resolution benchmarks; Cora is primarily citation coreference; RenoBench evaluates citation parsing; CiteTracer and CiteAudit contain controlled mutations or fabricated-reference cases; HalluCitation Matters, GhostCite, Topaz et al., Zhao et al. and Phantom References are primarily prevalence/audit studies; and the July 2026 Manual Reference Verification Dataset is a small human-verified detector-evaluation set. Evaluating a new matcher on only one category can give a seriously misleading impression of real-world quality. [3]

Your proposed author-certified correction stratum is therefore genuinely different from most existing resources: instead of asking an annotator to infer what a damaged reference was intended to cite, the correction can provide direct evidence from an author that the original erroneous string and corrected record refer to the same intended work. That is especially valuable for measuring the failure mode that fabricated-citation detectors struggle with: a real source that looks fake because its citation has been mangled. It should, however, complement rather than replace manually verified null/fabricated cases, because a correction corpus alone will be strongly enriched for references that do have a recoverable intended target.

The problem in beginner language

Imagine that a paper contains the reference below:

Messy reference:

Smth J, Lee A. 2021. Deep lerning for cite matchng. J Inform Retriev. 15:44-57. doi:10.1234/WRONG-ID

Suppose the real publication is actually:

Smith, John; Lee, Alice. 2020. Deep Learning for Citation Matching. Journal of Information Retrieval, 15, 44–57. DOI: 10.5678/correct-doi

A simplistic detector can make a disastrous mistake. It searches the exact title, gets no result, follows the supplied DOI to an unrelated paper and reports “fabricated citation”. A good matcher interprets the evidence differently: “the DOI conflicts with the title; the title has plausible spelling/OCR corruption; the authors roughly agree; the year differs by one; let me retrieve candidate publications using several independent clues.”

Illustrative placeholder showing a damaged citation entering a multi-stage matching pipeline before a fabrication decision
Suggested caption: Do not jump directly from “search failed” to “fabricated”. First recover candidates using redundant evidence, validate them and allow the system to abstain. Suggested alt text: A messy reference passes through multi-source search and calibrated validation before a final decision.

A step-by-step correction

Step A — preserve the raw string. Never throw away the original citation after parsing. PDF/OCR parsers can split authors, titles or page numbers incorrectly; HalluCitation Matters explicitly found enough extraction problems that the authors used MinerU to obtain bibliography blocks and then passed those blocks to GROBID rather than relying on a single extraction route. [4]

Step B — treat identifiers as strong evidence, not infallible truth. A DOI that resolves to a record whose title and authors strongly conflict with the citation is evidence of an identifier error, not proof that the whole citation is imaginary. Both the 2026 Manual Reference Verification Dataset and modern citation-auditing taxonomies explicitly recognise “DOI points to another publication” as a distinct bibliographic failure. [5]

Step C — retrieve candidates redundantly. Search the corrected/normalised title, character n-grams, author names and combinations of fields. Crossref’s strongest 2018 approach retrieved candidates using bibliographic search and then applied a separate validation stage; EXmatcher similarly found that combining the raw reference string with parsed segment features performed better than relying on one representation alone. [6]

Step D — compare fields independently. A title may be excellent while the year is wrong. The DOI may be wrong while the title and authors are right. Modern systems such as CiteTracer explicitly classify title, author, venue, year, identifier and peripheral-field errors separately rather than reducing everything to one string-similarity value. [7]

Step E — understand versions. A preprint can later become a proceedings or journal paper with a new year, venue, page range and sometimes a changed title. Phantom References deliberately excludes ordinary venue/year/publication-status drift from its hallucination definition for precisely this reason. [8]

Step F — abstain when evidence is insufficient. “I cannot verify this automatically” is not the same statement as “this is fabricated”. RefChecker explicitly keeps unverified, uncertain and hallucination outcomes separate, which is a much safer operational model. [8]

 flowchart LR A[Raw reference] --> B[Parse but preserve raw text] B --> C{Valid identifier?} C -->|Yes and metadata agrees| D[High-confidence match] C -->|Missing or conflicting| E[Multi-index candidate retrieval] E --> F[Lexical + character + dense retrieval] F --> G[Field-aware reranking] G --> H{Calibrated confidence} H -->|High| D H -->|Middle| I[Abstain / manual review] H -->|Low| J[Escalate to more databases and web evidence] J --> K{Independent evidence found?} K -->|Yes| D K -->|No| L[Unsupported candidate] L --> M{Strict fabrication criteria met?} M -->|No| I M -->|Yes| N[Likely fabricated] 

Mermaid flowchart for WordPress: render the block with a Mermaid-compatible plug-in or block. The important conceptual rule is that “likely fabricated” is the final branch, not the first response to a failed lookup.

Existing datasets and evaluation resources

The most useful way to compare these datasets is by asking what question the ground truth actually answers. A parser benchmark can tell you whether an author surname was extracted correctly; it cannot, by itself, tell you whether a noisy reference resolved to the correct DOI. Likewise, a collection containing only fabricated references is useful for measuring recall on fabrications but cannot tell you how often the detector falsely accuses valid references.

Crossref real-reference benchmark

Resource
“Reference matching: for real this time” / Crossref real-reference benchmark
Author / year
Dominika Tkaczyk, 2018.
Size
2,000 real unstructured reference strings.
Task
Resolve each string to the correct Crossref DOI, or correctly return no DOI.
Ground truth
Manually verified. Target DOI or null was assigned by checking algorithm-returned DOIs and/or manual searching.
Availability
Crossref blog and open evaluation framework.

This is arguably the most directly relevant existing benchmark for your intended project. Crossref first sampled 100,000 metadata records, selected unstructured references for which members had not already deposited a DOI, randomly chose 2,000 strings, and manually assigned the true DOI or null. That design deliberately exposes matchers to real typographical errors, extraction noise, incomplete references and imperfect Crossref metadata. [1]

Four systems were compared. The legacy parser-based matcher achieved precision 0.9895, recall 0.8685 and F1 0.9251. Search-Based Matching with Validation (SBMV) achieved precision 0.9809, recall 0.9456 and F1 0.9629. Of the 2,000 cases, SBMV correctly matched 1,129, correctly returned no match for 791, missed 58 true matches, returned seven wrong DOIs and returned a DOI for 15 references that should not have matched. [1]

How wrong references were gathered: they were not artificially selected as “wrong”. They were real deposited strings, meaning naturally occurring errors came along with them. Crossref reports markup contamination, multiple references merged into one string, spaces/typos, missing or incorrect reference information, incomplete target metadata and confusing related objects such as a book versus its chapter. [1]

Limitation: 2,000 is modest; manual verification constrained scale; the sample concerns unstructured references lacking member-supplied DOIs, so it is not a representative estimate of all Crossref citations. [1]

Crossref real-reference benchmark: F1 Legacy parser-based 0.9251 SBMV 0.9629 SBMV gains recall while giving up a small amount of precision.
Crossref’s 2018 real-reference evaluation. SBMV improved F1 from 0.9251 to 0.9629; the important lesson is that a separate validation stage helped search tolerate messy input. [1]

Crossref reference-matching evaluation framework and Marple

Resource
Crossref reference-matching evaluation framework and Marple
Year
Original framework associated with the 2018 work; Marple project created in the 2022 period and remains an active matching-service codebase.
Size
Not one fixed dataset.
Ground truth
Framework accepts/evaluates against ground-truth datasets; ground truth therefore depends on the supplied evaluation set.

The archived CrossRef/reference-matching-evaluation repository can generate artificial matching datasets, run matcher implementations and calculate evaluation results. The repository was later moved from GitHub to Crossref’s GitLab infrastructure. [9]

Marple is broader: Crossref describes it as a matching service able to run multiple matching tasks and strategies, build backend indexes and evaluate strategies against ground-truth datasets. The same architecture supports bibliographic-reference matching and other identifier-resolution tasks. The former crossref/labs/marple path now redirects to crossref/marple. [10]

Why it matters: this is infrastructure rather than a new adjudicated corpus. Reusing its evaluator or at least matching its semantics makes your results easier to compare with Crossref work.

EXmatcher gold standard

Paper
EXmatcher: Combining Features Based on Reference Strings and Segments to Enhance Citation Matching
Authors / year
Behnam Ghavimi, Wolfgang Otto and Philipp Mayr, 2019.
Gold-standard size
816 reference strings; 517 have at least one matching Sowiport record.
Target collection
Sowiport social-science metadata.
Ground truth
Human checked, but not described as independent multi-annotator adjudication.

EXmatcher first created candidates with a simple blocker and then had a trained human assessor check the results. When no candidate existed, the assessor manually constructed searches from correctly interpreted reference fields. Wrong candidates were removed, missing matches were added and duplicate valid target records could be retained. The resulting gold standard contains 816 strings, 517 with at least one match. [11]

EXmatcher’s method is a useful ancestor of modern two-stage systems: blocking creates a manageable candidate set, then a binary classifier decides whether each reference–record pair is a match. The paper compares features derived from the unstructured string, parsed segments and their combination. The combined representation was most useful. For the “one match per reference” experiment, an SVM configuration reached precision 0.972, recall 0.926 and F1 0.948 before accounting for full-pipeline blocking loss; corresponding pipeline F1 was 0.941. [12]

How wrong/misleading references were gathered: the source strings came from real extracted citations rather than synthetic mutation. Importantly, the repository notes that extraction errors were not necessarily manually repaired before matching, making the benchmark relevant to noisy input.

Limitation: one assessor is described; this is human-curated gold truth, but the accessible methods do not describe independent double annotation plus formal disagreement adjudication. It is also tied to an older, domain-specific target database.

Cora citation matching/coreference

Dataset
Cora Citation Matching
Creator
Andrew McCallum and collaborators; classic Cora resource.
Commonly used matching subset
1,295 citation strings grouped into 134 underlying papers.
Task
Coreference/entity resolution: decide which citation strings refer to the same publication.
Ground truth
Hand-clustered.

McCallum’s official data page describes the Cora citation-matching data as citation text hand-clustered into groups referring to the same paper. Later experimental descriptions of this matching corpus report 1,295 citations representing 134 papers. [13]

Do not confuse this with the other famous “Cora dataset” containing 2,708 papers and a citation network for node classification. That is a different task. The record-linkage version is useful because many noisy strings point to the same entity; however, its label answers “same work or different work?”, not necessarily “what is the canonical DOI in today’s scholarly graph?”. [14]

Limitation for your project: it is old, computer-science-heavy and lacks modern DOI/database coverage patterns. It remains useful as a coreference stress test, not as the main 2026 benchmark.

CiteTracer benchmark

Paper
Source or It Didn’t Happen: A Multi-Agent Framework for Citation Hallucination Detection
System
CiteTracer
Authors / year
Mingzhe Li, Zhiqiang Lin and Shiqing Ma, 2026.
Synthetic set
2,450 citations retained from 3,100 collected/generated entries.
Real-world fabricated set
957 citations: 807 from ICLR 2026 desk-rejected submissions and 150 from another conference.
Ground truth
Synthetic: known by controlled construction plus checks. Real-world: venue-chair fabrication flags, described by the paper as hallucinated “by construction”; not an independently re-adjudicated balanced set of real and fake citations.

CiteTracer defines a field-level taxonomy covering real formatting variants, ambiguous/potential cases and errors in title, authors, venue, year, identifiers and peripheral metadata. Synthetic cases begin with real BibTeX seeds; code-specific mutation operators change controlled fields, followed by round-trip checks, verifiability checks and manual boundary review for ambiguous name variants. [15]

Its pipeline uses extraction, a cascading evidence collector, deterministic field matching and specialist judgement stages. The synthetic benchmark produced 97.1% overall accuracy, with class-level F1 values of 97.0 for Real, 95.8 for Potential and 98.5 for Hallucinated. On the 957 conference-chair-flagged fabricated citations it detected 97.1% without abstaining. [16]

Strength: diagnostic field-specific error labels and genuinely difficult controlled corruptions.

Critical limitation: the 957-case real-world set contains fabricated positives, so its headline result is essentially positive-class recall. It cannot by itself tell you the false-fabrication rate on thousands of legitimate but mangled references — exactly the metric your project should emphasise. [17]

HalluCitation Matters appendix

Paper
HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences
Authors / year
Yusuke Sakai, Hidetaka Kamigaito and Taro Watanabe, 2026.
Corpus
17,842 ACL/NAACL/EMNLP PDFs from 2024–2025; 741,656 extracted citations.
Confirmed affected papers
295, despite the rounded “300” in the title.
Ground truth
Manual verification of automatic candidates at the paper level. It is not an exhaustive manual label for every reference in all papers.

This resolves an apparent discrepancy in your list: the title says “300 Hallucinated Papers”, but the paper’s result table reports exactly 295 HalluCited papers: 20 in 2024 and 275 in 2025. Their corresponding affected-paper rates were 0.28% and 2.59%; EMNLP 2025 alone contained 154 affected papers. [4]

The method is particularly relevant to matcher failure. MinerU extracts bibliography blocks, GROBID normalises them, and likely ACL/arXiv citations are compared with ACL Anthology, arXiv, DBLP and OpenAlex. A character-level normalised Levenshtein title similarity below 0.9 creates a candidate. Humans then investigate those candidates using identifiers, bibliographic details and web search. A citation is treated as HalluCitation when no corresponding work can be found or when a similar work disagrees in at least two important attributes under the authors’ conservative criteria. [4]

The paper itself explicitly notes that candidate noise arises from OCR, parsing errors and the limitations of simple fuzzy title matching. This is direct evidence for your thesis that matcher failure is a major source of false alarms. [4]

Important ground-truth nuance: once a paper has one manually verified HalluCitation, the authors stop checking the rest of that paper because their target outcome is “does this paper contain at least one?”. Consequently, the appendix is excellent for confirmed affected-paper labels but should not be treated as a complete reference-level census of every bad citation within those 295 papers. [4]

Manual Reference Verification Dataset

Dataset
Manual Reference Verification Dataset for Hallucinated and Suspicious Citation Detection Tools
Creators / year
Fidan Badalova, data collector; Philipp Mayr, supervisor; July 2026.
Size
104 references from three scholarly documents.
Labels
71 verified; 33 problematic.
Ground truth
Every reference manually verified. Independent double annotation is not stated in the public data description.

The three source documents contribute 24, 15 and 65 references. Verification considers title, authors, venue, year, DOI/identifier and evidence from databases, publishers, conferences and repositories. A reference is not called problematic merely because one database lacks it; problematic cases include non-verifiable works, identifiers pointing elsewhere and conflicting core bibliographic fields. [18]

This is especially useful for measuring false positives because it includes both valid and problematic references and stores standardised outputs from CheckIfExist, HalluCiteChecker, Hallucinator, HalRef and RefChecker. The authors explicitly warn that it is an evaluation set, not a dataset for estimating population prevalence. [18]

Version warning: the July 2026 Zenodo dataset reports 71 verified and 33 problematic references. The initial arXiv position-paper analysis used a different 84-real/20-problematic breakdown for the same total of 104 cases. Treat the Zenodo v1.0.0 labels as the released dataset state and record the exact version/DOI in experiments rather than silently combining counts from different versions. [19]

RenoBench

Paper
RenoBench: A Citation Parsing Benchmark
Authors / year
Parth Sarin, Juan Pablo Alperin, Adam Buttrick and Dione Mentis, 2026.
Size
10,000 plain-text references paired with JATS XML.
Starting pool
Approximately 161,000 annotated citations before automated validation and feature-based sampling.
Sources
SciELO, Redalyc, Public Knowledge Project and Open Research Europe.
Ground truth
Publisher-provided structured annotations aligned by an automated pipeline, not reference-by-reference human adjudication.

RenoBench is highly valuable, but it solves the stage immediately before your central task. It evaluates citation parsing: converting a plain-text citation into structured JATS fields. Its 10,000 examples are multilingual and heterogeneous and were extracted from public-domain PDFs, then matched to publisher-provided XML annotations. [20]

How wrong references were gathered: they were not curated as fabricated-reference cases. The benchmark is built from real publisher data and noisy PDF reference strings. That makes it very useful for testing whether your parser survives realistic input corruption, but it does not supply canonical target DOI/null labels for evaluating the final resolver.

Best use in your project: use RenoBench as a parsing auxiliary benchmark, and separately test entity resolution on Crossref/EXmatcher/your author-certified corpus.

Dataset comparison

Resource Approximate size What the label means Ground-truth route Public? Main strength Main weakness for your matcher
Crossref real benchmark 2,000 refs Exact DOI or null Manual DOI verification/search Yes Closest real-world resolution task Small; Crossref-specific sampling
Crossref framework / Marple Not fixed Depends on supplied evaluation set Framework infrastructure Yes Reusable matching/evaluation architecture Not itself a gold-standard corpus
EXmatcher 816 refs Sowiport matching record(s) Trained human assessor Yes Real noisy matching with blocking Older, domain-specific target database
Cora citation matching 1,295 refs / 134 entities Which strings refer to the same work Hand clustering Yes Classic entity-resolution test No modern canonical DOI/null target
CiteTracer synthetic 2,450 Field-specific Real/Potential/Hallucinated code Controlled mutation + checks Yes Excellent diagnostic error taxonomy Synthetic corruption distribution
CiteTracer real fabricated set 957 Fabricated Venue-chair flags / desk rejects Yes Real conference failures Positive-only; cannot estimate false-positive rate
HalluCitation Matters 17,842 papers; 295 confirmed affected Paper has at least one HalluCitation Automatic candidates + manual verification Paper appendix available Large real ACL-family audit Not exhaustive reference-level labels
Manual Verification Dataset 104 refs Verified / problematic Manual verification Zenodo Direct false-positive evaluation Very small; only three documents
RenoBench 10,000 refs Correct JATS parsing Publisher XML + automated alignment/validation Hugging Face Multilingual real-world parsing Not canonical record resolution
Proposed author-certified corrections To be collected Original damaged citation → author-confirmed intended record/correction Author certification Project-dependent Directly attacks matcher-failure false positives Selection bias; needs null/fabricated controls

Dataset sizes and ground-truth descriptions in this comparison follow the corresponding primary resource descriptions above. [21]

Reference-error and reference-matching research

Logan et al. — Reference accuracy of primary studies published in peer-reviewed scholarly journals

Authors / year
Samuel W. Logan, Uta Hussong-Christian, Layne Case and Samantha Noregaard; 2024 print issue, with earlier online publication.
Study type
Scoping review.
Included studies
105 primary reference-accuracy studies.
Reported general error rate
32.7% in the reference-accuracy literature synthesised by the review.
Ground truth
Heterogeneous. The review synthesises studies with their own verification methods; it is not one item-level gold-standard corpus.

This should indeed be read early because it establishes that ordinary bibliographic errors long predate generative AI. The review gathered primary studies that actually verified references and found a substantial error burden; a later study by the same research group explicitly describes the earlier review as containing 105 studies and reports the review’s general reference-accuracy error rate as 32.7%. Author names and titles are prominent error categories. [22]

Why it matters for matching: an error rate in bibliographic fields is not a fabrication rate. A valid work with a misspelt author or title is still a real work. A detector should therefore be trained and evaluated on the distinction between metadata corruption and non-existence.

Limitation: definitions and verification protocols vary substantially across the historical literature; combining them into one percentage does not create a modern resolution benchmark.

Broadus — An Investigation of the Validity of Bibliographic Citations

Author / year
Robert N. Broadus, 1983.
Question
Do authors sometimes copy a bibliographic citation from another citing publication without consulting the original?
Ground truth
Manual bibliographic investigation, not a released machine-learning benchmark.

Broadus examined recurring errors in references to Edward O. Wilson’s book and W. D. Hamilton’s papers as a way of testing the hypothesis that writers sometimes reproduce citations from other papers rather than consulting the originals. The ERIC record describes this design directly. [23]

A later analysis of this line of research reports a striking example: 23% of citing papers reproduced the same erroneous title in the examined citation lineage. This figure is useful as historical evidence that “citation hallucination-like” anomalies can propagate socially even without LLMs, but it is reported here through later literature rather than a directly machine-readable Broadus results table. [24]

Matcher lesson: repeated identical errors are not proof that the cited work is fake; citation strings can copy one another’s damage.

de Lacey, Record and Wade — How accurate are quotations and references in medical journals?

Authors / year
G. de Lacey, C. Record and J. Wade, 1985.
Venue
British Medical Journal, volume 291.
Ground truth
Manual reference/quotation checking in the medical literature.

There is an attribution error in the original reading list: the 1985 BMJ paper is by de Lacey, Record and Wade, not Evans et al. The BMJ record confirms the title, publication details and authorship. [25]

Its importance here is conceptual: it belongs to the long literature separating reference accuracy and quotation accuracy. A reference can point to a genuine paper yet contain bibliographic mistakes; conversely, a perfectly formatted citation can be used to support a claim that the source does not actually make. Those are separate validation layers.

Quantitative result: detailed numeric outcomes are marked unspecified in this review rather than inferred from later summaries, because the primary HTML record retrieved for this research did not expose the full article’s tables in a form suitable for reliable extraction.

Scheinfeld et al. — Assessing Formatting Accuracy of APA Style References

Authors / year
Laurel Scheinfeld, Sunny Chung, Christine Fena, Clara Tran, Chris Kretz and Myra R. Reisman, 2025.
Study
Scoping review of APA reference-formatting accuracy.
Included studies
32.
Ground truth
Study-level synthesis of formatting assessments; not a record-resolution gold standard.

The review’s goal is to synthesise how APA formatting accuracy has been defined and measured, identify common errors and assess reproducibility and assessment tools. The review includes 32 studies. [26]

Why it matters: it reinforces a critical distinction for your model: punctuation, capitalisation, italicisation and APA layout violations should normally have almost zero weight in deciding whether the underlying scholarly object exists. A resolver should normalise superficial formatting aggressively while remaining strict about identity-bearing evidence.

Aggregate numeric formatting-error result: no single cross-study percentage is reported here because the review studies use heterogeneous definitions; treating them as one pooled rate would be misleading.

Tkaczyk et al. — Machine Learning vs. Rules and Out-of-the-Box vs. Retrained

Authors / year
Dominika Tkaczyk, Andrew Collins, Paraic Sheridan and Joeran Beel, 2018.
Task
Bibliographic reference parsing, not DOI resolution.
Systems
Ten open-source parsers.
Ground truth
Task-specific labelled parsing data; this paper is not a fabricated-reference dataset.

The study compared Anystyle-Parser, Biblio, CERMINE, Citation, Citation-Parser, GROBID, ParsCit, PDFSSA4MET, Reference Tagger and Science Parse. The strongest out-of-the-box parser was GROBID at F1 0.89, followed by CERMINE at 0.83 and ParsCit at 0.75. Machine-learning systems and non-ML systems had similar average precision, 0.77 versus 0.76, but ML systems had much higher average recall, 0.66 versus 0.22. Retraining improved GROBID from 0.89 to 0.92, CERMINE from 0.83 to 0.92 and ParsCit from 0.75 to 0.87. [27]

Lesson: domain adaptation matters. If your references come from author corrections, OCR, XML, multilingual journals or a particular publishing workflow, a parser trained elsewhere should not automatically be treated as optimal.

Even more important: parser F1 is not matcher F1. A system can parse a title imperfectly but still resolve the correct DOI using the raw string; conversely, perfect parsing does not guarantee that the target database contains the right record.

Fedoryszak et al. — Methodology for Evaluating Citation Parsing and Matching

Authors / year
Mateusz Fedoryszak, Łukasz Bolikowski, Dominika Tkaczyk and Krzysztof Wojciechowski, 2013.
Contribution
Evaluation methodology for reference parsing and matching, applied to the YADDA2 platform.
Ground truth
Evaluation methodology rather than a single widely used released gold dataset.

This work is useful less for a headline score than for how it frames evaluation. Citation matching can be evaluated as grouping/coreference or as resolution against records, and parsing and matching should not be collapsed into one opaque end metric. The chapter develops such a methodology and applies it to the YADDA2 reference-processing module. [28]

Numeric result: unspecified here; the value for your project is methodological rather than a particular number.

Crossref — Reference Matching: For Real This Time

The Crossref work has already been described in the dataset section because it is both a benchmark and a matching study. The key result remains one of the most relevant empirical baselines in this whole literature: SBMV F1 0.9629 versus 0.9251 for the legacy approach on real unstructured strings, with the main gain coming from improved recall. [1]

EXmatcher

Likewise, EXmatcher is both a paper and a released gold standard. The central modelling result is that the best system combined evidence from the raw reference string and parsed segments; its reported one-match SVM experiment reached F1 0.948 before full-pipeline blocking effects and 0.941 at pipeline level. [12]

Sefid et al. — Cleaning Noisy and Heterogeneous Metadata for Record Linking Across Scholarly Big Datasets

Authors / year
Athar Sefid, Jian Wu, Allen C. Ge, Jing Zhao, Lu Liu, Cornelia Caragea, Prasenjit Mitra and C. Lee Giles, 2019.
Source data
CiteSeerX matched against Web of Science, PubMed and DBLP.
Method
BM25 candidate blocking in Elasticsearch plus supervised matching features from multiple metadata fields; citation information is also incorporated.
Ground truth
Evaluation test data used for record linkage; details of independent human adjudication are not specified in the accessible abstract record used here.

The important innovation is the same architectural principle that keeps reappearing: do not demand that a noisy title be perfectly cleaned before searching. First use BM25 to generate candidates, then let a supervised model combine the evidence contained in all available fields; citation relationships can provide additional evidence. The authors report that metadata plus citation evidence significantly outperformed their title-based baseline, although the exact quantitative table is marked unspecified here because it was not exposed in the primary HTML abstract retrieved during this review. [29]

2026 lesson: candidate retrieval should optimise recall; the expensive model should decide among a small number of candidates afterwards.

LLM-era fabricated-citation research

The LLM-era studies are easy to misread because they use different definitions. Some count only completely non-existent works. Some count severe author mismatch. Some treat a wrong DOI or year as a hallucination. Some report candidate matches before manual checking; others manually review every flagged case. Comparing prevalence percentages without reading these definitions is therefore unsafe.

Topaz et al. — Fabricated citations: an audit across 2.5 million biomedical papers

Authors / year
Maxim Topaz, Nir Roguin, Pallavi Gupta, Zhihong Zhang and Laura-Maria Peltonen, 2026.
Venue
The Lancet.
Corpus
Nearly 2.5 million PubMed Central Open Access papers, January 2023 to 18 February 2026.
Verified-reference pool
97.1 million references with verifiable identifiers in the reported analysis.
Fabricated references found
4,046 across 2,810 papers.
Ground truth
Automated/AI-assisted verification with multi-source checking; public summaries do not describe a benchmark-style independent double-adjudication protocol for all 97.1 million references.

The audit’s headline trend is striking: fabricated-reference prevalence was approximately four per 10,000 papers through 2023 and rose to roughly 57 per 10,000 by early 2026. The official Columbia summary reports 4,046 fake citations in 2,810 papers among 97.1 million verified references and a more than twelve-fold increase over the period. [30]

The workflow applied automated filters to eliminate formatting/title artefacts and then checked suspicious references against sources including PubMed, Crossref, OpenAlex and Google Scholar. The definition centres on a cited title corresponding to no real publication, making it narrower than “any metadata field is wrong”. [31]

Why it matters to your project: even a very low false-positive rate matters when processing nearly 100 million references. A matcher that is only 99.9% specific can still generate huge review burdens at this scale.

Limitation: this is a prevalence audit, not a balanced public matcher benchmark. It also does not prove that each fabricated citation was generated by an LLM; temporal association is not individual-level causal evidence. [32]

Topaz et al.: affected papers per 10,000 2023 early 2026 ~4 ~57 Schematic trend using the reported endpoints; intermediate curve is illustrative, not monthly data.
The official Columbia summary reports a rise from roughly 4 affected papers per 10,000 in 2023 to about 57 per 10,000 in early 2026. The connecting curve above is deliberately schematic rather than a reconstruction of unpublished monthly values. [33]

Zhao et al. — LLM hallucinations in the wild

Authors / year
Zhenyue Zhao, Yihe Wang, Toby Stuart, Mathijs De Vaan, Paul Ginsparg and Yian Yin, 2026.
Corpus
111 million references across 2.5 million papers from arXiv, bioRxiv, SSRN and PubMed Central.
Estimated 2025 hallucinated citations
146,932.
Ground truth
Not a per-reference manually adjudicated benchmark. The paper statistically separates excess unmatched references from a pre-LLM bibliographic-error baseline.

This study asks a different question from Crossref or EXmatcher. References are matched against bibliographic sources, but the authors then model unmatched citations as a mixture of ordinary historical matching/bibliographic failures and a post-LLM excess component. The abstract reports a conservative estimate of 146,932 hallucinated citations during 2025. [34]

A contemporaneous methodological review reports the paper’s estimated 2025 rates as 0.39% for arXiv, 0.21% for bioRxiv, 1.91% for SSRN and 0.27% for PubMed Central. [35]

Why this matters enormously for your work: “unmatched” is the raw observation; “LLM-hallucinated” is an inference after modelling a baseline. Better matching changes the unmatched pool and therefore can change the inferred prevalence. This is almost the purest illustration of why matcher error must be separately quantified.

Limitation: a statistical excess estimate cannot tell you with adjudicated certainty whether a particular individual unmatched citation is fabricated.

Sakai et al. — HalluCitation Matters

As described above, HalluCitation Matters used a deliberately high-recall screening stage followed by manual verification. The authors found 295 affected papers overall, with the proportion rising from 0.28% in 2024 to 2.59% in 2025. Crucially, they explicitly report that single-candidate cases frequently arise from OCR, parsing and fuzzy-matching noise. [4]

That makes HalluCitation Matters one of the strongest empirical motivations for your proposed study: the detector itself produces a candidate pool containing matcher failures, and manual verification must rescue real citations from that pool.

GhostCite

Paper
GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Models
Authors / year
Zuyao Xu and collaborators, 2026.
LLM experiment
13 models across 40 computer-science domains.
Archive audit
2.2 million citations from 56,381 papers, 2020–2025.
Confirmed affected papers
604 papers, or 1.07% of the audited papers.
Ground truth
Strong manual adjudication of flagged archival citations: 16 trained research assistants; each flagged citation checked independently at least twice.

GhostCite’s CiteVerifier uses a cascading verification process rather than relying on one database. The study reports model-generated citation hallucination rates ranging from 14.23% to 94.93% across the tested LLM/domain combinations. In the archival audit, 1.07% of 56,381 papers contained an invalid or fabricated citation, with an 80.9% increase reported in 2025. [36]

The strongest part of its ground truth is the manual stage. Sixteen trained research assistants reviewed 2,530 automatically flagged citations over approximately a month, and each was independently checked at least twice before classification as non-academic source, valid or invalid. [37]

Strength: this is far closer to real adjudication than simply declaring every unmatched reference fabricated.

Limitation: manual review concentrates on automatically flagged cases, so the quality of the candidate-generation stage still matters for estimating missed invalid references. Its “invalid” category is also not identical to every other paper’s definition of “fabricated”.

CiteAudit

Paper
CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era
Authors / year
Zhengqing Yuan, Kaiwen Shi, Zheyuan Zhang, Lichao Sun, Nitesh V. Chawla and Yanfang Ye, 2026, according to the current arXiv record.
Benchmark
Current 2026 versions describe a large human-validated benchmark incorporating synthetic and real-world reference cases.
Ground truth
Human validated.

The accessible 2026 manuscript version describes generated and real-world portions with 6,475 real and 2,967 fake entries overall, totalling 9,442, and a human-in-the-loop process in which retrieved evidence is manually cross-checked by the author team. Its synthetic component deliberately perturbs fields such as title, authors, venue, year and DOI; the real-world component draws from scholarly platforms and manuscripts. [38]

The current arXiv abstract describes a multi-agent verification pipeline spanning extraction, evidence retrieval, matching/reasoning and calibrated judgement, and characterises the dataset as human validated. [39]

Strength: unlike positive-only fabricated sets, CiteAudit contains both real and fake examples and therefore supports specificity/false-positive analysis.

Version caveat: the arXiv manuscript evolved during 2026; freeze and report the exact paper/dataset commit used for any comparison rather than assuming that all online descriptions refer to an identical snapshot.

Phantom References

Authors / year
Mark Russinovich, Ram Shankar Siva Kumar and Ahmed Salem, 2026.
System
RefChecker.
Corpus
48,095 accepted papers and 2,614,992 extracted references from ICLR, ICML, NeurIPS and USENIX Security across the studied years.
Definition
Non-existent works or substantial author-identity corruption; ordinary year/venue/publication-status drift is excluded.
Ground truth
Evidence-based automated audit with web-search escalation; results are intentionally called “likely hallucinated”, not a fully manually adjudicated corpus.

RefChecker searches Semantic Scholar, OpenAlex, Crossref, DBLP, ACL Anthology and identifiers/URLs, then escalates suspicious cases to constrained web-search verification. The authors preserve separate verified, error, warning, unverified and hallucination statuses; only cases surviving the stricter hallucination route are counted in their prevalence estimate. [8]

In 2025, its all-reference likely-hallucination rates were reported as 0.38% for ICLR, 0.54% ICML, 0.68% NeurIPS and 0.81% USENIX Security. When the denominator is restricted to academic-paper-like references, the corresponding values are 0.31%, 0.46%, 0.55% and 0.42%. The paper also reports that about one in twenty 2025 NeurIPS and USENIX Security papers contained at least two likely hallucinated academic-paper-like references. [8]

Yet the paper is unusually candid about the central limitation: PDF extraction noise, stale or incomplete databases and model behaviour can create both false positives and false negatives, and the authors explicitly describe their numbers as an auditable estimate rather than an exact count. [8]

That sentence should be central to your project. A system can have a sophisticated web-search agent and still need an external benchmark specifically designed to measure whether genuine damaged references are being mislabelled.

Badalova and Mayr — What Current Tools Can and Cannot Do

Paper
Detecting Hallucinated and Suspicious Citations: What Current Tools Can and Cannot Do
Authors / year
Fidan Badalova and Philipp Mayr, 2026.
Tools compared
CheckIfExist, HalluCiteChecker, Hallucinator, HalRef and RefChecker.
Evaluation scale
104 references.
Ground truth
Manual verification; released companion Zenodo data now contains 71 verified and 33 problematic items.

This is possibly the single most directly supportive paper for your project’s motivation. Its empirical finding is not merely “tools miss fake references”; rather, current detectors generate many false positives because extraction, matching, database coverage and interpretation can fail. The authors therefore argue for caution and manual verification rather than treating automated detector output as ground truth. [40]

The companion Zenodo dataset strengthens reproducibility because it exposes the individual reference labels and standardised tool outputs. [18]

Best role in your evaluation: treat it as a small but very important “false accusation” benchmark. A new matcher should sharply reduce valid references being flagged while preserving detection of the problematic references.

Glynn — Analysis of the Academ-AI Dataset

Paper
Suspected Undeclared Use of Artificial Intelligence in the Academic Literature: An Analysis of the Academ-AI Dataset
Author / year
Alex Glynn, 2024.
Analysed examples
First 500 Academ-AI cases.
Ground truth
Not a fabricated-citation ground-truth dataset. Cases are suspected undeclared AI use identified mainly through tell-tale textual artefacts.

Academ-AI is related contextual evidence, not a reference-matching benchmark. Glynn analyses 500 examples of suspected undeclared AI-assisted writing identified through characteristic chatbot language appearing in published scholarship. The paper emphasises that these visible examples are likely a small and highly selected subset of undeclared AI use. [41]

Relevance to citations: it demonstrates how generated artefacts can enter formal scholarship and survive publication processes, but it should not be used as ground truth for citation existence or matching accuracy.

What all of this evidence means

The literature is actually studying several different tasks

Layer Question Representative resources Typical failure
Extraction Did we find the bibliography entry in the PDF? HalluCitation pipeline, GhostCite, RefChecker Two references merged, line lost, OCR corruption
Parsing Which tokens are authors/title/year/etc.? Tkaczyk parser study, RenoBench Title accidentally parsed as venue
Candidate retrieval Did the true work enter the candidate list? Crossref SBMV, EXmatcher, Sefid et al. True match never retrieved
Record resolution Which candidate is the intended work? Crossref, EXmatcher, Cora Book matched instead of chapter
Bibliographic validation Which fields disagree with the canonical record? CiteTracer, CiteAudit, Manual Verification Dataset Wrong DOI but otherwise real citation
Existence / fabrication Does a corresponding scholarly work exist? Topaz, HalluCitation, GhostCite, Phantom Database miss mistaken for non-existence
Claim support Does the cited paper actually support the sentence? Separate citation-faithfulness problem Real citation attached to unsupported claim

The reviewed work makes clear that errors upstream propagate downstream. HalluCitation Matters reports OCR/parsing and fuzzy-title failures among its candidate false alarms; Crossref reports errors from noisy strings and incomplete target metadata; Phantom References explicitly acknowledges database and extraction error as sources of false positives; and the July 2026 manual audit is designed specifically around tools incorrectly flagging genuine references. [42]

Why “not found” is not a scientific ground truth

A bibliographic database is an observation system, not the universe of scholarship. Records may be missing, duplicated, partially populated, transliterated differently or separated into preprint and published versions. Even Crossref’s own benchmark identifies incomplete/incorrect target metadata as a cause of matcher failure. [1]

This means the following inference is invalid:

Crossref returned nothing ↓ therefore the reference is not real

A more defensible inference is:

Crossref returned nothing ↓ search additional structured sources ↓ try spelling-robust and author-aware retrieval ↓ check identifiers and version relationships ↓ perform evidence-constrained web search if necessary ↓ if still unresolved, output "unverified" ↓ only call "likely fabricated" under a stricter evidence rule

The strongest recent tools increasingly follow this distinction. RefChecker has an explicit unverified state; CiteTracer has a Potential class; GhostCite manually reviews flagged references; and HalluCitation Matters manually verifies candidates rather than automatically treating its title-match threshold as final truth. [43]

Why author-certified corrections are interesting

Human adjudicators usually work backwards: they see a damaged reference and try to infer what the author meant. An author-certified correction provides a different form of evidence:

original published reference + author's correction / corrected reference ↓ certified intended identity ↓ real positive pair: damaged string → actual scholarly object

This is unusually useful for testing false-fabrication behaviour. Consider a detector that says 95% of fabricated citations are caught. That sounds excellent. But suppose it also labels 3% of heavily mangled genuine citations “fabricated”. In a real journal where genuinely fabricated citations are rare and genuine citations are abundant, those false alarms can dominate the review queue. This is the classic reason precision and base rates matter more than recall alone.

But author certification is not automatically perfect ground truth. Authors can submit an incorrect correction, different authors may disagree about the intended version, or a correction may change more than bibliographic spelling. Store the evidence trail, require a canonical record/identifier where possible, preserve both original and corrected strings and manually audit a random subset. Most importantly, add genuine null/non-existent references from other adjudicated resources so that the system cannot learn that every difficult string must have a match.

A timeline of the methodological shift

 timeline title From citation errors to large-scale hallucination audits 1983 : Broadus studies propagation of erroneous bibliographic citations 1985 : de Lacey, Record and Wade audit quotations and references 2013 : Fedoryszak et al. formalise parsing/matching evaluation 2018 : Tkaczyk et al. compare parsers : Crossref evaluates 2,000 real unstructured references 2019 : EXmatcher releases manually checked matching gold standard : Sefid et al. combine BM25 blocking, metadata and citation evidence 2024 : Logan et al. synthesise reference-accuracy literature : Glynn analyses Academ-AI 2025 : APA formatting-accuracy scoping review 2026 : Topaz et al. biomedical audit : Zhao et al. 111M-reference unmatched-citation study : HalluCitation Matters : GhostCite : CiteAudit : CiteTracer : Phantom References / RefChecker : RenoBench : Badalova-Mayr manual verification dataset 

A rigorous evaluation protocol for a 2026 matcher

The model architecture I would build

Recommended system: a hybrid multi-stage scholarly entity resolver, with deterministic identifier logic, high-recall heterogeneous retrieval, a learned field-aware reranker, explicit publication-version modelling, calibrated confidence and a separate evidence-escalation system for fabrication decisions.

Canonical knowledge layer

Build a periodically refreshed canonical record graph from complementary sources rather than choosing one “master database”. Crossref is essential for DOI metadata; arXiv and ACL Anthology cover important preprint/NLP cases; DBLP is valuable in computer science; PubMed/Europe PMC matter for biomedicine; OpenAlex provides broad scholarly graph coverage; DataCite is important for non-Crossref DOI objects. Modern auditors already obtain robustness by combining several such sources. [44]

Represent version families: preprint, conference paper, accepted manuscript, journal version, chapter and parent book/proceedings should not be treated as unrelated records. Crossref’s observed “book instead of chapter” error and Phantom’s deliberate tolerance of publication-version drift show why this matters. [45]

Reference understanding layer

Keep two representations simultaneously:

Raw representation: exact original string, character n-grams and an embedding of the entire citation.

Structured representation: parsed title, author sequence, venue, year, pages, volume, issue, DOI, PMID, arXiv ID and URLs.

Do not force all subsequent matching through parsed fields. EXmatcher’s best results came from combining string and segment evidence, while Crossref’s successful search-based strategy was designed to survive cases where the parser lacked enough clean structure. [46]

Candidate generation layer

Use several retrievers in parallel and fuse their candidate lists:

Lexical BM25: excellent when several important words remain intact; Sefid et al. explicitly used BM25 for blocking. [47]

Character n-gram/fuzzy retrieval: robust to OCR and misspellings.

Dense multilingual retrieval: useful for transliteration and reordered wording, but never use it alone because semantically similar papers make dangerous false-positive candidates.

Author-focused retrieval: especially valuable when title OCR is poor.

Identifier retrieval: DOI/PMID/arXiv lookup as a high-value candidate generator, followed by metadata consistency checking rather than blind acceptance.

Measure candidate recall@k separately. If the true paper is absent from the top candidate pool, no reranker can repair the error.

Field-aware reranker

For each candidate, generate interpretable compatibility evidence:

{ "title_similarity": ..., "title_character_similarity": ..., "author_overlap": ..., "author_order_consistency": ..., "year_delta": ..., "venue_compatibility": ..., "page_overlap": ..., "doi_consistency": ..., "version_family_match": ..., "source_agreement_count": ... }

Feed these features plus text pairs into a cross-encoder or comparable learned record-linkage model. Train with hard negatives such as:

same first author + different paper same/similar title + different year conference version vs journal version book vs chapter correct title + deliberately wrong DOI paper A title + paper B authors near-duplicate proceedings records

These are much more useful negatives than randomly sampled unrelated papers because the real danger is not failing to distinguish “quantum physics” from “Victorian poetry”; it is choosing the wrong one of two very similar scholarly records.

Calibration and abstention

The production output should not be binary. Use at least:

MATCHED_HIGH_CONFIDENCE MATCHED_LOW_CONFIDENCE AMBIGUOUS_MULTIPLE_CANDIDATES UNVERIFIED LIKELY_FABRICATED

For a high-stakes research-integrity system, calibrate thresholds on held-out real data. A model with slightly lower raw recall but a trustworthy abstention region can be far safer than a nominally higher-accuracy system that confidently labels difficult valid references as fabricated. RefChecker’s multi-status design and CiteTracer’s Potential class already move in this direction. [48]

Evidence escalation

Only unresolved citations should invoke expensive search/agent stages. Check additional scholarly indices, publisher pages and repository records, then use web-search agents to find evidence. Crucially, the agent should return source evidence, not merely a yes/no judgement. GhostCite and RefChecker both use escalation after structured-source matching rather than using an LLM as the sole authority. [49]

Datasets to use in the experiment

A convincing 2026 paper should report separate results on several strata rather than merge them into one score:

Evaluation stratum Resource What it tests
Real DOI resolution Crossref 2,000 Messy string → true DOI/null
Real domain-specific resolution EXmatcher 816 Noisy reference → target scholarly record
Coreference robustness Cora Whether variants of one citation cluster together
Parser robustness RenoBench 10,000 Whether upstream field extraction is reliable
Controlled corruption CiteTracer 2,450 Specific title/author/year/DOI failure modes
Human-verified false-alarm test Manual Verification Dataset 104 Valid versus problematic references
Real fabricated positives CiteTracer 957 Recall on chair-flagged fabrications
Real audit replication HalluCitation / GhostCite subsets where obtainable Performance on genuine publication workflow failures
Primary new contribution Author-certified corrections Can a matcher recover genuinely real references even when the published string was wrong?

These sizes and task definitions come from the corresponding primary resource descriptions reviewed above. [50]

Metrics that should be reported

Exact resolution accuracy

For references that have one known canonical target:

Exact resolution accuracy = references assigned the correct target

all resolvable references

Use exact canonical identifiers after normalising equivalent DOI URL forms. Where multiple records represent legitimate versions, publish both a strict exact-record metric and a version-family metric.

Precision, recall and F1 of produced links

This preserves comparability with Crossref:

Precision = correct returned links / all returned links Recall = correct returned links / all true links F1 = harmonic mean of precision and recall

Crossref’s SBMV provides a concrete historical baseline: precision 98.09%, recall 94.56%, F1 96.29% on its 2,000-reference benchmark. [1]

Candidate recall at k

Report whether the true item appears among the first 1, 5, 10 and other pre-registered candidate depths. This cleanly separates retrieval failure from reranker failure.

False fabrication rate on genuine references

This should be a headline metric in your paper:

False fabrication rate = genuine references labelled "fabricated"

all genuine references

Report it separately for ordinary clean references and author-certified mangled references. The latter is your core contribution.

Fabrication recall

Fabrication recall = true fabricated references correctly flagged

all adjudicated fabricated references

The 957 CiteTracer real-world fabricated cases are useful here, but because they are positive cases they must not be the only benchmark. [17]

Precision of a fabrication accusation

Fabrication precision = true fabricated among items labelled fabricated

all items labelled fabricated

This metric is ethically important. A false negative means a bad citation may escape detection; a false positive means a real author/source is being labelled as fabricated. The latter can have reputational consequences, so a production verifier should be precision-first and able to abstain.

Risk–coverage curve

Instead of publishing one arbitrary threshold, show what happens as the system becomes more selective. At 100% coverage it must decide everything; at lower coverage it can send uncertain cases to review. Plot matching error versus percentage automatically resolved.

Calibration

When the system says “0.99 probability of a match”, approximately 99% of such predictions should actually be correct. Report a calibration metric such as Brier score and/or expected calibration error, alongside reliability plots.

Per-error-type metrics

Report performance separately for title corruption, author corruption, wrong year, wrong venue, incorrect DOI, missing DOI, OCR noise, transliteration, merged references, book/chapter confusion and preprint/published-version differences. CiteTracer’s taxonomy gives a useful starting point. [7]

Operational cost

Measure database calls, latency and the proportion reaching expensive web/LLM escalation. Phantom References reports that one venue-scale audit cost roughly four US cents per paper under the paper’s May 2026 API-pricing configuration, demonstrating why escalation rather than full web search for every citation is operationally attractive. [8]

Experimental protocol

Freeze every dataset version before development. This matters particularly for fast-moving 2026 repositories and for the Manual Verification Dataset, whose released Zenodo composition differs from the initial preprint table. [19]

Split by underlying scholarly work, not merely by citation string. Otherwise two damaged variants of the same paper can leak into train and test, creating unrealistically easy evaluation.

Freeze time-sensitive bibliographic snapshots where possible. A match that was impossible because a target was absent from a database in May might become easy after metadata is added in August.

Do not tune thresholds on the test datasets. Crossref explicitly selected matcher thresholds on a separate dataset; follow the same discipline. [1]

Blind the final author-certified test set. Construct training/dev material first, lock the evaluation IDs, then reveal the test labels only once the pipeline is fixed.

Report end-to-end and component metrics. At minimum: extraction success, parsing, candidate recall, reranker accuracy, exact resolution, abstention, and fabrication classification.

Manually investigate disagreements. For every system-vs-ground-truth disagreement in a manageable test set, record whether the error came from extraction, parsing, target-database coverage, candidate retrieval, ranking, canonicalisation/version logic or bad benchmark ground truth. This error taxonomy will probably be more scientifically valuable than another aggregate accuracy score.

Baselines

The evaluation should contain at least a deliberately simple lexical baseline, a Crossref-like bibliographic search baseline, a fuzzy-title threshold resembling HalluCitation’s screening stage, a learned EXmatcher/Sefid-style record-linkage model, and currently runnable multi-source verification systems such as CiteTracer/RefChecker where licensing and APIs permit fair reproduction. Crossref legacy/SBM/SBMV results should be quoted as historical comparators on the original benchmark rather than assumed to transfer unchanged to a new corpus. [51]

What effect size should you expect?

It would be scientifically inappropriate to invent a predicted percentage improvement before running your author-certified benchmark. The existing literature gives useful observed precedents, not a guaranteed effect for your data.

On Crossref’s real strings, moving from the legacy matcher to SBMV increased F1 from 0.9251 to 0.9629, an absolute gain of 0.0378; recall rose from 0.8685 to 0.9456 while precision declined slightly from 0.9895 to 0.9809. [1]

EXmatcher’s best combined string-plus-segment representation reached pipeline F1 0.941 in its reported one-match setting, supporting the idea that heterogeneous evidence helps. [12]

CiteTracer reaches very high scores on its controlled synthetic benchmark, but those values should not be used as the expected accuracy on naturally mangled author-certified citations because the data-generating processes differ. [7]

The most defensible hypothesis to pre-register is therefore directional:

Primary hypothesis: multi-source, field-aware, calibrated matching will reduce the proportion of author-certified genuine references falsely labelled unverified/fabricated relative to title-only, single-database and uncalibrated baselines, while preserving high exact-resolution precision.

Report the actual absolute risk reduction, relative risk reduction and confidence interval after the blind test. Do not select whichever statistic looks largest after seeing the results.

Final summary and FAQ

Final summary

The literature supports a fairly coherent story once parsing, matching and fabrication detection are separated.

First, reference errors are old. Broadus, de Lacey and decades of studies synthesised by Logan show that bibliographic inaccuracies existed long before modern LLMs. A malformed reference is therefore not inherently evidence of AI fabrication. [52]

Second, matching real references is already difficult. Crossref’s carefully verified 2,000-reference benchmark demonstrates that search-plus-validation can substantially improve recall, yet even a strong F1 of 0.9629 leaves errors. EXmatcher reaches the same architectural conclusion from another direction: combine raw strings and structured fields rather than trusting one representation. [6]

Third, the LLM-era phenomenon is real enough to warrant serious checking. Topaz et al., Zhao et al., HalluCitation Matters, GhostCite and Phantom References independently report rising or clearly detectable invalid/non-existent reference problems across biomedical, preprint and conference corpora, although their definitions and adjudication methods differ substantially. [53]

Fourth, matcher failure contaminates fabrication detection. HalluCitation explicitly observes OCR/parsing/fuzzy-match noise; RefChecker warns about false positives from extraction and database coverage; Crossref documents real matching failures caused by imperfect strings and metadata; and Badalova and Mayr show that current detectors can flag many valid references. [54]

That is why an author-certified correction benchmark is useful. It targets an important stratum that positive fabricated-citation datasets do not: “we know this awful-looking reference is real, and we know what the author intended it to point to.” The strongest 2026 paper would combine that new stratum with Crossref, EXmatcher, Cora, RenoBench, CiteTracer, the Manual Verification Dataset and adjudicated fabricated cases, then evaluate the full error cascade from PDF extraction through final fabrication verdict.

FAQ: How do I build the best reference-matching model in 2026?

Build a hybrid retrieval-and-reranking system, not a monolithic LLM prompt. Preserve the raw citation, parse it into fields, resolve identifiers, retrieve candidates with BM25 + character-level matching + dense retrieval from multiple scholarly databases, fuse candidates, rerank with a model that understands title/author/year/venue/identifier evidence and publication versions, calibrate its confidence and provide an abstention state. Escalate only unresolved cases to additional databases and evidence-constrained web search. This architecture follows the strongest recurring ideas in Crossref SBMV, EXmatcher, Sefid et al., CiteTracer, GhostCite and RefChecker. [55]

For training, favour hard real errors and hard negatives. Synthetic corruptions are useful for coverage, but they should supplement—not substitute for—naturally damaged references and blind real-world evaluation.

FAQ: How do I reduce false fabrication flags?

The single most important rule is:

Failure to match is a reason to investigate, not a fabrication label.

Use a three-stage interpretation: matched → unresolved/ambiguous → likely fabricated. The fabricated label should require stronger negative evidence than the unresolved label. Search several independent sources, test alternative spellings, handle transliteration, tolerate year/venue version drift, verify whether an incorrect DOI simply points to a different real object, and look for parent/child publication relations. RefChecker’s distinction between unverified and hallucination and CiteTracer’s Potential category are useful precedents. [48]

Then optimise and publish the false fabrication rate on author-certified real references. A detector should not be considered safe merely because it catches most synthetic fabrications.

FAQ: Should I train a giant language model from scratch?

No. The evidence reviewed here gives no reason to believe that training an end-to-end foundation model from scratch is the efficient path. The hard problem is not simply language understanding; it is retrieving and reconciling external bibliographic evidence. Crossref’s search-based approach, EXmatcher’s blocking/classification pipeline, Sefid et al.’s BM25 blocking, CiteTracer’s retrieval cascade and RefChecker’s multi-source verification all benefit from explicit retrieval infrastructure. [56]

Fine-tune relatively small specialised components where labelled data justifies it, and use large models where they add unique value—noisy parsing, difficult semantic comparison or constrained evidence-search escalation.

FAQ: Should DOI matching be the first thing I do?

Yes, but do not treat it as infallible. A DOI that resolves and whose metadata agrees strongly with the citation is nearly ideal evidence. A DOI that resolves to a completely different title should be classified as an identifier conflict and trigger candidate recovery. The Manual Reference Verification Dataset explicitly treats a DOI that points to another publication as a problematic-reference mode. [18]

FAQ: Is title similarity enough?

No. HalluCitation Matters uses a 0.9 normalised-Levenshtein title threshold as a screening mechanism, but its authors then manually verify candidates and report noise arising from fuzzy matching and extraction. EXmatcher found better performance from combining string and field evidence, and Crossref’s SBMV separately validates retrieved candidates. [57]

FAQ: Which dataset should be the primary benchmark?

For existing direct resolution ground truth, start with the Crossref 2,000-reference benchmark because it most closely asks “what DOI does this real messy string mean?”. Add EXmatcher for another manually checked resolution environment. [58]

For your new contribution, make the author-certified correction corpus the main stress-test stratum and report Crossref/EXmatcher as established comparators. Add the Manual Verification Dataset to test false flags, CiteTracer to test controlled error modes and genuine fabrications, RenoBench to isolate parser quality, and Cora to test coreference robustness. [59]

FAQ: What should the paper’s main metric be?

Do not use only one metric. For matching, make exact target accuracy plus link precision/recall/F1 primary. For your research-integrity contribution, prominently report false fabrication rate on author-certified genuine references, fabrication precision, fabrication recall and coverage at high precision.

The key scientific claim should not be “our model has 98.x% accuracy”. A much more meaningful claim would be: given genuinely real references that were damaged badly enough to require author correction, how often can the model still recover the intended scholarly object, and how rarely does it falsely call that object fabricated?

FAQ: What practical implementation details are easiest to overlook?

Cache external lookups, store provenance for every candidate, preserve raw strings, normalise Unicode, test transliterated author names, distinguish missing from conflicting fields, collapse known version families, detect book/chapter and proceedings/paper relations, version every external data snapshot, and make every final fabrication verdict inspectable by a human. CiteTracer reports that most benchmark cases can be resolved in deterministic/cached stages while a minority reach expensive judging, illustrating why a cascade is operationally sensible. [7]

Finally, never expose a naked “FAKE” label without its evidence. Return something closer to:

{ "status": "UNVERIFIED", "best_candidate": "...", "candidate_score": 0.61, "conflicts": ["DOI resolves to another work"], "supporting_sources": ["Crossref", "OpenAlex"], "sources_searched_without_match": ["DBLP", "ACL Anthology"], "recommended_action": "manual_review" }

That design makes the system useful to editors and researchers while acknowledging the epistemic reality documented throughout this literature: bibliographic infrastructure is incomplete, citations are noisy, and matcher failure is not the same thing as scientific fabrication.

Reference table with links

Reference / resource Authors / organisation Year Primary or official link Role in this review
Reference matching: for real this time Dominika Tkaczyk / Crossref 2018 Crossref 2,000 real references → DOI/null; direct matching benchmark
Reference Matching Evaluation Framework Crossref 2018–2019 era GitHub archive Reusable dataset generation/matching/evaluation code
Marple Crossref 2022– Crossref GitLab Modern multi-strategy matching infrastructure
Machine Learning vs. Rules and Out-of-the-Box vs. Retrained Dominika Tkaczyk, Andrew Collins, Paraic Sheridan, Joeran Beel 2018 arXiv Reference-parser comparison and retraining study
Methodology for Evaluating Citation Parsing and Matching Mateusz Fedoryszak, Łukasz Bolikowski, Dominika Tkaczyk, Krzysztof Wojciechowski 2013 DOI / Springer Evaluation methodology
EXmatcher: Combining Features Based on Reference Strings and Segments to Enhance Citation Matching Behnam Ghavimi, Wolfgang Otto, Philipp Mayr 2019 arXiv Reference matching method and 816-item gold standard
EXmatcher gold standard repository EXCITE Project 2019 GitHub Released manually checked matching data
Cora Citation Matching Andrew McCallum / UMass Classic corpus Official UMass data page Hand-clustered citation coreference benchmark
Cleaning Noisy and Heterogeneous Metadata for Record Linking Across Scholarly Big Datasets Athar Sefid et al. 2019 arXiv BM25 blocking + supervised multi-field record linkage
Reference accuracy of primary studies published in peer-reviewed scholarly journals: A scoping review Samuel W. Logan, Uta Hussong-Christian, Layne Case, Samantha Noregaard 2024 SAGE / DOI Historical reference-error evidence
An Investigation of the Validity of Bibliographic Citations Robert N. Broadus 1983 ERIC record Propagation/copying of erroneous citations
How accurate are quotations and references in medical journals? G. de Lacey, C. Record, J. Wade 1985 BMJ Classic reference/quotation accuracy study
Assessing Formatting Accuracy of APA Style References: A Scoping Review Laurel Scheinfeld et al. 2025 Érudit Formatting-error versus identity-error distinction
Source or It Didn’t Happen: A Multi-Agent Framework for Citation Hallucination Detection Mingzhe Li, Zhiqiang Lin, Shiqing Ma 2026 arXiv CiteTracer method, synthetic benchmark and real fabricated cases
CiteTracer repository CiteTracer authors 2026 GitHub Open implementation/data access
HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe 2026 ACL Anthology · arXiv ACL/NAACL/EMNLP audit; 295 confirmed affected papers
Manual Reference Verification Dataset for Hallucinated and Suspicious Citation Detection Tools Fidan Badalova, Philipp Mayr 2026 Zenodo 104 manually verified valid/problematic references
RenoBench: A Citation Parsing Benchmark Parth Sarin, Juan Pablo Alperin, Adam Buttrick, Dione Mentis 2026 arXiv Real multilingual citation-parsing benchmark
RenoBench / Reference Annotation Benchmark Public Knowledge Project 2026 Hugging Face 10,000 plaintext/JATS pairs
Fabricated citations: an audit across 2.5 million biomedical papers Maxim Topaz, Nir Roguin, Pallavi Gupta, Zhihong Zhang, Laura-Maria Peltonen 2026 The Lancet DOI · Columbia summary Large biomedical fabricated-citation audit
LLM hallucinations in the wild: Large-scale evidence from non-existent citations Zhenyue Zhao, Yihe Wang, Toby Stuart, Mathijs De Vaan, Paul Ginsparg, Yian Yin 2026 arXiv 111-million-reference statistical prevalence analysis
GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Models Zuyao Xu et al. 2026 arXiv LLM generation study plus manually reviewed archival audit
CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era Zhengqing Yuan, Kaiwen Shi, Zheyuan Zhang, Lichao Sun, Nitesh V. Chawla, Yanfang Ye 2026 arXiv Human-validated citation-verification benchmark and system
Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences Mark Russinovich, Ram Shankar Siva Kumar, Ahmed Salem 2026 arXiv RefChecker and accepted-paper audit
RefChecker Russinovich, Siva Kumar, Salem 2026 GitHub Open multi-source verification pipeline
Detecting Hallucinated and Suspicious Citations: What Current Tools Can and Cannot Do Fidan Badalova, Philipp Mayr 2026 arXiv Audit of detector false positives and limitations
Suspected Undeclared Use of Artificial Intelligence in the Academic Literature: An Analysis of the Academ-AI Dataset Alex Glynn 2024 arXiv Contextual evidence on visible undeclared AI artefacts
Academ-AI data dump Alex Glynn 2025 data release Figshare Dataset associated with Academ-AI research

Research status: sources were checked against primary or official pages where available, including Crossref, arXiv, ACL Anthology, GitHub/GitLab project repositories, Zenodo, Hugging Face, BMJ, SAGE, Érudit and institutional author/publisher pages. Fast-moving 2026 preprints and repositories may change after 13 August 2026; reproducible experiments should record exact dataset versions, commits and access dates. [60]

[1] [2] [3] [6] [21] [45] [50] [51] [55] [56] [58] https://www.crossref.org/blog/reference-matching-for-real-this-time/ https://www.crossref.org/blog/reference-matching-for-real-this-time/ [4] [42] [54] [57] https://arxiv.org/html/2601.18724 https://arxiv.org/html/2601.18724 [5] [18] [19] [59] https://zenodo.org/records/21457492 https://zenodo.org/records/21457492 [7] [15] [16] [17] [44] https://arxiv.org/html/2605.08583 https://arxiv.org/html/2605.08583 [8] [43] [48] https://arxiv.org/html/2607.00738 https://arxiv.org/html/2607.00738 [9] https://github.com/CrossRef/reference-matching-evaluation https://github.com/CrossRef/reference-matching-evaluation [10] https://gitlab.com/crossref/marple https://gitlab.com/crossref/marple [11] [12] [46] https://arxiv.org/html/1906.04484 https://arxiv.org/html/1906.04484 [13] https://people.cs.umass.edu/~mccallum/data.html https://people.cs.umass.edu/~mccallum/data.html [14] https://linkagelibrary.icpsr.umich.edu/linkagelibrary/project/109167/view https://linkagelibrary.icpsr.umich.edu/linkagelibrary/project/109167/view [20] https://arxiv.org/html/2603.25640 https://arxiv.org/html/2603.25640 [22] https://journals.sagepub.com/doi/abs/10.1177/09610006231177715 https://journals.sagepub.com/doi/abs/10.1177/09610006231177715 [23] [52] https://eric.ed.gov/?id=EJ279773 https://eric.ed.gov/?id=EJ279773 [24] https://arxiv.org/html/2502.20581v3 https://arxiv.org/html/2502.20581v3 [25] https://www.bmj.com/content/291/6499/884 https://www.bmj.com/content/291/6499/884 [26] https://www.erudit.org/en/journals/eblip/2025-v20-n4-eblip010557/1123056ar/ https://www.erudit.org/en/journals/eblip/2025-v20-n4-eblip010557/1123056ar/ [27] https://arxiv.org/abs/1802.01168 https://arxiv.org/abs/1802.01168 [28] Methodology for Evaluating Citation Parsing and Matching https://link.springer.com/chapter/10.1007/978-3-642-35647-6_11?utm_source=chatgpt.com [29] [47] https://arxiv.org/abs/1906.08470 https://arxiv.org/abs/1906.08470 [30] [31] [33] [53] https://www.nursing.columbia.edu/news/nearly-3-000-peer-reviewed-medical-papers-have-fake-citations-columbia-nursing-ai-assisted-audit-finds https://www.nursing.columbia.edu/news/nearly-3-000-peer-reviewed-medical-papers-have-fake-citations-columbia-nursing-ai-assisted-audit-finds [32] https://www.cuimc.columbia.edu/news/fraudulent-citations-blamed-ai-hallucinations-are-becoming-more-common-research-papers https://www.cuimc.columbia.edu/news/fraudulent-citations-blamed-ai-hallucinations-are-becoming-more-common-research-papers [34] https://arxiv.org/abs/2605.07723 https://arxiv.org/abs/2605.07723 [35] [40] https://arxiv.org/html/2607.22693v1 https://arxiv.org/html/2607.22693v1 [36] https://arxiv.org/abs/2602.06718 https://arxiv.org/abs/2602.06718 [37] [49] https://arxiv.org/html/2602.06718v2 https://arxiv.org/html/2602.06718v2 [38] CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era https://arxiv.org/html/2602.23452 [39] https://arxiv.org/abs/2602.23452 https://arxiv.org/abs/2602.23452 [41] https://arxiv.org/abs/2411.15218 https://arxiv.org/abs/2411.15218 [60] https://gitlab.com/crossref/labs/marple https://gitlab.com/crossref/labs/marple

Leave a reply

Your email address will not be published. Required fields are marked *