Executive summary: how to match messy references correctly — and avoid falsely accusing real citations of being fabricated
A modern reference-verification system has to solve two different problems that are often accidentally treated as one. First, it must answer “which real scholarly record does this messy reference mean?” Only after that has been attempted thoroughly should it answer “does the cited work appear not to exist?” The literature reviewed here shows why this ordering matters: real citations routinely contain misspellings, missing fields, OCR damage, wrong years, incomplete author lists, incorrect identifiers and version differences, while bibliographic databases themselves can contain missing or conflicting metadata. A detector that equates “my matcher failed” with “the citation is fabricated” will therefore create false accusations. Crossref’s own real-reference benchmark is a particularly clear demonstration: its stronger search-plus-validation matcher reached an F1 of 0.9629 on 2,000 messy references, yet still produced both missed true matches and incorrect matches. [1]
The central conclusion for a 2026 system: build a high-recall candidate retriever + evidence-aware reranker + calibrated abstention layer, not a binary “found/not found” script. Search several bibliographic sources, preserve the raw reference, model each bibliographic field separately, understand publication versions and parent/child records, and reserve the word fabricated for cases that remain unsupported after escalation. This design direction is consistent with Crossref’s Search-Based Matching with Validation, EXmatcher’s combined raw-string/field approach, CiteTracer’s multi-source cascade, GhostCite’s database-plus-web verification, and RefChecker’s deliberate distinction between unverified and likely hallucinated. [2]
The datasets in your list are not interchangeable. Crossref’s 2,000-reference set and EXmatcher are direct record-resolution benchmarks; Cora is primarily citation coreference; RenoBench evaluates citation parsing; CiteTracer and CiteAudit contain controlled mutations or fabricated-reference cases; HalluCitation Matters, GhostCite, Topaz et al., Zhao et al. and Phantom References are primarily prevalence/audit studies; and the July 2026 Manual Reference Verification Dataset is a small human-verified detector-evaluation set. Evaluating a new matcher on only one category can give a seriously misleading impression of real-world quality. [3]
Your proposed author-certified correction stratum is therefore genuinely different from most existing resources: instead of asking an annotator to infer what a damaged reference was intended to cite, the correction can provide direct evidence from an author that the original erroneous string and corrected record refer to the same intended work. That is especially valuable for measuring the failure mode that fabricated-citation detectors struggle with: a real source that looks fake because its citation has been mangled. It should, however, complement rather than replace manually verified null/fabricated cases, because a correction corpus alone will be strongly enriched for references that do have a recoverable intended target.
The problem in beginner language
Imagine that a paper contains the reference below:
Messy reference:
Smth J, Lee A. 2021. Deep lerning for cite matchng. J Inform Retriev. 15:44-57. doi:10.1234/WRONG-ID
Suppose the real publication is actually:
Smith, John; Lee, Alice. 2020. Deep Learning for Citation Matching. Journal of Information Retrieval, 15, 44–57. DOI: 10.5678/correct-doi
A simplistic detector can make a disastrous mistake. It searches the exact title, gets no result, follows the supplied DOI to an unrelated paper and reports “fabricated citation”. A good matcher interprets the evidence differently: “the DOI conflicts with the title; the title has plausible spelling/OCR corruption; the authors roughly agree; the year differs by one; let me retrieve candidate publications using several independent clues.”
A step-by-step correction
Step A — preserve the raw string. Never throw away the original citation after parsing. PDF/OCR parsers can split authors, titles or page numbers incorrectly; HalluCitation Matters explicitly found enough extraction problems that the authors used MinerU to obtain bibliography blocks and then passed those blocks to GROBID rather than relying on a single extraction route. [4]
Step B — treat identifiers as strong evidence, not infallible truth. A DOI that resolves to a record whose title and authors strongly conflict with the citation is evidence of an identifier error, not proof that the whole citation is imaginary. Both the 2026 Manual Reference Verification Dataset and modern citation-auditing taxonomies explicitly recognise “DOI points to another publication” as a distinct bibliographic failure. [5]
Step C — retrieve candidates redundantly. Search the corrected/normalised title, character n-grams, author names and combinations of fields. Crossref’s strongest 2018 approach retrieved candidates using bibliographic search and then applied a separate validation stage; EXmatcher similarly found that combining the raw reference string with parsed segment features performed better than relying on one representation alone. [6]
Step D — compare fields independently. A title may be excellent while the year is wrong. The DOI may be wrong while the title and authors are right. Modern systems such as CiteTracer explicitly classify title, author, venue, year, identifier and peripheral-field errors separately rather than reducing everything to one string-similarity value. [7]
Step E — understand versions. A preprint can later become a proceedings or journal paper with a new year, venue, page range and sometimes a changed title. Phantom References deliberately excludes ordinary venue/year/publication-status drift from its hallucination definition for precisely this reason. [8]
Step F — abstain when evidence is insufficient. “I cannot verify this automatically” is not the same statement as “this is fabricated”. RefChecker explicitly keeps unverified, uncertain and hallucination outcomes separate, which is a much safer operational model. [8]
flowchart LR A[Raw reference] --> B[Parse but preserve raw text] B --> C{Valid identifier?} C -->|Yes and metadata agrees| D[High-confidence match] C -->|Missing or conflicting| E[Multi-index candidate retrieval] E --> F[Lexical + character + dense retrieval] F --> G[Field-aware reranking] G --> H{Calibrated confidence} H -->|High| D H -->|Middle| I[Abstain / manual review] H -->|Low| J[Escalate to more databases and web evidence] J --> K{Independent evidence found?} K -->|Yes| D K -->|No| L[Unsupported candidate] L --> M{Strict fabrication criteria met?} M -->|No| I M -->|Yes| N[Likely fabricated]
Mermaid flowchart for WordPress: render the block with a Mermaid-compatible plug-in or block. The important conceptual rule is that “likely fabricated” is the final branch, not the first response to a failed lookup.
Existing datasets and evaluation resources
The most useful way to compare these datasets is by asking what question the ground truth actually answers. A parser benchmark can tell you whether an author surname was extracted correctly; it cannot, by itself, tell you whether a noisy reference resolved to the correct DOI. Likewise, a collection containing only fabricated references is useful for measuring recall on fabrications but cannot tell you how often the detector falsely accuses valid references.
Crossref real-reference benchmark
- Resource
- “Reference matching: for real this time” / Crossref real-reference benchmark
- Author / year
- Dominika Tkaczyk, 2018.
- Size
- 2,000 real unstructured reference strings.
- Task
- Resolve each string to the correct Crossref DOI, or correctly return no DOI.
- Ground truth
- Manually verified. Target DOI or null was assigned by checking algorithm-returned DOIs and/or manual searching.
- Availability
- Crossref blog and open evaluation framework.
This is arguably the most directly relevant existing benchmark for your intended project. Crossref first sampled 100,000 metadata records, selected unstructured references for which members had not already deposited a DOI, randomly chose 2,000 strings, and manually assigned the true DOI or null. That design deliberately exposes matchers to real typographical errors, extraction noise, incomplete references and imperfect Crossref metadata. [1]
Four systems were compared. The legacy parser-based matcher achieved precision 0.9895, recall 0.8685 and F1 0.9251. Search-Based Matching with Validation (SBMV) achieved precision 0.9809, recall 0.9456 and F1 0.9629. Of the 2,000 cases, SBMV correctly matched 1,129, correctly returned no match for 791, missed 58 true matches, returned seven wrong DOIs and returned a DOI for 15 references that should not have matched. [1]
How wrong references were gathered: they were not artificially selected as “wrong”. They were real deposited strings, meaning naturally occurring errors came along with them. Crossref reports markup contamination, multiple references merged into one string, spaces/typos, missing or incorrect reference information, incomplete target metadata and confusing related objects such as a book versus its chapter. [1]
Limitation: 2,000 is modest; manual verification constrained scale; the sample concerns unstructured references lacking member-supplied DOIs, so it is not a representative estimate of all Crossref citations. [1]
Crossref reference-matching evaluation framework and Marple
- Resource
- Crossref reference-matching evaluation framework and Marple
- Year
- Original framework associated with the 2018 work; Marple project created in the 2022 period and remains an active matching-service codebase.
- Size
- Not one fixed dataset.
- Ground truth
- Framework accepts/evaluates against ground-truth datasets; ground truth therefore depends on the supplied evaluation set.
The archived CrossRef/reference-matching-evaluation repository can generate artificial matching datasets, run matcher implementations and calculate evaluation results. The repository was later moved from GitHub to Crossref’s GitLab infrastructure. [9]
Marple is broader: Crossref describes it as a matching service able to run multiple matching tasks and strategies, build backend indexes and evaluate strategies against ground-truth datasets. The same architecture supports bibliographic-reference matching and other identifier-resolution tasks. The former crossref/labs/marple path now redirects to crossref/marple. [10]
Why it matters: this is infrastructure rather than a new adjudicated corpus. Reusing its evaluator or at least matching its semantics makes your results easier to compare with Crossref work.
EXmatcher gold standard
- Paper
- EXmatcher: Combining Features Based on Reference Strings and Segments to Enhance Citation Matching
- Authors / year
- Behnam Ghavimi, Wolfgang Otto and Philipp Mayr, 2019.
- Gold-standard size
- 816 reference strings; 517 have at least one matching Sowiport record.
- Target collection
- Sowiport social-science metadata.
- Ground truth
- Human checked, but not described as independent multi-annotator adjudication.
EXmatcher first created candidates with a simple blocker and then had a trained human assessor check the results. When no candidate existed, the assessor manually constructed searches from correctly interpreted reference fields. Wrong candidates were removed, missing matches were added and duplicate valid target records could be retained. The resulting gold standard contains 816 strings, 517 with at least one match. [11]
EXmatcher’s method is a useful ancestor of modern two-stage systems: blocking creates a manageable candidate set, then a binary classifier decides whether each reference–record pair is a match. The paper compares features derived from the unstructured string, parsed segments and their combination. The combined representation was most useful. For the “one match per reference” experiment, an SVM configuration reached precision 0.972, recall 0.926 and F1 0.948 before accounting for full-pipeline blocking loss; corresponding pipeline F1 was 0.941. [12]
How wrong/misleading references were gathered: the source strings came from real extracted citations rather than synthetic mutation. Importantly, the repository notes that extraction errors were not necessarily manually repaired before matching, making the benchmark relevant to noisy input.
Limitation: one assessor is described; this is human-curated gold truth, but the accessible methods do not describe independent double annotation plus formal disagreement adjudication. It is also tied to an older, domain-specific target database.
Cora citation matching/coreference
- Dataset
- Cora Citation Matching
- Creator
- Andrew McCallum and collaborators; classic Cora resource.
- Commonly used matching subset
- 1,295 citation strings grouped into 134 underlying papers.
- Task
- Coreference/entity resolution: decide which citation strings refer to the same publication.
- Ground truth
- Hand-clustered.
McCallum’s official data page describes the Cora citation-matching data as citation text hand-clustered into groups referring to the same paper. Later experimental descriptions of this matching corpus report 1,295 citations representing 134 papers. [13]
Do not confuse this with the other famous “Cora dataset” containing 2,708 papers and a citation network for node classification. That is a different task. The record-linkage version is useful because many noisy strings point to the same entity; however, its label answers “same work or different work?”, not necessarily “what is the canonical DOI in today’s scholarly graph?”. [14]
Limitation for your project: it is old, computer-science-heavy and lacks modern DOI/database coverage patterns. It remains useful as a coreference stress test, not as the main 2026 benchmark.
CiteTracer benchmark
- Paper
- Source or It Didn’t Happen: A Multi-Agent Framework for Citation Hallucination Detection
- System
- CiteTracer
- Authors / year
- Mingzhe Li, Zhiqiang Lin and Shiqing Ma, 2026.
- Synthetic set
- 2,450 citations retained from 3,100 collected/generated entries.
- Real-world fabricated set
- 957 citations: 807 from ICLR 2026 desk-rejected submissions and 150 from another conference.
- Ground truth
- Synthetic: known by controlled construction plus checks. Real-world: venue-chair fabrication flags, described by the paper as hallucinated “by construction”; not an independently re-adjudicated balanced set of real and fake citations.
CiteTracer defines a field-level taxonomy covering real formatting variants, ambiguous/potential cases and errors in title, authors, venue, year, identifiers and peripheral metadata. Synthetic cases begin with real BibTeX seeds; code-specific mutation operators change controlled fields, followed by round-trip checks, verifiability checks and manual boundary review for ambiguous name variants. [15]
Its pipeline uses extraction, a cascading evidence collector, deterministic field matching and specialist judgement stages. The synthetic benchmark produced 97.1% overall accuracy, with class-level F1 values of 97.0 for Real, 95.8 for Potential and 98.5 for Hallucinated. On the 957 conference-chair-flagged fabricated citations it detected 97.1% without abstaining. [16]
Strength: diagnostic field-specific error labels and genuinely difficult controlled corruptions.
Critical limitation: the 957-case real-world set contains fabricated positives, so its headline result is essentially positive-class recall. It cannot by itself tell you the false-fabrication rate on thousands of legitimate but mangled references — exactly the metric your project should emphasise. [17]
HalluCitation Matters appendix
- Paper
- HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences
- Authors / year
- Yusuke Sakai, Hidetaka Kamigaito and Taro Watanabe, 2026.
- Corpus
- 17,842 ACL/NAACL/EMNLP PDFs from 2024–2025; 741,656 extracted citations.
- Confirmed affected papers
- 295, despite the rounded “300” in the title.
- Ground truth
- Manual verification of automatic candidates at the paper level. It is not an exhaustive manual label for every reference in all papers.
This resolves an apparent discrepancy in your list: the title says “300 Hallucinated Papers”, but the paper’s result table reports exactly 295 HalluCited papers: 20 in 2024 and 275 in 2025. Their corresponding affected-paper rates were 0.28% and 2.59%; EMNLP 2025 alone contained 154 affected papers. [4]
The method is particularly relevant to matcher failure. MinerU extracts bibliography blocks, GROBID normalises them, and likely ACL/arXiv citations are compared with ACL Anthology, arXiv, DBLP and OpenAlex. A character-level normalised Levenshtein title similarity below 0.9 creates a candidate. Humans then investigate those candidates using identifiers, bibliographic details and web search. A citation is treated as HalluCitation when no corresponding work can be found or when a similar work disagrees in at least two important attributes under the authors’ conservative criteria. [4]
The paper itself explicitly notes that candidate noise arises from OCR, parsing errors and the limitations of simple fuzzy title matching. This is direct evidence for your thesis that matcher failure is a major source of false alarms. [4]
Important ground-truth nuance: once a paper has one manually verified HalluCitation, the authors stop checking the rest of that paper because their target outcome is “does this paper contain at least one?”. Consequently, the appendix is excellent for confirmed affected-paper labels but should not be treated as a complete reference-level census of every bad citation within those 295 papers. [4]
Manual Reference Verification Dataset
- Dataset
- Manual Reference Verification Dataset for Hallucinated and Suspicious Citation Detection Tools
- Creators / year
- Fidan Badalova, data collector; Philipp Mayr, supervisor; July 2026.
- Size
- 104 references from three scholarly documents.
- Labels
- 71 verified; 33 problematic.
- Ground truth
- Every reference manually verified. Independent double annotation is not stated in the public data description.
The three source documents contribute 24, 15 and 65 references. Verification considers title, authors, venue, year, DOI/identifier and evidence from databases, publishers, conferences and repositories. A reference is not called problematic merely because one database lacks it; problematic cases include non-verifiable works, identifiers pointing elsewhere and conflicting core bibliographic fields. [18]
This is especially useful for measuring false positives because it includes both valid and problematic references and stores standardised outputs from CheckIfExist, HalluCiteChecker, Hallucinator, HalRef and RefChecker. The authors explicitly warn that it is an evaluation set, not a dataset for estimating population prevalence. [18]
Version warning: the July 2026 Zenodo dataset reports 71 verified and 33 problematic references. The initial arXiv position-paper analysis used a different 84-real/20-problematic breakdown for the same total of 104 cases. Treat the Zenodo v1.0.0 labels as the released dataset state and record the exact version/DOI in experiments rather than silently combining counts from different versions. [19]
RenoBench
- Paper
- RenoBench: A Citation Parsing Benchmark
- Authors / year
- Parth Sarin, Juan Pablo Alperin, Adam Buttrick and Dione Mentis, 2026.
- Size
- 10,000 plain-text references paired with JATS XML.
- Starting pool
- Approximately 161,000 annotated citations before automated validation and feature-based sampling.
- Sources
- SciELO, Redalyc, Public Knowledge Project and Open Research Europe.
- Ground truth
- Publisher-provided structured annotations aligned by an automated pipeline, not reference-by-reference human adjudication.
RenoBench is highly valuable, but it solves the stage immediately before your central task. It evaluates citation parsing: converting a plain-text citation into structured JATS fields. Its 10,000 examples are multilingual and heterogeneous and were extracted from public-domain PDFs, then matched to publisher-provided XML annotations. [20]
How wrong references were gathered: they were not curated as fabricated-reference cases. The benchmark is built from real publisher data and noisy PDF reference strings. That makes it very useful for testing whether your parser survives realistic input corruption, but it does not supply canonical target DOI/null labels for evaluating the final resolver.
Best use in your project: use RenoBench as a parsing auxiliary benchmark, and separately test entity resolution on Crossref/EXmatcher/your author-certified corpus.
Dataset comparison
| Resource | Approximate size | What the label means | Ground-truth route | Public? | Main strength | Main weakness for your matcher |
|---|---|---|---|---|---|---|
| Crossref real benchmark | 2,000 refs | Exact DOI or null | Manual DOI verification/search | Yes | Closest real-world resolution task | Small; Crossref-specific sampling |
| Crossref framework / Marple | Not fixed | Depends on supplied evaluation set | Framework infrastructure | Yes | Reusable matching/evaluation architecture | Not itself a gold-standard corpus |
| EXmatcher | 816 refs | Sowiport matching record(s) | Trained human assessor | Yes | Real noisy matching with blocking | Older, domain-specific target database |
| Cora citation matching | 1,295 refs / 134 entities | Which strings refer to the same work | Hand clustering | Yes | Classic entity-resolution test | No modern canonical DOI/null target |
| CiteTracer synthetic | 2,450 | Field-specific Real/Potential/Hallucinated code | Controlled mutation + checks | Yes | Excellent diagnostic error taxonomy | Synthetic corruption distribution |
| CiteTracer real fabricated set | 957 | Fabricated | Venue-chair flags / desk rejects | Yes | Real conference failures | Positive-only; cannot estimate false-positive rate |
| HalluCitation Matters | 17,842 papers; 295 confirmed affected | Paper has at least one HalluCitation | Automatic candidates + manual verification | Paper appendix available | Large real ACL-family audit | Not exhaustive reference-level labels |
| Manual Verification Dataset | 104 refs | Verified / problematic | Manual verification | Zenodo | Direct false-positive evaluation | Very small; only three documents |
| RenoBench | 10,000 refs | Correct JATS parsing | Publisher XML + automated alignment/validation | Hugging Face | Multilingual real-world parsing | Not canonical record resolution |
| Proposed author-certified corrections | To be collected | Original damaged citation → author-confirmed intended record/correction | Author certification | Project-dependent | Directly attacks matcher-failure false positives | Selection bias; needs null/fabricated controls |
Dataset sizes and ground-truth descriptions in this comparison follow the corresponding primary resource descriptions above. [21]
Reference-error and reference-matching research
Logan et al. — Reference accuracy of primary studies published in peer-reviewed scholarly journals
- Authors / year
- Samuel W. Logan, Uta Hussong-Christian, Layne Case and Samantha Noregaard; 2024 print issue, with earlier online publication.
- Study type
- Scoping review.
- Included studies
- 105 primary reference-accuracy studies.
- Reported general error rate
- 32.7% in the reference-accuracy literature synthesised by the review.
- Ground truth
- Heterogeneous. The review synthesises studies with their own verification methods; it is not one item-level gold-standard corpus.
This should indeed be read early because it establishes that ordinary bibliographic errors long predate generative AI. The review gathered primary studies that actually verified references and found a substantial error burden; a later study by the same research group explicitly describes the earlier review as containing 105 studies and reports the review’s general reference-accuracy error rate as 32.7%. Author names and titles are prominent error categories. [22]
Why it matters for matching: an error rate in bibliographic fields is not a fabrication rate. A valid work with a misspelt author or title is still a real work. A detector should therefore be trained and evaluated on the distinction between metadata corruption and non-existence.
Limitation: definitions and verification protocols vary substantially across the historical literature; combining them into one percentage does not create a modern resolution benchmark.
Broadus — An Investigation of the Validity of Bibliographic Citations
- Author / year
- Robert N. Broadus, 1983.
- Question
- Do authors sometimes copy a bibliographic citation from another citing publication without consulting the original?
- Ground truth
- Manual bibliographic investigation, not a released machine-learning benchmark.
Broadus examined recurring errors in references to Edward O. Wilson’s book and W. D. Hamilton’s papers as a way of testing the hypothesis that writers sometimes reproduce citations from other papers rather than consulting the originals. The ERIC record describes this design directly. [23]
A later analysis of this line of research reports a striking example: 23% of citing papers reproduced the same erroneous title in the examined citation lineage. This figure is useful as historical evidence that “citation hallucination-like” anomalies can propagate socially even without LLMs, but it is reported here through later literature rather than a directly machine-readable Broadus results table. [24]
Matcher lesson: repeated identical errors are not proof that the cited work is fake; citation strings can copy one another’s damage.
de Lacey, Record and Wade — How accurate are quotations and references in medical journals?
- Authors / year
- G. de Lacey, C. Record and J. Wade, 1985.
- Venue
- British Medical Journal, volume 291.
- Ground truth
- Manual reference/quotation checking in the medical literature.
There is an attribution error in the original reading list: the 1985 BMJ paper is by de Lacey, Record and Wade, not Evans et al. The BMJ record confirms the title, publication details and authorship. [25]
Its importance here is conceptual: it belongs to the long literature separating reference accuracy and quotation accuracy. A reference can point to a genuine paper yet contain bibliographic mistakes; conversely, a perfectly formatted citation can be used to support a claim that the source does not actually make. Those are separate validation layers.
Quantitative result: detailed numeric outcomes are marked unspecified in this review rather than inferred from later summaries, because the primary HTML record retrieved for this research did not expose the full article’s tables in a form suitable for reliable extraction.
Scheinfeld et al. — Assessing Formatting Accuracy of APA Style References
- Authors / year
- Laurel Scheinfeld, Sunny Chung, Christine Fena, Clara Tran, Chris Kretz and Myra R. Reisman, 2025.
- Study
- Scoping review of APA reference-formatting accuracy.
- Included studies
- 32.
- Ground truth
- Study-level synthesis of formatting assessments; not a record-resolution gold standard.
The review’s goal is to synthesise how APA formatting accuracy has been defined and measured, identify common errors and assess reproducibility and assessment tools. The review includes 32 studies. [26]
Why it matters: it reinforces a critical distinction for your model: punctuation, capitalisation, italicisation and APA layout violations should normally have almost zero weight in deciding whether the underlying scholarly object exists. A resolver should normalise superficial formatting aggressively while remaining strict about identity-bearing evidence.
Aggregate numeric formatting-error result: no single cross-study percentage is reported here because the review studies use heterogeneous definitions; treating them as one pooled rate would be misleading.
Tkaczyk et al. — Machine Learning vs. Rules and Out-of-the-Box vs. Retrained
- Authors / year
- Dominika Tkaczyk, Andrew Collins, Paraic Sheridan and Joeran Beel, 2018.
- Task
- Bibliographic reference parsing, not DOI resolution.
- Systems
- Ten open-source parsers.
- Ground truth
- Task-specific labelled parsing data; this paper is not a fabricated-reference dataset.
The study compared Anystyle-Parser, Biblio, CERMINE, Citation, Citation-Parser, GROBID, ParsCit, PDFSSA4MET, Reference Tagger and Science Parse. The strongest out-of-the-box parser was GROBID at F1 0.89, followed by CERMINE at 0.83 and ParsCit at 0.75. Machine-learning systems and non-ML systems had similar average precision, 0.77 versus 0.76, but ML systems had much higher average recall, 0.66 versus 0.22. Retraining improved GROBID from 0.89 to 0.92, CERMINE from 0.83 to 0.92 and ParsCit from 0.75 to 0.87. [27]
Lesson: domain adaptation matters. If your references come from author corrections, OCR, XML, multilingual journals or a particular publishing workflow, a parser trained elsewhere should not automatically be treated as optimal.
Even more important: parser F1 is not matcher F1. A system can parse a title imperfectly but still resolve the correct DOI using the raw string; conversely, perfect parsing does not guarantee that the target database contains the right record.
Fedoryszak et al. — Methodology for Evaluating Citation Parsing and Matching
- Authors / year
- Mateusz Fedoryszak, Łukasz Bolikowski, Dominika Tkaczyk and Krzysztof Wojciechowski, 2013.
- Contribution
- Evaluation methodology for reference parsing and matching, applied to the YADDA2 platform.
- Ground truth
- Evaluation methodology rather than a single widely used released gold dataset.
This work is useful less for a headline score than for how it frames evaluation. Citation matching can be evaluated as grouping/coreference or as resolution against records, and parsing and matching should not be collapsed into one opaque end metric. The chapter develops such a methodology and applies it to the YADDA2 reference-processing module. [28]
Numeric result: unspecified here; the value for your project is methodological rather than a particular number.
Crossref — Reference Matching: For Real This Time
The Crossref work has already been described in the dataset section because it is both a benchmark and a matching study. The key result remains one of the most relevant empirical baselines in this whole literature: SBMV F1 0.9629 versus 0.9251 for the legacy approach on real unstructured strings, with the main gain coming from improved recall. [1]
EXmatcher
Likewise, EXmatcher is both a paper and a released gold standard. The central modelling result is that the best system combined evidence from the raw reference string and parsed segments; its reported one-match SVM experiment reached F1 0.948 before full-pipeline blocking effects and 0.941 at pipeline level. [12]
Sefid et al. — Cleaning Noisy and Heterogeneous Metadata for Record Linking Across Scholarly Big Datasets
- Authors / year
- Athar Sefid, Jian Wu, Allen C. Ge, Jing Zhao, Lu Liu, Cornelia Caragea, Prasenjit Mitra and C. Lee Giles, 2019.
- Source data
- CiteSeerX matched against Web of Science, PubMed and DBLP.
- Method
- BM25 candidate blocking in Elasticsearch plus supervised matching features from multiple metadata fields; citation information is also incorporated.
- Ground truth
- Evaluation test data used for record linkage; details of independent human adjudication are not specified in the accessible abstract record used here.
The important innovation is the same architectural principle that keeps reappearing: do not demand that a noisy title be perfectly cleaned before searching. First use BM25 to generate candidates, then let a supervised model combine the evidence contained in all available fields; citation relationships can provide additional evidence. The authors report that metadata plus citation evidence significantly outperformed their title-based baseline, although the exact quantitative table is marked unspecified here because it was not exposed in the primary HTML abstract retrieved during this review. [29]
2026 lesson: candidate retrieval should optimise recall; the expensive model should decide among a small number of candidates afterwards.
LLM-era fabricated-citation research
The LLM-era studies are easy to misread because they use different definitions. Some count only completely non-existent works. Some count severe author mismatch. Some treat a wrong DOI or year as a hallucination. Some report candidate matches before manual checking; others manually review every flagged case. Comparing prevalence percentages without reading these definitions is therefore unsafe.
Topaz et al. — Fabricated citations: an audit across 2.5 million biomedical papers
- Authors / year
- Maxim Topaz, Nir Roguin, Pallavi Gupta, Zhihong Zhang and Laura-Maria Peltonen, 2026.
- Venue
- The Lancet.
- Corpus
- Nearly 2.5 million PubMed Central Open Access papers, January 2023 to 18 February 2026.
- Verified-reference pool
- 97.1 million references with verifiable identifiers in the reported analysis.
- Fabricated references found
- 4,046 across 2,810 papers.
- Ground truth
- Automated/AI-assisted verification with multi-source checking; public summaries do not describe a benchmark-style independent double-adjudication protocol for all 97.1 million references.
The audit’s headline trend is striking: fabricated-reference prevalence was approximately four per 10,000 papers through 2023 and rose to roughly 57 per 10,000 by early 2026. The official Columbia summary reports 4,046 fake citations in 2,810 papers among 97.1 million verified references and a more than twelve-fold increase over the period. [30]
The workflow applied automated filters to eliminate formatting/title artefacts and then checked suspicious references against sources including PubMed, Crossref, OpenAlex and Google Scholar. The definition centres on a cited title corresponding to no real publication, making it narrower than “any metadata field is wrong”. [31]
Why it matters to your project: even a very low false-positive rate matters when processing nearly 100 million references. A matcher that is only 99.9% specific can still generate huge review burdens at this scale.
Limitation: this is a prevalence audit, not a balanced public matcher benchmark. It also does not prove that each fabricated citation was generated by an LLM; temporal association is not individual-level causal evidence. [32]
Zhao et al. — LLM hallucinations in the wild
- Authors / year
- Zhenyue Zhao, Yihe Wang, Toby Stuart, Mathijs De Vaan, Paul Ginsparg and Yian Yin, 2026.
- Corpus
- 111 million references across 2.5 million papers from arXiv, bioRxiv, SSRN and PubMed Central.
- Estimated 2025 hallucinated citations
- 146,932.
- Ground truth
- Not a per-reference manually adjudicated benchmark. The paper statistically separates excess unmatched references from a pre-LLM bibliographic-error baseline.
This study asks a different question from Crossref or EXmatcher. References are matched against bibliographic sources, but the authors then model unmatched citations as a mixture of ordinary historical matching/bibliographic failures and a post-LLM excess component. The abstract reports a conservative estimate of 146,932 hallucinated citations during 2025. [34]
A contemporaneous methodological review reports the paper’s estimated 2025 rates as 0.39% for arXiv, 0.21% for bioRxiv, 1.91% for SSRN and 0.27% for PubMed Central. [35]
Why this matters enormously for your work: “unmatched” is the raw observation; “LLM-hallucinated” is an inference after modelling a baseline. Better matching changes the unmatched pool and therefore can change the inferred prevalence. This is almost the purest illustration of why matcher error must be separately quantified.
Limitation: a statistical excess estimate cannot tell you with adjudicated certainty whether a particular individual unmatched citation is fabricated.
Sakai et al. — HalluCitation Matters
As described above, HalluCitation Matters used a deliberately high-recall screening stage followed by manual verification. The authors found 295 affected papers overall, with the proportion rising from 0.28% in 2024 to 2.59% in 2025. Crucially, they explicitly report that single-candidate cases frequently arise from OCR, parsing and fuzzy-matching noise. [4]
That makes HalluCitation Matters one of the strongest empirical motivations for your proposed study: the detector itself produces a candidate pool containing matcher failures, and manual verification must rescue real citations from that pool.
GhostCite
- Paper
- GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Models
- Authors / year
- Zuyao Xu and collaborators, 2026.
- LLM experiment
- 13 models across 40 computer-science domains.
- Archive audit
- 2.2 million citations from 56,381 papers, 2020–2025.
- Confirmed affected papers
- 604 papers, or 1.07% of the audited papers.
- Ground truth
- Strong manual adjudication of flagged archival citations: 16 trained research assistants; each flagged citation checked independently at least twice.
GhostCite’s CiteVerifier uses a cascading verification process rather than relying on one database. The study reports model-generated citation hallucination rates ranging from 14.23% to 94.93% across the tested LLM/domain combinations. In the archival audit, 1.07% of 56,381 papers contained an invalid or fabricated citation, with an 80.9% increase reported in 2025. [36]
The strongest part of its ground truth is the manual stage. Sixteen trained research assistants reviewed 2,530 automatically flagged citations over approximately a month, and each was independently checked at least twice before classification as non-academic source, valid or invalid. [37]
Strength: this is far closer to real adjudication than simply declaring every unmatched reference fabricated.
Limitation: manual review concentrates on automatically flagged cases, so the quality of the candidate-generation stage still matters for estimating missed invalid references. Its “invalid” category is also not identical to every other paper’s definition of “fabricated”.
CiteAudit
- Paper
- CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era
- Authors / year
- Zhengqing Yuan, Kaiwen Shi, Zheyuan Zhang, Lichao Sun, Nitesh V. Chawla and Yanfang Ye, 2026, according to the current arXiv record.
- Benchmark
- Current 2026 versions describe a large human-validated benchmark incorporating synthetic and real-world reference cases.
- Ground truth
- Human validated.
The accessible 2026 manuscript version describes generated and real-world portions with 6,475 real and 2,967 fake entries overall, totalling 9,442, and a human-in-the-loop process in which retrieved evidence is manually cross-checked by the author team. Its synthetic component deliberately perturbs fields such as title, authors, venue, year and DOI; the real-world component draws from scholarly platforms and manuscripts. [38]
The current arXiv abstract describes a multi-agent verification pipeline spanning extraction, evidence retrieval, matching/reasoning and calibrated judgement, and characterises the dataset as human validated. [39]
Strength: unlike positive-only fabricated sets, CiteAudit contains both real and fake examples and therefore supports specificity/false-positive analysis.
Version caveat: the arXiv manuscript evolved during 2026; freeze and report the exact paper/dataset commit used for any comparison rather than assuming that all online descriptions refer to an identical snapshot.
Phantom References
- Authors / year
- Mark Russinovich, Ram Shankar Siva Kumar and Ahmed Salem, 2026.
- System
- RefChecker.
- Corpus
- 48,095 accepted papers and 2,614,992 extracted references from ICLR, ICML, NeurIPS and USENIX Security across the studied years.
- Definition
- Non-existent works or substantial author-identity corruption; ordinary year/venue/publication-status drift is excluded.
- Ground truth
- Evidence-based automated audit with web-search escalation; results are intentionally called “likely hallucinated”, not a fully manually adjudicated corpus.
RefChecker searches Semantic Scholar, OpenAlex, Crossref, DBLP, ACL Anthology and identifiers/URLs, then escalates suspicious cases to constrained web-search verification. The authors preserve separate verified, error, warning, unverified and hallucination statuses; only cases surviving the stricter hallucination route are counted in their prevalence estimate. [8]
In 2025, its all-reference likely-hallucination rates were reported as 0.38% for ICLR, 0.54% ICML, 0.68% NeurIPS and 0.81% USENIX Security. When the denominator is restricted to academic-paper-like references, the corresponding values are 0.31%, 0.46%, 0.55% and 0.42%. The paper also reports that about one in twenty 2025 NeurIPS and USENIX Security papers contained at least two likely hallucinated academic-paper-like references. [8]
Yet the paper is unusually candid about the central limitation: PDF extraction noise, stale or incomplete databases and model behaviour can create both false positives and false negatives, and the authors explicitly describe their numbers as an auditable estimate rather than an exact count. [8]
That sentence should be central to your project. A system can have a sophisticated web-search agent and still need an external benchmark specifically designed to measure whether genuine damaged references are being mislabelled.
Badalova and Mayr — What Current Tools Can and Cannot Do
- Paper
- Detecting Hallucinated and Suspicious Citations: What Current Tools Can and Cannot Do
- Authors / year
- Fidan Badalova and Philipp Mayr, 2026.
- Tools compared
- CheckIfExist, HalluCiteChecker, Hallucinator, HalRef and RefChecker.
- Evaluation scale
- 104 references.
- Ground truth
- Manual verification; released companion Zenodo data now contains 71 verified and 33 problematic items.
This is possibly the single most directly supportive paper for your project’s motivation. Its empirical finding is not merely “tools miss fake references”; rather, current detectors generate many false positives because extraction, matching, database coverage and interpretation can fail. The authors therefore argue for caution and manual verification rather than treating automated detector output as ground truth. [40]
The companion Zenodo dataset strengthens reproducibility because it exposes the individual reference labels and standardised tool outputs. [18]
Best role in your evaluation: treat it as a small but very important “false accusation” benchmark. A new matcher should sharply reduce valid references being flagged while preserving detection of the problematic references.
Glynn — Analysis of the Academ-AI Dataset
- Paper
- Suspected Undeclared Use of Artificial Intelligence in the Academic Literature: An Analysis of the Academ-AI Dataset
- Author / year
- Alex Glynn, 2024.
- Analysed examples
- First 500 Academ-AI cases.
- Ground truth
- Not a fabricated-citation ground-truth dataset. Cases are suspected undeclared AI use identified mainly through tell-tale textual artefacts.
Academ-AI is related contextual evidence, not a reference-matching benchmark. Glynn analyses 500 examples of suspected undeclared AI-assisted writing identified through characteristic chatbot language appearing in published scholarship. The paper emphasises that these visible examples are likely a small and highly selected subset of undeclared AI use. [41]
Relevance to citations: it demonstrates how generated artefacts can enter formal scholarship and survive publication processes, but it should not be used as ground truth for citation existence or matching accuracy.
What all of this evidence means
The literature is actually studying several different tasks
| Layer | Question | Representative resources | Typical failure |
|---|---|---|---|
| Extraction | Did we find the bibliography entry in the PDF? | HalluCitation pipeline, GhostCite, RefChecker | Two references merged, line lost, OCR corruption |
| Parsing | Which tokens are authors/title/year/etc.? | Tkaczyk parser study, RenoBench | Title accidentally parsed as venue |
| Candidate retrieval | Did the true work enter the candidate list? | Crossref SBMV, EXmatcher, Sefid et al. | True match never retrieved |
| Record resolution | Which candidate is the intended work? | Crossref, EXmatcher, Cora | Book matched instead of chapter |
| Bibliographic validation | Which fields disagree with the canonical record? | CiteTracer, CiteAudit, Manual Verification Dataset | Wrong DOI but otherwise real citation |
| Existence / fabrication | Does a corresponding scholarly work exist? | Topaz, HalluCitation, GhostCite, Phantom | Database miss mistaken for non-existence |
| Claim support | Does the cited paper actually support the sentence? | Separate citation-faithfulness problem | Real citation attached to unsupported claim |
The reviewed work makes clear that errors upstream propagate downstream. HalluCitation Matters reports OCR/parsing and fuzzy-title failures among its candidate false alarms; Crossref reports errors from noisy strings and incomplete target metadata; Phantom References explicitly acknowledges database and extraction error as sources of false positives; and the July 2026 manual audit is designed specifically around tools incorrectly flagging genuine references. [42]
Why “not found” is not a scientific ground truth
A bibliographic database is an observation system, not the universe of scholarship. Records may be missing, duplicated, partially populated, transliterated differently or separated into preprint and published versions. Even Crossref’s own benchmark identifies incomplete/incorrect target metadata as a cause of matcher failure. [1]
This means the following inference is invalid:
Crossref returned nothing ↓ therefore the reference is not real
A more defensible inference is:
Crossref returned nothing ↓ search additional structured sources ↓ try spelling-robust and author-aware retrieval ↓ check identifiers and version relationships ↓ perform evidence-constrained web search if necessary ↓ if still unresolved, output "unverified" ↓ only call "likely fabricated" under a stricter evidence rule
The strongest recent tools increasingly follow this distinction. RefChecker has an explicit unverified state; CiteTracer has a Potential class; GhostCite manually reviews flagged references; and HalluCitation Matters manually verifies candidates rather than automatically treating its title-match threshold as final truth. [43]
Why author-certified corrections are interesting
Human adjudicators usually work backwards: they see a damaged reference and try to infer what the author meant. An author-certified correction provides a different form of evidence:
original published reference + author's correction / corrected reference ↓ certified intended identity ↓ real positive pair: damaged string → actual scholarly object
This is unusually useful for testing false-fabrication behaviour. Consider a detector that says 95% of fabricated citations are caught. That sounds excellent. But suppose it also labels 3% of heavily mangled genuine citations “fabricated”. In a real journal where genuinely fabricated citations are rare and genuine citations are abundant, those false alarms can dominate the review queue. This is the classic reason precision and base rates matter more than recall alone.
But author certification is not automatically perfect ground truth. Authors can submit an incorrect correction, different authors may disagree about the intended version, or a correction may change more than bibliographic spelling. Store the evidence trail, require a canonical record/identifier where possible, preserve both original and corrected strings and manually audit a random subset. Most importantly, add genuine null/non-existent references from other adjudicated resources so that the system cannot learn that every difficult string must have a match.
A timeline of the methodological shift
timeline title From citation errors to large-scale hallucination audits 1983 : Broadus studies propagation of erroneous bibliographic citations 1985 : de Lacey, Record and Wade audit quotations and references 2013 : Fedoryszak et al. formalise parsing/matching evaluation 2018 : Tkaczyk et al. compare parsers : Crossref evaluates 2,000 real unstructured references 2019 : EXmatcher releases manually checked matching gold standard : Sefid et al. combine BM25 blocking, metadata and citation evidence 2024 : Logan et al. synthesise reference-accuracy literature : Glynn analyses Academ-AI 2025 : APA formatting-accuracy scoping review 2026 : Topaz et al. biomedical audit : Zhao et al. 111M-reference unmatched-citation study : HalluCitation Matters : GhostCite : CiteAudit : CiteTracer : Phantom References / RefChecker : RenoBench : Badalova-Mayr manual verification dataset
A rigorous evaluation protocol for a 2026 matcher
The model architecture I would build
Recommended system: a hybrid multi-stage scholarly entity resolver, with deterministic identifier logic, high-recall heterogeneous retrieval, a learned field-aware reranker, explicit publication-version modelling, calibrated confidence and a separate evidence-escalation system for fabrication decisions.
Canonical knowledge layer
Build a periodically refreshed canonical record graph from complementary sources rather than choosing one “master database”. Crossref is essential for DOI metadata; arXiv and ACL Anthology cover important preprint/NLP cases; DBLP is valuable in computer science; PubMed/Europe PMC matter for biomedicine; OpenAlex provides broad scholarly graph coverage; DataCite is important for non-Crossref DOI objects. Modern auditors already obtain robustness by combining several such sources. [44]
Represent version families: preprint, conference paper, accepted manuscript, journal version, chapter and parent book/proceedings should not be treated as unrelated records. Crossref’s observed “book instead of chapter” error and Phantom’s deliberate tolerance of publication-version drift show why this matters. [45]
Reference understanding layer
Keep two representations simultaneously:
Raw representation: exact original string, character n-grams and an embedding of the entire citation.
Structured representation: parsed title, author sequence, venue, year, pages, volume, issue, DOI, PMID, arXiv ID and URLs.
Do not force all subsequent matching through parsed fields. EXmatcher’s best results came from combining string and segment evidence, while Crossref’s successful search-based strategy was designed to survive cases where the parser lacked enough clean structure. [46]
Candidate generation layer
Use several retrievers in parallel and fuse their candidate lists:
Lexical BM25: excellent when several important words remain intact; Sefid et al. explicitly used BM25 for blocking. [47]
Character n-gram/fuzzy retrieval: robust to OCR and misspellings.
Dense multilingual retrieval: useful for transliteration and reordered wording, but never use it alone because semantically similar papers make dangerous false-positive candidates.
Author-focused retrieval: especially valuable when title OCR is poor.
Identifier retrieval: DOI/PMID/arXiv lookup as a high-value candidate generator, followed by metadata consistency checking rather than blind acceptance.
Measure candidate recall@k separately. If the true paper is absent from the top candidate pool, no reranker can repair the error.
Field-aware reranker
For each candidate, generate interpretable compatibility evidence:
{ "title_similarity": ..., "title_character_similarity": ..., "author_overlap": ..., "author_order_consistency": ..., "year_delta": ..., "venue_compatibility": ..., "page_overlap": ..., "doi_consistency": ..., "version_family_match": ..., "source_agreement_count": ... }
Feed these features plus text pairs into a cross-encoder or comparable learned record-linkage model. Train with hard negatives such as:
same first author + different paper same/similar title + different year conference version vs journal version book vs chapter correct title + deliberately wrong DOI paper A title + paper B authors near-duplicate proceedings records
These are much more useful negatives than randomly sampled unrelated papers because the real danger is not failing to distinguish “quantum physics” from “Victorian poetry”; it is choosing the wrong one of two very similar scholarly records.
Calibration and abstention
The production output should not be binary. Use at least:
MATCHED_HIGH_CONFIDENCE MATCHED_LOW_CONFIDENCE AMBIGUOUS_MULTIPLE_CANDIDATES UNVERIFIED LIKELY_FABRICATED
For a high-stakes research-integrity system, calibrate thresholds on held-out real data. A model with slightly lower raw recall but a trustworthy abstention region can be far safer than a nominally higher-accuracy system that confidently labels difficult valid references as fabricated. RefChecker’s multi-status design and CiteTracer’s Potential class already move in this direction. [48]
Evidence escalation
Only unresolved citations should invoke expensive search/agent stages. Check additional scholarly indices, publisher pages and repository records, then use web-search agents to find evidence. Crucially, the agent should return source evidence, not merely a yes/no judgement. GhostCite and RefChecker both use escalation after structured-source matching rather than using an LLM as the sole authority. [49]
Datasets to use in the experiment
A convincing 2026 paper should report separate results on several strata rather than merge them into one score:
| Evaluation stratum | Resource | What it tests |
|---|---|---|
| Real DOI resolution | Crossref 2,000 | Messy string → true DOI/null |
| Real domain-specific resolution | EXmatcher 816 | Noisy reference → target scholarly record |
| Coreference robustness | Cora | Whether variants of one citation cluster together |
| Parser robustness | RenoBench 10,000 | Whether upstream field extraction is reliable |
| Controlled corruption | CiteTracer 2,450 | Specific title/author/year/DOI failure modes |
| Human-verified false-alarm test | Manual Verification Dataset 104 | Valid versus problematic references |
| Real fabricated positives | CiteTracer 957 | Recall on chair-flagged fabrications |
| Real audit replication | HalluCitation / GhostCite subsets where obtainable | Performance on genuine publication workflow failures |
| Primary new contribution | Author-certified corrections | Can a matcher recover genuinely real references even when the published string was wrong? |
These sizes and task definitions come from the corresponding primary resource descriptions reviewed above. [50]
Metrics that should be reported
Exact resolution accuracy
For references that have one known canonical target:
Exact resolution accuracy = references assigned the correct target all resolvable references
Use exact canonical identifiers after normalising equivalent DOI URL forms. Where multiple records represent legitimate versions, publish both a strict exact-record metric and a version-family metric.
Precision, recall and F1 of produced links
This preserves comparability with Crossref:
Precision = correct returned links / all returned links Recall = correct returned links / all true links F1 = harmonic mean of precision and recall
Crossref’s SBMV provides a concrete historical baseline: precision 98.09%, recall 94.56%, F1 96.29% on its 2,000-reference benchmark. [1]
Candidate recall at k
Report whether the true item appears among the first 1, 5, 10 and other pre-registered candidate depths. This cleanly separates retrieval failure from reranker failure.
False fabrication rate on genuine references
This should be a headline metric in your paper:
False fabrication rate = genuine references labelled "fabricated" all genuine references
Report it separately for ordinary clean references and author-certified mangled references. The latter is your core contribution.
Fabrication recall
Fabrication recall = true fabricated references correctly flagged all adjudicated fabricated references
The 957 CiteTracer real-world fabricated cases are useful here, but because they are positive cases they must not be the only benchmark. [17]
Precision of a fabrication accusation
Fabrication precision = true fabricated among items labelled fabricated all items labelled fabricated
This metric is ethically important. A false negative means a bad citation may escape detection; a false positive means a real author/source is being labelled as fabricated. The latter can have reputational consequences, so a production verifier should be precision-first and able to abstain.
Risk–coverage curve
Instead of publishing one arbitrary threshold, show what happens as the system becomes more selective. At 100% coverage it must decide everything; at lower coverage it can send uncertain cases to review. Plot matching error versus percentage automatically resolved.
Calibration
When the system says “0.99 probability of a match”, approximately 99% of such predictions should actually be correct. Report a calibration metric such as Brier score and/or expected calibration error, alongside reliability plots.
Per-error-type metrics
Report performance separately for title corruption, author corruption, wrong year, wrong venue, incorrect DOI, missing DOI, OCR noise, transliteration, merged references, book/chapter confusion and preprint/published-version differences. CiteTracer’s taxonomy gives a useful starting point. [7]
Operational cost
Measure database calls, latency and the proportion reaching expensive web/LLM escalation. Phantom References reports that one venue-scale audit cost roughly four US cents per paper under the paper’s May 2026 API-pricing configuration, demonstrating why escalation rather than full web search for every citation is operationally attractive. [8]
Experimental protocol
Freeze every dataset version before development. This matters particularly for fast-moving 2026 repositories and for the Manual Verification Dataset, whose released Zenodo composition differs from the initial preprint table. [19]
Split by underlying scholarly work, not merely by citation string. Otherwise two damaged variants of the same paper can leak into train and test, creating unrealistically easy evaluation.
Freeze time-sensitive bibliographic snapshots where possible. A match that was impossible because a target was absent from a database in May might become easy after metadata is added in August.
Do not tune thresholds on the test datasets. Crossref explicitly selected matcher thresholds on a separate dataset; follow the same discipline. [1]
Blind the final author-certified test set. Construct training/dev material first, lock the evaluation IDs, then reveal the test labels only once the pipeline is fixed.
Report end-to-end and component metrics. At minimum: extraction success, parsing, candidate recall, reranker accuracy, exact resolution, abstention, and fabrication classification.
Manually investigate disagreements. For every system-vs-ground-truth disagreement in a manageable test set, record whether the error came from extraction, parsing, target-database coverage, candidate retrieval, ranking, canonicalisation/version logic or bad benchmark ground truth. This error taxonomy will probably be more scientifically valuable than another aggregate accuracy score.
Baselines
The evaluation should contain at least a deliberately simple lexical baseline, a Crossref-like bibliographic search baseline, a fuzzy-title threshold resembling HalluCitation’s screening stage, a learned EXmatcher/Sefid-style record-linkage model, and currently runnable multi-source verification systems such as CiteTracer/RefChecker where licensing and APIs permit fair reproduction. Crossref legacy/SBM/SBMV results should be quoted as historical comparators on the original benchmark rather than assumed to transfer unchanged to a new corpus. [51]
What effect size should you expect?
It would be scientifically inappropriate to invent a predicted percentage improvement before running your author-certified benchmark. The existing literature gives useful observed precedents, not a guaranteed effect for your data.
On Crossref’s real strings, moving from the legacy matcher to SBMV increased F1 from 0.9251 to 0.9629, an absolute gain of 0.0378; recall rose from 0.8685 to 0.9456 while precision declined slightly from 0.9895 to 0.9809. [1]
EXmatcher’s best combined string-plus-segment representation reached pipeline F1 0.941 in its reported one-match setting, supporting the idea that heterogeneous evidence helps. [12]
CiteTracer reaches very high scores on its controlled synthetic benchmark, but those values should not be used as the expected accuracy on naturally mangled author-certified citations because the data-generating processes differ. [7]
The most defensible hypothesis to pre-register is therefore directional:
Primary hypothesis: multi-source, field-aware, calibrated matching will reduce the proportion of author-certified genuine references falsely labelled unverified/fabricated relative to title-only, single-database and uncalibrated baselines, while preserving high exact-resolution precision.
Report the actual absolute risk reduction, relative risk reduction and confidence interval after the blind test. Do not select whichever statistic looks largest after seeing the results.
Final summary and FAQ
Final summary
The literature supports a fairly coherent story once parsing, matching and fabrication detection are separated.
First, reference errors are old. Broadus, de Lacey and decades of studies synthesised by Logan show that bibliographic inaccuracies existed long before modern LLMs. A malformed reference is therefore not inherently evidence of AI fabrication. [52]
Second, matching real references is already difficult. Crossref’s carefully verified 2,000-reference benchmark demonstrates that search-plus-validation can substantially improve recall, yet even a strong F1 of 0.9629 leaves errors. EXmatcher reaches the same architectural conclusion from another direction: combine raw strings and structured fields rather than trusting one representation. [6]
Third, the LLM-era phenomenon is real enough to warrant serious checking. Topaz et al., Zhao et al., HalluCitation Matters, GhostCite and Phantom References independently report rising or clearly detectable invalid/non-existent reference problems across biomedical, preprint and conference corpora, although their definitions and adjudication methods differ substantially. [53]
Fourth, matcher failure contaminates fabrication detection. HalluCitation explicitly observes OCR/parsing/fuzzy-match noise; RefChecker warns about false positives from extraction and database coverage; Crossref documents real matching failures caused by imperfect strings and metadata; and Badalova and Mayr show that current detectors can flag many valid references. [54]
That is why an author-certified correction benchmark is useful. It targets an important stratum that positive fabricated-citation datasets do not: “we know this awful-looking reference is real, and we know what the author intended it to point to.” The strongest 2026 paper would combine that new stratum with Crossref, EXmatcher, Cora, RenoBench, CiteTracer, the Manual Verification Dataset and adjudicated fabricated cases, then evaluate the full error cascade from PDF extraction through final fabrication verdict.
FAQ: How do I build the best reference-matching model in 2026?
Build a hybrid retrieval-and-reranking system, not a monolithic LLM prompt. Preserve the raw citation, parse it into fields, resolve identifiers, retrieve candidates with BM25 + character-level matching + dense retrieval from multiple scholarly databases, fuse candidates, rerank with a model that understands title/author/year/venue/identifier evidence and publication versions, calibrate its confidence and provide an abstention state. Escalate only unresolved cases to additional databases and evidence-constrained web search. This architecture follows the strongest recurring ideas in Crossref SBMV, EXmatcher, Sefid et al., CiteTracer, GhostCite and RefChecker. [55]
For training, favour hard real errors and hard negatives. Synthetic corruptions are useful for coverage, but they should supplement—not substitute for—naturally damaged references and blind real-world evaluation.
FAQ: How do I reduce false fabrication flags?
The single most important rule is:
Failure to match is a reason to investigate, not a fabrication label.
Use a three-stage interpretation: matched → unresolved/ambiguous → likely fabricated. The fabricated label should require stronger negative evidence than the unresolved label. Search several independent sources, test alternative spellings, handle transliteration, tolerate year/venue version drift, verify whether an incorrect DOI simply points to a different real object, and look for parent/child publication relations. RefChecker’s distinction between unverified and hallucination and CiteTracer’s Potential category are useful precedents. [48]
Then optimise and publish the false fabrication rate on author-certified real references. A detector should not be considered safe merely because it catches most synthetic fabrications.
FAQ: Should I train a giant language model from scratch?
No. The evidence reviewed here gives no reason to believe that training an end-to-end foundation model from scratch is the efficient path. The hard problem is not simply language understanding; it is retrieving and reconciling external bibliographic evidence. Crossref’s search-based approach, EXmatcher’s blocking/classification pipeline, Sefid et al.’s BM25 blocking, CiteTracer’s retrieval cascade and RefChecker’s multi-source verification all benefit from explicit retrieval infrastructure. [56]
Fine-tune relatively small specialised components where labelled data justifies it, and use large models where they add unique value—noisy parsing, difficult semantic comparison or constrained evidence-search escalation.
FAQ: Should DOI matching be the first thing I do?
Yes, but do not treat it as infallible. A DOI that resolves and whose metadata agrees strongly with the citation is nearly ideal evidence. A DOI that resolves to a completely different title should be classified as an identifier conflict and trigger candidate recovery. The Manual Reference Verification Dataset explicitly treats a DOI that points to another publication as a problematic-reference mode. [18]
FAQ: Is title similarity enough?
No. HalluCitation Matters uses a 0.9 normalised-Levenshtein title threshold as a screening mechanism, but its authors then manually verify candidates and report noise arising from fuzzy matching and extraction. EXmatcher found better performance from combining string and field evidence, and Crossref’s SBMV separately validates retrieved candidates. [57]
FAQ: Which dataset should be the primary benchmark?
For existing direct resolution ground truth, start with the Crossref 2,000-reference benchmark because it most closely asks “what DOI does this real messy string mean?”. Add EXmatcher for another manually checked resolution environment. [58]
For your new contribution, make the author-certified correction corpus the main stress-test stratum and report Crossref/EXmatcher as established comparators. Add the Manual Verification Dataset to test false flags, CiteTracer to test controlled error modes and genuine fabrications, RenoBench to isolate parser quality, and Cora to test coreference robustness. [59]
FAQ: What should the paper’s main metric be?
Do not use only one metric. For matching, make exact target accuracy plus link precision/recall/F1 primary. For your research-integrity contribution, prominently report false fabrication rate on author-certified genuine references, fabrication precision, fabrication recall and coverage at high precision.
The key scientific claim should not be “our model has 98.x% accuracy”. A much more meaningful claim would be: given genuinely real references that were damaged badly enough to require author correction, how often can the model still recover the intended scholarly object, and how rarely does it falsely call that object fabricated?
FAQ: What practical implementation details are easiest to overlook?
Cache external lookups, store provenance for every candidate, preserve raw strings, normalise Unicode, test transliterated author names, distinguish missing from conflicting fields, collapse known version families, detect book/chapter and proceedings/paper relations, version every external data snapshot, and make every final fabrication verdict inspectable by a human. CiteTracer reports that most benchmark cases can be resolved in deterministic/cached stages while a minority reach expensive judging, illustrating why a cascade is operationally sensible. [7]
Finally, never expose a naked “FAKE” label without its evidence. Return something closer to:
{ "status": "UNVERIFIED", "best_candidate": "...", "candidate_score": 0.61, "conflicts": ["DOI resolves to another work"], "supporting_sources": ["Crossref", "OpenAlex"], "sources_searched_without_match": ["DBLP", "ACL Anthology"], "recommended_action": "manual_review" }
That design makes the system useful to editors and researchers while acknowledging the epistemic reality documented throughout this literature: bibliographic infrastructure is incomplete, citations are noisy, and matcher failure is not the same thing as scientific fabrication.
Reference table with links
| Reference / resource | Authors / organisation | Year | Primary or official link | Role in this review |
|---|---|---|---|---|
| Reference matching: for real this time | Dominika Tkaczyk / Crossref | 2018 | Crossref | 2,000 real references → DOI/null; direct matching benchmark |
| Reference Matching Evaluation Framework | Crossref | 2018–2019 era | GitHub archive | Reusable dataset generation/matching/evaluation code |
| Marple | Crossref | 2022– | Crossref GitLab | Modern multi-strategy matching infrastructure |
| Machine Learning vs. Rules and Out-of-the-Box vs. Retrained | Dominika Tkaczyk, Andrew Collins, Paraic Sheridan, Joeran Beel | 2018 | arXiv | Reference-parser comparison and retraining study |
| Methodology for Evaluating Citation Parsing and Matching | Mateusz Fedoryszak, Łukasz Bolikowski, Dominika Tkaczyk, Krzysztof Wojciechowski | 2013 | DOI / Springer | Evaluation methodology |
| EXmatcher: Combining Features Based on Reference Strings and Segments to Enhance Citation Matching | Behnam Ghavimi, Wolfgang Otto, Philipp Mayr | 2019 | arXiv | Reference matching method and 816-item gold standard |
| EXmatcher gold standard repository | EXCITE Project | 2019 | GitHub | Released manually checked matching data |
| Cora Citation Matching | Andrew McCallum / UMass | Classic corpus | Official UMass data page | Hand-clustered citation coreference benchmark |
| Cleaning Noisy and Heterogeneous Metadata for Record Linking Across Scholarly Big Datasets | Athar Sefid et al. | 2019 | arXiv | BM25 blocking + supervised multi-field record linkage |
| Reference accuracy of primary studies published in peer-reviewed scholarly journals: A scoping review | Samuel W. Logan, Uta Hussong-Christian, Layne Case, Samantha Noregaard | 2024 | SAGE / DOI | Historical reference-error evidence |
| An Investigation of the Validity of Bibliographic Citations | Robert N. Broadus | 1983 | ERIC record | Propagation/copying of erroneous citations |
| How accurate are quotations and references in medical journals? | G. de Lacey, C. Record, J. Wade | 1985 | BMJ | Classic reference/quotation accuracy study |
| Assessing Formatting Accuracy of APA Style References: A Scoping Review | Laurel Scheinfeld et al. | 2025 | Érudit | Formatting-error versus identity-error distinction |
| Source or It Didn’t Happen: A Multi-Agent Framework for Citation Hallucination Detection | Mingzhe Li, Zhiqiang Lin, Shiqing Ma | 2026 | arXiv | CiteTracer method, synthetic benchmark and real fabricated cases |
| CiteTracer repository | CiteTracer authors | 2026 | GitHub | Open implementation/data access |
| HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferences | Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe | 2026 | ACL Anthology · arXiv | ACL/NAACL/EMNLP audit; 295 confirmed affected papers |
| Manual Reference Verification Dataset for Hallucinated and Suspicious Citation Detection Tools | Fidan Badalova, Philipp Mayr | 2026 | Zenodo | 104 manually verified valid/problematic references |
| RenoBench: A Citation Parsing Benchmark | Parth Sarin, Juan Pablo Alperin, Adam Buttrick, Dione Mentis | 2026 | arXiv | Real multilingual citation-parsing benchmark |
| RenoBench / Reference Annotation Benchmark | Public Knowledge Project | 2026 | Hugging Face | 10,000 plaintext/JATS pairs |
| Fabricated citations: an audit across 2.5 million biomedical papers | Maxim Topaz, Nir Roguin, Pallavi Gupta, Zhihong Zhang, Laura-Maria Peltonen | 2026 | The Lancet DOI · Columbia summary | Large biomedical fabricated-citation audit |
| LLM hallucinations in the wild: Large-scale evidence from non-existent citations | Zhenyue Zhao, Yihe Wang, Toby Stuart, Mathijs De Vaan, Paul Ginsparg, Yian Yin | 2026 | arXiv | 111-million-reference statistical prevalence analysis |
| GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Models | Zuyao Xu et al. | 2026 | arXiv | LLM generation study plus manually reviewed archival audit |
| CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era | Zhengqing Yuan, Kaiwen Shi, Zheyuan Zhang, Lichao Sun, Nitesh V. Chawla, Yanfang Ye | 2026 | arXiv | Human-validated citation-verification benchmark and system |
| Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences | Mark Russinovich, Ram Shankar Siva Kumar, Ahmed Salem | 2026 | arXiv | RefChecker and accepted-paper audit |
| RefChecker | Russinovich, Siva Kumar, Salem | 2026 | GitHub | Open multi-source verification pipeline |
| Detecting Hallucinated and Suspicious Citations: What Current Tools Can and Cannot Do | Fidan Badalova, Philipp Mayr | 2026 | arXiv | Audit of detector false positives and limitations |
| Suspected Undeclared Use of Artificial Intelligence in the Academic Literature: An Analysis of the Academ-AI Dataset | Alex Glynn | 2024 | arXiv | Contextual evidence on visible undeclared AI artefacts |
| Academ-AI data dump | Alex Glynn | 2025 data release | Figshare | Dataset associated with Academ-AI research |
Research status: sources were checked against primary or official pages where available, including Crossref, arXiv, ACL Anthology, GitHub/GitLab project repositories, Zenodo, Hugging Face, BMJ, SAGE, Érudit and institutional author/publisher pages. Fast-moving 2026 preprints and repositories may change after 13 August 2026; reproducible experiments should record exact dataset versions, commits and access dates. [60]
