DataRank v6.0 Citation-Only 1-Hop

• Data papers — the paper is the dataset—
• Papers that deposit data — a finding paper that released a dataset—
• No data released — no deposit detectable from the record—

The classes are disjoint: a data paper that also deposits a dataset is counted once, as a data paper. Counts are live and shift as classification and deposit detection continue across the corpus. Only the first class is ranked by DataRank.

How we identify them — DrPaper

DrPaper is the AI classifier we built for the job. Under the hood it's a fine-tuned SciBERT — a 110M-parameter language model pre-trained on scientific text — that we re-trained on roughly 4,000 hand-labeled papers. On validation it reaches an F1 of 0.9153.

We then stress-tested it on 38 known edge cases spanning 15 categories — consortium genomics, brain atlases, cohort profiles, database updates, benchmarks, clinical databases, genome resources — and DrPaper got all 38 right. That includes landmark resources like GTEx, TCGA, the Framingham Heart Study, UK Biobank, and the RCSB Protein Data Bank, all correctly flagged as data papers because their citations come primarily from data reuse. It also correctly rejected influential methods papers — AlphaFold, DESeq2, and BWA — which are cited heavily for their algorithms, not their data.

The model, the labeled training set, and the fine-tuning notebook are all open. See Resources for direct links to the HuggingFace model, the Kaggle notebook, and the labeled dataset.

How a score is produced

The rubric is a fixed taxonomy of 18 atomic criteria, each anchored to external frameworks — the RDA FAIR maturity indicators, F-UJI, and the NIH Data Management & Sharing Policy — and weighted by RDA priority (Essential 2.0, Important 1.0, Useful 0.5). Each criterion asks one question of the same shape: which kind of artefact does this paper exhibit for its own data? The model names the strongest class it can find — yes, partial, or no — and must supply the verbatim quote that is that artefact. The quote is verified against the text in code; a positive verdict whose quote isn't actually in the paper is downgraded one step. Each question also demands that the quote assert the artefact of this study's data, so a journal's own CC-BY footer or a third-party code link can't inflate the verdict.

The model never emits a number. Verdicts map to 1.0 / 0.5 / 0.0 and every score is computed deterministically in code — identical verdicts produce byte-identical scores, whichever model produced them. If an assessment fails for any reason (a transport error, a truncated response, a missing criterion), no score is produced or stored: a failed assessment is never a score.

Headline vs. advisory criteria

We measured how well each criterion reproduces when the same paper is assessed by different language models. Nine criteria are fact-shaped — a persistent identifier, a named repository, a formal dataset citation, open access, an open file format, a reuse licence, versioning, code availability, and funding attribution — and reproduce reliably across models. The other nine require judgment calls (is the data availability statement adequate? is the documentation sufficient?) and reproduce measurably worse.

The published headline score is therefore a weighted mean over the nine reproducible criteria only. The judgment-heavy nine are still assessed and shown as an advisory “full FAIR picture” with per-dimension F/A/I/R sub-scores — but they never move the headline number or the percentile. We would rather publish a narrower score that is stable than a broader one that changes with the model behind it.

How reproducible is the score?

A score produced by a language model is only meaningful if the same paper gets the same answer twice. Because the number is computed in code from the verdicts, all of the instability lives in the verdicts — so that is where we measure it, rather than on the composite, which is an average and can hide two flips that cancel out. We report Krippendorff's α (chance-corrected, so a rubric that answers “no” to everything isn't flattered; α ≥ 0.8 is the conventional bar for reliable) and the flip rate — how often a criterion ever changed answer across repeats.

Repeating the same paper, same model

A single read is not reproducible enough to publish: about a third of criteria change answer between two runs of the same paper at temperature 0, because borderline cases coin-flip on the partial/yes boundary. Reading each paper five times and taking the majority verdict per criterion is what earns the published number. We also checked that voting can work here at all — every one of the 18 criteria shows a decisive lean rather than a true 50/50, so no criterion is so ambiguous that more passes couldn't settle it.

Same paper, a different model

Reproducibility within one model isn't enough — a model can be perfectly self-consistent and consistently idiosyncratic. Scoring the same papers with two independent models agrees at α 0.883 on the nine headline criteria versus 0.789 on the nine advisory ones. That gap is the entire justification for the split: the fact-shaped criteria travel between models, the judgment-heavy ones travel less well, so only the former set the published number.

Controlling what else could move a score

Hosted model endpoints are not reproducible by default. Requests are pinned to a single provider with fallbacks disabled (an unannounced reroute is silently a different model), to a fixed numerical precision, and to grammar-constrained decoding so the verdict values cannot drift into free text. Temperature is 0. Without these controls a measurement records the routing layer as if it were the model — an earlier internal run compared two models across two and seven providers respectively, and its conclusion did not survive re-testing under matched conditions.

Measured on 10 open-access data papers × 6 repeats × 2 models (240 evaluations across two independent runs). A small sample: it constrains the reliability of the instrument, not the score of any individual paper.

Does FAIR just re-measure citations?

If a FAIR score simply tracked citation counts it would be redundant, so we test it against citations as an external check — never as an input. Across 2,850 data papers published 2021 or later and read in full text, the rank correlation between FAIR score and citation count is ρ = 0.07. It is statistically significant only because the sample is large; it explains well under 1% of the variation in how often a paper is cited.

We report that plainly because the honest reading is the useful one: sharing data well and being cited a lot are close to unrelated in this corpus. That is an argument for measuring data sharing separately rather than assuming citations already capture it — which is the reason DataRank and FAIR are two numbers and not one. The result holds when papers whose own availability statement says they generated no dataset are included or excluded.

What the score picks up: funding mechanism

A quality metric should distinguish things we independently believe differ. NIH award numbers encode the funding mechanism that paid for the work — an S10 buys shared instrumentation, a P20 seeds a centre, an R00 supports someone starting their own lab, a D43 funds international training. These have genuinely different expectations about producing shareable data, and the score separates them sharply.

Papers are funded by several awards more often than not — only 40% of NIH-funded papers here carry a single mechanism. So rather than assigning each paper to one bucket, we fit a regression with one indicator per mechanism across all 2,796 papers. Each coefficient is that mechanism’s marginal association, holding co-funding constant: a paper backed by both an R01 and a U24 informs both estimates without contaminating either. The outcome is a paper’s FAIR percentile within its own publication year, so the comparison is year-controlled by construction.

Every row shown is significant after Holm–Bonferroni correction across all 33 mechanisms modelled; the other 24 were not and are omitted. The ordering is the one you would predict from the mechanisms themselves: awards whose purpose is shared infrastructure scores highest, and training and career-transition awards — small, short, usually without a data-sharing mandate — sit far below. The rubric was built without any knowledge of funding, so agreeing with it is evidence the score tracks something real about data sharing rather than writing style.

It is not a time trend. Re-fitting the same model on raw FAIR, with the year control removed, barely moves the estimates (S10 +18.6 rather than +21.3, R35 +10.4 rather than +12.0) and changes no ordering. That is consistent with the separate finding that FAIR itself is flat across 2020–2025.

What the score does not pick up: time

The mechanism differences above are structural, not dynamic. Across 2020–2025 the score is flat: an S10-funded paper scored the same in 2020 as in 2025, and so did everything else. Of fourteen mechanisms with 120–1,402 papers each, none shows a year trend after correction for multiple testing, and neither does the data-availability-statement criterion on its own.

The raw year correlation is slightly negative (Spearman −0.055, p = 0.002), but that is corpus composition, not practice. Our scored population widened from 132 distinct journals in 2020 to 301 in 2025 as ingestion reached further into the literature, and the papers added later sit in smaller venues. Comparing papers to others in the same journal, the trend disappears entirely (−0.007, p = 0.81). We report the within-journal figure because the raw one mostly measures which journals we had reached.