Andy Garcia · freelance dev
FRENES
Back to the blog
Science & Evidence

The BMJ Published 21 Checks for Untrustworthy Clinical Trials. Check One: Was It Retracted?

October 19, 2026 7 min read

On September 24, 2026, The BMJ published INSPECT-SR, a tool for deciding whether a randomized controlled trial can be trusted at all. The paper has 54 authors, from Cochrane, The Lancet, The BMJ and universities from Manchester to Beijing, and it comes down to 21 checks in four domains, each worded so that a "yes" flags a potential problem. Cochrane had already endorsed it in October 2025, as an optional tool for authors of its reviews.

The problem it targets has a name in Cochrane's policy: problematic studies, "where there are serious questions about the trustworthiness of the data or findings." The cause can be fraud. It can also be an honest, fatal error, such as miscoding which patients got the treatment and which got the placebo. Either way, the numbers are not real. And the paper says the standard risk-of-bias tools "are not designed to identify problems of this nature, nor do they appear to be successful in doing so."

That is the sentence I keep coming back to. The guidance notes that many reports of problematic studies "describe sound methodology." A study can read as well designed and still rest on data that does not exist. Quality and authenticity are two different questions, and most of us only ever ask the first one.

Why a checklist, and why now

The paper opens with two numbers, both from other work, so I read the originals.

The first is VITALITY Study I, published in The BMJ on April 23, 2025. Its authors started from 1,330 retracted trials and found 847 systematic reviews that had pooled at least one of them, for 3,902 meta-analyses they could recompute. Removing the retracted trials reversed the direction of the pooled effect in 8.4% of them, changed its statistical significance in 16.0%, and moved its size by more than half in 15.7%. Evidence from 68 reviews with distorted conclusions had been used in 157 guideline documents. Meta-analyses with fewer studies were hit hardest, which makes sense: one bad trial weighs more in a small pool. A linked BMJ editorial, published on September 28, adds a quieter cost: even when fake trials do not bias a result, adding their participants "could inflate the precision of the findings."

The second number is softer than it reads. INSPECT-SR says Carlisle "estimated that 26% of randomised controlled trials submitted to the journal Anaesthesia were problematic." In Carlisle's 2021 paper, that 26% is 40 of the 153 submissions for which he examined individual patient data, trials he categorized as "zombie." He found false data in 44% of those 153. Across all 526 trials submitted from February 2017 to March 2020, he identified false data in 14%. Still alarming. But it describes one journal's submissions, measured on the subset whose raw spreadsheets he examined (at first requested only from trials that had raised suspicion), not published trials in general.

Retraction is also slow. The paper notes it can take years, "if it is retracted at all," so a reviewer cannot simply filter on retraction status.

The 21 checks

The four domains are ordered so that the cheap, definitive checks come first:

  • Post-publication notices (3 checks): is the study retracted, does it carry an expression of concern, and do the team's other studies carry such notices?
  • Conduct, governance and transparency (5): concerns about ethical approval, the timing or absence of trial registration, inconsistencies between the paper and the registry, implausible recruitment, and methods that are implausible given the reported resources.
  • Text and figures (2): duplicated or incompatible text, and manipulated or duplicated figures.
  • Results (11): from implausible baseline data to errors in statistical results, plus inconsistencies across the publications that describe the same trial.

What struck me in the guidance is how much of it is arithmetic. Three of its worked examples:

  • Recruitment. A trial reports 2,200 patients with a ruptured abdominal aortic aneurysm over two years, from a catchment of about 500,000 people. At the incidence the guidance cites, 5.6 to 17.5 per 100,000 person-years, that population produces roughly 56 to 175 cases in two years. The claim is more than 12 times the upper bound.
  • Allocation. A single-center trial says it used blocked randomization with a fixed block size of 4 and no stratification, then reports 5 more patients in the control arm. With blocks of 4, the imbalance cannot exceed 2.
  • Impossible means. A trial reports a mean Apgar score of 8.96 for a group of 30 newborns. Apgar scores are whole numbers, so the group total must be one too: 268 gives a mean of 8.93, 269 gives 8.97. Nothing gives 8.96. This is the GRIM test, and the guidance links web apps that run it.

None of these needs the raw data. They need someone who reads the numbers as numbers.

Each domain ends in a judgment, "no concerns," "some concerns" or "serious concerns," and the tool deliberately has no scoring rule. Trials with serious concerns should be excluded. Trials with some concerns should go into a sensitivity analysis. The authors recommend running the tool before risk-of-bias assessment and data extraction, so nobody spends hours on a trial that will be thrown out, and repeating the checks for retractions and expressions of concern shortly before the review is finished.

What the paper does not show

This is a consensus tool, and it should be read as one.

The 76 candidate checks became 69, then 29 after a Delphi survey (148 participants in the first round, 134 in the second, 39% of them from the UK and 7% from low and middle income countries), then 21 after eight consensus meetings in June and July 2024. That establishes what experts agree on. It does not measure how often the tool is right.

The user testing was small. Forty people gave feedback. The median assessment took 45 minutes per trial (interquartile range 27 to 74), 70% said they would use the tool in future work, and 93% found the judgments reasonable. The authors note that the testers were self-selected, so acceptability "may therefore be overestimated." The paper reports no inter-rater reliability; the authors list it as future evaluation. And they warn against validating the tool against retraction status, because whether a paper is retracted is "a subjective decision that is arbitrary to some degree."

They are also explicit that it is not a fraud detector. Concerns "do not constitute an accusation of research misconduct." The tool does not assess risk of bias, generalizability or conflicts of interest either.

The last open question is the one closest to my own work. The authors ask whether forgers could use the checklist to produce more convincing fakes, and say it is unclear whether efforts to get around it, "either by manual fabrication or using large language models," will succeed. On balance, they expect it to detect and deter many more fake trials than it creates. Meanwhile, a project called INSPECT-AI is exploring how language models could assist with the checks.

What I would borrow

I am not a systematic reviewer. I am the CTO of a LegalTech built on Claude. I built BidForge, an AI that reads French public tender files for small businesses. And I fact-check my own posts with Jev, a model I use as a judge, before they go out. All three run into the same question: a document can be quoted correctly and still be wrong at the source.

So here is what I would take from INSPECT-SR for anyone whose product, deck or post leans on a study.

Ask the trust question before the quality question. Before debating a study's method, check that it still stands: retraction, expression of concern, notices on the authors' other papers. The guidance points to the Retraction Watch database for this. Then check again right before you publish, because notices arrive late.

Run the arithmetic. Do the totals add up? Could the sample have been recruited where and when the authors say? Can a mean of whole numbers land on that value? These checks need no special access, only attention.

Weight by leverage. If one study carries your whole claim, it deserves the full check. VITALITY's finding says thin evidence is where a single bad trial does the most damage.

If a model helps, let it flag, not rule. INSPECT-SR asks for two independent reviewers and written reasons for every judgment. That is a good spec for an AI-assisted check too: the model points to the inconsistency, a person decides, and the reason is recorded.

A checklist will not end research fraud. But it moves the default from "published, therefore real" to "real, if it survives 21 questions." That default is worth stealing.

Sources

A project like this one?

I design and deploy products like this. Let's talk.

Let's talk