Science & Product

A Meta-Analysis Put Nudges at d = 0.43. Corrected for Publication Bias, a Re-analysis Found 0.04

October 19, 2026 7 min read

On December 30, 2021, PNAS published a meta-analysis of "choice architecture", the family of interventions better known as nudges, by Stephanie Mertens, Mario Herberz, Ulf Hahnel and Tobias Brosch of the University of Geneva. It pooled more than 200 randomized studies and reported an average effect on behavior of Cohen's d = 0.45, "small to medium" by convention. Defaults were the strongest technique, food the most responsive domain. The conclusion called choice architecture "an effective and widely applicable behavior change tool."

On July 19, 2022, the same journal published three letters arguing that the headline was the wrong number, along with a reply from the authors. One letter carried a title that traveled much further than its tables: "No evidence for nudging after adjusting for publication bias."

This is an older debate, and it is still going: a reanalysis posted on September 7, 2026 argues that a single average, on either side, was the wrong number to lead with. I am going back to it because I ship iOS apps and a strategy game, and every default and every reminder I put in a product is choice architecture in this paper's sense. If you A/B test nudges, these papers are worth reading past their titles.

What the meta-analysis measured

The literature search closed in June 2019. To get in, a study had to be a randomized experiment, measure behavior in a real or hypothetical choice, use individuals as the unit, be published in English, and test choice architecture on its own. Studies that combined techniques, or added significant incentives or education, were excluded. Six domains: health, food, environment, finance, prosocial behavior and "other".

On May 4, 2022, PNAS published a correction. The sample had included four observations from a 2012 paper by Shu et al. that had since been retracted, erroneous values for two observations from another paper, and several coding errors. Corrected figures: 447 effect sizes from 212 publications, 2,148,439 participants, d = 0.43 (95% CI 0.38 to 0.48). The authors wrote that none of the changes affected the pattern of results or the conclusions.

Three details matter more than the average:

The spread was enormous. After correction, I² was 99.52% and the 95% prediction interval ran from -0.36 to 1.22. The paper estimated that about 15% of interventions were likely to backfire.
Technique mattered. Defaults scored d = 0.62 over 128 effect sizes, reminders 0.29, commitment devices 0.23. As categories, "decision structure" scored 0.54, against 0.34 for information and 0.28 for assistance.
More recent publications reported smaller effects than older ones.

The authors flagged the bias themselves. In the corrected figures, Egger's test found asymmetry (b = 2.10), and their sensitivity analysis put the effect at d = 0.31 under moderate publication bias and d = 0.08 under severe bias (0.03 before the correction), a scenario they judged "only partially supported by the funnel plot."

Three letters and a reply

Maximilian Maier, František Bartoš, T. D. Stanley, David Shanks, Adam Harris and Eric-Jan Wagenmakers reran the corrected data with robust Bayesian meta-analysis (RoBMA), which averages across several bias-correction models instead of picking one. Adjusted average: d = 0.04 (credible interval 0.00 to 0.14), with a Bayes factor BF01 of 0.95. A Bayes factor near 1 means the data are about as likely with no average effect as with one. The letter's own text calls it "an absence of evidence" for an overall effect, which is not the same as evidence of absence.

The detail is sharper. For "information" and "assistance" nudges, the evidence pointed against an average effect (BF01 = 33.84 and 9.05). "Structure", where defaults sit, was undecided (0.12, BF01 = 1.12). Using only the most precise estimate from each study, they found weak evidence for an overall effect (0.11, BF01 = 0.31). And every category and every domain except finance showed heterogeneity, "which implies that some nudges might be effective, even when there is evidence against the mean effect."

Barnabas Szaszi and eight co-authors, including Andrew Gelman and Elizabeth Tipton, applied three other corrections to the same data: Andrews-Kasy gave d = -0.01, WAAP 0.07, trim and fill 0.08. Their title is the practitioner's version: "No reason to expect large and consistent effects of nudge interventions." They allowed for "a few exceptions", citing defaults as an example.

Jonathan Bakdash and Laura Marusich, of the US Army Research Laboratory, plotted the effects by domain and found a "cliff": in four domains, the distribution stops sharply at or just below zero, and they call the suppression of unfavorable results "a plausible mechanism" for it. Nearly two thirds of all effects (62.64%) sat below the pooled mean.

Mertens and colleagues replied that they agreed the publication bias was "problematic", that their paper was never meant as "the final word", and that the field needs appropriately powered studies, preregistered reports and published null results.

Coverage compressed all of this. PsyPost's headline on July 27, 2022: "Re-analysis of meta-analysis reveals no evidence for nudging once accounting for publication bias." Then, on September 7, 2026, Ulrich Schimmack argued on his blog, Replicability-Index, that the near-zero means follow from assuming effects are normally distributed, and that the same models imply that effects in the intended direction average d = 0.44 (step-function model) or 0.26 (RoBMA). It is a blog post, not a peer-reviewed paper. The line I would keep from it: "There is no single effect of nudging."

What none of this shows

It does not show that nudges do not work: the main Bayes factor is undecided, and every letter stressed large variation. It does not show that they work at d = 0.43 either. And it says little directly about your funnel. There is no separate estimate for commerce or software, studies that combined techniques were excluded by design while a redesigned screen often changes several things at once, and hypothetical choices counted as behavior.

The number I would bring to a planning meeting

It comes from a different paper. In Econometrica in January 2022, Stefano DellaVigna and Elizabeth Linos assembled 126 trials covering 23 million people, including every trial run by two of the largest Nudge Units in the United States, and compared them with nudge trials published in academic journals. In the journals, the average nudge raised take-up by 8.7 percentage points, a 33.4% increase. In the Nudge Units sample, by 1.4 points, an 8.0% increase: still highly significant, and about a sixth of the journal effect in percentage points. Their model attributes about 70% of the gap to selective publication, made worse by low statistical power.

Now the arithmetic. Take a signup step converting at 10%. To detect a 33.4% relative lift (10% to 13.3%) with 80% power at a 5% two-sided threshold, you need about 1,450 users per variant. To detect 8.0% (10% to 10.8%), about 22,850 per variant, sixteen times as many. A test sized for the published effect would detect the realistic one only about 11% of the time. And when an underpowered test does crown a winner, it is more likely to have caught an overestimate: the mechanism DellaVigna and Linos measured in the literature, running inside your own company.

What I would do before shipping a nudge

Size the test for the at-scale effect, not the published one. If your traffic cannot detect an 8% lift, the honest result is "not testable yet", not "it won".
Write down the metric, the sample size and the stopping rule before launch. That is a preregistration with one reader.
Keep a log of every test, nulls and losses included. A dashboard that lists only shipped winners is a file drawer with a nicer interface.
Test two-sided and add a guardrail metric, such as refunds, churn or support tickets. The meta-analysis's own prediction interval goes below zero.
With one test slot, try a default before a reminder. In this literature, structure is the category the critics did not knock down, and information and assistance are where the evidence turned against an average effect. A hypothesis, not a promise.
Rerun a winner before stacking the next nudge on top of it.

The useful output of this debate is not a number for a slide. It is the size of the spread, and the reminder that the only effect that counts is the one you measure on your own users.

Sources

Sample sizes and power: author's calculation, two-sided test at 5%, normal approximation, n = 7.85 × [p1(1-p1) + p2(1-p2)] / (p2-p1)², giving 1,447 per variant for 10% to 13.34% and 22,852 for 10% to 10.8%; at 1,447 per variant, power to detect 10% to 10.8% is about 11%.

A project like this one?

I design and deploy products like this. Let's talk.

Let's talk