Publication bias
Funnel plots and the Egger, Begg and Harbord tests for small-study effects.
What it is
Small studies with unimpressive results are less likely to be published, so the studies that reach a meta-analysis can overstate the effect. This is one cause of small-study effects: smaller studies showing larger effects than larger ones.
The funnel plot places each study by its effect (horizontal axis) and its standard error (vertical axis). Without bias, studies scatter symmetrically around the pooled effect, widening as they get smaller, like an inverted funnel. A gap in one bottom corner suggests missing small studies.
Statistical tests quantify the asymmetry: Egger's regression for most measures, Harbord's modification for odds ratios, and Begg's rank correlation as a less powerful complement. Asymmetry is not proof of publication bias: heterogeneity, chance or lower quality of small studies can produce it too.
No single method settles the question. Visual inspection of the funnel plot is subjective, and different readers often disagree about the same plot; the tests are objective but have low power. They complement each other: the judgement comes from the funnel, the tests and the clinical context together.
| Test | Statistic | P |
|---|---|---|
| Egger | t = −0.18 | 0.86 |
| Begg | z = 0.09 | 0.93 |
| Harbord | t = −0.35 | 0.73 |
With only 10 informative studies the tests have little power, so the conclusion is "no evidence", not "no bias".
When to use it
Use it when
- The meta-analysis compares two groups and has at least 10 studies; with fewer, the tests cannot tell true asymmetry from chance.
- The studies differ in size, so that small and large studies can be compared.
- The assessment feeds the publication bias domain of GRADE.
Choose another tool when
- There are fewer than 10 studies: report that publication bias could not be assessed reliably, because the tests cannot distinguish chance from true asymmetry (Lau et al., 2006).
- The meta-analysis is single-arm (proportions or means): funnel plots are not valid there, because there is no expected relation between study size and the pooled rate (Hunter et al., 2014). See single-arm binary and single-arm continuous.
- All studies have a similar size: the funnel has no spread to show asymmetry.
The template, parameter by parameter
Click a highlighted argument to see what it does and which values to use. Picking a value updates the code.
# m: a pairwise model with at least 10 studies
metabias(m, = "Egger", = 10)
metabias(m, = "Begg", = 10)
# funnel plot: see section 4The funnel plot, parameter by parameter
These arguments change only how the result is shown, not the result itself. Click one to see the options.
png("funnel.png", width = 3500, height = 3000, res = 500)
funnel(m,
common = FALSE, = TRUE,
= c(0.90, 0.95, 0.99),
col.contour = c("grey85", "grey70", "grey55"),
= FALSE,
= "navyblue")
dev.off()How to read the results
The funnel plot
- Visual judgement is hard to make with confidence, especially with few studies; do not decide on the plot alone.
- Symmetric, with studies on both sides at every size: no sign of small-study effects.
- Empty bottom corner on the side of no effect: small negative studies may be missing.
- Use the contours to see whether the missing studies would have been non-significant.
The tests
- Read them together with the funnel and with each other: concordant results are more convincing than any single one.
- P < 0.10 is the threshold, as in Egger's original paper, because the tests have low power; Cochrane does not fix one, so state it in the methods.
- Choose the test by effect measure: Egger for RR and MD, Harbord (or Peters) for OR.
- A non-significant test does not rule out publication bias, especially near 10 studies.
Interpretation
- Asymmetry can also come from heterogeneity, chance or poorer quality of small studies.
- Trim-and-fill (
trimfill()) can be shown as a sensitivity analysis, never as the corrected result. - Report the judgement in the publication bias domain of GRADE.
In my work
A publication bias assessment from a published review, next to the calls behind it. The funnel plot was redrawn from the published study data; the Egger and Begg results reproduce the reported values, and the Harbord test is added because the measure is an odds ratio. Click the figure to enlarge it.
metabias(m, method.bias = "Egger") # t = -0.18, P = 0.86
metabias(m, method.bias = "Begg") # P = 0.93
metabias(m, method.bias = "Harbord") # P = 0.73 (OR)
funnel(m,
contour.levels = c(0.90, 0.95, 0.99),
bg = "navyblue")References
- Higgins JPT, Thomas J, Chandler J, et al. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. Chapter 13 (assessing risk of bias due to missing results).
- Sterne JAC, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002.
- Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315:629-634.
- Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50:1088-1101.
- Harbord RM, Egger M, Sterne JAC. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006;25:3443-3457.
- Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008;61:991-996.
- Lau J, Ioannidis JPA, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ. 2006;333:597-600.
- Hunter JP, Saratzis A, Sutton AJ, Boucher RH, Sayers RD, Bown MJ. In meta-analyses of proportion studies, funnel plots were found to be an inaccurate method of assessing publication bias. J Clin Epidemiol. 2014;67:897-903.
- Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56:455-463.