Deakin University
Browse

File(s) under embargo

Estimating the extent of selective reporting: An application to economics

journal contribution
posted on 2024-03-05, 03:09 authored by Stephan B Bruns, Teshome K Deressa, Tom StanleyTom Stanley, Chris Doucouliagos, John PA Ioannidis
AbstractUsing a sample of 70,399 published p‐values from 192 meta‐analyses, we empirically estimate the counterfactual distribution of p‐values in the absence of any biases. Comparing observed p‐values with counterfactually expected p‐values allows us to estimate how many p‐values are published as being statistically significant when they should have been published as non‐significant. We estimate the extent of selectively reported p‐values to range between 57.7% and 71.9% of the significant p‐values. The counterfactual p‐value distribution also allows us to assess shifts of p‐values along the entire distribution of published p‐values, revealing that particularly very small p‐values (p < 0.001) are unexpectedly abundant in the published literature. Subsample analysis suggests that the extent of selective reporting is reduced in research fields that use experimental designs, analyze microeconomics research questions, and have at least some adequately powered studies.

History

Journal

Research Synthesis Methods

Pagination

1-13

Location

England

ISSN

1759-2879

eISSN

1759-2887

Language

en

Publisher

Wiley