Subjective p intervals researchers underestimate the variability of p values over replication

Suppose you obtain p =.02 in an experiment, then replicate the experiment with new samples. What p value might you obtain, and what interval has an 80% chance of including that replication p? Under conservative assumptions the answer is, perhaps surprisingly (.0003,.30). The authors report three email surveys that asked authors of articles published in leading journals in psychology, medicine, or statistics to estimate such intervals. Overall response rate (7%) was low, but responses from 360 researchers gave intervals with an average 40% to 50% chance of including replication p, rather than the target 80%. Results were similar for all three disciplines. Respondents generally found the task unfamiliar and difficult. There was great variability over respondents, but almost all of them gave intervals that were too short. This widespread, and often severe, underestimation of the variability of p may help to explain why researchers place too much interpretive weight on single p values. © 2011 Hogrefe Publishing.