The paper Redefine Statistical Significance reveals an inconvenient truth: p-values near .05 are evidentially weak. Such p-values should not be used “for sanctification, for the preservation of conclusions from all criticism, for the granting of an imprimatur.” (Tukey, 1962, p. 13 — NB: Tukey was referring to statistical procedures in general, not to p-values or p-just-below-.05 results specifically). Unfortunately, in…

In our previous posts about the paper “Redefine Statistical Significance”, two concrete examples corroborated the general claim that p-just-below-.05 results constitute weak evidence against the null hypothesis. We compared the predictive performance of H0 (effect size = 0) to the predictive performance of H1 (specified by a range of different prior distributions on effect size) and found that for a…

In previous posts we provided detailed Bayesian reanalyses of two “p-just-below-.05” experiments (i.e., red, rank, and romance, and flag-priming). For both experiments, the evidence against the null hypothesis was relatively weak, and this supported the main claim from the paper “Redefine Statistical Significance” (and the 2016 claim by the American Statistical Association, and the claim made by statisticians throughout the last…

In the two previous posts on the paper “Redefine Statistical Significance”, we reanalyzed Experiment 1 from “Red, Rank, and Romance in Women Viewing Men” (Elliot et al., 2010). Female undergrads rated the attractiveness of a single male from a black-and-white photo. Ten women saw the photo on a red background, and eleven saw the photo on a white background. The…

Has the common criterion for statistical significance –“1-in-20”– tempted researchers into making strong claims from weak evidence? Should p-values near .05 be considered only suggestive? Are researchers caught in a bad romance? Last year, the American Statistical Association stated that “a p-value near 0.05 taken by itself offers only weak evidence against the null hypothesis” (Wasserstein and Lazar, 2016, p.…

In the previous post we discussed the paper “Redefine Statistical Significance”. The key point of that paper was that p-values near .05 provide (at best) only weak evidence against the null hypothesis. This contradicts current practice, where p-values slightly lower than .05 bring forth an epistemic jamboree, one where researchers merrily draw bold conclusions such as “we reject the null…

Statisticians have worried about the evidential impact of p-values for over 60 years. Again and again, they reported that p-values slightly lower than .05 provide only weak evidence against the null hypothesis (e.g., Edwards, Lindman, & Savage, 1963; Berger & Delampady, 1987; Sellke, Bayarri, & Berger, 2001; Johnson, 2013). Last year, the American Statistical Association (ASA) issued a statementon p-values…