This Tuesday, one of us [EJ] participated in a debate about –you guessed it– the α = .005 recommendation from the paper ‘Redefine Statistical Significance’. The debate was organized as part of the Annual Meeting of the Berkeley Initiative for Transparency in the Social Sciences (BITSS), and the two other discussants were Simine Vazire and Daniel Lakens.
The debate was live-streamed and taped so that you can view a recording: the debate starts at about 32:30 and lasts until 1:40:30. The discussion starts at around 01:13:00.
Barbecue Chicken Alert!
In the opening statement, I [EJ] wanted to emphasize the weakness of evidence for p-just-below-.05 results. To drive the point home, I used a popular phrase from basketball great Shaquille O’Neal — a.k.a. ‘The big Aristotle’. For those of you who do not know the Diesel, he is a 325 pound (147 kg), 7 ft 1 in (2.16 m) force of nature. In his prime, he reduced the wonderful game of basketball to a boring display of raw power: Shaq would catch the ball in the post (i.e., near the basket), back up into his defender, let the poor fellow taste “some of the elbow juice”, and then dunk the ball with authority, occasionally shattering the backboard for good measure (a funny example is here, “is that all you got?”). The way in which Shaq would demolish the defense felt both inevitable and unfair. At any rate, the Diesel had a phrase to describe the inevitability of success in the face of weak opposition: “barbecue chicken”.
If you look up the phrase on the urban dictionary, you will find two definitions, and both I believe are missing the point. Here’s the first one:
And here’s the second one:
We won’t know for certain until the big Aristotle explains exactly what he meant, but I believe the expression applies in general, and is meant to describe any situation in which a person of superior skill applies a series of routine moves to overwhelm an ineffective resistance. In Shaq’s analogy, there is no need to chase the chicken, pluck the chicken, and cook the chicken; no, the chicken has already been prepared and the only thing that you need to do is reach out and eat it. A phrase that is semantically close is ‘shooting fish in a barrel’.
At any rate, what I [EJ] wanted to convey is that, whenever a p-just-below-.05 result is presented, it is a routine exercise to open JASP, tick a few boxes in the Summary Stats module, and reveal that the evidence against the null hypothesis is surprisingly weak. Concrete demonstrations are provided in earlier blog posts, here and here.
So, whether encountered in the published literature or in a preprint, any p-just-below-.05 result should raise a red flag — it will be easy to show that the evidence is disappointingly low. In the words of the big Aristotle: “barbecue chicken alert!”
The Upshot
So who won the debate? You can judge for yourself, but in the end we hope the real winners are the researchers who viewed the debate and are now more aware that the .05 threshold is anything but sacred.
At the end of the workshop, the 40+ participants were offered a choice between three alternatives:
- Stick to α = .05 (one vote);
- Move to α = .005 (three votes, including that of Simine and myself);
- Something else / don’t know (all the other votes).
Would the third option be ‘justify your own alpha and ignore the concept of evidence altogether?’; would it be ‘let’s just be Bayesian, who needs this alpha?’; or perhaps ‘let’s use estimation instead of testing’; or perhaps even ‘let’s think carefully and take all information into consideration before arriving at a judgement that everybody will consider wise and appropriate, after which world peace is declared’.
We don’t know, but we do know that the debate itself was worthwhile and that there is an increasing need to better understand how correct conclusions can be drawn from noisy data.
Like this post?
Subscribe to the JASP newsletter to receive regular updates about JASP including the latest Bayesian Spectacles blog posts! You can unsubscribe at any time.
References
Wagenmakers, E.-J. (2017). Barbecue chicken alert! Invited presentation for the plenary discussion session (with Simine Vazire and Daniel Lakens) at the Annual Meeting of the Berkeley Initiative for Transparency in the Social Sciences (BITSS), Berkeley, USA, December 2017. Slides are here.
About The Authors
Eric-Jan Wagenmakers
Eric-Jan (EJ) Wagenmakers is professor at the Psychological Methods Group at the University of Amsterdam.
Quentin Gronau
Quentin is a PhD candidate at the Psychological Methods Group of the University of Amsterdam.
