Back in 2011, Daryl Bem shocked the academic world by publishing an article in which he claimed to present empirical evidence for the existence of precognition, that is, the ability of people to “look into the future” (Bem, 2011). Particularly shocking was the fact that Bem had managed to publish this claim in the flagship journal of social psychology, the Journal of Personality and Social Psychology (JPSP).
After learning about the Bem paper, together with several colleagues at the Psychological Methods Unit we wrote a reply titled “Why psychologists must change the way they analyze their data: The case of psi” (Wagenmakers et al., 2011). In this reply, we pointed to the exploratory elements in Bem’s article, and we showed with a Bayesian re-analyses how p-values just below .05 offer little evidence against the null hypothesis (a message recently repeated in the paper “Redefine Statistical Significance”, which is the topic of an ongoing series of posts on this blog).
Next, Bem and two colleagues produced a rejoinder, “Must psychologists change the way they analyze their data?” (Bem, Utts, & Johnson, 2011), in which –unsurprisingly– they argued that we were sorely mistaken. At this point JPSP had had enough, but the discussion was not finished. We felt that several key claims in the Bem et al. rejoinder were sufficiently inaccurate to warrant further comment. So we produced a rebuttal, titled “Yes, psychologists must change the way they analyze their data: Clarifications for Bem, Utts, and Johnson (2011)”, which we never submitted for publication. Instead, I made this rebuttal available through my public dropbox folder. However, times have changed, and a much better way to make this information permanently available is by uploading it to PsyArXiv;. Rereading the paper I am still happy with our arguments. At any rate, you can read our rebuttal here. This is the abstract:
“Does psi exist? In a widely publicized article featuring nine experiments with over one thousand participants, Bem (2011) claimed that future events retroactively affect people’s responses. In a response, we pointed out that Bem’s analyses were partly exploratory. Moreover, we reanalyzed Bem’s data using a default Bayesian t-test and showed that Bem’s evidence for psi is weak to nonexistent. A robustness analysis confirmed our skeptical conclusions. Recently, Bem, Utts, and Johnson (2011) question several aspects of our analysis. In this brief reply we clarify our analysis procedure and demonstrate that our arguments still hold.”
References
Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-425.
Bem, D. J., Utts, J., & Johnson, W. O. (2011). Must psychologists change the way they analyze their data? Journal of Personality and Social Psychology, 101, 716-719.
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi. Journal of Personality and Social Psychology, 100, 426-432.
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., Kievit, R. & van der Maas, H. L. J. (2011). Yes, psychologists must change the way they analyze their data: Clarifications for Bem, Utts, and Johnson (2011). ArXiv Preprint.
About The Author
