For the past 18 months I have served on a committee that was tasked to write a report on how to improve the replicability of the empirical sciences. The report came out this Monday, and you can find it here. Apart from the advice to conduct more replication studies, the committee’s general recommendations are as follows (pp. 47-48 of the report):
“IMPROVE STUDY METHODS
Researchers should conduct research more rigorously by strengthening standardisation, quality control, evidence-based guidelines and checklists, validation studies and internal replications. Institutions should provide researchers with more training and support for rigorous study design, research practices that improve reproducibility, and the appropriate analysis and interpretation of the results of studies.IMPROVE STUDY REPORTING
Funding agencies and journals should require preregistration of hypothesis-testing studies. Journals should issue detailed evidence-based guidelines and checklists for reporting studies and ensure compliance with them. Journals and funding agencies should require storage of study data and methods in accessible repositories.CREATE PROPER INCENTIVES
Journals should be more open to publishing studies with null results and incentivise researchers to report such results. Rather than reward researchers mainly for ‘high-impact’ publications, ‘innovative’ studies and inflated claims, institutions, funding agencies and journals should also offer them incentives for conducting rigorous studies and producing reproducible research results.”
The other committee members were Johan Mackenbach (chair), Jean Philippe de Jong (secretary), Cock van Duijn, Harry Büller, Aad van der Vaart, Patricia Dankers, and Lex Bouter. It was an honor to work with this group of people, and I am happy with the end result. In addition, it was gratifying to see the level of public attention that this topic now generates. Only a decade ago, it would have unthinkable for the prestigious Academy to commission a report on replicability, for one of the largest Dutch newspapers –Trouw– to dedicate their entire front page to that report, and for the main TV news show in the Netherlands to include an two-minute feature on the report (see 17:25-19:25 here).
The front page of Trouw. The title and subtitle read: “The Royal Netherlands Academy of Arts and Sciences: Substantially more money needed for replication studies. Replicating scientific research is necessary to find out which studies pass muster and which do not. False findings are currently elaborated upon too often”.
My experiences in the committee weren’t all positive, however. My main gripe is that the other members of the committee (and, truth be told, the students in my own lab) failed to appreciate my artistic interpretation of the report’s contents.
Specifically, the cover of the report now looks like this:
A little boring perhaps, and I don’t believe replication research is boring at all — if anything, it is exciting and even scary; this explains why so few researchers publicly announce a replication study of their own work, even when that work has been called into question.
So I came up with a different concept, and as usual our graphical artist Viktor Beekman delivered the goods:
The problem with this illustration is that nobody gets it, that is, nobody has been able to figure out what it purports to show. Can you?
What the figure shows, of course, is replication research as Cinderella (note the missing shoe); initially despised, abused, and neglected, she eventually overcomes adversity to take her rightful place in the court of science. Cinderella’s broom symbolizes the potential for replication research to clean up some of the pollution that innovative research inevitably generates.
At any rate, I hope that the committee’s report will stimulate funders, institutions, universities, editors, and researchers to pay more attention to the replicability of studies in the empirical sciences.
Like this post?
Subscribe to the JASP newsletter to receive regular updates about JASP including the latest Bayesian Spectacles blog posts! You can unsubscribe at any time.
About The Author
Eric-Jan Wagenmakers
Eric-Jan (EJ) Wagenmakers is professor at the Psychological Methods Group at the University of Amsterdam.
