Preprint: “Because it is the Right Thing to Do”: Taking Stock of the Peer Reviewers’ Openness Initiative

August 08 - 2019

This post is an extended synopsis of Dahrendorf, M., Hoffmann, T., Mittenbühler, M., Wiechert, S., Sarafoglou, A., Matzke, D., & Wagenmakers, E.-J. (2019). “Because it is the Right Thing to Do”: Taking Stock of the Peer Reviewers’ Openness Initiative. Preprint available on PsyArXiv: https://psyarxiv.com/h39jt

Abstract

In recent years, multiple initiatives have sought to improve the transparency and reproducibility of psychological research. One example is the Peer Reviewers’ Openness Initiative (PRO), which aims to promote the sharing of data and code. PRO signatories pledge to provide a full review only for manuscripts that publicly share data and code, or include a justification why sharing is not possible. The punitive element of this approach attracted criticism: PRO may be perceived as coercive and could lead to negative repercussions for its signatories. Therefore we conducted a survey to assess signatories’ experiences with PRO. Contrary to the criticism, the reported experiences were predominantly positive, and 92% (117/127) of the signatories who completed the survey indicated that they would sign the initiative again today. Only 15% (19/127) of the respondents experienced negative reactions related to their commitment to PRO. Almost 50 respondents suggested ways in which PRO could be improved. We conclude that, from the signatories’ perspective, the benefits of the PRO initiative outweigh its drawbacks.

Summary and Figures

The Peer Reviewers’ Openness Initiative (PRO) was first introduced in 2016 by Morey and colleagues in order to change the status quo and accelerate the adoption of data sharing practices. When signing PRO, researchers agree to provide a full review only for manuscripts that publicly share data and code, or else provide a clear reason why sharing is not possible. In its almost two-year existence PRO has enjoyed a mixed reception – the initiative has attracted criticism for being too strict, for being opposed to APA guidelines, and for frustrating scientific progress (e.g., Bishop, 2016; Naik, 2017). Almost two years after PRO was launched we sought to take stock of the initiative by surveying signatories on their subjective experiences and opinions.

Overall, the signatories’ feedback shows that PRO has potential pitfalls, can meet resistance, and shows room for improvement. However, as can be seen from Figures 1 and 2, the majority of signatories are positive about PRO and its ability to contribute to a more open and transparent science. About 40% of the respondents (i.e., 51/127) indicated that data had been made available, about 30% reported to have received praise from colleagues (i.e., 40/127), and about 25% indicated to have been able to provide a higher-quality review (i.e., 33/127; Figure 1). In contrast, only a small fraction of respondents reported negative experiences, that is, being blacklisted by a journal (i.e., 4/127) or being criticized by colleagues (i.e., 9/127) or journals (i.e., 10/127; Figure 2). In addition, the majority of respondents, expressed their support for PRO: the most commonly stated reasons to still sign today was the belief in the initiative’s contribution to science and its values and ideals (n = 59). Five respondents used the exact phrase: “the right thing to do”.

Figure 1. Signatories’ positive experiences with the PRO initiative. The bars visualize how many respondents indicated to have had each of the positive experiences (multiple answers were possible).

Figure 2. Signatories’ negative experiences with the PRO initiative. The bars visualize how many respondents indicated to have had each of the negative experiences (multiple answers were possible).

Even though this survey only concerns the experiences of a relatively small and highly selective sample it shows that at least from the signatories’ perspective, the PRO initiative has demonstrated that it is a reasonable approach to improving science.

References

Bishop, D. V. M. (2016). Open research practices: Unintended consequences and suggestions for averting them. (Commentary on the Peer Reviewers’ Openness Initiative). Royal Society Open Science, 3, 160109.

Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., … & Vanpaemel, W. (2016). The Peer Reviewers’ Openness Initiative: Incentivizing open research practices through peer review. Royal Society Open Science, 3, 150547.

Naik, G. (2017). Peer-review activists push psychology journals towards open data. Nature News, 543, 161.

About The Authors

Maike Dahrendorf

Maike Dahrendorf is a second-year Research Master student at the University of Amsterdam majoring in psychological methods and social psychology.

Tabea Hoffmann

Tabea Hoffmann is student of Psychology in her second year in the Research Master program at the University of Amsterdam.

Max Mittenbühler

Max Mittenbühler is a Psychology Research Master student at the University of Amsterdam majoring in neuro-/cognitive psychology and psychological methods.