This week marked the launch of a new diamond open-access journal: the Journal of Robustness Reports. This journal will publish short reanalyses of published empirical research findings. This is the abstract of the editorial:
The vast majority of empirical research articles report a single primary analysis outcome that is the result of a single analysis plan, executed by a single analysis team (usually the team that also designed the experiment and collected the data). However, recent many-analyst projects have demonstrated that different analysis teams generally adopt a unique approach and that there exists considerable variability in the associated conclusions. There appears to be no single optimal statistical analysis plan, and different plausible plans need not lead to the same conclusion. A high variability in outcomes signals that the conclusions are relatively fragile and dependent on the specifics of the analysis plan. Crucially, without multiple teams analyzing the data, it is difficult to gauge the extent to which the conclusions are robust. We have recently proposed that empirical articles of particular scientific interest or societal importance are accompanied by two or three short reports that summarize the results of alternative analyses conducted by independent experts [1]. In order to showcase the practical feasibility and epistemic benefits of this approach we have founded the Journal of Robustness Reports, which is dedicated to publishing short reanalyses of empirical findings. This editorial describes the scope and the workflow of the Journal of Robustness Reports including the type and format of the published articles. We hope that the Journal of Robustness Reports will help make reanalyses of published findings the norm across the empirical sciences.
JRR Article Format
As detailed in the editorial, Robustness Reports in JRR consist of the following sections:
- An Abstract section that does not exceed two lines in print. This is equivalent to 180 characters (including spaces).
- A Goal section, which outlines the question that the reanalysis is trying to address.
- A Methods section, which provides background information about the reanalysis.
- A Results section, which describes the main outcomes of the reanalysis.
- A Conclusion section, which compares the results from the reanalysis to those from the original analysis and assesses the degree to which the reanalysis corroborates or undercuts the conclusions from the original analysis.
- An Acknowledgments and Disclosures section, which may consist of the following subsections:
• A Reproducibility subsection. The purpose of this subsection is to declare whether or not the reanalysis team was able to reproduce the original analysis, along with an explanation in those cases where this proved to be impossible.
• A Code and Data Availability subsection. At a minimum, this subsection must include a link to a public FAIR repository that contains the code and data used for the reanalysis. All reanalyses must be fully reproducible. The Robustness Report authors might choose to supply the original findings with additional data, which also need to be made publicly available in the same FAIR repository. The repository may also contain a more extensive version of the Robustness Report.
• An Author Contribution subsection. For multiple-author articles, this subsection may be used to document the contributorship of the different authors.
• A Funding subsection.
• A Conflict of Interest subsection. - A References section.
JRR limits Robustness Reports to only 500 words (excluding the Acknowledgments and Disclosures section, the References section, figure/table captions, and the title page) and one display element (table/figure), with additional material presented as online supplements. A constructive response of the original authors and the editorial summary are also limited to 500 words and one display element, but need not feature the sections outlined above. Article templates are available online in either Word or LaTeX. The length restrictions are in place because one of the goals of JRR is to convince mainstream empirical journals to adopt the format and arrange their own robustness reports to accompany articles of substantial interest — an investment of 500 words is minimal and similar to that of a comment or a letter to the editor (which which many journals are already familiar).
SciPost Infrastructure
JRR uses the infrastructure of SciPost, an initiative for diamond open-access journals that originated in physics but is now expanding to other disciplines as well. The collaboration between JRR and SciPost means that all of the article-handling logistics is already in place. The JRR journal website looks like this:
In the near future we will start a companion website that is meant for JRR-related blog posts.
Two Example Reports
As you can see from the JRR website screenshot above, the editorial came with two example Robustness Reports. The first example report is by Suzanne Hoogeveen:
The second example report is by Alexandra Sarafoglou:
JRR Logo
The JRR logo was designed by Johan van der Woude at Studio Black and is available under a CC-BY license. The logo is a reference to an iceberg (where the part that is submerged refers to the uncertainty that is hidden from view) and to a diamond (because of the diamond open-access nature of JRR).
Plans for the Immediate Future
In order for JRR to become a success, we hope that statisticians, methodologists, data-analysts, and researchers will be interested in contributing to the journal. Those who have the motivation and the editorial experience may be invited to join the JRR Editorial College, which we hope to expand greatly over the next months. Those who appreciate the journal’s mission can help JRR by simply spreading the word.
Funding
The work that resulted in the launch of JRR was made possible in part by a an NWO Open Science grant (OSF23.1.030) and a 2024 Ammodo Science Award. The JRR initiative itself was also supported by the UvA diamond open-access fund.
References
Bartoš, F., Sarafoglou, A., Aczel, B., Hoogeveen, S., Chambers, C., & Wagenmakers, E.-J. (2025). Introducing the Journal of Robustness Reports. Journal of Robustness Reports.
Hoogeveen, S. (2025). Water tasting: A Bayesian multiverse analysis. Journal of Robustness Reports.
Sarafoglou, A. (2025). Preference for Chaudfontaine or bias towards the preferred option? Journal of Robustness Reports.
