Preprint: A Bayesian Reanalysis of the Effects of Hydroxychloroquine and Azithromycin on Viral Carriage in Patients with COVID-19 (Reply to Gautret et al. 2020)

Below is a summary of a preprint that features a Bayesian reanalysis of the famous/infamous Gautret et al. data. What I like about this preprint is (a) the multiverse analysis; (b) the Bayesian conclusions — they are so easy to obtain with JASP, and provide much more information then just “p<.05” or “p>.05”; but what I like most of all is (c) the emphasis on the fact that in the end, design always beats analysis — the Gautret et al. case strikes me as a schoolbook example of this principle. The preprint is hosted on the Open Science Framework with materials. The work is explained in a series of tweets.

Abstract

Gautret and colleagues reported results of a non-randomised open-label case series which examined the effects of hydroxychloroquine and azithromycin on viral load in the upper respiratory tract of severe acute respiratory syndrome-Coronavirus-2 (SARS-CoV-2) patients. The authors report that hydroxychloroquine (HCQ) had significant virus reducing effects, and that dual treatment of both HCQ and azithromycin further enhanced virus reduction. These data have triggered speculation whether these drugs should be considered as candidates for the treatment of severe COVID-19. However, questions have been raised regarding the study’s data integrity, statistical analyses, and experimental design. We therefore reanalysed the original data to interrogate the main claims of the paper. Here we apply Bayesian statistics to assess the robustness of the original paper’s claims by testing four variants of the data: 1) The original data; 2) Data including patients who deteriorated; 3) Data including patients who deteriorated with exclusion of untested patients in the comparison group; 4) Data that includes patients who deteriorated with the assumption that untested patients were negative. To ask if HCQ monotherapy is effective, we performed an A/B test for a model which assumes a positive effect, compared to a model of no effect. We find that the statistical evidence is highly sensitive to these data variants. Statistical evidence for the positive effect model ranged from strong for the original data (BF+0 ~11), to moderate when including patients who deteriorated (BF+0 ~4.35), to anecdotal when excluding untested patients (BF+0 ~2), and to anecdotal negative evidence if untested patients were assumed positive (BF+0 ~0.6). To assess whether HCQ is more effective when combined with AZ, we performed the same tests, and found only anecdotal evidence for the positive effect model for the original data (BF+0 ~2.8), and moderate evidence for all other variants of the data (BF+0 ~5.6). Our analyses only explore the effects of different assumptions about excluded and untested patients. These assumptions are not adequately reported, nor are they justified in the original paper, and we find that varying them causes substantive changes to the evidential support for the main claims of the original paper. This statistical uncertainty is exacerbated by the fact that the treatments were not randomised, and subject to several confounding variables including the patients consent to treatment, different care centres, and clinical decision-making. Furthermore, while the viral load measurements were noisy, showing multiple reversals between test outcomes, there is greater certainty around other clinical outcomes such as the 4 patients who seriously deteriorated. The fact that all of these belonged to the HCQ group should be assigned greater weight when evaluating the potential clinical efficacy of HCQ. Randomised controlled trials are currently underway, and will be critical in resolving this uncertainty as to whether HCQ and AZ are effective as a treatment for COVID-19.

Posterior Distributions for the Four Data Variants

 

References

O. J. Hulme, Wagenmakers, E.-J., Damkier. P., Madelung, C. F., Siebner, H. R., Helweg-Larsen, J., Gronau, Q. F., Benfield, T., &, Madsen, K. H. (2020). Reply to Gautret et al. 2020: A Bayesian reanalysis of the effects of hydroxychloroquine and azithromycin on viral carriage in patients with COVID-19. Manuscript submitted for publication.

About The Author

Eric-Jan Wagenmakers

Eric-Jan (EJ) Wagenmakers is professor at the Psychological Methods Group at the University of Amsterdam.