There are many lessons to learn from the way studies are conducted and results are presented. With the use of only RRRs, and omitting ARRs, reporting bias is introduced, which affects the interpretation of vaccine efficacy.10
When communicating about vaccine efficacy, especially for public health decisions such as choosing the type of vaccines to purchase and deploy, having a full picture of what the data actually show is important, and ensuring comparisons are based on the combined evidence that puts vaccine trial results in context and not just looking at one summary measure, is also important. Such decisions should be properly informed by detailed understanding of study results, requiring access to full datasets and independent scrutiny and analyses.
Unfortunately, comparing vaccines on the basis of currently available trial (interim) data is made even more difficult by disparate study protocols, including primary endpoints (such as what is considered a COVID-19 case, and when is this assessed), types of placebo, study populations, background risks of COVID-19 during the study, duration of exposure, and different definitions of populations for analyses both within and between studies, as well as definitions of endpoints and statistical methods for efficacy.
Importantly, we are left with the unanswered question as to whether a vaccine with a given efficacy in the study population will have the same efficacy in another population with different levels of background risk of COVID-19. This is not a trivial question because transmission intensity varies between countries, affected by factors such as public health interventions and virus variants.
The only reported indication of vaccine effectiveness is the Israeli mass vaccination campaign using the Pfizer–BioNTech product. Although the design and methodology are radically different from the randomised trial,2 Dagan and colleagues11 report an RRR of 94%, which is essentially the same as the RRR of the phase 3 trial (95%) but with an ARR of 0·46%, which translates into an NNV of 217 (when the ARR was 0·84% and the NNV was 119 in the phase 3 trial). This means in a real-life setting, 1·8 times more subjects might need to be vaccinated to prevent one more case of COVID-19 than predicted in the corresponding clinical trial.
Uncoordinated phase 3 trials do not satisfy public health requirements; platform trials designed to address public health relevant questions with a common protocol will allow decisions to be made, informed by common criteria and uniform assessment. These considerations on efficacy and effectiveness are based on studies measuring prevention of mild to moderate COVID-19 infection; they were not designed to conclude on prevention of hospitalisation, severe disease, or death, or on prevention of infection and transmission potential.
Assessing the suitability of vaccines must consider all indicators, and involve safety, deployability, availability, and costs. We declare no competing interests.Copyright © 2021 The Author(s). Published by Elsevier Ltd.
This is an Open Access article under the CC BY-NC-ND 4.0 license. *Piero Olliaro, Els Torreele, Michel Vaillant[email protected]Centre for Tropical Medicine and Global Health, Nuffield Department of Medicine, University of Oxford, Oxford OX3 7FZ, UK (PO); Institute for Innovation and Public Purpose, University College London, London, UK (ET); Competence Center for Methodology and Statistics, Luxembourg Institute of Health, Strassen, Luxemburg (MV)1Zimmer C, Corum J, Wee S-L. Covid-19
Vaccine Tracker. https://www.nytimes.com/interactive/2020/science/coronavirus-vaccine-tracker.html (accessed March 10, 2021).2Polack FP, Thomas SJ, Kitchin N, et al. Safety and efficacy of the BNT162b2 mRNA COVID-19 Vaccine. N Engl J Med 2020; 383: 2603–15.3Baden LR, El Sahly HM, Essink B, et al.
Efficacy and safety of the mRNA-1273 SARS-CoV-2 Vaccine. N Engl J Med 2021; 384: 403–16.4Voysey M, Clemens SAC, Madhi SA, et al. Safety and efficacy of the ChAdOx1 nCoV-19 vaccine (AZD1222) against SARS-CoV-2: an interim analysis of four randomised controlled trials in Brazil, South Africa, and the UK.
Lancet 2021; 397: 99–111.5Logunov DY, Dolzhikova IV, Shcheblyakov DV, et al. Safety and efficacy of an rAd26 and rAd5 vector-based heterologous prime-boost COVID-19 vaccine: an interim analysis of a randomised controlled phase 3 trial in Russia. Lancet2021; 397: 671–81.6US Food and Drug Administration.
Vaccines and Related Biological Products Advisory Committee meeting: FDA briefing document. Dec 10, 2020. https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-december-10-2020-meeting-announcement (accessed March 10, 2021).7US Food and Drug Administration.
Vaccines and Related Biological Products Advisory Committee meeting: FDA briefing document. Dec 17, 2020. https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-december-17-2020-meeting-announcement (accessed March 10, 2021).8US Food and Drug Administration.
Vaccines and Related Biological Products Advisory Committee meeting: FDA briefing document. Feb 26, 2021. https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-february-26-2021-meeting-announcement (accessed March 10, 2021).9Olliaro P.
What does 95% COVID-19 vaccine efficacy really mean? Lancet Infect Dis 2021; published online Feb 17. https://doi.org/10.1016/S1473-3099(21)00075-X.10Brown RB. Outcome reporting bias in COVID-19 mRNA vaccine clinical trials.
Medicina (Kaunas) 2021; 57: 199.11Dagan N, Barda N, Kepten E, et al. BNT162b2 mRNA COVID-19 vaccine in a nationwide mass vaccination setting. N Engl J Med 2021; published online Feb 24. https://doi.org/10.1056/NEJMoa2101765.