Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength.
1800; 12 (1): e053820
INTRODUCTION: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment.METHODS: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation.RESULTS: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.
View details for DOI 10.1136/bmjopen-2021-053820
View details for PubMedID 35017250
Association between convalescent plasma treatment and mortality in COVID-19: a collaborative systematic review and meta-analysis of randomized clinical trials.
BMC infectious diseases
2021; 21 (1): 1170
BACKGROUND: Convalescent plasma has been widely used to treat COVID-19 and is under investigation in numerous randomized clinical trials, but results are publicly available only for a small number of trials. The objective of this study was to assess the benefits of convalescent plasma treatment compared to placebo or no treatment and all-cause mortality in patients with COVID-19, using data from all available randomized clinical trials, including unpublished and ongoing trials (Open Science Framework, https://doi.org/10.17605/OSF.IO/GEHFX ).METHODS: In this collaborative systematic review and meta-analysis, clinical trial registries (ClinicalTrials.gov, WHO International Clinical Trials Registry Platform), the Cochrane COVID-19 register, the LOVE database, and PubMed were searched until April 8, 2021. Investigators of trials registered by March 1, 2021, without published results were contacted via email. Eligible were ongoing, discontinued and completed randomized clinical trials that compared convalescent plasma with placebo or no treatment in COVID-19 patients, regardless of setting or treatment schedule. Aggregated mortality data were extracted from publications or provided by investigators of unpublished trials and combined using the Hartung-Knapp-Sidik-Jonkman random effects model. We investigated the contribution of unpublished trials to the overall evidence.RESULTS: A total of 16,477 patients were included in 33 trials (20 unpublished with 3190 patients, 13 published with 13,287 patients). 32 trials enrolled only hospitalized patients (including 3 with only intensive care unit patients). Risk of bias was low for 29/33 trials. Of 8495 patients who received convalescent plasma, 1997 died (23%), and of 7982 control patients, 1952 died (24%). The combined risk ratio for all-cause mortality was 0.97 (95% confidence interval: 0.92; 1.02) with between-study heterogeneity not beyond chance (I2=0%). The RECOVERY trial had 69.8% and the unpublished evidence 25.3% of the weight in the meta-analysis.CONCLUSIONS: Convalescent plasma treatment of patients with COVID-19 did not reduce all-cause mortality. These results provide strong evidence that convalescent plasma treatment for patients with COVID-19 should not be used outside of randomized trials. Evidence synthesis from collaborations among trial investigators can inform both evidence generation and evidence application in patient care.
View details for DOI 10.1186/s12879-021-06829-7
View details for PubMedID 34800996
- Medical journal requirements for clinical trial data sharing: Ripe for improvement. PLoS medicine 2021; 18 (10): e1003844
- Author Correction: Mortality outcomes with hydroxychloroquine and chloroquine in COVID-19 from an international collaborative meta-analysis of randomized trials. Nature communications 2021; 12 (1): 3001
Association of Convalescent Plasma Treatment With Clinical Outcomes in Patients With COVID-19: A Systematic Review and Meta-analysis.
Importance: Convalescent plasma is a proposed treatment for COVID-19.Objective: To assess clinical outcomes with convalescent plasma treatment vs placebo or standard of care in peer-reviewed and preprint publications or press releases of randomized clinical trials (RCTs).Data Sources: PubMed, the Cochrane COVID-19 trial registry, and the Living Overview of Evidence platform were searched until January 29, 2021.Study Selection: The RCTs selected compared any type of convalescent plasma vs placebo or standard of care for patients with confirmed or suspected COVID-19 in any treatment setting.Data Extraction and Synthesis: Two reviewers independently extracted data on relevant clinical outcomes, trial characteristics, and patient characteristics and used the Cochrane Risk of Bias Assessment Tool. The primary analysis included peer-reviewed publications of RCTs only, whereas the secondary analysis included all publicly available RCT data (peer-reviewed publications, preprints, and press releases). Inverse variance-weighted meta-analyses were conducted to summarize the treatment effects. The certainty of the evidence was assessed using the Grading of Recommendations Assessment, Development, and Evaluation.Main Outcomes and Measures: All-cause mortality, length of hospital stay, clinical improvement, clinical deterioration, mechanical ventilation use, and serious adverse events.Results: A total of 1060 patients from 4 peer-reviewed RCTs and 10 722 patients from 6 other publicly available RCTs were included. The summary risk ratio (RR) for all-cause mortality with convalescent plasma in the 4 peer-reviewed RCTs was 0.93 (95% CI, 0.63 to 1.38), the absolute risk difference was -1.21% (95% CI, -5.29% to 2.88%), and there was low certainty of the evidence due to imprecision. Across all 10 RCTs, the summary RR was 1.02 (95% CI, 0.92 to 1.12) and there was moderate certainty of the evidence due to inclusion of unpublished data. Among the peer-reviewed RCTs, the summary hazard ratio was 1.17 (95% CI, 0.07 to 20.34) for length of hospital stay, the summary RR was 0.76 (95% CI, 0.20 to 2.87) for mechanical ventilation use (the absolute risk difference for mechanical ventilation use was -2.56% [95% CI, -13.16% to 8.05%]), and there was low certainty of the evidence due to imprecision for both outcomes. Limited data on clinical improvement, clinical deterioration, and serious adverse events showed no significant differences.Conclusions and Relevance: Treatment with convalescent plasma compared with placebo or standard of care was not significantly associated with a decrease in all-cause mortality or with any benefit for other clinical outcomes. The certainty of the evidence was low to moderate for all-cause mortality and low for other outcomes.
View details for DOI 10.1001/jama.2021.2747
View details for PubMedID 33635310
Second versus first wave of COVID-19 deaths: shifts in age distribution and in nursing home fatalities.
OBJECTIVE: To examine whether the age distribution of COVID-19 deaths and the share of deaths in nursing homes changed in the second versus the first pandemic wave.ELIGIBLE DATA: We considered all countries that had at least 4000 COVID-19 deaths occurring as of January 14, 2020, at least 200 COVID-19 deaths occurring in each of the two epidemic wave periods; and which had sufficiently detailed information available on the age distribution of these deaths. We also considered countries with data available on COVID-19 deaths of nursing home residents for the two waves.MAIN OUTCOME MEASURES: Change in the second wave versus the first wave in the proportion of COVID-19 deaths occurring in people <50 years ("young deaths") among all COVID-19 deaths and among COVID-19 deaths in people <70 years old; and change in the proportion of COVID-19 deaths in nursing home residents among all COVID-19 deaths.RESULTS: Data on age distribution were available for 14 eligible countries. Individuals <50 years old had small absolute difference in their share of the total COVID-19 deaths in the two waves across 13 high-income countries (absolute differences 0.0-0.4%). Their proportion was higher in Ukraine, but it decreased markedly in the second wave. The odds of young deaths was lower in the second versus the first wave (summary prevalence ratio 0.81, 95% CI 0.71-0.92) with large between-country heterogeneity. The odds of young deaths among deaths <70 years did not differ significantly across the two waves (summary prevalence ratio 0.96, 95% CI 0.86-1.06). Eligible data on nursing home COVID-19 deaths were available for 11 countries. The share of COVID-19 deaths that were accounted by nursing home residents decreased in the second wave significantly and substantially in 8 countries (prevalence ratio estimates: 0.36 to 0.78), remained the same in Denmark and Norway and markedly increased in Australia.CONCLUSIONS: In the examined countries, age distribution of COVID-19 deaths has been fairly similar in the second versus the first wave, but the contribution of COVID-19 deaths in nursing home residents to total fatalities has decreased in most countries in the second wave.
View details for DOI 10.1016/j.envres.2021.110856
View details for PubMedID 33581086
Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation (PEACHPIE): A systematic strength of methods review.
medRxiv : the preprint server for health sciences
The impact of policies on COVID-19 outcomes is one of the most important questions of our time. Unfortunately, there are substantial concerns about the strength and quality of the literature examining policy impacts. This study systematically assessed the currently published COVID-19 policy impact literature for a checklist of study design elements and methodological issues.We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on November 26 or earlier and screening, all studies were reviewed by three reviewers independently and in consensus. The review tool was based on review guidance for assessing COVID-19 health policy impact evaluation analyses, including first identifying the assumptions behind the methods used, followed by assessing graphical display of outcomes data, functional form for the outcomes, timing between policy and impact, concurrent changes to the outcomes, and an overall rating.After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. The majority (n=23/36) of studies in our sample examined the impact of stay-at-home requirements. Nine studies were set aside due to inappropriate study design (n=8 pre/post; n=1 cross-section), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 1/27 studies passed all of the above checks, and 4/27 were rated as overall appropriate. Including the 9 studies set aside, we found that only four (or by a stricter standard, only one) of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.The current literature directly evaluating the impact of COVID-19 policies largely fails to meet key design criteria for useful inference. This may be partially due to the circumstances for evaluation being particularly difficult, as well as a context with desire for rapid publication, the importance of the topic, and weak peer review processes. Importantly, weak evidence is non-informative and does not indicate how effective these policies were on COVID-19 outcomes.
View details for DOI 10.1101/2021.01.21.21250243
View details for PubMedID 33501457
View details for PubMedCentralID PMC7836129
- Recruitment and Results Reporting of COVID-19 Randomized Clinical Trials Registered in the First 100 Days of the Pandemic. JAMA network open 2021; 4 (3): e210330
Change in age distribution of COVID-19 deaths with the introduction of COVID-19 vaccination.
Most countries initially deployed COVID-19 vaccines preferentially in elderly populations. We aimed to evaluate whether population-level vaccine effectiveness is heralded by an increase in the relative proportion of deaths among non-elderly populations that were less covered by vaccination programs.We collected data from 40 countries on age-stratified COVID-19 deaths during the vaccination period (1/14/2021-5/31/2021) and two control periods (entire pre-vaccination period and excluding the first wave).We meta-analyzed the proportion of deaths in different age groups in vaccination versus control periods in countries with low vaccination rates; (2) countries with age-independent vaccination policies; and (3) countries with standard age-dependent vaccination policies.Countries that prioritized vaccination among older people saw an increasing share of deaths among 0-69 year old people in the vaccination versus the two control periods (summary proportion ratio 1.32 [95 CI% 1.24-1.41] and 1.35 [95 CI% 1.26-1.44)]. No such change was seen on average in countries with age-independent vaccination policies (1.05 [95 CI% 0.78-1.41 and 0.97 [95 CI% 0.95-1.00], respectively) and limited vaccination (0.93 [95 CI% 0.85-1.01] and 0.95 [95 CI% 0.87-1.03], respectively). Proportion ratios were associated with the difference of vaccination rates in elderly versus non-elderly people. No significant changes occurred in the share of deaths in age 0-49 among all 0-69 deaths in the vaccination versus pre-vaccination periods.The substantial shift in the age distribution of COVID-19 deaths in countries that rapidly implemented vaccination predominantly among elderly provides evidence for the population level-effectiveness of COVID-19 vaccination and a favorable evolution of the pandemic towards endemicity with fewer elderly deaths.
View details for DOI 10.1016/j.envres.2021.112342
View details for PubMedID 34748775
Mortality outcomes with hydroxychloroquine and chloroquine in COVID-19 from an international collaborative meta-analysis of randomized trials.
2021; 12 (1): 2349
Substantial COVID-19 research investment has been allocated to randomized clinical trials (RCTs) on hydroxychloroquine/chloroquine, which currently face recruitment challenges or early discontinuation. We aim to estimate the effects of hydroxychloroquine and chloroquine on survival in COVID-19 from all currently available RCT evidence, published and unpublished. We present a rapid meta-analysis of ongoing, completed, or discontinued RCTs on hydroxychloroquine or chloroquine treatment for any COVID-19 patients (protocol: https://osf.io/QESV4/ ). We systematically identified unpublished RCTs (ClinicalTrials.gov, WHO International Clinical Trials Registry Platform, Cochrane COVID-registry up to June 11, 2020), and published RCTs (PubMed, medRxiv and bioRxiv up to October 16, 2020). All-cause mortality has been extracted (publications/preprints) or requested from investigators and combined in random-effects meta-analyses, calculating odds ratios (ORs) with 95% confidence intervals (CIs), separately for hydroxychloroquine and chloroquine. Prespecified subgroup analyses include patient setting, diagnostic confirmation, control type, and publication status. Sixty-three trials were potentially eligible. We included 14 unpublished trials (1308 patients) and 14 publications/preprints (9011 patients). Results for hydroxychloroquine are dominated by RECOVERY and WHO SOLIDARITY, two highly pragmatic trials, which employed relatively high doses and included 4716 and 1853 patients, respectively (67% of the total sample size). The combined OR on all-cause mortality for hydroxychloroquine is1.11 (95% CI: 1.02, 1.20; I=0%; 26 trials; 10,012 patients) and for chloroquine 1.77 (95%CI: 0.15, 21.13, I=0%; 4 trials; 307 patients). We identified no subgroup effects. We found that treatment with hydroxychloroquine is associated with increased mortality in COVID-19 patients, and there is no benefit of chloroquine. Findings have unclear generalizability to outpatients, children, pregnant women, and people with comorbidities.
View details for DOI 10.1038/s41467-021-22446-z
View details for PubMedID 33859192
Population-level COVID-19 mortality risk for non-elderly individuals overall and for non-elderly individuals without underlying diseases in pandemic epicenters.
2020; 188: 109890
OBJECTIVE: To provide estimates of the relative rate of COVID-19 death in people <65 years old versus older individuals in the general population, the absolute risk of COVID-19 death at the population level during the first epidemic wave, and the proportion of COVID-19 deaths in non-elderly people without underlying diseases in epicenters of the pandemic.ELIGIBLE DATA: Cross-sectional survey of countries and US states with at least 800 COVID-19 deaths as of April 24, 2020 and with information on the number of deaths in people with age <65. Data were available for 14 countries (Belgium, Canada, France, Germany, India, Ireland, Italy, Mexico, Netherlands, Portugal, Spain, Sweden, Switzerland, UK) and 13 US states (California, Connecticut, Florida, Georgia, Illinois, Indiana, Louisiana, Maryland, Massachusetts, Michigan, New Jersey, New York, Pennsylvania). We also examined available data on COVID-19 deaths in people with age <65 and no underlying diseases.MAIN OUTCOME MEASURES: Proportion of COVID-19 deaths in people <65 years old; relative mortality rate of COVID-19 death in people <65 versus ≥65 years old; absolute risk of COVID-19 death in people <65 and in those ≥80 years old in the general population as of June 17, 2020; absolute COVID-19 mortality rate expressed as equivalent of mortality rate from driving a motor vehicle.RESULTS: Individuals with age <65 account for 4.5-11.2% of all COVID-19 deaths in European countries and Canada, 8.3-22.7% in the US locations, and were the majority in India and Mexico. People <65 years old had 30- to 100-fold lower risk of COVID-19 death than those ≥65 years old in 11 European countries and Canada, 16- to 52-fold lower risk in US locations, and less than 10-fold in India and Mexico. The absolute risk of COVID-19 death as of June 17, 2020 for people <65 years old in high-income countries ranged from 10 (Germany) to 349 per million (New Jersey) and it was 5 per million in India and 96 per million in Mexico. The absolute risk of COVID-19 death for people ≥80 years old ranged from 0.6 (Florida) to 17.5 per thousand (Connecticut). The COVID-19 mortality rate in people <65 years old during the period of fatalities from the epidemic was equivalent to the mortality rate from driving between 4 and 82 miles per day for 13 countries and 5 states, and was higher (equivalent to the mortality rate from driving 106-483 miles per day) for 8 other states and the UK. People <65 years old without underlying predisposing conditions accounted for only 0.7-3.6% of all COVID-19 deaths in France, Italy, Netherlands, Sweden, Georgia, and New York City and 17.7% in Mexico.CONCLUSIONS: People <65 years old have very small risks of COVID-19 death even in pandemic epicenters and deaths for people <65 years without underlying predisposing conditions are remarkably uncommon. Strategies focusing specifically on protecting high-risk elderly individuals should be considered in managing the pandemic.
View details for DOI 10.1016/j.envres.2020.109890
View details for PubMedID 32846654
The worldwide clinical trial research response to the COVID-19 pandemic - the first 100 days.
2020; 9: 1193
Background: Never before have clinical trials drawn as much public attention as those testing interventions for COVID-19. We aimed to describe the worldwide COVID-19 clinical research response and its evolution over the first 100 days of the pandemic. Methods: Descriptive analysis of planned, ongoing or completed trials by April 9, 2020 testing any intervention to treat or prevent COVID-19, systematically identified in trial registries, preprint servers, and literature databases. A survey was conducted of all trials to assess their recruitment status up to July 6, 2020. Results: Most of the 689 trials (overall target sample size 396,366) were small (median sample size 120; interquartile range [IQR] 60-300) but randomized (75.8%; n=522) and were often conducted in China (51.1%; n=352) or the USA (11%; n=76). 525 trials (76.2%) planned to include 155,571 hospitalized patients, and 25 (3.6%) planned to include 96,821 health-care workers. Treatments were evaluated in 607 trials (88.1%), frequently antivirals (n=144) or antimalarials (n=112); 78 trials (11.3%) focused on prevention, including 14 vaccine trials. No trial investigated social distancing. Interventions tested in 11 trials with >5,000 participants were also tested in 169 smaller trials (median sample size 273; IQR 90-700). Hydroxychloroquine alone was investigated in 110 trials. While 414 trials (60.0%) expected completion in 2020, only 35 trials (4.1%; 3,071 participants) were completed by July 6. Of 112 trials with detailed recruitment information, 55 had recruited <20% of the targeted sample; 27 between 20-50%; and 30 over 50% (median 14.8% [IQR 2.0-62.0%]). Conclusions: The size and speed of the COVID-19 clinical trials agenda is unprecedented. However, most trials were small investigating a small fraction of treatment options. The feasibility of this research agenda is questionable, and many trials may end in futility, wasting research resources. Much better coordination is needed to respond to global health threats.
View details for DOI 10.12688/f1000research.26707.1
View details for PubMedID 33082937