That Newsweek Article: Review of Yale Epidemiologist’s Key to Defeating COVID
After writing about Dr. Stella Immanuel’s viral video, the most common request I got was to assess this Newsweek Opinion piece circulating by Dr. Harvey Risch, a Professor of Epidemiology at Yale, claiming that we already have the key to defeating COVID (hydroxychloroquine), and we need to start using it. So let’s assess his argument and see if it holds merit.
He argues that hydroxychloroquine has proven to be effective against COVID, in particular when it is given early on in the disease course and when combined with azithromycin (or doxycycline, another antibiotic) and zinc. This is based on 5 studies summarized in his publication in the American Journal of Epidemiology (AJE) Early Outpatient Treatment of Symptomatic, High-Risk Covid-19 Patients that Should be Ramped-Up Immediately as Key to the Pandemic Crisis and 7 more studies published in a follow-up letter. He additionally points to two examples of correlation between hydroxychloroquine prescription and mortality rate in Pará, Brazil and Switzerland. He further argues that the reason that other studies have not shown benefit is that they were not used in the proper setting: it should be given early in the course of disease to high risk patients (although he does point to two studies done in hospitalized patients that show benefit; so it seems he is also arguing that there is some efficacy even when the drug is given later in the disease course after patients are already quite sick).
Now, let me clarify the purpose of this blog post. My goal is to evaluate Dr. Risch’s claim that we already have evidence that this treatment is effective based on the studies he has cited. He is not saying ‘maybe this works let’s study this more,’ he is arguing that we already have enough evidence to show that it works (at least enough evidence for a pandemic setting), so we need to start prescribing it now. Therefore, my goal is to evaluate that claim. Do we have enough evidence to show that it works? Do the studies he cited truly demonstrate efficacy of the drug(s)? Can we reliably say that the hydroxychloroquine drug combo is the “key to defeating COVID-19” based on the data he cited? These are the questions this post is tackling, not whether or not more hydroxychloroquine combo studies are warranted. That is another discussion for another day.
First, let’s nail down what treatment combo he is saying is effective. In his Newsweek editorial he seems to argue that hydroxychloroquine + azithromycin (or doxycycline) + zinc + given early (outpatient) + high risk patients is the best combo. To clarify a few terms — outpatient describes patients who are treated outside of a hospital (like at a clinic/doctor’s office), and inpatient describes patients who are admitted to a hospital for treatment (i.e. they have their own bed). Generally, outpatient patients are less sick / early in their disease course, and inpatients are quite sick / later in their disease course. ‘High risk’ is not explicitly defined, but he seems to mean patients that are older or have underlying conditions (those who are at higher risk of dying from COVID). However, given that the criteria for what ‘high risk’ means isn’t precisely defined, I’m not going to try to determine if the studies he cites really study a ‘high risk’ population, as without clear criteria of precisely what that means it becomes a bit subjective.
While he strongly advocates for use of the hydroxychloroquine drug combo for outpatients, his comments about use of hydroxychloroquine alone or in hospitalized patients are conflicting… sometimes he says that data is irrelevant: “Evidence about use of hydroxychloroquine alone, or of hydroxychloroquine+azithromycin in inpatients, is irrelevant concerning efficacy of the pair in early high-risk outpatient disease” and sometimes he seems to cite the data as evidence to support his argument “Even so, it has demonstrated significant benefit in large hospital studies in Michigan and New York City when started within the first 24 to 48 hours after admission.” So his opinion on hydroxychloroquine +/- other drugs in a hospital setting (very sick patients) is a little unclear, but perhaps he thinks there is some benefit. Additionally, in his original publication in the AJE, he doesn’t emphasize the importance of zinc: “all of these reviews have omitted the two critical aspects of reasoning about these drugs: use of HCQ combined with AZ or with doxycycline, and use in the outpatient setting,” but in his Newsweek editorial he does. So for each study below, I will evaluate three different scenarios:
- Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?
- Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?
- Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?
One of the main things we learn to do in MD-PhD training is to evaluate study design. One of my favorite classes in grad school was “Method and Logic,” where we ripped apart studies and evaluated whether or not the data they provide actually supports the conclusions they made. Before taking that class, I naively thought that nearly every scientific paper’s claims were reliably supported by their data. But that is not true — many do support it (perhaps with a few minor weaknesses), but a surprising number have significant and sometimes severe methodological flaws. It is our job as scientists to not just blindly accept the conclusions provided by the authors of the study, but to see if their data and their study design really support the conclusions they are making. So that is what I’m going to do with the studies cited by this epidemiologist.
Before we look at the studies, let’s talk about a few things that are essential for a scientific study. Note I’m not even talking about the strengths of different types of studies (observational versus randomized), I’m talking about what are basic criteria that any type of study needs to have in order to be considered valid. These are more like bare minimum standards:
1. The details of the data are made available.
2. We know the people being studied actually have the disease we’re trying to study.
3. Patients aren’t eliminated from analysis because they got sick or died.
4. The statistics are sound.
5. There is an adequate control group.
What is an ‘adequate’ control group and why is it important? A study must have a control group (a similar group of patients who did not get the treatment) to know if any benefit you see is actually from the drug(s) and not from something else about the population you are studying. As a hypothetical example, if you did an analysis of lollypop consumption in all COVID patients, you would likely find that it is associated with good COVID outcomes. Would that mean lollypops cure COVID? No, it just means that children are more likely eat lollypops than adults, and for reasons unrelated to lollypop consumption, they are also less likely to get severely sick from COVID. This example is obvious because nobody actually thinks lollypops can cure COVID, but similar things can happen with drug treatments. Maybe doctors were more likely to give hydroxychloroquine drug combos to patients who were less sick? Or more sick? Or maybe the particular hospital or clinic giving the drugs serves a different demographic of patients with different underlying conditions? All of these can impact clinical outcomes, thus an adequate control group is essential to make any meaningful conclusions about whether or not a drug really works. So what do I mean by ‘adequate control group?’ I mean that we can be reasonably convinced that the patients in the control group are similar enough to the patients in the treatment group that we can think of them as roughly equal groups of people (at least, equal in terms of factors that impact how sick they become / their risk of dying).
Now, here are the studies:
Citation #1: Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial
Type of Study: Observational
Outcome: Positive for Virus after 6 days
Sample Size: 36 patients
Hospitalized or Outpatient: Mix*
Treatment Regimen: Hydroxychloroquine +/- Azithromycin
Summary: This was the first hydroxychloroquine study to get a lot of attention, published by Dr. Raoult back in March. They gave hydroxychloroquine +/- azithromycin to 20 people, and included 16 people that either refused treatment or were from a different medical center as controls. They tested for the presence of the virus in nasal swabs, and concluded that after several days, those who got the treatment were more likely to test negative for the virus.
There are many issues with this study and I will not be able to address all of them (check out this post for a more detailed discussion of some of them.) However I will highlight what I consider to be the biggest flaw: there were originally 26 people in the hydroxychloroquine (+/- azithromycin) group, but 6 of those people weren’t included in the final data analysis. Why? Because they didn’t finish the treatment. Why didn’t they finish the treatment? Because one got too nauseous from the medication, one went home, three were transferred to the ICU (meaning they got very, very sick), and one of them died. That means 15% of people who got the drug(s) had very bad outcomes (ICU or death), compared to 0% in the control group. If people are clinically deteriorating in the treatment group and not the control group, I don’t really care if there are differences in whether or not their nasal swabs were positive for the virus. Clinical outcomes are what ultimately matters, not nasal PCR tests. Notably, the person who died was PCR-negative the day before he/she died, which also makes me question whether or not the results of their PCR test (which is what their entire results are based on) are reliable and/or have anything to do with how sick the people were. Additionally, the control group was not a good control group. The controls were either children who were not sick, or adults from another medical center. Why does it matter if they were from another medical center? Maybe that medical center served a different demographic group and there were more people with pre-existing conditions. Maybe the PCR test used to measure the presence of the virus was different at the other medical center (from an earlier version of the paper, the way the PCR results were reported was different for some of the control subjects, making me think they used a different method. This is not good). In short, we know very little about how comparable the treatment and control groups actually were, and what we do know is not comforting. An experiment is only as good as its controls. You simply cannot conclude anything without adequate controls in your experiment.
*While the study states it was done it a “hospital setting”, it sounds like some in the control group may have not been hospitalized. Additionally, some patients in both the treatment and control group were asymptomatic, which makes me think they were not hospitalized.
Conclusion:
1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?
No – not tested
2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?
No – not tested (not outpatient)
3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?
Not really – main outcome (viral shedding) was not a measure of clinical severity, and more patients in treatment group had bad clinical outcomes (ICU/death). Also lots of other problems.
Citation #2: Early Treatment of COVID-19 Patients With Hydroxychloroquine and Azithromycin: A Retrospective Analysis of 1061 Cases in Marseille, France
Type of Study: Uncontrolled
Outcome: Death, Clinical Worsening, Viral Shedding
Sample Size: 1061 patients
Outpatient or Hospitalized: Mixed (Inpatient and ‘day-care’ hospital)
Treatment Regimen: Hydroxychloroquine + Azithromycin
Summary: This study was run by the same group as Study #1. They gave hydroxychloroquine and azithromycin to 1061 patients and tracked a variety of parameters. However, this study has no control group to compare to (patients who did not get the treatment), so it does not provide any useful evidence about the effect of the treatment.
Conclusion:
1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?
No – not tested
2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?
No – not exclusively outpatient, and no control group
3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?
No – no control group
Citation #3: Dr. Zelenko’s Clinical Protocol (google doc)
Type of Study: Uncontrolled
Outcome: Death/Hospitalization/Intubation
Sample Size: 405
Population: Outpatient, some high risk
Treatment Regimen: Hydroxychloroquine + Azithromycin + Zinc
Summary: This isn’t a publication, it’s a google doc by Dr. Zelenko describing his treatment regimen. He does provide a few sentences about the patients he has treated — he reports treating 405 cases in an outpatient setting that are either confirmed or suspected to have COVID. He argues that treatment should be started before the diagnosis is confirmed, so it is not clear whether every patient was eventually confirmed to have COVID or not. Two patients died, six were hospitalized, and four were intubated. It is unclear how robust his follow-up is to track the outcome of patients. There is no control group.
Conclusion:
1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?
No – no control group, uncertain if patients had COVID
2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?
No – not tested
3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?
No – no control group, uncertain if patients had COVID
Citation #4: Empirical treatment with hydroxychloroquine and azithromycin for suspected cases of COVID-19 followed-up by telemedicine
Type of Study: Observational
Outcome: Need for Hospitalization
Sample Size: 636
Outpatient or Hospitalized: Outpatient (Telemedicine)
Treatment Regimen: Hydroxychloroquine + Azithromycin
Summary: This non-peer reviewed study evaluated patients who had mild COVID-like symptoms (but the diagnosis was not confirmed) in an outpatient setting (via telemedicine). All patients were offered hydroxychloroquine/azithromycin, and those who refused served as the control group (224 patients) while those who accepted were the treatment group (412 patients). They report lower hospitalization in the treatment group (1.9%) versus control group (5.4%). They also look at differences in how early patients in the treatment group started their treatment, and report that those who started it earlier (< 7 days from start of symptoms) were less likely to go to the hospital (1.17%) than those who started it later (3.2%).
On first glance this study looks much better than the previous two, as it includes a control group. This is not an ideal control group as there may be significant differences in people who refused treatment versus not, but it’s certainly better than nothing. However, as I dug into the study, I found a couple things that were pretty funky and didn’t quite add up. First, they do not actually confirm that their patients had COVID. They enrolled anyone with “flu-like symptoms,” and do not do any diagnostic testing. Some of the patients underwent CT scans, and they found results of 40% of scanned patients in the control group were suggestive of COVID versus 70% in the treatment group. Over half of the patients in the study were not scanned, so all we really know about them is that they had flu-like symptoms for at least 2-3 days and weren’t sick enough to go the hospital. Given that lots of viruses cause mild flu-like symptoms, it is very likely that not all of these patients actually had COVID. If the percent of true COVID patients was different between the two groups, this could invalidate the results.
There are also some funky things with their statistics. The main conclusion they report is that 1.9% of the patients who got the treatment went to the hospital versus 5.4% of those who didn’t get treatment, with a reported p-value of p < 0.0001. It is standard in studies like this to not only report the percentage of patients, but to also include the actual number of patients who were hospitalized in each group. The authors do this for most of their data with the exception of their main results, which I find odd. For their main results, they only report the percentage and a p-value. But assuming this is the percentage of the total patients in each group (which is what the study implies), we can calculate it: 412 x 1.9% = 8 hospitalized patients in the treatment group and 224 x 5.4% = 12 hospitalized patients in the control group. They don’t explicitly state which statistical test they use to get the p-value < 0.0001, but the only two tests included in their methods that are used for this type of analysis would be the Fisher Exact Test or the Chi-square Test. I ran these statistical tests on their data, and the p-values they return are p = 0.02 and p = 0.03, respectively. So something is very wrong, because that is a very different result than the p < 0.0001 they report. It is still modestly significant, but it gives me a significant pause as to how they ran their analysis and the transparency of their data. We additionally do not get the numbers for their analysis of hospitalization rates in the early versus late treatment groups. But thankfully, with a little high school algebra, we can back-calculate them as well based on a total of 8 hospitalizations in the treatment group and the percentages provided in Figure 2. This gets us 255 patients treated early (3 hospitalized) and 157 treated late (5 hospitalized). If we run a chi-square on this data, though they report p < 0.0001, the result is not significant (p = 0.15). This is very alarming; something is very wrong. Some of their p-values in Table 1 seem to have a similar problem.
In conclusion, my main criticism of this study is that we don’t know if all the patients actually had COVID, which really limits the conclusions we can draw from it. My secondary criticism is that some of the numbers and statistics don’t add up. If the authors could be more transparent in their data and calculations, that could solve the second problem, but not the first.
Conclusion:
1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?
No – not tested
2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?
No – unknown if patients had COVID, stats questionable
3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?
No – unknown if patients had COVID, stats questionable
Citation #5: Long Island Long-Term Care Facility
Type of Study: Uncontrolled
Outcome: Hospitalization / Death
Sample Size: 45 (news report), 200 (unofficial update)
Outpatient or Hospitalized: Outpatient (Long-term care facility)
Treatment Regimen: Hydroxychloroquine + Doxycycline
Summary: This is a news report as well as personal correspondence about a long-term care facility who is giving hydroxychloroquine + doxychycline to its residents. It sounds like they are giving it to residents diagnosed with COVID (rather than prophylactically), but I am not certain, as there are not very many details provided. 5.6% and 4.5% of patients died according to the news report and unofficial update, respectively. This does not have a control group.
Conclusion:
1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?
No – no not tested
2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin (or doxycycline) + given early (outpatient) has a clinical benefit?
No – no control group
3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?
No – no control group
Type of Study: Uncontrolled
Outcome: Hospitalization / Death
Sample Size: 54
Outpatient or Hospitalized: Outpatient (Long-term care facility)
Treatment Regimen: Hydroxychloroquine + Doxycycline
Summary: This is a non-peer reviewed case study (that I believe includes some of the same patients in Study #5, which is why I called this Study #5.5). Residents of a long-term care facility who were diagnosed or presumed to have COVID were given hydroxychloroquine + doxycycline. 11% went to the hospital and 6% died. This does not have a control group.
Conclusion:
1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?
No – no not tested
2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin (or doxycycline) + given early (outpatient) has a clinical benefit?
No – no control group
3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?
No – no control group
Citation #6: Outcomes of 3,737 COVID-19 patients treated with hydroxychloroquine/azithromycin and other regimens in Marseille, France: A retrospective analysis
Type of Study: Observational
Outcome: Death / Hospital Stay ≥ 10 days / Transfer to ICU / Viral Shedding
Sample Size: 3737 patients
Outpatient or Hospitalized: Mixed — some Hospitalized, Some in Hospital “day-care” (don’t stay overnight)
Treatment Regimen: Hydroxychloroquine + Azithromycin
Summary: This is a study done by the same group as Study #1 and Study #2 in France. It is a retrospective study, meaning there was no design set up beforehand, rather they just looked back to see what happened with their patients. In that hospital they were trying to give hydroxychloroquine + azithromycin (HCQ-AZ) to nearly everyone with COVID, and most got that combo for ≥ 3 days (83%), some got it < 3 days (6%), some got just hydroxychlroqouine (3%), some got just azithromycin (4%), and some got neither (4%). They compare various outcomes between these groups. To make it simple, let’s look at their summary category: poor clinical outcome (Death, ICU, and/or Hospitalization ≥ 10 days). They looked at what percent of people in each of the groups above had at least one of these bad outcomes, and found people treated with HCQ-AZ ≥ 3 days had the fewest percent of people with a bad outcome (3.9%), followed by HCQ-only (7.9%), neither drug (8%), HCQ-AZ < 3 days (23.4%), and azithromycin only (27%). They sliced and diced the data quite a few ways, but going through all of that is beyond the scope of this post.
First off, after reading all the other studies cited, this one is a breath of fresh air. This is an actual study. It still has significant flaws/limitations, but they are the normal kinds of flaws and limitations we expect from science, not weird things like no control group, statistics that don’t add up, and dropping out people from analysis because they had a bad outcome. So what do we make of this study? How strong is the evidence that hydroxychloroquine/azithromycin works?
It is an observational study, meaning the authors didn’t assign people to study groups ahead of time, rather they just let life happen and then look back to see what happened organically. These studies are nice because they are easier to do than randomized trials, but they provide much weaker evidence than a randomized trial because there is no guarantee that people who got the treatment versus the control group are truly equivalent groups of people. And that is what we find in this study — while the people who did not get either drug had worse outcomes than those who got the HCQ-AZ combo for ≥ 3 days, those people were also older, had more underlying conditions (heart disease, high blood pressure), were already sicker, and a higher percentage of them were already hospitalized compared to the people who got HCQ-AZ ≥ 3 days. This could very likely explain why these people had worse outcomes, rather than any effect of the drug treatment. The authors do acknowledge this imbalance in the groups and try to correct for it by doing various types of adjusted statistical analysis. (Adjusted just means they controlled for some of these variables, then looked to see if there was still an effect associated with treatment). They report a significant benefit of HCQ-AZ for ≥ 3 days; however, when they did this analysis, they grouped everyone else together into one group (so those who got one drug, no drugs, or the drugs for < 3 days were all lumped together and treated equally). They likely did this in order to have enough subject numbers to run their analysis, which I can understand, but it also means it’s very hard to determine what the difference between drug(s) and no drugs really is, since it mixes a lot of different treatment regimens together. They do adjust for a score that is a measure of how sick the patients were before treatment (the NEWS score), which is a good step. Statistical “adjustments” like these are supposed to correct for skew between groups, but they are only as good as the clinical measure being used, and they’re not a guarantee that they’ll truly capture all the differences in severity between the two groups. Given all these considerations and caveats, I find the results of this study to be only modestly convincing. I’d give it a ‘hmmmm… maybe.’ There could be something there, but it’s also possible the results could be explained by factors other than the drugs.
Finally, this study does not seem to really be reflective of an outpatient setting / people early in their disease course. Some of the patients were hospitalized in inpatient units (they stayed overnight), and the rest were inpatients in the ‘day-care’ hospital. If they needed to stay the full day in the hospital, even if they didn’t spend the night, that makes me think they were probably sicker than somebody who is early in their disease and goes to their doctor’s office. But they do say many were “mild” cases, so it could be that some of the patients were comparable to an outpatient setting. However, the results aren’t analyzed separately for mild versus severe cases, so we can’t necessarily apply their results to mild cases only. This is important because Dr. Risch’s main argument is that this drug combo is effective when given early. We already have studies showing hydroxychloroquine + azithromycin doesn’t work when given to hospitalized patients, so if that is the group of people this study looked at, then we have to take into consideration all those other studies that say it doesn’t work as well.
Conclusion:
1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?
No – no not tested
2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?
No – some patients were hospitalized, and no analysis of mild cases only was provided
3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?
Maybe? Something could be there, but results could potentially be explained by underlying differences between the study groups.
Citation #7: Dr. Crawford’s patients (radio interview)
Type of Study: Uncontrolled
Outcome: Death
Sample Size: 52
Population: Nursing Home
Treatment Regimen: Hydroxychloroquine + rehydration
Summary: 52 patients COVID-19 patients (it sounds like they were confirmed infections) at a nursing home were given hydroxychloroquine early on in their disease course, and they report 0% died. No more details are available, and there is no control group.
1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?
No – no not tested
2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?
No – no not tested
3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?
No – no control group
This study wasn’t actually cited in the NewsWeek editorial or linked articles, but I know someone is going to ask about it so I included it.
Study #8: COVID-19 Outpatients – Early Risk-Stratified Treatment with Zinc Plus Low Dose Hydroxychloroquine and Azithromycin: A Retrospective Case Series Study
Type of Study: Observational
Outcome: Hospitalization / Death
Sample Size: 518
Population: Outpatient (General Practice)
Treatment Regimen: Hydroxychloroquine + Azithromycin + Zinc
Summary: This is another study presumably run by Dr. Zelenko (he is the senior author) in which 141 risk-stratified COVID-19 patients were given the treatment regimen, and 4 were hospitalized and 1 died. As a control group, they use “Independent public reference data from 377 confirmed COVID-19 patients of the same community,” which had a significantly higher rate of hospitalization but not death. My big, big issue with this study is we do not have any information about the control group. We don’t know how sick they were, their age, their underlying conditions, etc. So while the hospitalization rate was higher in that group (15.4% vs 2.8%), we have no way of knowing if this has anything to do with the treatment. Maybe that reference group was already sicker? (Notably, subjects older than 60 didn’t even have to have symptoms to be included in Dr. Zelenko’s treatment group). Maybe the reference group was older? Maybe they had more underlying conditions? It is standard in observational studies to include a table comparing the relevant characteristics of the treatment and control groups to see if the groups are comparable, and that was impossible to provide in this study, since they had no information about the control patients. Overall, this makes the results essentially impossible to interpret.
Conclusion:
1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?
No – adequate control group not provided
2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?
No – adequate control group not provided
3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?
No – adequate control group not provided
Those are all the data I could find cited in Dr. Risch’s articles. In his letter, 6 other citations of results reported by “Personal Communication” are also included. Most are uncontrolled observations like #7 above, and as the data are not available, I have not included them here.
The end of the editorial also addresses two other claims of evidence: ‘natural experiments’ where regulatory changes in use of hydroxychloroquine +/- azithromycin or shipment of drug doses to the region is correlated with changes in regional deaths in Pará, Brazil and Switzerland. No links to the data are made available, so these claims are difficult to assess. But on a general level, there are so many variables that can affect case/death rates on a population level (lockdowns, mask use, social distancing measures, other changes in treatment, etc.) that it is not feasible to confidently attribute changes like these to a single event. Check out this website to see more examples of correlation ≠ causation.
So, what did we find? Overall, the evidence was very underwhelming. All but one of the studies failed to meet basic scientific standards of confirming the subjects actually had COVID, not dropping out subjects who got sick or died, including a control group, and having accurate statistics. Much of the data cited do not have a control group — some have argued it’s unethical to do a control group during a pandemic if we have enough evidence to say that the treatment works. However, this becomes a bit of a circular argument, because we can only know if it works with an adequate control group. (Or in reality, at least a couple studies with adequate control groups.) You can’t simultaneously argue that a study done without a control group was justified because we already know it works and also argue we know it works because of those same studies without control groups. In his AJE article, Dr. Risch tries to get around the need for a control group by making an estimate of the mortality in a similar population, but as Dr. Fleury points out in a published rebuttal to the article, this leads to many problematic assumptions. Simply put, it is very, very difficult to reliably ‘estimate’ a mortality rate that is truly reflective of a relatively small group of individuals (< 1000). There are so many variables that come into play that could affect that number. This is why a control group of people who are part of the same group of individuals being treated is essential. That is the most reliable way to assess what the mortality rate would have been without treatment. Control groups are Science 101. You have to have them, and it is very much possible to include them, even during a pandemic.
Dr. Risch does acknowledge at least some of the weakness of the studies (though not all that I have highlighted here) and argues this is acceptable because we are in a pandemic and don’t have the luxury of perfect data. He states “Each piece of evidence, contained in each study, must be carefully considered and not dismissed because in an ideal world such evidence would fall in a lower part of the evidence-quality triangle.” While it is true we are in a pandemic setting which makes everything more challenging, it’s not true that we are limited to very, very poorly designed studies. It is very much possible to do a reasonably good study, even during a pandemic — there are many that have already been done. We should (and will) give more weight to better designed studies and give minimal weight to the results of critically flawed studies. Yes, just as with non-pandemic times, no study is perfect, but some are far, far, far worse than others.
Based on the above discussed data, Dr. Risch argues that high-risk patients should get the hydroxychloroquine drug combo immediately upon clinical suspicion of COVID-19: ‘These medications need to be widely available and promoted immediately for physicians to prescribe.’ He is not saying ‘this might work let’s wait for more research’, he is saying that we already have enough data to justify prescribing these medications, presumably to all high risk patients with suspected COVID infection. However, all but one of the studies cited were critically flawed, failing to meet basic standards of scientific enquiry. Furthermore, most of these studies did not even test the proposed regimen (hydroxychloroquine + azithromycin + zinc + outpatient setting). Only one study was not critically flawed (Study #6), and does provide modest evidence of the efficacy of hydroxychloroquine + azithromycin. However, it was done in a hospital setting where the underlying conditions and disease severity were quite skewed between the treatment and control groups, and requires us to trust that the statistical methods used to adjust for these differences were fully adequate. Furthermore, this study was not done in an outpatient setting, many of the patients were not mild cases, and those that were mild cases were not analyzed separately. Thus, these results cannot be applied as evidence the drugs work for mild cases and should be consider along with the multiple studies failing to find an effect of this regimen in a hospital setting. Overall, the cited data is not nearly sufficient evidence to declare hydroxychloroquine + azithromycin +/- zinc as the “key to defeating COVID-19” nor justify prescribing it to all high-risk mild/outpatient COVID cases, which over the coming months would amount to thousands, if not millions, of Americans. A higher standard of evidence is required given the risk of side effects (even rare side effects become a significant burden once many, many people get the drugs) and the risk of breeding antibiotic resistance to azithromycin. We need a higher standard of evidence to justify these risks, and that standard is attainable, even during a pandemic.
This blog is intended to help people understand the scientific literature and is not intended to provide medical advice. Please consult with your physician for any questions about health concerns or medical treatments. The American College of Physician’s statement on hydroxychloroquine for COVID-19 can be found here.
