That Newsweek Article: Review of Yale Epidemiologist’s Key to Defeating COVID

That Newsweek Article: Review of Yale Epidemiologist’s Key to Defeating COVID
By Kristen Panthagani

After writing about Dr. Stella Immanuel’s viral video, the most common request I got was to assess this Newsweek Opinion piece circulating by Dr. Harvey Risch, a Professor of Epidemiology at Yale, claiming that we already have the key to defeating COVID (hydroxychloroquine), and we need to start using it. So let’s assess his argument and see if it holds merit.

 

He argues that hydroxychloroquine has proven to be effective against COVID, in particular when it is given early on in the disease course and when combined with azithromycin (or doxycycline, another antibiotic) and zinc. This is based on 5 studies summarized in his publication in the American Journal of Epidemiology (AJE) Early Outpatient Treatment of Symptomatic, High-Risk Covid-19 Patients that Should be Ramped-Up Immediately as Key to the Pandemic Crisis and 7 more studies published in a follow-up letter. He additionally points to two examples of correlation between hydroxychloroquine prescription and mortality rate in Pará, Brazil and Switzerland. He further argues that the reason that other studies have not shown benefit is that they were not used in the proper setting: it should be given early in the course of disease to high risk patients (although he does point to two studies done in hospitalized patients that show benefit; so it seems he is also arguing that there is some efficacy even when the drug is given later in the disease course after patients are already quite sick).

 

Now, let me clarify the purpose of this blog post. My goal is to evaluate Dr. Risch’s claim that we already have evidence that this treatment is effective based on the studies he has cited. He is not saying ‘maybe this works let’s study this more,’ he is arguing that we already have enough evidence to show that it works (at least enough evidence for a pandemic setting), so we need to start prescribing it now. Therefore, my goal is to evaluate that claim. Do we have enough evidence to show that it works? Do the studies he cited truly demonstrate efficacy of the drug(s)? Can we reliably say that the hydroxychloroquine drug combo is the “key to defeating COVID-19” based on the data he cited? These are the questions this post is tackling, not whether or not more hydroxychloroquine combo studies are warranted. That is another discussion for another day.

 

First, let’s nail down what treatment combo he is saying is effective. In his Newsweek editorial he seems to argue that hydroxychloroquine + azithromycin (or doxycycline) + zinc + given early (outpatient) + high risk patients is the best combo. To clarify a few terms — outpatient describes patients who are treated outside of a hospital (like at a clinic/doctor’s office), and inpatient describes patients who are admitted to a hospital for treatment (i.e. they have their own bed). Generally, outpatient patients are less sick / early in their disease course, and inpatients are quite sick / later in their disease course. ‘High risk’ is not explicitly defined, but he seems to mean patients that are older or have underlying conditions (those who are at higher risk of dying from COVID). However, given that the criteria for what ‘high risk’ means isn’t precisely defined, I’m not going to try to determine if the studies he cites really study a ‘high risk’ population, as without clear criteria of precisely what that means it becomes a bit subjective.

 

While he strongly advocates for use of the hydroxychloroquine drug combo for outpatients, his comments about use of hydroxychloroquine alone or in hospitalized patients are conflicting… sometimes he says that data is irrelevant: “Evidence about use of hydroxychloroquine alone, or of hydroxychloroquine+azithromycin in inpatients, is irrelevant concerning efficacy of the pair in early high-risk outpatient disease” and sometimes he seems to cite the data as evidence to support his argument “Even so, it has demonstrated significant benefit in large hospital studies in Michigan and New York City when started within the first 24 to 48 hours after admission.” So his opinion on hydroxychloroquine +/- other drugs in a hospital setting (very sick patients) is a little unclear, but perhaps he thinks there is some benefit. Additionally, in his original publication in the AJE, he doesn’t emphasize the importance of zinc: “all of these reviews have omitted the two critical aspects of reasoning about these drugs: use of HCQ combined with AZ or with doxycycline, and use in the outpatient setting,” but in his Newsweek editorial he does. So for each study below, I will evaluate three different scenarios:

 

  1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?
  2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?
  3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?

One of the main things we learn to do in MD-PhD training is to evaluate study design. One of my favorite classes in grad school was “Method and Logic,” where we ripped apart studies and evaluated whether or not the data they provide actually supports the conclusions they made. Before taking that class, I naively thought that nearly every scientific paper’s claims were reliably supported by their data. But that is not true — many do support it (perhaps with a few minor weaknesses), but a surprising number have significant and sometimes severe methodological flaws. It is our job as scientists to not just blindly accept the conclusions provided by the authors of the study, but to see if their data and their study design really support the conclusions they are making. So that is what I’m going to do with the studies cited by this epidemiologist.

 

Before we look at the studies, let’s talk about a few things that are essential for a scientific study. Note I’m not even talking about the strengths of different types of studies (observational versus randomized), I’m talking about what are basic criteria that any type of study needs to have in order to be considered valid. These are more like bare minimum standards:

 

1. The details of the data are made available.

2. We know the people being studied actually have the disease we’re trying to study.

3. Patients aren’t eliminated from analysis because they got sick or died.

4. The statistics are sound.

5. There is an adequate control group.

 

What is an ‘adequate’ control group and why is it important? A study must have a control group (a similar group of patients who did not get the treatment) to know if any benefit you see is actually from the drug(s) and not from something else about the population you are studying. As a hypothetical example, if you did an analysis of lollypop consumption in all COVID patients, you would likely find that it is associated with good COVID outcomes. Would that mean lollypops cure COVID? No, it just means that children are more likely eat lollypops than adults, and for reasons unrelated to lollypop consumption, they are also less likely to get severely sick from COVID. This example is obvious because nobody actually thinks lollypops can cure COVID, but similar things can happen with drug treatments. Maybe doctors were more likely to give hydroxychloroquine drug combos to patients who were less sick? Or more sick? Or maybe the particular hospital or clinic giving the drugs serves a different demographic of patients with different underlying conditions? All of these can impact clinical outcomes, thus an adequate control group is essential to make any meaningful conclusions about whether or not a drug really works. So what do I mean by ‘adequate control group?’ I mean that we can be reasonably convinced that the patients in the control group are similar enough to the patients in the treatment group that we can think of them as roughly equal groups of people (at least, equal in terms of factors that impact how sick they become / their risk of dying).

Now, here are the studies:

Citation #1: Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial

Type of Study: Observational

Outcome: Positive for Virus after 6 days

Sample Size: 36 patients

Hospitalized or Outpatient: Mix*

Treatment Regimen: Hydroxychloroquine +/- Azithromycin

Summary: This was the first hydroxychloroquine study to get a lot of attention, published by Dr. Raoult back in March. They gave hydroxychloroquine +/- azithromycin to 20 people, and included 16 people that either refused treatment or were from a different medical center as controls. They tested for the presence of the virus in nasal swabs, and concluded that after several days, those who got the treatment were more likely to test negative for the virus.

 

There are many issues with this study and I will not be able to address all of them (check out this post for a more detailed discussion of some of them.) However I will highlight what I consider to be the biggest flaw: there were originally 26 people in the hydroxychloroquine (+/- azithromycin) group, but 6 of those people weren’t included in the final data analysis. Why? Because they didn’t finish the treatment. Why didn’t they finish the treatment? Because one got too nauseous from the medication, one went home, three were transferred to the ICU (meaning they got very, very sick), and one of them died. That means 15% of people who got the drug(s) had very bad outcomes (ICU or death), compared to 0% in the control group. If people are clinically deteriorating in the treatment group and not the control group, I don’t really care if there are differences in whether or not their nasal swabs were positive for the virus. Clinical outcomes are what ultimately matters, not nasal PCR tests. Notably, the person who died was PCR-negative the day before he/she died, which also makes me question whether or not the results of their PCR test (which is what their entire results are based on) are reliable and/or have anything to do with how sick the people were. Additionally, the control group was not a good control group. The controls were either children who were not sick, or adults from another medical center. Why does it matter if they were from another medical center? Maybe that medical center served a different demographic group and there were more people with pre-existing conditions. Maybe the PCR test used to measure the presence of the virus was different at the other medical center (from an earlier version of the paper, the way the PCR results were reported was different for some of the control subjects, making me think they used a different method. This is not good). In short, we know very little about how comparable the treatment and control groups actually were, and what we do know is not comforting. An experiment is only as good as its controls. You simply cannot conclude anything without adequate controls in your experiment.

 

*While the study states it was done it a “hospital setting”, it sounds like some in the control group may have not been hospitalized. Additionally, some patients in both the treatment and control group were asymptomatic, which makes me think they were not hospitalized.

 

Conclusion:

1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?

No – not tested

2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?

No – not tested (not outpatient)

3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?

Not really – main outcome (viral shedding) was not a measure of clinical severity, and more patients in treatment group had bad clinical outcomes (ICU/death). Also lots of other problems.

Citation #2: Early Treatment of COVID-19 Patients With Hydroxychloroquine and Azithromycin: A Retrospective Analysis of 1061 Cases in Marseille, France

Type of Study: Uncontrolled

Outcome: Death, Clinical Worsening, Viral Shedding

Sample Size: 1061 patients

Outpatient or Hospitalized: Mixed (Inpatient and ‘day-care’ hospital)

Treatment Regimen: Hydroxychloroquine + Azithromycin

Summary: This study was run by the same group as Study #1. They gave hydroxychloroquine and azithromycin to 1061 patients and tracked a variety of parameters. However, this study has no control group to compare to (patients who did not get the treatment), so it does not provide any useful evidence about the effect of the treatment.

 

Conclusion:

1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?

No – not tested

2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?

No – not exclusively outpatient, and no control group

3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?

No – no control group

Citation #3: Dr. Zelenko’s Clinical Protocol (google doc)

Type of Study: Uncontrolled

Outcome: Death/Hospitalization/Intubation

Sample Size: 405

Population: Outpatient, some high risk

Treatment Regimen: Hydroxychloroquine + Azithromycin + Zinc

Summary: This isn’t a publication, it’s a google doc by Dr. Zelenko describing his treatment regimen. He does provide a few sentences about the patients he has treated — he reports treating 405 cases in an outpatient setting that are either confirmed or suspected to have COVID. He argues that treatment should be started before the diagnosis is confirmed, so it is not clear whether every patient was eventually confirmed to have COVID or not. Two patients died, six were hospitalized, and four were intubated. It is unclear how robust his follow-up is to track the outcome of patients. There is no control group.

 

Conclusion:

1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?

No – no control group, uncertain if patients had COVID

2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?

No – not tested

3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?

No – no control group, uncertain if patients had COVID

Citation #4: Empirical treatment with hydroxychloroquine and azithromycin for suspected cases of COVID-19 followed-up by telemedicine

Type of Study: Observational

Outcome: Need for Hospitalization

Sample Size: 636

Outpatient or Hospitalized: Outpatient (Telemedicine)

Treatment Regimen: Hydroxychloroquine + Azithromycin

Summary: This non-peer reviewed study evaluated patients who had mild COVID-like symptoms (but the diagnosis was not confirmed) in an outpatient setting (via telemedicine). All patients were offered hydroxychloroquine/azithromycin, and those who refused served as the control group (224 patients) while those who accepted were the treatment group (412 patients). They report lower hospitalization in the treatment group (1.9%) versus control group (5.4%). They also look at differences in how early patients in the treatment group started their treatment, and report that those who started it earlier (< 7 days from start of symptoms) were less likely to go to the hospital (1.17%) than those who started it later (3.2%).

 

On first glance this study looks much better than the previous two, as it includes a control group. This is not an ideal control group as there may be significant differences in people who refused treatment versus not, but it’s certainly better than nothing. However, as I dug into the study, I found a couple things that were pretty funky and didn’t quite add up. First, they do not actually confirm that their patients had COVID. They enrolled anyone with “flu-like symptoms,” and do not do any diagnostic testing. Some of the patients underwent CT scans, and they found results of 40% of scanned patients in the control group were suggestive of COVID versus 70% in the treatment group. Over half of the patients in the study were not scanned, so all we really know about them is that they had flu-like symptoms for at least 2-3 days and weren’t sick enough to go the hospital. Given that lots of viruses cause mild flu-like symptoms, it is very likely that not all of these patients actually had COVID. If the percent of true COVID patients was different between the two groups, this could invalidate the results.

 

There are also some funky things with their statistics. The main conclusion they report is that 1.9% of the patients who got the treatment went to the hospital versus 5.4% of those who didn’t get treatment, with a reported p-value of p < 0.0001. It is standard in studies like this to not only report the percentage of patients, but to also include the actual number of patients who were hospitalized in each group. The authors do this for most of their data with the exception of their main results, which I find odd. For their main results, they only report the percentage and a p-value. But assuming this is the percentage of the total patients in each group (which is what the study implies), we can calculate it: 412 x 1.9% = 8 hospitalized patients in the treatment group and 224 x 5.4% = 12 hospitalized patients in the control group. They don’t explicitly state which statistical test they use to get the p-value < 0.0001, but the only two tests included in their methods that are used for this type of analysis would be the Fisher Exact Test or the Chi-square Test. I ran these statistical tests on their data, and the p-values they return are p = 0.02 and p = 0.03, respectively. So something is very wrong, because that is a very different result than the p < 0.0001 they report. It is still modestly significant, but it gives me a significant pause as to how they ran their analysis and the transparency of their data. We additionally do not get the numbers for their analysis of hospitalization rates in the early versus late treatment groups. But thankfully, with a little high school algebra, we can back-calculate them as well based on a total of 8 hospitalizations in the treatment group and the percentages provided in Figure 2. This gets us 255 patients treated early (3 hospitalized) and 157 treated late (5 hospitalized). If we run a chi-square on this data, though they report p < 0.0001, the result is not significant (p = 0.15). This is very alarming; something is very wrong. Some of their p-values in Table 1 seem to have a similar problem.

 

In conclusion, my main criticism of this study is that we don’t know if all the patients actually had COVID, which really limits the conclusions we can draw from it. My secondary criticism is that some of the numbers and statistics don’t add up. If the authors could be more transparent in their data and calculations, that could solve the second problem, but not the first.

 

Conclusion:

1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?

No – not tested

2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?

No – unknown if patients had COVID, stats questionable

3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?

No – unknown if patients had COVID, stats questionable

Citation #5: Long Island Long-Term Care Facility

Type of Study: Uncontrolled

Outcome: Hospitalization / Death

Sample Size: 45 (news report), 200 (unofficial update)

Outpatient or Hospitalized: Outpatient (Long-term care facility)

Treatment Regimen: Hydroxychloroquine + Doxycycline

Summary: This is a news report as well as personal correspondence about a long-term care facility who is giving hydroxychloroquine + doxychycline to its residents. It sounds like they are giving it to residents diagnosed with COVID (rather than prophylactically), but I am not certain, as there are not very many details provided. 5.6% and 4.5% of patients died according to the news report and unofficial update, respectively. This does not have a control group.

 

Conclusion:

1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?

No – no not tested

2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin (or doxycycline) + given early (outpatient) has a clinical benefit?

No – no control group

3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?

No – no control group

Citation #5.5: Doxycycline and Hydroxychloroquine as Treatment for High-Risk COVID-19 Patients: Experience from Case Series of 54 Patients in Long-Term Care Facilities

Type of Study: Uncontrolled

Outcome: Hospitalization / Death

Sample Size: 54

Outpatient or Hospitalized: Outpatient (Long-term care facility)

Treatment Regimen: Hydroxychloroquine + Doxycycline

Summary: This is a non-peer reviewed case study (that I believe includes some of the same patients in Study #5, which is why I called this Study #5.5). Residents of a long-term care facility who were diagnosed or presumed to have COVID were given hydroxychloroquine + doxycycline. 11% went to the hospital and 6% died. This does not have a control group.

 

Conclusion:

1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?

No – no not tested

2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin (or doxycycline) + given early (outpatient) has a clinical benefit?

No – no control group

3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?

No – no control group

Citation #6: Outcomes of 3,737 COVID-19 patients treated with hydroxychloroquine/azithromycin and other regimens in Marseille, France: A retrospective analysis Type of Study: Observational

Outcome: Death / Hospital Stay ≥ 10 days / Transfer to ICU / Viral Shedding

Sample Size: 3737 patients

Outpatient or Hospitalized: Mixed — some Hospitalized, Some in Hospital “day-care” (don’t stay overnight)

Treatment Regimen: Hydroxychloroquine + Azithromycin

Summary: This is a study done by the same group as Study #1 and Study #2 in France. It is a retrospective study, meaning there was no design set up beforehand, rather they just looked back to see what happened with their patients. In that hospital they were trying to give hydroxychloroquine + azithromycin (HCQ-AZ) to nearly everyone with COVID, and most got that combo for ≥ 3 days (83%), some got it < 3 days (6%), some got just hydroxychlroqouine (3%), some got just azithromycin (4%), and some got neither (4%). They compare various outcomes between these groups. To make it simple, let’s look at their summary category: poor clinical outcome (Death, ICU, and/or Hospitalization ≥ 10 days). They looked at what percent of people in each of the groups above had at least one of these bad outcomes, and found people treated with HCQ-AZ ≥ 3 days had the fewest percent of people with a bad outcome (3.9%), followed by HCQ-only (7.9%), neither drug (8%), HCQ-AZ < 3 days (23.4%), and azithromycin only (27%). They sliced and diced the data quite a few ways, but going through all of that is beyond the scope of this post.

 

First off, after reading all the other studies cited, this one is a breath of fresh air. This is an actual study. It still has significant flaws/limitations, but they are the normal kinds of flaws and limitations we expect from science, not weird things like no control group, statistics that don’t add up, and dropping out people from analysis because they had a bad outcome. So what do we make of this study? How strong is the evidence that hydroxychloroquine/azithromycin works?

 

It is an observational study, meaning the authors didn’t assign people to study groups ahead of time, rather they just let life happen and then look back to see what happened organically. These studies are nice because they are easier to do than randomized trials, but they provide much weaker evidence than a randomized trial because there is no guarantee that people who got the treatment versus the control group are truly equivalent groups of people. And that is what we find in this study — while the people who did not get either drug had worse outcomes than those who got the HCQ-AZ combo for ≥ 3 days, those people were also older, had more underlying conditions (heart disease, high blood pressure), were already sicker, and a higher percentage of them were already hospitalized compared to the people who got HCQ-AZ ≥ 3 days. This could very likely explain why these people had worse outcomes, rather than any effect of the drug treatment. The authors do acknowledge this imbalance in the groups and try to correct for it by doing various types of adjusted statistical analysis. (Adjusted just means they controlled for some of these variables, then looked to see if there was still an effect associated with treatment). They report a significant benefit of HCQ-AZ for ≥ 3 days; however, when they did this analysis, they grouped everyone else together into one group (so those who got one drug, no drugs, or the drugs for < 3 days were all lumped together and treated equally). They likely did this in order to have enough subject numbers to run their analysis, which I can understand, but it also means it’s very hard to determine what the difference between drug(s) and no drugs really is, since it mixes a lot of different treatment regimens together. They do adjust for a score that is a measure of how sick the patients were before treatment (the NEWS score), which is a good step. Statistical “adjustments” like these are supposed to correct for skew between groups, but they are only as good as the clinical measure being used, and they’re not a guarantee that they’ll truly capture all the differences in severity between the two groups. Given all these considerations and caveats, I find the results of this study to be only modestly convincing. I’d give it a ‘hmmmm… maybe.’ There could be something there, but it’s also possible the results could be explained by factors other than the drugs.

 

Finally, this study does not seem to really be reflective of an outpatient setting / people early in their disease course. Some of the patients were hospitalized in inpatient units (they stayed overnight), and the rest were inpatients in the ‘day-care’ hospital. If they needed to stay the full day in the hospital, even if they didn’t spend the night, that makes me think they were probably sicker than somebody who is early in their disease and goes to their doctor’s office. But they do say many were “mild” cases, so it could be that some of the patients were comparable to an outpatient setting. However, the results aren’t analyzed separately for mild versus severe cases, so we can’t necessarily apply their results to mild cases only. This is important because Dr. Risch’s main argument is that this drug combo is effective when given early. We already have studies showing hydroxychloroquine + azithromycin doesn’t work when given to hospitalized patients, so if that is the group of people this study looked at, then we have to take into consideration all those other studies that say it doesn’t work as well.

 

Conclusion:

1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?

No – no not tested

2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?

No – some patients were hospitalized, and no analysis of mild cases only was provided

3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?

Maybe? Something could be there, but results could potentially be explained by underlying differences between the study groups.

Citation #7: Dr. Crawford’s patients (radio interview)

Type of Study: Uncontrolled

Outcome: Death

Sample Size: 52

Population: Nursing Home

Treatment Regimen: Hydroxychloroquine + rehydration

Summary: 52 patients COVID-19 patients (it sounds like they were confirmed infections) at a nursing home were given hydroxychloroquine early on in their disease course, and they report 0% died. No more details are available, and there is no control group.

 

1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?

No – no not tested

2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?

No – no not tested

3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?

No – no control group

 

This study wasn’t actually cited in the NewsWeek editorial or linked articles, but I know someone is going to ask about it so I included it.

Study #8: COVID-19 Outpatients – Early Risk-Stratified Treatment with Zinc Plus Low Dose Hydroxychloroquine and Azithromycin: A Retrospective Case Series Study

Type of Study: Observational

Outcome: Hospitalization / Death

Sample Size: 518

Population: Outpatient (General Practice)

Treatment Regimen: Hydroxychloroquine + Azithromycin + Zinc

Summary: This is another study presumably run by Dr. Zelenko (he is the senior author) in which 141 risk-stratified COVID-19 patients were given the treatment regimen, and 4 were hospitalized and 1 died. As a control group, they use “Independent public reference data from 377 confirmed COVID-19 patients of the same community,” which had a significantly higher rate of hospitalization but not death. My big, big issue with this study is we do not have any information about the control group. We don’t know how sick they were, their age, their underlying conditions, etc. So while the hospitalization rate was higher in that group (15.4% vs 2.8%), we have no way of knowing if this has anything to do with the treatment. Maybe that reference group was already sicker? (Notably, subjects older than 60 didn’t even have to have symptoms to be included in Dr. Zelenko’s treatment group). Maybe the reference group was older? Maybe they had more underlying conditions? It is standard in observational studies to include a table comparing the relevant characteristics of the treatment and control groups to see if the groups are comparable, and that was impossible to provide in this study, since they had no information about the control patients. Overall, this makes the results essentially impossible to interpret.

 

Conclusion:

1. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + zinc + given early (outpatient) has a clinical benefit?

No – adequate control group not provided

2. Does this study provide reliable evidence that hydroxychloroquine + azithromycin + given early (outpatient) has a clinical benefit?

No – adequate control group not provided

3. Does this study provide reliable evidence that hydroxychloroquine (any combo, any population) has a clinical benefit?

No – adequate control group not provided

Those are all the data I could find cited in Dr. Risch’s articles. In his letter, 6 other citations of results reported by “Personal Communication” are also included. Most are uncontrolled observations like #7 above, and as the data are not available, I have not included them here.

 

The end of the editorial also addresses two other claims of evidence: ‘natural experiments’ where regulatory changes in use of hydroxychloroquine +/- azithromycin or shipment of drug doses to the region is correlated with changes in regional deaths in Pará, Brazil and Switzerland. No links to the data are made available, so these claims are difficult to assess. But on a general level, there are so many variables that can affect case/death rates on a population level (lockdowns, mask use, social distancing measures, other changes in treatment, etc.) that it is not feasible to confidently attribute changes like these to a single event. Check out this website to see more examples of correlation ≠ causation.

 

So, what did we find? Overall, the evidence was very underwhelming. All but one of the studies failed to meet basic scientific standards of confirming the subjects actually had COVID, not dropping out subjects who got sick or died, including a control group, and having accurate statistics. Much of the data cited do not have a control group — some have argued it’s unethical to do a control group during a pandemic if we have enough evidence to say that the treatment works. However, this becomes a bit of a circular argument, because we can only know if it works with an adequate control group. (Or in reality, at least a couple studies with adequate control groups.) You can’t simultaneously argue that a study done without a control group was justified because we already know it works and also argue we know it works because of those same studies without control groups. In his AJE article, Dr. Risch tries to get around the need for a control group by making an estimate of the mortality in a similar population, but as Dr. Fleury points out in a published rebuttal to the article, this leads to many problematic assumptions. Simply put, it is very, very difficult to reliably ‘estimate’ a mortality rate that is truly reflective of a relatively small group of individuals (< 1000). There are so many variables that come into play that could affect that number. This is why a control group of people who are part of the same group of individuals being treated is essential. That is the most reliable way to assess what the mortality rate would have been without treatment. Control groups are Science 101. You have to have them, and it is very much possible to include them, even during a pandemic.

 

Dr. Risch does acknowledge at least some of the weakness of the studies (though not all that I have highlighted here) and argues this is acceptable because we are in a pandemic and don’t have the luxury of perfect data. He states “Each piece of evidence, contained in each study, must be carefully considered and not dismissed because in an ideal world such evidence would fall in a lower part of the evidence-quality triangle.” While it is true we are in a pandemic setting which makes everything more challenging, it’s not true that we are limited to very, very poorly designed studies. It is very much possible to do a reasonably good study, even during a pandemic — there are many that have already been done. We should (and will) give more weight to better designed studies and give minimal weight to the results of critically flawed studies. Yes, just as with non-pandemic times, no study is perfect, but some are far, far, far worse than others.

 

Based on the above discussed data, Dr. Risch argues that high-risk patients should get the hydroxychloroquine drug combo immediately upon clinical suspicion of COVID-19: ‘These medications need to be widely available and promoted immediately for physicians to prescribe.’ He is not saying ‘this might work let’s wait for more research’, he is saying that we already have enough data to justify prescribing these medications, presumably to all high risk patients with suspected COVID infection. However, all but one of the studies cited were critically flawed, failing to meet basic standards of scientific enquiry. Furthermore, most of these studies did not even test the proposed regimen (hydroxychloroquine + azithromycin + zinc + outpatient setting). Only one study was not critically flawed (Study #6), and does provide modest evidence of the efficacy of hydroxychloroquine + azithromycin. However, it was done in a hospital setting where the underlying conditions and disease severity were quite skewed between the treatment and control groups, and requires us to trust that the statistical methods used to adjust for these differences were fully adequate. Furthermore, this study was not done in an outpatient setting, many of the patients were not mild cases, and those that were mild cases were not analyzed separately. Thus, these results cannot be applied as evidence the drugs work for mild cases and should be consider along with the multiple studies failing to find an effect of this regimen in a hospital setting. Overall, the cited data is not nearly sufficient evidence* to declare hydroxychloroquine + azithromycin +/- zinc as the “key to defeating COVID-19” nor justify prescribing it to all high-risk mild/outpatient COVID cases, which over the coming months would amount to thousands, if not millions, of Americans. A higher standard of evidence is required* given the risk of side effects (even rare side effects become a significant burden once many, many people get the drugs) and the risk of breeding antibiotic resistance to azithromycin. We need a higher standard of evidence* to justify these risks, and that standard is attainable, even during a pandemic.

 

*This blog is intended to help people understand the scientific literature and is NOT intended to provide medical advice. Please consult with your physician for any questions about health concerns or medical treatments. The American College of Physician’s statement on hydroxychloroquine for COVID-19 can be found here.

Fact-check: Dr. Stella Immanuel’s hydroxychloroquine cure

Fact-check: Dr. Stella Immanuel’s hydroxychloroquine cure
By Kristen Panthagani

This morning I got a request to address one of the latest viral videos going around from a doctor claiming that we already have a cure for COVID-19: hydroxychloroquine, azithromycin, and zinc. While the video has been taken down on many platforms (and for the record, I have very mixed feelings about this type of censorship — that is a whole other discussion), it has rekindled the hydroxychloroquine fire set by Dr. Raoult back in March, the idea that hydroxychloroquine is the silver bullet for COVID-19, and all these masks and lockdowns are unnecessary. So let’s take a look at her claims and see if what she’s saying has any merit.

Her basic arguments are this:

1. She has treated over 350 COVID-19 patients with hydroxychloroquine + azithromycin + zinc, and none of them have died, therefore this treatment is a cure.

 

2. She and her staff and some other doctors have been taking this drug combo as prophylaxis and none of them have gotten sick, therefore it is also effective as a prophylaxis.

She then goes on to say that any study saying otherwise is fake science, it’s unethical not to give the drug now because people are dying, and doctors who are standing by, and not giving this treatment are like the ‘good Germans’ who stood by and let the holocaust happen. We’ll tackle some of these follow-up claims in a minute, but first let’s look at her dataset that she is basing these claims on.

 

Her argument is that treating 350 COVID-19 patients and all of them surviving is evidence that the treatment is a cure. Usually we want a control group to compare to, but it seems Dr. Immanuel believes this would be unethical, so we have none. This is big flaw #1 of her data set. But, let’s work with we what we got: whenever we look at any outcome in science, we always first look to see what is the probability of getting that outcome by pure chance. So what is the chance of having 350 COVID-19 patients in a row all survive? While the COVID-19 mortality rate is a tricky number to nail down, let’s use an estimate of 1% (i.e. on average, across the entire population, 1% of people who contract COVID die from it). If we look at 350 COVID-19 patients at random, the chance of having every one of them survive is (1-0.01)^350 = ~3%. Seems small right? Not necessarily — if we consider the fact that millions of people are getting this disease all across the country, the chances of this happening at least once becomes quite large. As of today there are ~4.38 million total confirmed COVID-19 cases in the US — if you broke all of those people into groups of 350 patients (that’s about ~12,000 groups of 350 patients), we would expect ~360 of those groups to be all patients who survive. So this result is expected to happen by chance 360 times across the US. This indicates her data set really isn’t strong evidence of anything, as the chances of this happening aren’t too improbable when you are looking at a disease that is so prevalent across the US.

 

But, as everybody knows, the mortality rate is highly dependent on the population you are looking at. So what patients is she treating? Are they representative of the average population? Is 1% mortality a reasonable estimate for them?

 

To listen to her talk, you might think she is working in a hospital taking care of very sick COVID-19 patients and miraculously seeing them all get better with the hydroxychloroquine combo. But she keeps using the word “clinic”.. which is not where sick hospital patients are treated. ‘Clinic’ generally refers to an outpatient doctor’s office or perhaps an urgent care center. So what “clinic” is she talking about?

 

After doing a little googling, I found that she works at Rehoboth Medical Center, which, though the name sounds like it might be a rather large medical operation, is in fact a walk-in clinic in a Houston strip mall.

Google street view of strip mall with “Rehoboth Medical Center.” This seems to be the right image — it matches the video on the clinic’s facebook page, where the name of the medical center appears to be added electronically.

Edit: earlier version of this post included an image a few shops down, which is what google pulls up for “Rehoboth Medical Center.” However, based on closer review of the clinic’s facebook video, I believe this is the correct image of the clinic.

 

So these are 350 COVID-19 patients who came to her walk-in clinic. This very much skews her data set. First, it means that the people she is studying are not very sick patients (because they are going to a walk-in clinic for treatment, not a hospital.) This is confirmed by the video on the clinic’s facebook page, where she says they “screen and treat mild cases of COVID-19.” The chances of having 350 mild COVID-19 patients all survive is much, much higher than the chances of 350 very sick hospitalized COVID-19 patients all survive. Second, and perhaps more problematic, it is unlikely that she is able to follow-up with all of her patients to see whether they did well or not. Is she regularly calling all the patients who came to her clinic to see if they went to the hospital and died? I guarantee you medical records are not coordinated enough for her to follow up with them that way. If she has a patient who comes in on Tuesday, gets his hydroxychloroquine/azithromycin/zinc combo, then falls very ill on a Friday and goes to a hospital across town, how would Dr. Immanuel know? Unless she is faithfully following up with every walk-in patient and has backup plans if those patients become too sick to speak on the phone, it is unlikely she could rigorously track whether or not her patients became sick and died. So in essence, it seems like Dr. Immanuel may be saying that nobody died at her walk in clinic, or called to let her know that one of her patients died. The fact that this happened for 350 people in a row now becomes highly, highly probable, not improbable.

 

And now her prophylaxis argument. She adds that masks are not necessary because we already have a COVID-19 prophylaxis: hydroxychloroquine + azithromycin + zinc. It is a little confusing watching the viral video of her making this claim and then watching the video on her clinic’s facebook page where she is encouraging everyone to wear masks, stay 6 feet away, and use hand sanitizer. But, let’s address her argument. She argues that because herself and her staff and some other doctors have used the hydroxychloroquine drug combo as prophylaxis and they haven’t gotten sick, that proves that the drug is effective as a prophylaxis for everybody. But how many staff does she have? Based on the picture of the clinic, this is a fairly small operation, and they likely only have a few staff. Maybe ~10 staff work there. That is a very small data set to make such a bold claim. She said ‘some other doctors’ are taking it too… how many other doctors is she referring to? We can only guess, but let’s say it’s as many as 20. The chances of 20 health care workers not getting sick from COVID, if they are wearing masks and other PPE as the staff in her clinic are in the video, is not that small.

 

In summary, her “evidence” that hydroxychloroquine/azithromycin/zinc is a cure and prophylaxis for COVID-19 does not hold up at all. We would expect these same results by pure chance.

 

Now let’s look at the details of a few of her other claims.

 

She argues there is a 2005 NIH study that says ‘it works.’

While there are numerous in vitro studies looking at the effect of hydroxychloroquine on various viruses, I guarantee you that whatever 2005 study she is referring to was not studying SARS-CoV-2, as the virus did not exist back then. I’m not sure what study she is referring to (perhaps it was this 2005 in vitro study of chloroquine efficacy against SARS), but please remember that different types of studies carry different levels of weight. In vitro studies are considered very, very preliminary, and you can’t conclude a drug works in humans just because it worked in an in vitro study.

 

She says the NIH knows that hydroxychloroquine works because of a COVID hiccup study. “If the NIH knows that treating a patient with hydroxychloroquine proves that hiccups is a symptom of COVID then they definitely know that hydroxychloroquine works.”

She says to google hiccups and COVID to see what she is talking about, so I did. This is the study that came up: it is a case report (description of a single patient) of a man in China who presented with hiccups as an atypical presentation of COVID. That man was given hydroxychloroquine, and his hiccups did go away. However, I hope this doesn’t need to be said — this is not a study, it’s a story about what happened to a single patient. You can’t make sweeping conclusions about the efficacy of a drug based on one patient. If that were true, then any single patient who got the hydroxychloroquine drug combo and died would be evidence that it’s 100% lethal. This is considered anecdotal evidence and is not proof of anything.

 

She says she sees people sitting in her office knowing that this is a death sentence.

This is a very dramatic claim for someone who treats mild COVID patients. Not everyone who gets COVID-19 dies. Yes, an upsetting percentage of them do… but “death sentence” is over the top.

 

She says there is no way she can treat 350 patients and they all live, but other doctors/scientists are going to tell her that they treated 20 people, 40 people and it didn’t work.

This, I believe, is her criticism of other studies showing that hydroxychloroquine doesn’t work, which she asserts are fake science. She seems to be arguing that she has the biggest study of hydroxychloroquine effectiveness, and that studies of 20 – 40 people aren’t strong evidence to show lack of efficacy. While she is correct that studies of 20 – 40 people aren’t very strong evidence, she is mistaken in thinking that this is the sample size of hydroxychloroquine studies to date. Here is a randomized controlled trial of 4716 patients showing no benefit of hydroxychloroquine treatment, and here is a meta-analysis of 26 different studies (including a total of 103,486 patients) showing no clinical benefit of hydroxychloroquine treatment (with or without azithromycin). These are two of the strongest studies we have on hydroxychloroquine for COVID-19 to date. Check out this post for more details, as well as other published studies on hydroxychloroquine +/- azithromycin for COVID-19.

 

She says you don’t need masks — there is a prevention and a cure.

Again, her data “proving” that the hydroxychloroquine drug combo works as a prophylaxis is based on herself and her staff and some unknown number of other doctors, which is not very many people. Here is a randomized double-blinded placebo-controlled trial of hydroxyhcloroquine prophylaxis (studying 821 people) demonstrating that hydroxychloroquine prophylaxis did not protect against COVID-19. Check out this post for more details on hydroxychloroquine prophylaxis studies.

 

She says that for all the doctors waiting for data — if 6 months down the road they find out the drugs work, its unethical not to have treated them now. She also compares doctors standing by watching patients die to the ‘good Germans’ standing by letting the holocaust happen.

No. First, we already have lots of data on hydroxychloroquine and COVID-19, and there is not strong evidence to suggest it works against COVID-19 (see studies in previous paragraphs). But even if we didn’t have this data yet — that doesn’t mean it would be unethical to withhold hydroxychloroquine treatment until we know if it works or not. The way doctors decide whether or not to give any treatment is by weighing the benefits versus the risks. For benefit, we look at the evidence that the drug works (which is very little). For the risks, we look at the side effects (which include risks of heart problems). If you have a drug that lacks evidence that it works and has side effects, it is not unethical to avoid prescribing it.

 

In conclusion, this doctor is making claims based on a deeply flawed data set and ignores the other studies on hydroxychloroquine that contradict her conclusions. This is not helpful. I am not sure why she is doing this — it is very possible that she genuinely believes what she is saying and is trying to get the word out. But that doesn’t make her arguments valid.

 

Disclaimer: This blog is intended to help people understand scientific concepts and is NOT intended to provide medical advice. Please consult with your physician for any questions about health concerns or medical treatments. The American College of Physicians’ statement on hydroxychloroquine for COVID-19 can be found here.

 

Edit: But what about the Yale epidemiologist’s Newsweek article calling hydroxychloroquine the key to defeating COVID? Read about that here.

Does hydroxychloroquine work? Here’s what the studies say so far…

Does hydroxychloroquine work? Here’s what the studies say so far…
Kristen Panthagani, PhD

There has been lots of excitement about hydroxychloroquine as a treatment option for COVID-19; early on, this excitement was based on a few small studies and anecdotal reports from physicians. Since then, more studies have come out looking at the effectiveness of hydroxychloroquine in COVID-19 patients. Below is a summary of the study results so far.

For each study, I provide a simple Yes/No answer to did patients who got hydroxychloroquine do better than the patients who didn’t receive it? But this is very much an over-simplification: the type of study, details of the study population, details of the statistical analyses performed, and other strengths and weaknesses of the study should always be taken into account when interpreting the “Yes/No” conclusions (discussion of these is beyond the scope of this post.) Additionally, I only included studies that were (1) done in humans (not in cell lines) and (2) details of the results were made available.

First – a quick refresher on how to evaluate studies. More details are provided at the end of this post, but as you go through these studies, please remember two very important things:

1. Randomized >>> Observational

2. Big Sample Size >>> Small Sample Size

Hydroxychloroquine Study Results

(ordered by sample size) – last updated July 2020

Note: I have done my best to find all the studies to date, but I could have missed some. If you see one I missed, please send it to me and I’ll update this post.

Links to studies are provided in every title!

This is by far the best study we have to date, as it is the largest and it is randomized:

Effect of Hydroxychloroquine in Hospitalized Patients with COVID-19: Preliminary results from a multi-centre, randomized, controlled trial

Type of Study: Randomized

Outcome: Death within 28 days

Sample Size: 4716 patients

Important notes: not yet peer-reviewed

Did patients who got hydroxychloroquine do better? No

Hydroxychloroquine with or without azithromycin and in-hospital mortality or discharge in patients hospitalized for COVID-19 infection: a cohort study of 4,642 in-patients in France

Type of Study: Observational

Outcome: Death within 28 days

Sample Size: 4642 patients

Important notes: not yet peer-reviewed, some patients got azithromycin

Did patients who got hydroxychloroquine do better? No*

*they did see a slight reduction in % of patients discharged with hydroxychloroquine, but no difference in mortality. They also saw a trending increase risk in mortality in hydroxychloroquine + azithromycin group.

 

Outcomes of 3,737 COVID-19 patients treated with hydroxychloroquine/azithromycin and other regimens in Marseille, France: A retrospective analysis Type of Study: Observational

Outcome: Death / Hospital Stay ≥ 10 days / Transfer to ICU / Viral Shedding

Sample Size: 3737 patients

Important notes: patients also got azithromycin, patients who didn’t get drugs were older and sicker

Did patients who got hydroxychloroquine do better? Yes

 

Outcomes of Hydroxychloroquine Treatment Among Hospitalized COVID-19 Patients in the United States- Real-World Evidence From a Federated Electronic Medical Record Network

Type of Study: Observational

Outcome: Death / Need for Mechanical Ventilation

Sample Size: 3,372 patients (though main analysis is 1820 patients)

Important Notes: not yet peer-reviewed, some patients also got azithromycin

Did patients who got hydroxychloroquine do better? No

Treatment with hydroxychloroquine, azithromycin, and combination in patients hospitalized with COVID-19

Type of Study: Observational

Outcome: In hospital mortality

Sample Size: 2541 patients

Important Notes: Some patients also got azithromycin, discussion of confounders here.

Did patients who got hydroxychloroquine do better? Yes

Hydroxychloroquine and Tocilizumab Therapy in COVID-19 Patients – An Observational Study

Type of Study: Observational

Outcome: Death

Sample Size: 2512 patients

Important Notes: Some patients got azithromycin and tocilizumab, not yet peer-reviewed.

Did patients who got hydroxychloroquine do better? No

Association of Treatment With Hydroxychloroquine or Azithromycin With In-Hospital Mortality in Patients With COVID-19 in New York State

Type of Study: Observational

Outcome: Death

Sample Size: 1438 patients

Important Notes: Some patients also got azithromycin

Did patients who got hydroxychloroquine do better? No

Observational Study of Hydroxychloroquine in Hospitalized Patients with Covid-19

Type of Study: Observational

Outcome: Intubation or Death

Sample Size: 1376 patients

Important Notes: Patients who got hydroxychloroquine were already sicker prior to treatment.

Did patients who got hydroxychloroquine do better? No

Early Treatment of COVID-19 Patients With Hydroxychloroquine and Azithromycin: A Retrospective Analysis of 1061 Cases in Marseille, France

Outcome: Death, Clinical Worsening, Viral Shedding

Sample Size: 1061 patients

Important Notes: No Control Group

Did patients who got hydroxychloroquine do better? No evidence provided (no control group)

Empirical treatment with hydroxychloroquine and azithromycin for suspected cases of COVID-19 followed-up by telemedicine

Type of Study: Observational

Outcome: Hospitalization

Sample Size: 636 patients

Important Notes: Not all patients were confirmed to have COVID, and stats are weird.

Did patients who got hydroxychloroquine do better? Inconclusive*

*patients were not tested for COVID. also stats are funky in this paper. I do not trust it. not

Hydroxychloroquine application is associated with a decreased mortality in critically ill patients with COVID-19

Type of Study: Observational

Outcome: Death

Sample Size: 568 patients

Important Notes: not peer-reviewed

Did patients who got hydroxychloroquine do better? Yes

Low dose of hydroxychloroquine reduces fatality of critically ill patients with COVID-19

Type of Study: Observational

Outcome: Death, measures of inflammation

Sample Size: 550 patients

Important Notes: issues about statistics reported have been raised on pubpeer

Did patients who got hydroxychloroquine do better? Yes

COVID-19 Outpatients – Early Risk-Stratified Treatment with Zinc Plus Low Dose Hydroxychloroquine and Azithromycin: A Retrospective Case Series Study

Type of Study: Observational

Outcome: Hospitalization / Death

Sample Size: 518

Important Notes: No way to know if control group was comparable; no details provided

Did patients who got hydroxychloroquine do better? Inconclusive

Outcomes of hydroxychloroquine usage in United States veterans hospitalized with Covid-19

Type of Study: Observational

Outcome: Use of Ventilator and/or Death

Sample Size: 368 patients

Important notes: Not yet peer-reviewed

Did patients who got hydroxychloroquine do better? No

Hydroxychloroquine for Early Treatment of Adults with Mild Covid-19: A Randomized-Controlled Trial

Type of Study: Randomized

Outcome: viral RNA load in nose up to 7 days after treatment start, disease progression, time to complete resolution of symptoms

Sample Size: 293

Did patients who got hydroxychloroquine do better? No

Clinical efficacy of hydroxychloroquine in patients with covid-19 pneumonia who require oxygen: observational comparative study using routine care data

Type of Study: Observational

Outcome: Transfer to ICU and/or Death

Sample Size: 181 patients

Did patients who got hydroxychloroquine do better? No

Early Hydroxychloroquine Is Associated with an Increase of Survival in COVID-19 Patients: An Observational Study

Type of Study: Observational

Outcome: Death

Sample Size: 166 patients

Important notes: Not yet peer-reviewed

Did patients who got hydroxychloroquine do better? Yes

Hydroxychloroquine in patients with mainly mild to moderate coronavirus disease 2019: open label, randomised controlled trial

Type of Study: Randomized Trial

Outcome: Positive for virus after 28 days

Sample Size: 150 patients

Did patients who got hydroxychloroquine do better? No

Compassionate use of hydroxychloroquine in clinical practice for patients with mild to severe Covid-19 in a French university hospital

Type of Study: Observational

Outcome: time to death, ICU admission, or withdrawal of supportive care

Sample Size: 89 patients

Did patients who got hydroxychloroquine do better? No

Clinical and microbiological effect of a combination of hydroxychloroquine and azithromycin in 80 COVID-19 patients with at least a six-day follow up: A pilot observational study

Type of Study: Observational

Outcome: Positive for Virus, Length of Hospital Stay, Clinical Outcome

Sample Size: 80 patients

Important notes: No control group

Did patients who got hydroxychloroquine do better? No evidence provided (no control group)

Clinical outcomes of hydroxychloroquine in hospitalized patients with COVID-19 : a quasi-randomized comparative study

Type of Study: Quasi-randomized (I would put this more as observational)

Outcomes: mortality, respiratory status

Sample Size: 63 patients

Important notes: this study was written in French and this is based off the google-translated abstract

Did patients who got hydroxychloroquine do better? Mixed Results (mortality was higher but improvement in respiratory status was better in the hydroxychloroquine group )

Efficacy of hydroxychloroquine in patients with COVID-19: results of a randomized clinical trial

Type of Study: Randomized Trial

Outcome: Time to Clinical Recovery

Sample Size: 62 patients

Important notes: Not yet peer-reviewed

Did patients who got hydroxychloroquine do better? Yes (modest effect)

Efficacy and safety of chloroquine or hydroxychloroquine in moderate type of COVID-19: a prospective open-label randomized controlled study. Type of Study: Randomized Trial

Outcome: Time to Clinical Recovery

Sample Size: 48 patients

Important notes: Not yet peer-reviewed, also included chloroquine group

Did patients who got hydroxychloroquine do better? Yes (modest effect)

Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial

Type of Study: Observational

Outcome: Positive for Virus after 6 days

Sample Size: 36 patients

Important notes: major issues with study design

Did patients who got hydroxychloroquine do better? Mehhhhhh*

*the authors conclude there is an effect, but the study design had so many issues that I wrote a whole other post about it.

Hydroxychloroquine and azithromycin as potential treatments for COVID-19; clinical status impacts the outcome

Type of Study: Observational

Outcome: Positive for Virus

Sample Size: 36 patients

Important notes: I think they may have artificially inflated their statistical power in the way they did their analysis, but I’d have to dig into it more to be sure.

Did patients who got hydroxychloroquine do better? Yes

Hydroxychloroquine is associated with slower viral clearance in clinical COVID-19 patients with mild to moderate disease: A retrospective study

Type of Study: Observational

Outcome: Positive for Virus

Sample Size: 34 patients

Important notes: not yet peer-reviewed

Did patients who got hydroxychloroquine do better? No (those who got the drug did worse)

A pilot study of hydroxychloroquine in treatment of patients with common coronavirus disease-19 (COVID-19)

Type of Study: Randomized Trial

Outcome: Positive for virus after 7 days of treatment

Sample Size: 30 patients

Did patients who got hydroxychloroquine do better? No

Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis

Type of Study: Observational

Outcome: Death (in the hospital)

Sample Size: 96,032 patients

Important Notes: Also includes analysis of chloroquine and macrolides (such as azithromycin)

Did patients who got hydroxychloroquine do better? No (those who got the drug did worse)

STUDY RETRACTED – leaving here for record purposes only.

So, does it work?

Most studies (13 studies) showed no improvement with hydroxychloroquine treatment, including the best designed study we have as well as 6 big studies (>1000 people). However, some studies (8) did show an effect of hydroxychloroquine (though these were often smaller studies, with only 2 studies with >1000 people). How do we interpret this? One way is to go back and do a meta-analysis — where someone pulls all the available data from the studies and re-analyzes the data from multiple studies together. This study did just that, here is what they found:

The Role of Hydroxychloroquine in the Age of COVID-19: A Periodic Systematic Review and Meta-Analysis Sample Size: 21 studies, 103,486 patients

Important Notes: not yet peer reviewed, some patients also received azithromycin

Did patients who got hydroxychloroquine do better? No

Studies of Hydroxychloroquine as Prophylaxis

(ordered by sample size)

prophylaxis: a treatment that is given to prevent a disease from developing (rather than treat a disease that somebody already has)

A Cluster-Randomized Trial of Hydroxychloroquine as Prevention of Covid-19 Transmission and Disease

Type of Study: Randomized Trial

Outcome: PCR-confirmed symptomatic Covid-19 within 14 days

Sample Size: 2314 patients

Important notes: not yet peer-reviewed

Were patients who got hydroxychloroquine less likely to develop COVID? No

A Randomized Trial of Hydroxychloroquine as Postexposure Prophylaxis for Covid-19

Type of Study: Randomized Trial

Outcome: laboratory-confirmed COVID-19, or illness compatible with COVID-19, within 14 days

Sample Size: 821 patients

Were patients who got hydroxychloroquine less likely to develop COVID? No

Pre exposure Hydroxychloroquine Prophylaxis for COVID-19 in healthcare workers: a retrospective cohort

Type of Study: Observational

Outcome: positive for COVID-19

Sample Size: 106

Important notes: not yet peer-reviewed

Were patients who got hydroxychloroquine less likely to develop COVID? Yes

What do all these words describing studies mean?

Type of Study: All of the studies reported here are some form of either an observational or randomized trial. What’s the difference? An observational study is a study that is done by looking back at what happened organically in the hospital. These studies usually use hospital records to see who got the treatment, who didn’t, and how these groups of patients did over the course of their hospital stay. These studies are important and provide much better evidence than anecdotal reports by physicians, but there are down sides — the main one is that there is no guarantee that those who got the treatment and those who didn’t are truly comparable groups of patients. For example, maybe patients who got hydroxychloroquine were already sicker than those who didn’t — this would effect the differences in outcomes between the treatment and control groups. There are some fancy statistical methods to help correct for these differences, but they are not perfect. Observational studies also are prone to the placebo effect, where patients improve because of the hope of treatment (and possibly other more complicated reasons), and not due to the actual effect of the drug. A randomized study takes care of one of these problems — patients are randomized to either the treatment or control group at the beginning, which usually takes care of differences in populations between the two groups. The control group in these cases is usually the standard-of-care (everything the hospital would normally do for a COVID-19 patient). If it is a randomized placebo-controlled study, then that study also takes care of the placebo effect problem by giving the control group a placebo in addition to the standard-of-care. So if randomized placebo-controlled studies are clearly the best, why don’t we do that for every study? Because they are much harder to do (they take time to get up and running, they cost more, they require much more time on the part of the researchers, etc.)

Outcome: This is simply the patient outcome (positive for virus after 7 days, fever, transfer to ICU, death, etc.) the study used to measure if there was an effect of the treatment. All of these in some way measure how sick the patient is.

Sample Size: How many patients were included in the study (more is always better).

Disclaimer: This post is not intended to provide medical advice or guide treatment decisions. Please consult your physician for questions about medical treatments. COVID-19 treatment guidelines provided by the CDC can be found here.