A critical review of Florida’s new vaccine analysis
Yesterday, the Florida State Surgeon General issued new guidance around the COVID vaccines based on an analysis they performed. They say the results show “an increased risk of cardiac-related death among men 18-39” for the mRNA vaccines, and recommend this group not receive these vaccines.
This is a bold recommendation. Is it justified? Let’s look at the data…
The link to the analysis takes you here, not a published study, but a pdf of a word document. A few notes right off the bat:
1. This analysis is not published and is not peer reviewed.
2. There are no authors listed, which is a bit odd.
Now, to the data. They performed a self-controlled case series — basically they look to see if there is an increased incidence of bad outcomes shortly after vaccination (first 28 days after vaccination) compared to a control period (29 days – 25 weeks after vaccination.)
They did a couple of analyses, but let’s focus on the finding upon which they based their new vaccine recommendations: an increased incidence of cardiac-related deaths in men age 18-39 after mRNA vaccination.
Flaw #1: a hodge-podge of diagnoses
How did they assess cardiac-related deaths? They looked at death certificates, and any death that had an ICD 10 code under “Other forms of heart disease” (ICD I30-I52) was included.
Here are the individual ICD 10 codes included:
A couple things to note about this methodology: first, ICD-10 codes are not necessarily always accurate or specific; they are primarily used for billing.
Second, this list doesn’t even include all cardiac related ICD 10 codes, it only includes the “Other forms of heart disease” section. There isn’t a clear rationale why they included these specific ICD-10 codes vs other cardiac-related codes (i.e. ischemic heart disease is not included?)
Third, “cardiac arrest” is included, which simply means “the heart stopped” and can be the terminal event for many different diseases, not just cardiac issues.
Finally, many of these diagnoses are most likely related to ongoing disease processes (like heart failure) rather than an acute side effect of vaccination.
Overall, it is a random, incomplete list defined not by a hypothesis, but by the arbitrary structure of hospital billing.
Flaw #2: Diagnoses weren't verified
To get around this limitation, did they dig through medical records and confirm that the ICD-10 codes were accurate and that there wasn’t another obvious cause leading to that diagnosis and death? (For example, someone who died from bacterial endocarditis, something that could not plausibly be connected to a vaccine?)
No, they didn’t.
The anonymous author(s) themselves note these significant limitations, stating that they “cannot determine the causative nature of a participant’s death” and “the underlying cause of death may not be cardiac related.”
Contrast that with this robust, peer-reviewed study published in JAMA, that instead of lumping a hodge-podge of unverified cardiac diagnoses together, looked at 1. specific diagnoses (myocarditis, pericarditis) and, 2. confirmed them via medical record review.
That’s how you do it.
Flaw #3: Sample size is quite small
The results for males age 18-39 who received an mRNA vaccine included a grand total of 20 deaths during the risk period.
If even just a few of these have another cause (which is highly likely), they lose statistical significance.
Flaw #4: Confounding by COVID
Those who had COVID should be excluded from this analysis, as the protective effect of the vaccine would confound the results. And the authors (rightly) exclude deaths for which COVID was listed as a cause on the death certificate.
In the methods, in addition to mentioning the exclusion of people with COVID on their death certificate, they say they excluded people with a documented COVID infection.
This makes it sound like they had data on who had tested positive for COVID, right? But later on, they say COVID testing status was unknown, and the only info they had on COVID status was based on the death certificates.
So they didn’t know whether or not people had COVID, unless it was listed on their death certificate. (As a side note, this sneaky use of language in the methods section is a red flag. I don’t like it.)
Overall, this means that susceptibility to COVID confounded their analysis, and COVID (not COVID vaccination) very well could have contributed to some of these deaths.
Why?
Recall that after vaccination, it takes a little time for immunity to kick in. The “risk” period in the study (first 28 days) partially overlaps with the period of time vaccinated people are still building their immunity to COVID, and the “control” period occurs after, when people have become fully vaccinated.
If someone gets one of the coagulopathic or vascular complications from COVID (say, a massive pulmonary embolism that ultimately leads to cardiac arrest), it’s not always as obvious in the moment that COVID is to blame, and “COVID” may not be on the death certificate.
Deaths like these wouldn’t have been excluded from analysis and may have been skewed toward the “risk” group, because people in the risk group (on average) likely had lower immunity to COVID than those in the control group.
In short, because immunity from the vaccine doesn’t develop on day 1 after vaccination, susceptibility to COVID itself confounds this analysis. For this study, a dive into medical records would be warranted for many reasons, including to know whether or not people had tested positive shortly before they died.
Other limitations: listen to the authors
The anonymous authors include a fairly lengthy limitations section, which points out other issues with the analysis. Their text is included below (mild formatting adjustments included for clarity):
“These data are preliminary, based on surveillance data, and should be interpreted with caution. The results have several limitations:
While this method has been used to assess risk of death following COVID-19 vaccination, it violates the assumption that an event does not affect subsequent exposure (for mRNA vaccines), which may introduce bias. Further, it does not consider the multidose vaccination schedule required for mRNA vaccination.
This study cannot determine the causative nature of a participant’s death. We used death certificate data and not medical records. COVID testing status was unknown for those who did not die of/with COVID. Cardiac-related deaths were ascertained if an ACME code of I3-I52 were on their death certificate, thus, the underlying cause of death may not be cardiac-related.
The finding that the Janssen vaccine was more protective than mRNA vaccine against mortality within 28 days of vaccination could be due to confounding and needs to be further evaluated. It is likely that the populations who received COVID-19 mRNA vaccine and the Johnson vaccine are different, something we were not able to ascertain in this analysis. It is possible that the population who received the Johnson vaccine was younger and healthier than those receiving the mRNA vaccines. The Pfizer and Moderna mRNA vaccines were released more than 2 months earlier than the Janssen vaccine when the recommendations were limited to those 65 and older.
Additional studies should be conducted to further understand the risks and benefits of vaccination of males between 25-39. Increased risk in the primary analysis for the 25 – 39 age group was based on a small sample size. Additionally, significant mortality from diagnosed COVID-19 infection occurred among all adult age groups. COVID-19 mortality among asymptomatic or undiagnosed COVID-19 infection is less clear. However, excess overall mortality among 25–44-year-old Americans was significant in a study1 looking at mortality from January 2020-October 2020. The largest increases were among Hispanic and Latino. It is unclear what the contribution of asymptomatic or undiagnosed COVID-19 infection is to mortality risk, and how this contributes to excess mortality
Finally, it should be noted that whenever you perform a risk assessment of vaccination, you compare the risks of the vaccine vs the disease it protects against. This analysis failed to do that, as it did not include any information on the risk of COVID for this age group.
Given the arbitrary selection of cardiac outcomes, the lack of validation, the small sample size, and the failure to control for COVID infection as a confounding variable, overall these results are largely unreliable, and personally I would not base any decisions on them.
Furthermore, while we can’t say for sure, this whole analysis smells of p-hacking, which occurs when people slice and dice the data a whole bunch of ways until they find a result they like.
Update, Monday Oct 10, 2022
Dr. Joseph Ladapo, the Florida State Surgeon General who originally put out this analysis, has provided a response to some of the criticisms of the analysis, including some I made here. Overall, his rebuttals do not satisfactorily address the points I raised, and my concerns about the analysis remain. Below are his tweets and a brief explanation why I find them unsatisfactory.
The argument here is quite vague. He seems to be arguing that because they used the same method across different subgroups and only some were significant (with young men being more so), that means the methodological concerns are minimal.
This is not a valid rebuttal. One does not decide if a method is valid based on whether or not it yielded statistically significant results. Varying levels of risk/statistical significance across subgroups is also not validation that the methods were sound — this can occur even when the data are random noise. Methods validate results, not the other way around.
It was clear that the methods section states that subjects were excluded if they had a documented COVID-19 infection. My concern arises from the limitations section, which states the exact opposite: “COVID testing status was unknown for those who did not die of/with COVID.” This blaring contradiction is not addressed.
If COVID testing was in fact used to exclude participants, details should be included regarding: 1. How they linked COVID testing data to death certificates, 2. How robust their COVID testing data are (would it be likely to capture the majority of people who actually tested positive?), and 3. how many subjects were excluded due to a positive test. This type of information would undoubtedly be asked for in peer review. As it stands, the current analysis does not provide sufficient evidence that they controlled for COVID infection as a confounder.
For 3a, the argument seems to be that the results from one subgroup analysis (men age > 60) are evidence of reproducibility of the result in the smaller subgroup (age 18-39). This is not correct.
If they wanted to show reproducibility, they would need to reproduce the result in the relevant age group. For example, a second cohort from a different state showing the same result in the same age group for the same vaccines would be a mark of reproducibility. Furthermore, the 40-59 age group does not show a significant association. As it stands, two subgroups show one result, and the third shows the opposite. Overall this is not particularly robust.
In short, the result in the age > 60 subgroup does not overcome the issues with small sample size in the age 18-39 subgroup.
Regarding 3b: for the analysis of men age 18-39 who received a mRNA vaccine, there were 20 events during the “risk” period (the window assessed for vaccine side effects) and 52 events during the control period. My original criticism remains – if even just a few of those 20 men had deaths that were misclassified and/or clearly unconnected to the vaccine, the statistically significant result will likely vanish.
Regarding 3c, self-controlled case series are not immune to the problems of low statistical power (small sample size). Statistical significance (which helps assess if “clustering” is likely due to “chance”) is essentially always influenced by sample size, inclusive of this method.
Different types of heart problems are not interchangeable. This rebuttal seems to be switching back and forth between myocarditis and “cardiac death” as if these are highly related outcomes, but in reality there are hundreds of different diseases that affect the heart, and a signal surrounding “cardiac death” would not necessarily imply anything about myocarditis specifically.
If the hypothesis is that myocarditis leads to an increase in cardiac deaths after vaccination, then deaths caused by myocarditis should be evaluated. This analysis was not performed; instead the two ICD 10 codes for myocarditis and two ICD 10 codes for pericarditis were lumped in with a bunch of other unrelated cardiac diagnoses and the results were analyzed in aggregate.
We have no idea if myocarditis was the cause of any of the deaths included in this analysis because the ICD 10 codes listed on the death certificates are not provided. The word “myocarditis” is not present anywhere in the analysis. As it stands, this analysis tells us absolutely nothing about myocarditis after vaccination.
