Peer review (weird version)
An incoherent rant about interpreting an interesting study about COVID deaths
It’s not easy to get most people to read and share academic papers when they’re not either being held at gunpoint or hours away from a deadline for their final. But when “Excess Death Rates for Republicans and Democrats During the COVID-19 Pandemic” was first published, it was hard to scroll very far on Twitter without encountering the study. “The Right’s Anti-Vaxxers Are Killing Republicans”, said the Intercept. Other outlets were more even-tempered in reporting the study, but the message was spread far and wide: Republicans are facing the consequences of their ill-advised beliefs.
The most eye-catching figure in the paper, republished by a number of news outlets, is this:
Put simply, excess deaths for Democrats and Republicans were more or less equivalent until the pandemic. Then, the Republicans began to see slightly more deaths than their compatriots until around the time of the vaccine rollout. That’s when the gap truly started to widen, and by the end of 2021, Republicans were dying in far greater numbers than Democrats.
Being before anything a party pooper, I knew that I needed to figure out myself whether this study was actually any good, or if it was some mumbo-jumbo that the libs were sharing while buzzed on confirmation bias.
Upon my first reading, only one methodological problem stuck out to me. Granted, I’m not the most perceptive academic reviewer, and the equations could have been gibberish for all I’m aware. In any case, I was concerned by the threshold the researchers selected for their analysis. The authors state:
We selected April 5, 2021 as a cutoff for when vaccines were widely available to adults in Florida and Ohio. The federal deadline for when states had to open eligibility for vaccines for all adults was April 24, 2021, but Florida and Ohio did so earlier. Florida made vaccines available to all adults on April 5, 2021… because our data is monthly, we use April 2021 data on and onwards to reflect this period, which corresponds closely to when vaccines were widely available.
However, this is a clearly inappropriate threshold; a cursory look at the Ohio and Florida vaccination uptake statistics1 should reveal the issue.
By April 5th, roughly half of all people who would go on to receive a vaccine in the next year and a half had already received their first dose in Ohio. The situation in Florida is slightly less drastic, but the fraction is still well over a third. Likewise, a third of Ohio residents who would go on to complete their vaccine regimens had already gotten both shots, with Florida once again not far behind in proportion.
This is a significant issue. Nevertheless, it isn’t fatal to the conclusions of the paper; the authors simply need to incorporate a sensitivity analysis for their chosen threshold in the final paper. By a simple visual examination, I think that the conclusion still may very well hold, but it must be tested. This is a pre-print, after all: it’s meant to be subject to revision!
It wasn’t enough for me to just read the paper, though; I had to see what the other side was saying in response. In the several hours that I've spent online, I’ve learnt that there is a rambunctious community of people who love nothing more than to loudly contest research that shows vaccines as efficacious. Surely they’d be all over this paper! Surprisingly, though, I only came across two critiques of any substance.2
This is strange, because this study went incredibly viral, and I’m sure that many of the head honchos of the anti-vaccine community encountered it. If I could find such a weird design choice in a cursory examination of the paper, then why didn’t one of them thoroughly dissect it like a lab frog? Perhaps, even with the flaw I identified, the study was so robust that it withstood even opponents with the most motivated reasoning known to man.
In any case, the critiques that I did find, few though they were, did make valid points. First, the weaker critique: the blogger “Modern Discontent” makes the observation that the study only assessed Ohio and Florida, and used party registration from a pre-pandemic year (2017) for the analysis. Additionally, the Substacker noted how Figure 3 of the study used the arbitrary measure of one-dose vaccination on June 6, 2021 to sort counties, and the “bins” used to adjust by age were fairly wide and imprecise. Finally, there was a critique that is based upon a misreading of the study that I needn’t address here.
The stronger critique here comes from Mary Pat Campbell, who actually has some valid credentials as an actuary. In fact, she is in no sense an anti-vaxxer; she’s just a conservative aggrieved at the partisan underpinnings of this study, which is somewhat understandable. She has frankly done some valuable work analyzing the drivers of the rise in all-cause mortality over the past year and a half, which I recommend you check out.
Campbell, being fairly level-headed, doesn’t hang the study out to dry. She begins by noting that it doesn’t make the mistakes that many other similar analyses do, which fail to adjust for crucial covariates like age! Frankly, checking to see if someone has adjusted for age in their analysis of COVID trends is a good way to tell whether to discount their work entirely. I see this mistake made frequently by anti-vaxxers who don’t account for age differences in mortality and vaccine uptake, but the fallacy can come from the pro-vaccine side as well.
The strengths of the study, Campbell continues, don’t excuse its deficiencies. As it focuses only on registered voter data from Ohio and Florida (which “Modern Discontent” already noted), the study is subject to the whims of the states’ registration laws. She points out that affiliated voters in Ohio must vote in their party’s primary in the last two years, meaning that fully 78 percent of voters in the state are unaffiliated! It is left unaddressed by the authors how this law affects their sample.
Of course, Campbell couldn’t just complete her critique without one strange non-sequitur claim. She states in response to the assumptions that Democrats in this analysis are more vaccinated that she “would not assume that Republicans had a different uptake of vaccines in Florida than Democrats did… people make this a political thing where some of the conspiracies are of a more medical nature”.
This is frankly a ridiculous claim for someone as otherwise well-informed as Campbell to make. Not only has county-by-county analysis shown a strong (and increasing) correlation between partisanship and vaccination, but survey data testifies to the very same trend! While partisanship certainly isn’t a substitute for vaccination status, it certainly can serve as a decent proxy.
In conclusion, there are a number of significant shortcomings of the study. It uses an arbitrary discontinuity, its measure of party registration is imperfect, and it’s restricted to only two out of the fifty states. With any luck, though, these qualms will be addressed after the paper goes through a much-needed formal review process. Critics of the study raised good points, but ultimately nothing fundamentally discrediting.
Unfortunately, I don’t have a clear lesson to leave you with after all of this. I suppose it could be that the fact-checker’s binary of “true” or “false” is hard to square with the nuances of science, though the generally high-quality fact checking website HealthFeedback did a decent job dissecting this study. We would all stand to benefit if we approach data- whether it agrees or disagrees with our suppositions- with both intellectual humility and healthy skepticism.
Lame!
While I was able to find a readable graph of cumulative uptake on Ohio’s official website, Florida’s portal was so unusable (they only display doses taken in the past year!) that I decided to simplify everything and use OurWorldInData for both states’ data. Surely that has nothing to do with the policy choices that Florida has made.
It’s possible that I missed some other worthy critique(s), because I essentially restricted my search to the Substack domain. While that may appear to be an arbitrary choice, this website really is the “intellectual” center of the anti-vaccine movement, and if it isn’t on here, I don’t know where else it would be.
Interesting topic. Well-written and concise. You have a knack at this. Will be interested to see if there are any updates
@dudeasincool on Twitter
I haven’t read the study other than in the popular press. Non peer-reviewed scholarship can be fraught with bias as you clearly point out. Unless the R and D cohorts are matched for multiple comorbidities as well as age, statistical validity is questionable. Has this since been peer reviewed and published?