What does the Flaxman controversy say about the U.S. CDC?

3 minute read

While steelmanning whether children 6 months through 4 years should get the Pfizer vaccine or not, a controversy was highlighted over the Flaxman et al. (2022) paper. This paper was used prominently by the U.S. CDC during its vaccine approval process (U.S. CDC, 2022).

The controversy started after “a mother who reviews this data on my personal time” reported an error about the reported death rates on Twitter to the study’s main author, Dr. Flaxman (Kelley, 2022; Kelley, 2022b). Dr. Flaxman admitted to the error on Twitter (Flaxman, 2022). The study was subsequently fixed and re-published, and significantly changed the results: “We have fixed an error” (Flaxman et al., 2022c).

One might think, “that’s unfortunate but that’s just science in action”. The paper was used by the CDC prominently, but one might think, “that’s just an unfortunate mistake in a fluid situation and the vaccine would have been approved anyway due to other evidence”.

However, some facts about the controversy raise some questions:

  1. It was a paper about the United States that was authored by a team mostly out of the United Kingdom. The team made basic mistakes mixing data from incompatible databases which is something a U.S. team accustomed to such data would not do, particularly the CDC itself which publishes the raw data.
  2. The paper originally mixed cumulative COVID-19 deaths and annualized deaths for other causes. No study was used to justify this (Flaxman et al., 2022d), and it was removed in the final version. However, the CDC knowingly mixed annualized and cumulative deaths as it noted on its slides (U.S. CDC, 2022b; U.S. CDC, 2022c).
  3. The paper was a pre-print meaning it was not yet peer-reviewed. This is not an inherent problem but suggests extra caution.
  4. The claims were repeated at least three times by three different CDC doctors: Dr. Daley, Dr. Fleming-Dutra, and Dr. Oliver (U.S. CDC, 2022f; U.S. CDC, 2022g; U.S. CDC, 2022).

Here’s the steelman defending the CDC:

  1. The CDC are under extreme pressure in an emergency situation with limited resources so they depended on an external team of scientists and didn’t have the time to investigate it in detail due to competing demands for time. The lead author, Dr. Flaxman, has co-authored other research on COVID-19 epidemiology in the United States so he was being trusted (Monod et al., 2021).
  2. Despite the cumulative statistic ultimately being removed from the study, cumulative mortality is a plausible way to compare COVID-19 deaths to other deaths because lockdowns, masks, and social distancing might have reduced the death rate that otherwise would have occurred. The use of the cumulative statistics were clearly noted on the slides.
  3. Even if COVID-19 ranks lower than flu & pneumonia, there are still a significant number of COVID-19 deaths for this age group and the basic point stands that childhood deaths are a significant justification for approving the vaccine as the flu vaccine is similarly approved for this age range (U.S. CDC, 2022e).

Here’s the skeptic steelman:

  1. Instead of analyzing their own data which is not particularly complicated (U.S. CDC, 2022d), the CDC used the analysis of a non-peer reviewed study by a team from the United Kingdom who were so unfamiliar with the data that they made a basic and significant mistake that, when fixed, largely changed the story the CDC presented at the vaccine approval meeting. The fixed study shows half the rates as seen in the CDC slides, ranked COVID-19 7th instead of 4th/5th, and ranked flu & pneumonia as worse mortality than COVID-19 (Flaxman et al., 2022b).
  2. The CDC knowingly used an uncommon comparison of death rates (cumulative vs. annualized), later removed from the study, that had purely speculative assumptions which were not even mentioned. The original study reported annualized COVID-19 statistics but the CDC consciously chose to use the cumulative statistics. The presentations also emphasized that COVID-19 was the top infectious cause of death which is no longer true in the study results.
  3. The CDC could have presented mortality much more simply, gotten the point across that COVID-19 kills lots of children, and the vaccine would have still likely been approved, so what was the point of such gratuitous exaggerations? At best, this is a sign of general incompetence; at worst, motivated reasoning and vaccine zealotry. In either case, this makes one wonder about the CDC’s other data and interpretations.

We reached out with questions to Dr. Flaxman, Dr. Daley, Dr. Fleming-Dutra, and Dr. Oliver but have not heard back for days. If they respond, we’ll update this post.

Below are the slides the CDC used during its presentation to the approval committees (U.S. CDC, 2022; U.S. CDC, 2022b; U.S. CDC, 2022c).

References

15 references
  1. (Flaxman, 2022):

    “We have received some feedback and criticism along several dimensions. We are planning to update the preprint to take into account some of this feedback, primarily by focusing on Covid as an underlying cause of death using CDC WONDER Provisional Mortality Statistics.”

     

    Flaxman, S. [@flaxter]. (2022, June 19). We have received some feedback […] [Tweet]. Twitter. https://twitter.com/flaxter/status/1538533542543646720

  2. (Flaxman et al., 2022):

    Flaxman, S., Whittaker, C., Semenova, E., Rashid, T., Parks, R., Blenkinsop, A., … & Ratmann, O. (2022). Covid-19 is a leading cause of death in children and young people ages 0-19 years in the United States. medRxiv. DOI: 10.1101/2022.05.23.22275458. https://doi.org/10.1101/2022.05.23.22275458 ; Recommended: https://www.medrxiv.org/content/10.1101/2022.05.23.22275458v3.full.pdf#page=5

  3. (Flaxman et al., 2022b):

    “Table 1(a) Age: < 1 year […] #Influenza and pneumonia (J09-J18): 4.1 […] #COVID-19 (U07.1): 3.5

    Table 1(b) Age: 1-4 year olds […] #Influenza and pneumonia (J09-J18): 0.8 […] #COVID-19 (U07.1): 0.5” (Pages 5-6)

     

    Flaxman, S., Whittaker, C., Semenova, E., Rashid, T., Parks, R., Blenkinsop, A., … & Ratmann, O. (2022b). Covid-19 is a leading cause of death in children and young people ages 0-19 years in the United States. medRxiv. DOI: 10.1101/2022.05.23.22275458. https://doi.org/10.1101/2022.05.23.22275458 ; Recommended: https://www.medrxiv.org/content/10.1101/2022.05.23.22275458v3.full.pdf#page=5

  4. (Flaxman et al., 2022c):

    “We have fixed an error: our comparisons now use Covid-19 underlying cause of death data obtained from CDC Wonder.”

     

    Flaxman, S., Whittaker, C., Semenova, E., Rashid, T., Parks, R., Blenkinsop, A., … & Ratmann, O. (2022c). Covid-19 is a leading cause of death in children and young people ages 0-19 years in the United States. medRxiv. https://www.medrxiv.org/content/10.1101/2022.05.23.22275458v3.article-info

  5. (Flaxman et al., 2022d):

    “Cumulative mortality is a useful comparison as the first two years of the pandemic included significant non-pharmaceutical interventions which limited spread in the pre-Omicron period, while annualized mortality may represent a lower bound on the future burden in the absence of widespread vaccination in pediatric age groups.” (Page 2)

     

    Flaxman, S., Whittaker, C., Semenova, E., Rashid, T., Parks, R., Blenkinsop, A., … & Ratmann, O. (2022c). Covid-19 is a leading cause of death in children and young people ages 0-19 years in the United States. medRxiv. https://www.medrxiv.org/content/10.1101/2022.05.23.22275458v2.full.pdf#page=2

  6. (Kelley, 2022):

    “This slide and the claims that Covid is a top 5 cause of death in young children are false. There are major data issues with this slide put out by @cdcgov for FDA VRBPAC and ACIP for their childhood vaccine approval discussions.”

     

    Kelley, K. [@KelleyKga]. (2022, June 17). This slide and […] [Tweet]. Twitter. https://twitter.com/KelleyKga/status/1538014817838956546

  7. (Kelley, 2022b):

    “How did I uncover these issues, instead of them being identified by someone whose job it is to evaluate this kind of data? I am a mother who reviews this data on my personal time, and yet I seem to be much more knowledgeable about Covid deaths in children than most academics and public health officials working with Covid-19.”

     

    Kelley (2022b). Fact Check: Covid as a Leading Cause of Death in Children. Retrieved July, 2022, from https://www.covid-georgia.com/pediatric-news/fact-check-covid-is-a-leading-cause-of-death-in-children/

  8. (Monod et al., 2021):

    Monod, M., Blenkinsop, A., Xi, X., Hebert, D., Bershan, S., Tietze, S., … & Imperial College COVID-19 Response Team. (2021). Age groups that sustain resurging COVID-19 epidemics in the United States. Science, 371(6536). DOI: 10.1126/science.abe8372. https://doi.org/10.1126/science.abe8372

  9. (U.S. CDC, 2022):

    “COVID-19 was a leading cause of death among children ages 0 – 4 years” (Page 16)

     

    U.S. CDC (2022). Evidence to Recommendation Framework: Moderna COVID-19 vaccine in children ages 6 months – 5 years & Pfizer-BioNTech COVID-19 vaccine in children ages 6 months – 4 years. Retrieved July, 2022, from https://www.cdc.gov/vaccines/acip/meetings/downloads/slides-2022-06-17-18/03-COVID-Oliver-508.pdf#page=16

  10. (U.S. CDC, 2022b):

    “COVID-19 is a leading cause of death among infants age <1 year” (Page 17)

     

    U.S. CDC (2022b). Evidence to Recommendation Framework: Moderna COVID-19 vaccine in children ages 6 months – 5 years & Pfizer-BioNTech COVID-19 vaccine in children ages 6 months – 4 years. Retrieved July, 2022, from https://www.cdc.gov/vaccines/acip/meetings/downloads/slides-2022-06-17-18/03-COVID-Oliver-508.pdf#page=17

  11. (U.S. CDC, 2022c):

    “COVID-19 is a leading cause of death among children ages 1–4 years” (Page 18)

     

    U.S. CDC (2022c). Evidence to Recommendation Framework: Moderna COVID-19 vaccine in children ages 6 months – 5 years & Pfizer-BioNTech COVID-19 vaccine in children ages 6 months – 4 years. Retrieved July, 2022, from https://www.cdc.gov/vaccines/acip/meetings/downloads/slides-2022-06-17-18/03-COVID-Oliver-508.pdf#page=18

  12. (U.S. CDC, 2022d):

    Group results by: Year; Ten-Year Age Groups: < 1 year and 1-4 years; Select underlying cause of death: U07.1 (COVID-19) versus (J09-J18) (influenza and pneumonia)

     

    U.S. CDC (2022d). Provisional mortality statistics, 2018 through last month request: Deaths occurring through June 25, 2022 as of July 06, 2022. Retrieved July, 2022, from http://wonder.cdc.gov/mcd-icd10-provisional.html

  13. (U.S. CDC, 2022e):

    “CDC recommends that everyone 6 months and older get a seasonal flu vaccine each year”

     

    U.S. CDC (2022e). Flu & Young Children. Retrieved July, 2022, from https://www.cdc.gov/flu/highrisk/children.htm

  14. (U.S. CDC, 2022f):

    “Among people ages 1-4, COVID-19 is the fifth most common of all causes of death” (Page 4)

     

    U.S. CDC (2022f). COVID-19 death rate among children by age, United States, March 1, 2020—April 30, 2022. Retrieved July, 2022, from https://www.cdc.gov/vaccines/acip/meetings/downloads/slides-2022-06-17-18/01-COVID-Daley-508.pdf#page=4

  15. (U.S. CDC, 2022g):

    “COVID-19 is a leading cause of death among children ages 0–19 years” (Page 26)

     

    U.S. CDC (2022g). COVID-19 epidemiology in children ages 6 months–4 years. Retrieved July, 2022, from https://www.cdc.gov/vaccines/acip/meetings/downloads/slides-2022-06-17-18/02-COVID-Fleming-Dutra-508.pdf#page=26

You may also enjoy

Our First Steelmen

4 minute read

This post is a bit academic, and will be unlike most other posts, but it seems fitting that our first steelmanning exercise should be to consider potential p...

Welcome to the Steelman Anything Newsletter

1 minute read

Steelmanning is another name for the principle of charity in argumentation that tries to make the strongest possible argument for someone by empathizing with...