Methodology

Our methodology

Below are the core principles we try to follow. This might change over time as we steelman the techniques but we had to start somewhere.

  1. Use structured reasons and arguments (Dutilh Novaes & Zalta, 2021; Walton, 1988).
  2. If possible, use arguments based on evidence (Kelly & Zalta, 2016), science (Hepburn et al., 2021), and rationalism (Markie et al., 2021).
  3. Use a hierarchy of evidence (Schünemann et al., 2022) that generally values certain types of evidence higher than others in the following order:
    1. Experimental studies (Franklin et al., 2021) by experts in peer-reviewed, scientific journals
      1. Meta-analyses, systematic reviews (Lasserson et al., 2022) and umbrella reviews (Ioannidis, 2009) of the following types of studies:
      2. Randomized controlled trial (RCT) experiments (Reiss et al., 2022; Kendall, 2003): Patients are randomly assigned to an intervention group or a control group (not receiving the intervention) and the groups are compared, hopefully controlling for confounding variables (Lu, 2009):
        1. Placebo-controlled: A placebo is assumed to have no or minimal effect
        2. Non-placebo-controlled
        3. Versions of the above two types:
          1. Triple blinded (patient, experimenters, and data analyst)
          2. Double blinded (patient and experimenters)
          3. Single blinded (patient)
          4. Unblinded
      3. Experiments without a control group
        1. Experiments on groups
        2. Experiments on individuals (case studies)
      4. Experiments exploring a mechanism of action (Craver et al., 2019)
    2. Correlational/observational studies by experts in peer-reviewed, scientific journals
      1. Meta-analyses, systematic reviews, and umbrella reviews of the following types of studies:
      2. Cohort studies (Euser et al., 2009): Follow one exposed group and a non-exposed group (control) and compare outcomes.
        1. Prospective: Baseline is assessed and then researchers actively follow patients to perform a follow-up: More accurate data collection
        2. Retrospective: Historical analysis of existing data
      3. Case-control studies (Lu, 2009): Follow one group with an outcome and another without an outcome (control) and compare exposure.
      4. Cross-sectional studies (Lu, 2009): Analyze whether individuals were exposed and whether they had certain outcomes and compare to those that didn’t (control).
      5. Observations without a control group
        1. Observations of groups
        2. Observations of individuals (case studies)
      6. Ecological studies (Lu, 2009): Similar to cross-sectional studies but groups are analyzed instead of individuals
    3. Simulated model (Frigg et al., 2020) results by experts in peer-reviewed, scientific journals
    4. Opinions by experts in peer-reviewed, scientific journals
      1. Groups of experts
      2. Individual experts
    5. All of the above but not in peer-reviewed, scientific journals
    6. All of the above but not by experts
  4. Use a burden of proof

Burden of Proof

A burden of proof is an expectation by one side about the strength of argument required by another side for persuasion. A burden of proof may be useful to establish the core of a debate and avoid an argument going on indefinitely (Walton, 1988b). A burden of proof may assert controversial philosophical or ethical premises, but we think this is still valuable in clarifying the context and exit criteria of an argument.

Other general points

  1. We use the heuristic that mistakes are generally made due to incompetence rather than malice (Bloch, 2003), although the latter is certainly possible.
  2. When comparing arguments, the number of points doesn’t necessarily matter.

Caveats

  1. There are potential issues with the concept of a hierarchy of evidence (Anglemyer et al., 2014; Frieden, 2017; Stegenga, 2018; Blunt, 2015).
  2. A lack of evidence higher in the hierarchy is not necessarily problematic due to infeasibility, ethical issues, etc.
  3. There are potential issues with specific types of evidence; for examples:
    1. RCTs: Simpson’s Paradox (Sprenger et al., 2021)
  4. In some cases, evidence lower in the hierarchy may be stronger; for example, a well-done experiment might be stronger than a poorly done RCT.
  5. There are potential issues with all types of evidence (Ioannidis, 2005); for examples:
    1. Lacking or poor reproduction, or too much reproduction (Ioannidis & Trikalinos, 2007)
    2. Statistically significant but false effects due to low statistical power (Button et al., 2013)
    3. Non-random sampling (Carlisle, 2017)
    4. Small effect sizes
    5. Poor study design or methodology
    6. Poor external validity (Reiss et al., 2022)
    7. Incorrect statistical controls (Westfall & Yarkoni, 2016)
    8. Mistakes (data entry, variable coding, statistics, over-interpretation, etc.)
    9. Multiple comparisons (Bennett et al., 2009)
    10. p-hacking, researcher degrees of freedom, multiple potential comparisons, or fishing expeditions (Simmons et al., 2016; Humphreys et al., 2013; Gelman & Loken, 2013)
    11. Fraud (Smith, 2021; Van Noorden, 2022; Adam, 2019)
    12. Conflicts of interest (Lundh et al., 2010; Angell, 2009)
    13. Poor incentives (Nosek et al., 2012)
    14. Poor instrument or method reliability (Vul et al., 2009)

References

39 references
  1. (Adam, 2019):

    Adam, D. (2019). How a data detective exposed suspicious medical trials. Nature, 571(7766), 462-465. DOI: https://doi.org/10.1038/d41586-019-02241-z. Source: https://www.nature.com/articles/d41586-019-02241-z

  2. (Angell, 2009):

    Angell, M. (2009). Drug companies & doctors: A story of corruption. The New York Review of Books, 56(1), 8-12. Retrieved August, 2022 from https://www.nybooks.com/articles/2009/01/15/drug-companies-doctorsa-story-of-corruption/

  3. (Anglemyer et al., 2014):

    Anglemyer, A., Horvath, H. T., & Bero, L. (2014). Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database of Systematic Reviews, (4). DOI: https://doi.org/10.1002/14651858.MR000034.pub2. Source: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.MR000034.pub2/epdf/full/en

  4. (Bennett et al., 2009):

    Bennett, C. M., Baird, A. A., Miller, M. B., and Wolford, G. L. (2009). Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction. Poster presented at Human Brain Mapping conference. https://tauruspet.med.yale.edu/staff/edm42/courses/ENAS_880_2018/papers/Bennett-Salmon-2009.pdf

  5. (Bloch, 2003):

    Bloch, A. (2003). Murphy’s law. Penguin. https://archive.org/details/murphyslawbooktw00bloc/page/52/mode/2up

  6. (Blunt, 2015):

    Blunt, C. (2015). Hierarchies of evidence in evidence-based medicine (Doctoral dissertation, London School of Economics and Political Science). Retrieved July, 2022, from https://etheses.lse.ac.uk/3284/1/Blunt_heirachies_of_evidence.pdf

  7. (Button et al., 2013):

    Button, K. S., Ioannidis, J., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature reviews neuroscience, 14(5), 365-376. DOI: https://doi.org/10.1038/nrn3475. Source: https://www.nature.com/articles/nrn3475.pdf

  8. (Carlisle, 2017):

    Carlisle, J. B. (2017). Data fabrication and other reasons for non‐random sampling in 5087 randomised, controlled trials in anaesthetic and general medical journals. Anaesthesia, 72(8), 944-952. DOI: https://doi.org/10.1111/anae.13938. Source: https://associationofanaesthetists-publications.onlinelibrary.wiley.com/doi/pdfdirect/10.1111/anae.13938?download=true

  9. (Craver et al., 2019):

    Craver, C., Tabery, J., & Zalta, E. (Ed.) (2019). Mechanisms in Science. The Stanford Encyclopedia of Philosophy (Summer 2019 Edition). https://plato.stanford.edu/archives/sum2019/entries/science-mechanisms/

  10. (Dutilh Novaes & Zalta, 2021):

    Dutilh Novaes, C., & Zalta, E. (Ed.) (2021). Argument and Argumentation. The Stanford Encyclopedia of Philosophy (Fall 2021 Edition). https://plato.stanford.edu/archives/fall2021/entries/argument/

  11. (Euser et al., 2009):

    Euser, A. M., Zoccali, C., Jager, K. J., & Dekker, F. W. (2009). Cohort studies: prospective versus retrospective. Nephron Clinical Practice, 113(3), c214-c217. https://doi.org/10.1159/000235241

  12. (Franklin et al., 2021):

    Franklin, A., Perovic, S., & Zalta, E. (Ed.) (2021). Experiment in Physics. The Stanford Encyclopedia of Philosophy (Summer 2021 Edition). https://plato.stanford.edu/archives/sum2021/entries/physics-experiment/

  13. (Frieden, 2017):

    Frieden, T. R. (2017). Evidence for health decision making—beyond randomized, controlled trials. New England Journal of Medicine, 377(5), 465-475. DOI: https://doi.org/10.1056/NEJMra1614394. Source: https://www.nejm.org/doi/pdf/10.1056/NEJMra1614394

  14. (Frigg et al., 2020):

    Frigg, R., Hartmann, S., & Zalta, E. (Ed.) (2020). Models in Science. The Stanford Encyclopedia of Philosophy (Spring 2020 Edition). https://plato.stanford.edu/archives/spr2020/entries/models-science/

  15. (Gelman & Loken, 2013):

    “P-values are a method of protecting researchers from declaring truth based on patterns in noise, and so it is ironic that, by way of data-dependent analyses, p-values are often used to lend credence to noisy claims based on small samples. To put it another way: without modern statistics, we find it unlikely that people would take seriously a claim about the general population of women, based on two survey questions asked to 100 volunteers on the internet and 24 college students. But with the p-value, a result can be declared significant and deemed worth publishing in a leading journal in psychology.”

    “absent pre-registration, our data analysis choices will be data-dependent, even when they are motivated directly from theoretical concerns. When pre-registered replication is difficult or impossible (as in much research in social science and public health), we believe the best strategy is to move toward an analysis of all the data rather than a focus on a single comparison or small set of comparisons”

    “In fields where new data can readily be gathered (such as in all four of the examples discussed above), perhaps the two-part structure of Nosek et al. (2013) will be a standard for future research. Instead of the current norm in which several different studies are performed, each with statistical significance but each with analyses that are contingent on data, perhaps researchers can perform half as many original experiments in each paper and just pair each new experiment with a pre-registered replication.”

     

    Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University, 348, 1-17. https://stat.columbia.edu/~gelman/research/unpublished/forking.pdf

  16. (Hepburn et al., 2021):

    Hepburn, B., Andersen, H., & Zalta, E. (Ed.) (2021). Scientific Method. The Stanford Encyclopedia of Philosophy (Summer 2021 Edition). https://plato.stanford.edu/archives/sum2021/entries/scientific-method/

  17. (Humphreys et al., 2013):

    Humphreys, M., De la Sierra, R. S., & Van der Windt, P. (2013). Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration. Political Analysis, 21(1), 1-20. https://doi.org/10.1093/pan/mps021

  18. (Ioannidis, 2005):

    Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124

  19. (Ioannidis, 2009):

    Ioannidis, J. P. (2009). Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. Cmaj, 181(8), 488-493. DOI: https://doi.org/10.1503/cmaj.081086. Source: https://www.cmaj.ca/content/cmaj/181/8/488.full.pdf

  20. (Ioannidis & Trikalinos, 2007):

    Ioannidis, J. P., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical trials, 4(3), 245-253. https://doi.org/10.1177/1740774507079441

  21. (Kelly & Zalta, 2016):

    Kelly, T., & Zalta, E. (Ed.) (2016). Evidence. The Stanford Encyclopedia of Philosophy (Winter 2016 Edition). https://plato.stanford.edu/archives/win2016/entries/evidence/

  22. (Kendall, 2003):

    Kendall, J. (2003). Designing a research project: randomised controlled trials and their principles. Emergency medicine journal: EMJ, 20(2), 164. DOI: https://doi.org/10.1136/emj.20.2.164. Source: https://emj.bmj.com/content/emermed/20/2/164.full.pdf

  23. (Lasserson et al., 2022):

    Lasserson, TJ., Thomas, J., & Higgins, JPT. (2022). Cochrane handbook for systematic reviews of interventions. Cochrane. Retrieved July, 2022, from https://training.cochrane.org/handbook/current/chapter-01#section-1-1

  24. (Lu, 2009):

    Lu, C. Y. (2009). Observational studies: a review of study designs, challenges and strategies to reduce confounding. International journal of clinical practice, 63(5), 691-697. https://doi.org/10.1111/j.1742-1241.2009.02056.x

  25. (Lundh et al., 2010):

    Lundh, A., Barbateskovic, M., Hróbjartsson, A., & Gøtzsche, P. C. (2010). Conflicts of interest at medical journals: the influence of industry-supported randomised trials on journal impact factors and revenue–cohort study. PLoS medicine, 7(10), e1000354. DOI: https://doi.org/10.1371/annotation/7e5c299c-2db7-4ddf-8eff-ab793511eccd. Source: https://journals.plos.org/plosmedicine/article/file?id=10.1371/journal.pmed.1000354&type=printable

  26. (Markie et al., 2021):

    Markie, P., Folescu, M., & Zalta, E. (Ed.) (2021). Rationalism vs. Empiricism. The Stanford Encyclopedia of Philosophy (Fall 2021 Edition). https://plato.stanford.edu/archives/fall2021/entries/rationalism-empiricism/

  27. (Nosek et al., 2012):

    Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615-631. DOI: https://doi.org/10.1177/1745691612459058. Source: https://journals.sagepub.com/doi/pdf/10.1177/1745691612459058

  28. (Reiss et al., 2022):

    Reiss, J., Ankeny, R., & Zalta, E. (Ed.) (2022). Philosophy of Medicine. The Stanford Encyclopedia of Philosophy (Summer 2022 Edition). https://plato.stanford.edu/archives/spr2022/entries/medicine/

  29. (Schunemann et al., 2022):

    “Not downgrading [Non-randomized Studies of Interventions] from high to low certainty needs transparent and detailed justification for what mitigates concerns about confounding and selection bias (Schünemann et al 2018). Very few examples of where not rating down by two levels is appropriate currently exist.”

     

    Schünemann, HJ., Higgins, JPT., Vist, GE., Glasziou, P., Akl, EA., Skoetz, N., & Guyatt, GH. (2022). Cochrane handbook for systematic reviews of interventions. Cochrane. Retrieved July, 2022, from https://training.cochrane.org/handbook/current/chapter-14#section-14-2

  30. (Simmons et al., 2016):

    Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2016). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. https://psycnet.apa.org/doi/10.1037/14805-033

  31. (Smith, 2021):

    Smith, R. (2021). Time to assume that health research is fraudulent until proven otherwise. The BMJ Opinion. Retrieved August, 2022, from https://blogs.bmj.com/bmj/2021/07/05/time-to-assume-that-health-research-is-fraudulent-until-proved-otherwise/

  32. (Smith & Pell, 2003):

    “Stephen Lock, my predecessor as editor of The BMJ, became worried about research fraud in the 1980s, but people thought his concerns eccentric. Research authorities insisted that fraud was rare, didn’t matter because science was self-correcting, and that no patients had suffered because of scientific fraud. All those reasons for not taking research fraud seriously have proved to be false, and, 40 years on from Lock’s concerns, we are realising that the problem is huge, the system encourages fraud, and we have no adequate way to respond. It may be time to move from assuming that research has been honestly conducted and reported to assuming it to be untrustworthy until there is some evidence to the contrary.

    Richard Smith was the editor of The BMJ until 2004.”

     

    Smith, G. C., & Pell, J. P. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ, 327(7429), 1459-1461. https://doi.org/10.1136/bmj.327.7429.1459

  33. (Sprenger et al., 2021):

    Sprenger, J., Weinberger, N., & Zalta, E. (Ed.) (2021). Simpson’s Paradox. The Stanford Encyclopedia of Philosophy (Summer 2021 Edition). https://plato.stanford.edu/archives/sum2021/entries/paradox-simpson/

  34. (Stegenga, 2018):

    Stegenga, J. (2018). Medical nihilism. Oxford University Press. https://doi.org/10.1093/oso/9780198747048.001.0001

  35. (Van Noorden, 2022):

    Van Noorden, R. (2022). Exclusive: investigators found plagiarism and data falsification in work from prominent cancer lab. Nature, 607(7920), 650-652. DOI: https://doi.org/10.1038/d41586-022-02002-5. Source: https://media.nature.com/original/magazine-assets/d41586-022-02002-5/d41586-022-02002-5.pdf

  36. (Vul et al., 2009):

    Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on psychological science, 4(3), 274-290. https://doi.org/10.1111/j.1745-6924.2009.01125.x

  37. (Walton, 1988):

    “This description of reasoned dialogue as a process of deepened insight into one’s own position on a controversial issue is consistent with the Socratic view of dialogue as a means to attain self-knowledge. For Socrates, the process of learning was an ascent from the depths of the cave towards the clearer light of self-knowledge through the process of reasoned, and primarily verbal, dialogue with another discussant, on controversial issues. What Socrates emphasized as a most important benefit or gain of dialogue was self-knowledge. It was somehow through the process of articulation and testing of one’s best arguments against an able opponent in dialogue that real knowledge was to be gained.

    This Socratic point of view draws our attention to the more hidden and subtle benefit of good, reasoned dialogue. Not only does it enable one to rationally persuade an opponent or co-participant in discussion, but it is also the vehicle that enables one to come to better understand one’s own position on important issues, one’s own reasoned basis behind one’s deeply held convictions. It is the concept of burden of proof that makes such shifts of rational persuasion possible, and thereby enables dialogue to contribute to knowledge.”

     

    Walton, D. N. (1988). Burden of proof. Argumentation, 2(2), 233-254. https://doi.org/10.1007/BF00178024

  38. (Walton, 1988b):

    “One of the most trenchant and fundamental criticisms of reasoned dialogue as a method of arriving at a conclusion is that argument on a controversial issue can go on and on, back and forth, without a decisive conclusion ever being determined by the argument. The only defence against this criticism lies in the use of the concept of the burden of proof within reasoned dialogue. Once a burden of proof is set externally, then it can be determined, after a finite number of moves in the dialogue, whether the burden has been met or not. Only by this device can we forestall an argument from going on indefinitely, and thereby arrive at a definite conclusion for or against the thesis at issue.”

     

    Walton, D. N. (1988b). Burden of proof. Argumentation, 2(2), 233-254. https://doi.org/10.1007/BF00178024

  39. (Westfall & Yarkoni, 2016):

    Westfall, J., & Yarkoni, T. (2016). Statistically controlling for confounding constructs is harder than you think. PloS one, 11(3), e0152719. DOI: https://doi.org/10.1371/journal.pone.0152719. Source: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0152719&type=printable

 

Other topics: