QML - Week 9

Open Research

Stefano Coretta

Attendance

Built-in control




“But scientific power is like inherited wealth: attained without discipline. You read what others have done, and you take the next step. […] There is no discipline lasting many decades. […] There is no humility before nature.”

Ian Malcom, chaotician

RESEARCH

“A crisis of confidence?” (Pashler and Wagenmakers 2012)





REASERCH CRISES

Part of the polycrisis (Morin and Kern 1999; Tooze 2023)

Research reliability

From The Turing Way.

Research reliability

From The Turing Way.

Reproducibility: shareability

Miske et al. (2026)

Reproducibility: computational

Miske et al. (2026)

Reproducibility: procedural

96%

of researchers have never shared data (DuBois 2024)

Claiming data collector is the only legitimate knowledge gatekeeper is at odds with epistemic plurality (Lamb et al. 2024)

Data are not epistemically exhausted by first analysis (Heaton 2008)

Replicability

Open Science Collaboration (2015)

Robustness


50-200% of original effect

in 69% of studies.

Economy and political science (Brodeur et al. 2026)


Not invariance across all methods, but insensitivity within a justified class of models.

(Auspurg 2025)

Generalisability

Uninterpretability of conceptual models (Meehl 1990)

Lack of meaningful connection between verbal generalisations and evidence (Scheel 2022; Yarkoni 2022)

Replicable results do not imply correct conditional inference and non-replicable results do not imply incorrect conditional inference (Devezer et al. 2019)

Open Research in practice

Quarto projects and documents.

Version control with git and GitHub/GitLab.

Sharing research compendia on Zenodo.

Sharing manuscripts on preprint servers.

License your work for reuse.

A (bad) example

https://doi.org/10.1080/10255842.2026.2751469, Computer Methods in Biomechanics and Biomedical Engineering

References

Auspurg, Katrin. 2025. “Robustness Is Better Assessed with a Few Thoughtful Models Than with Billions of Regressions.” Proceedings of the National Academy of Sciences 122 (43): e2521917122. https://doi.org/10.1073/pnas.2521917122.
Brodeur, Abel, Derek Mikola, Nikolai Cook, et al. 2026. “Reproducibility and Robustness of Economics and Political Science Research.” Nature 652 (8108): 151–56. https://doi.org/10.1038/s41586-026-10251-x.
Devezer, Berna, Luis G Nardin, Bert Baumgaertner, and Erkan Ozge Buzbas. 2019. “Scientific Discovery in a Model-Centric Framework: Reproducibility, Innovation, and Epistemic Diversity.” PloS One 14 (5): e0216125. https://doi.org/10.1371/journal.pone.0216125.
DuBois, James M. 2024. Sharing Qualitative Research Data: Survey of Qualitative Researchers, United States, 2019: Version 1. ICPSR - Interuniversity Consortium for Political and Social Research. https://doi.org/10.3886/ICPSR38957.V1.
Heaton, Janet. 2008. “Secondary Analysis of Qualitative Data: An Overview.” Historical Social Research 33 (3). https://www.jstor.org/stable/20762299.
Lamb, Danielle, Amy Russell, Nicola Morant, and Fiona Stevenson. 2024. “The Challenges of Open Data Sharing for Qualitative Researchers.” Journal of Health Psychology 29 (7): 659–64. https://doi.org/10.1177/13591053241237620.
Meehl, Paul E. 1990. “Why Summaries of Research on Psychological Theories Are Often Uninterpretable.” Psychological Reports 66 (1): 195–244. https://doi.org/10.2466/pr0.1990.66.1.195.
Miske, Olivia, Anna Lou Abatayo, Mason Daley, et al. 2026. “Investigating the Reproducibility of the Social and Behavioural Sciences.” Nature 652 (8108): 126–34. https://doi.org/10.1038/s41586-026-10203-5.
Morin, Edgar, and Anne Brigitte Kern. 1999. Homeland earth: a manifesto for the new millenium. Advances in systems theory, complexity, and the human sciences. Hampton Press.
Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349 (6251): aac4716. https://doi.org/10.1126/science.aac4716.
Pashler, Harold, and Eric-Jan Wagenmakers. 2012. “Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?” Perspectives on Psychological Science 7 (6): 528530. https://doi.org/10.1177/1745691612465253.
Scheel, Anne M. 2022. “Why Most Psychological Research Findings Are Not Even Wrong.” Infant and Child Development 31 (1): e2295. https://doi.org/10.1002/icd.2295.
Tooze, Adam. 2023. “Welcome to the World of the Polycrisis.” Financial Times 28 (10). https://www.ft.com/content/498398e7-11b1-494b-9cd3-6d669dc3de33.
Yarkoni, Tal. 2022. “The Generalizability Crisis.” Behavioral and Brain Sciences 45. https://doi.org/10.1017/s0140525x20001685.