QML - Week 4

The research cycle and questionable research practices

Stefano Coretta

The Research Cycle

Questionable Research Practices (QRPs)

Group activity

Instructions

Discuss the following practices: are they questionable? Why?

1. A researcher tests several different statistical models on the same dataset but only reports the one that gives the strongest effect.

2. A researcher preregisters their study design, runs the study as planned, and publishes the results even though they found no effect.

3. After running an experiment, a researcher changes the stated hypothesis to match the observed results and rewrites the paper introduction.

4. A research team decides to increase their sample size mid-study because they realise the initial sample lacks the statistical power to detect medium effects. They clearly document this change in their published article.

5. An author declines to share their raw data when requested by other researchers, without offering any clear reason (e.g. confidentiality, ethical restrictions).

Share your thoughts

Group activity

Instructions

• Go to https://stefanocoretta.github.io/qrp/.

• Play the game three times to publish three papers: the goal is to publish all three!

• You need to balance Integrity and Career points and beat the reviewers lottery.

Share your final paper count

Peer-review for better or worse?

Little empirical evidence that peer review ensures quality.

Changes to manuscript contents are limited.

Statistical content increases after review.

Training reviewers does not improve manuscript.

Most suggestions are ignored when re-submitting to a different journal.

Smith (2006); White (2003); Jefferson et al. (2002); Jefferson et al. (2007); Bruce et al. (2016); Garcia-Costa et al. (2022); Stephen (2022)

Quality assurance icons created by Slamlabs - Flaticon. Register icons created by Pixel perfect - Flaticon. Statistics icons created by Freepik - Flaticon. Practice icons created by Elzicon - Flaticon. Cancel icons created by Freepik - Flaticon

Registered Reports

Registered Reports

References

Bruce, Rachel, Anthony Chauvin, Ludovic Trinquart, Philippe Ravaud, and Isabelle Boutron. 2016. “Impact of Interventions to Improve the Quality of Peer Review of Biomedical Journals: A Systematic Review and Meta-Analysis.” BMC Medicine 14 (1): 85. https://doi.org/10.1186/s12916-016-0631-5.

Garcia-Costa, Daniel, Anabel Forte, Emilia Lòpez-Iñesta, Flaminio Squazzoni, and Francisco Grimaldo. 2022. “Does Peer Review Improve the Statistical Content of Manuscripts? A Study on 27 467 Submissions to Four Journals.” Royal Society Open Science 9 (9): 210681. https://doi.org/10.1098/rsos.210681.

Jefferson, Tom, Philip Alderson, Elizabeth Wager, and Frank Davidoff. 2002. “Effects of Editorial Peer Review: A Systematic Review.” JAMA 287 (21): 2784. https://doi.org/10.1001/jama.287.21.2784.

Jefferson, Tom, Melanie Rudin, Suzanne Brodney Folse, and Frank Davidoff. 2007. “Editorial Peer Review for Improving the Quality of Reports of Biomedical Studies.” Cochrane Database of Systematic Reviews 2010 (1). https://doi.org/10.1002/14651858.MR000016.pub3.

Smith, Richard. 2006. “Peer Review: A Flawed Process at the Heart of Science and Journals.” Journal of the Royal Society of Medicine 99 (4): 178–82. https://doi.org/10.1177/014107680609900414.

Stephen, Dimity. 2022. “Peer Reviewers Equally Critique Theory, Method, and Writing, with Limited Effect on the Final Content of Accepted Manuscripts.” Scientometrics 127 (6): 3413–35. https://doi.org/10.1007/s11192-022-04357-y.

White, C. 2003. “Little Evidence for Effectiveness of Scientific Peer Review.” BMJ 326 (7383): 241a–241. https://doi.org/10.1136/bmj.326.7383.241/a.