Data Analysis for LEL - Week 2

Research process

Stefano Coretta

University of Edinburgh

Rate the following from 1 (strongly disagree) to 5 (strongly agree) about your experience with Week 1 of the QML course

List things from Week 1 that you feel still unsure about

Research process

Which of the following are research hypotheses?

Research rationale

Research questions and hypotheses

Research question

• Research questions are testable questions whose answers directly address the research problem.

Research hypotheses

• Research hypotheses are testable statements (not questions) about the research problem.

• The hypotheses must be falsifiable (there can be in principle an outcome that shows them to be false).

• Hypotheses can never be true nor confirmed. We can only corroborate hypothesis, and it’s a long term process.

Research cycle

Which part of a research study do you believe should be beyond the control of the researcher?

Which part of a research study do you believe is most important for advancing a researcher’s career?

Research cycle: the dangers

Replication in linguistics

[O]nly 1 in 1,250 experimental linguistic articles contains an independent direct replication.

The observed smaller number of citations of replication studies compared to corresponding initial studies is also in line with the lack of perceived value of replication studies reported in other fields.

—Korback and Roettger 2023

Statistical power in linguistics

An opportunistic review of seven papers from the November 2017 issue of this journal containing statistical analyses of acoustic speech production data found the number of participants to range from 11 to 39.

—Kirby and Sonderegger 2018

In group differences research, the median sample sizes (n = 18 and n = 14 for case and control groups, respectively) were insufficient for detecting large, medium, or small effect sizes.

—Gaeta and Brydges 2020

[O]f the 1004 studies reviewed by Lehtonen et al. (2018) 878 had sample sizes smaller than 50 participants per group (i.e., 87%) and 987 had sample sizes smaller than 100 (98%).

—Brysbaert 2020

Case study: participant number in prosody research

The dataset contains data from 113 studies, published between 1955 and 2017 (the bulk of studies is within the range 1990-2017 though). The median number of speakers per study is 5. (An estimate of number of speakers per study in phonetics, Coretta 2019).

Case study: participant number in prosody research

Reproducibility in linguistics

35% of the articles were published open access and the rates of sharing materials, data, and protocols were below 10%. None of the articles reported preregistrations, 1% reported replications, and 10% had conflict of interest statements. These rates have not increased noticeably between 2008/2009 and 2018/2019.

—Bochynska et al. 2023

Activity

• Search for a paper you are interested in or that you have recently read.

• Fill in the following spreadsheet with info on the paper you picked.

  • Spreadsheet.

Open Scholarship practices

• Pre-registration and Registered Reports.

• Share the Research Compendium (data, materials, code, etc).

• Ensure your study is reproducible.

• Think about sample size.

Be open, go Open!