QML - Week 7

Frequentist statistics and the Null Ritual

Stefano Coretta

Statistical significance

Suppose you have a treatment that you suspect may alter performance on a certain task. You compare the means of your control and experimental groups (say 20 subjects in each sample). Further, suppose a significance test indicates a significant difference, with a p-value of 0.01. Which of the following statements are FALSE?

  1. You have absolutely disproved the null hypothesis (that is, there is no difference between the population means).

  2. You have found the probability of the null hypothesis being true.

  3. You have absolutely proved your experimental hypothesis (that there is a difference between the population means).

  4. You can deduce the probability of the experimental hypothesis being true.

  5. You know, if you decide to reject the null hypothesis, the probability that you are making the wrong decision.

  6. You have a reliable experimental finding in the sense that if, hypothetically, the experiment were repeated a great number of times, you would obtain a significant result on 99% of occasions.

Which statements are false?

Statistical significance is not meaningful

  • Simulate N observations from two populations that have very similar means. Their difference is not linguistically meaningful.

  • N is sample size: from 50 to 500.

  • For each N, simulate N observations 100 times (100 draws). For each draw get the p-value of the difference. So 100 p-values for each N.

Statistical significance is not meaningful

Group activity

Focus on the results sections of these papers and discuss how they compare with one another.

How are the statistical approaches different? Which papers focus more on general directional patterns vs more precise estimation? How are the results weighted and use as evidence?