Reporting Effect Sizes

What is effect size and why is it important?

P-values are a measure of statistical significance to quantify evidence against the null hypothesis.

In other words, null-hypothesis significance testing using p-values only tells us that there is an effect to be observed (i.e., the observed phenomenon is not due to chance).

The problem with p-values is that:

Does not tell you the magnitude of the effect (i.e., whether the effect large enough to have a meaningful impact in the real world) - this is AKA practical significance.
Statistical significance can be misleading as increasing the sample sizing always makes it more likely to be statistically significant.

Therefore, we want to measure effect size, why?

Effect sizes are independent of sample size
Should be reported with the p-value to indicate the practical significance of the finding.

Distinguishing p-value and effect size

Consider a drug test:

p-values tells you that a treatment works
effect size tells you how much the treatment works

There are different effect sizes measures that can be used to quantify the magnitude of the effect for each different statistical tests:

Effect Size for T-tests: Cohen's d, Pearson's r