1. Introduction to ANOVAs

What are ANOVAs?

Analysis of Variances (ANOVAs) is an inferential statistical technique used to compare the difference in the means of three or more groups. Two types:

Can ANOVAs be used to compare the means of two groups?

Yes - ANOVA can also be used to compare 2 groups but a independent t-test is usually used instead.

Why not perform multiple t-test?

i.e., to compare the means of 3 groups: $μ_{1} = μ_{2}, μ_{1} = μ_{3}, μ_{2} = μ_{3}$ ?

need to conduct independent T tests for each possible pairs
however, this is not sufficient to compare the difference among all groups because the test is only conducted in pairs, and therefore, will only provide results for the 2 groups in each test
cannot check the difference AMONG ALL GROUPS

Running multiple t tests like this leads to an unacceptable inflation of Type I Error of rejecting the null hypothesis when it is True.

In other words, this leads to you INCORRECTLY REJECTING THE NULL when it is TRUE.

With ANOVA, the Type I Error remains at 0.05 - recall, the type 1 error is directly related to the alpha significance level of the maximum ALLOWABLE probability of rejecting the null when it is True.

Problems with ANOVA

ANOVA only tells us whether a statistically significant difference exists. Does NOT tell us where the difference lie (i.e., which groups differ). Therefore, should a significant result be produced, we need to follow up with a Post-Hoc Test e.g., Tukey HSD, to determine which group differ.

Assumptions and Conditions

To use ANOVA, the following assumptions and conditions about the population from which the sample was taken from must be satisfied:

Assumptions Check

The dependent variable must be measured at a continuous level (e.g., years of education, score, salary)
The independent variable must consists of 3 or more categorical, INDEPENDENT groups (note: when there are TWO independent variables -> two-way ANOVA)
Independence of observations i.e., observations are mutually independent of each other, such that there is no relationship between the participants in any of the groups (e.g., the selection of participants in the control group has no effect on individual in the treatment group; each group is INDEPENDENT OF EACH OTHER)^[1]

Remaining 3 assumptions related to the shape of the population distribution and how the data fits ANOVA

No significant outliers in the groups of your IDV in terms of the DV
Values of the DV should approximate a NORMAL DISTRIBUTION (if violated, consider: Transformation for Normality)
Homogeneity of the variances (Heterodescasity) i.e., the variance of the DV (spread) in EACH CATEGORICAL GROUP of the IDV is the SAME. Specifically, with respect to the residuals (actual - predicted), for each group^[2], but we can run it before. Alternative: use Welch's ANOVA for One-Way ANOVA
Residuals are IID (independent and identically distributed) i.e., each random variable has the same probability distribution as the others but are all mutually independent of each other.

How to check for i.i.d.

We can check for the i.i.d. assumption by creating a scatterplot of the actual vs predicted values. This should be random with NO discerning pattern.

The Multiple Hypothesis Problem

The multiple hypothesis problem in ANOVA (Analysis of Variance) refers to the issue of conducting multiple comparisons between the means of two or more groups.

This is a known feature with ANOVAs, such that as the number of k elements in a categorical predictor increases, the probability of making a Type 1 Error increases.
This is an additive/cumulative effect^[3], such that the the probability of making a Type I error increases with each additional hypothesis tested.

Why is this the case?

WIth each hypothesis test, there is an inherent risk of rejecting the null when it is true (Type I Error).
The probability of making a Type I error for a single test is equal to the significance level alpha (e.g., 0.05), but the more tests you perform, the greater the chance of making at least one Type I error.

Example:

If you perform 10 independent tests at the 0.05 significance level, the overall probability of making at least one Type I error is:

Type 1 Error = 1 - (1 - 0.05)^{10} = 0.401

This means that there is a 40.1% chance of falsely rejecting at least one null hypothesis, even if all the null hypotheses are true.

Solution

Adjust the alpha significance level
Correction methods to account for multiple comparisons

Examples: Bonferroni Correction

This can otherwise lead to confounding variables ↩︎
The Shapiro-Wilk test or other normality tests can be used to assess whether the residuals (i.e., the differences between the observed values and the model's predicted values) are normally distributed. ↩︎
The multiple hypothesis problem in ANOVA is often referred to as a "cumulative error" or "cumulative effect" problem because the probability of making a Type I error increases with each additional hypothesis tested. ↩︎