Analysis of Variance (ANOVA)

Analysis of Variance (ANOVA) is a collection of statistical models and their associated estimation procedures used to analyze the differences among group means in a sample. ANOVA was developed by statistician and evolutionary biologist Ronald Fisher. In its simplest form, ANOVA provides a statistical test of whether two or more population means are equal, and therefore generalizes the $t$-test beyond two means.

While the $t$-test is limited to comparing two groups, applying multiple $t$-tests across several groups exponentially increases the Type I error rate (false positives). ANOVA controls this error rate by evaluating the entire set of groups simultaneously, partitioning the observed variance in a particular variable into components attributable to different sources of variation.

The Logic of Variance Partitioning

The fundamental mechanism of ANOVA is the partitioning of total variance into two primary components:

1. Between-Group Variance: The variance of the group means around the grand mean. This reflects the effect of the independent variable(s) plus error.
2. Within-Group Variance: The variance of individual scores around their respective group means. This reflects pure error (unexplained variance).

If the between-group variance is significantly larger than the within-group variance, it indicates that the independent variable has a significant effect on the dependent variable.

Assumptions of ANOVA

The validity of ANOVA relies on three core assumptions:

1. Independence of Observations: The residuals must be mutually independent. This is fundamentally a design issue handled through random sampling and random assignment.
2. Normality: The residuals of the model are normally distributed. While ANOVA is robust to moderate violations of normality (especially with large, equal sample sizes due to the Central Limit Theorem), severe skewness or outliers can compromise the $F$-test.
3. Homogeneity of Variances (Homoscedasticity): The variances of the populations from which the samples are drawn are equal. This is tested using Levene’s Test or Bartlett’s Test. Welch’s ANOVA can be used if this assumption is heavily violated.

One-Way ANOVA

A One-Way ANOVA involves a single independent variable (factor) with three or more categorical levels. The model for an observation $y_{ij}$ (the $i$-th observation in the $j$-th group) is given by:

$y_{ij} = \mu + \tau_j + \varepsilon_{ij}$

Where:

• $\mu$ is the grand mean.
• $\tau_j$ is the treatment effect for the $j$-th group (where $\sum \tau_j = 0$).
• $\varepsilon_{ij}$ is the random error associated with the $i$-th observation in the $j$-th group, assumed to be $\mathcal{N}(0, \sigma^2)$.

Hypotheses

The null hypothesis ($H_0$) states that all group population means are equal (or equivalently, all treatment effects are zero): $H_0: \mu_1 = \mu_2 = \dots = \mu_k \quad \text{or} \quad \tau_1 = \tau_2 = \dots = \tau_k = 0$

The alternative hypothesis ($H_a$) states that at least one population mean is different: $H_a: \exists \ i, j \text{ such that } \mu_i \neq \mu_j$

Sums of Squares

The Total Sum of Squares ($SST$) is partitioned into the Sum of Squares Between ($SSB$) and the Sum of Squares Within ($SSW$, also known as Error Sum of Squares, $SSE$).

$SST = SSB + SSW$

Total Sum of Squares (SST) measures the total variation in the data: $SST = \sum_{j=1}^{k} \sum_{i=1}^{n_j} (y_{ij} - \bar{y}_{..})^2$ where $\bar{y}_{..}$ is the grand mean.

Sum of Squares Between (SSB) measures the variation of group means around the grand mean: $SSB = \sum_{j=1}^{k} n_j (\bar{y}_{.j} - \bar{y}_{..})^2$ where $\bar{y}_{.j}$ is the mean of the $j$-th group and $n_j$ is the number of observations in the $j$-th group.

Sum of Squares Within (SSW) measures the variation of individual observations around their respective group means: $SSW = \sum_{j=1}^{k} \sum_{i=1}^{n_j} (y_{ij} - \bar{y}_{.j})^2$

Degrees of Freedom and Mean Squares

Degrees of freedom ($df$) are required to convert sums of squares into variances (mean squares). Let $N$ be the total sample size and $k$ be the number of groups.

• $df_{Total} = N - 1$
• $df_{Between} = k - 1$
• $df_{Within} = N - k$

The Mean Squares ($MS$) are calculated by dividing the Sum of Squares by their respective degrees of freedom:

$MSB = \frac{SSB}{k - 1}$ $MSW = \frac{SSW}{N - k}$

The F-Statistic

The test statistic for ANOVA is the ratio of the Mean Square Between to the Mean Square Within. Under the null hypothesis, both $MSB$ and $MSW$ are independent estimates of the population variance $\sigma^2$, so their ratio follows an $F$-distribution with $k-1$ and $N-k$ degrees of freedom.

$F = \frac{MSB}{MSW}$

If the $F$-statistic is significantly larger than 1 (specifically, greater than the critical value from the $F$-distribution for a given alpha level), the null hypothesis is rejected.

Two-Way ANOVA

A Two-Way ANOVA analyzes the effect of two independent categorical variables (factors) on a continuous dependent variable. It fundamentally differs from running two independent One-Way ANOVAs because it evaluates the interaction effect between the two variables.

The statistical model for a Two-Way ANOVA with factors $A$ and $B$, fixed effects, and with replication ($n$ observations per cell) is:

$y_{ijk} = \mu + \alpha_i + \beta_j + (\alpha\beta)_{ij} + \varepsilon_{ijk}$

Where:

• $y_{ijk}$ is the $k$-th observation in the $i$-th level of factor $A$ and $j$-th level of factor $B$.
• $\mu$ is the overall population grand mean.
• $\alpha_i$ is the main effect of factor A at level $i$.
• $\beta_j$ is the main effect of factor B at level $j$.
• $(\alpha\beta)_{ij}$ is the interaction effect between level $i$ of A and level $j$ of B.
• $\varepsilon_{ijk}$ is the random error term, $\sim \mathcal{N}(0, \sigma^2)$.

Interaction Effects

An interaction effect occurs when the effect of one independent variable on the dependent variable changes depending on the level of the other independent variable. Graphically, this is observed when the lines representing the means across levels of factors are not parallel (they may cross or diverge).

If the interaction effect is significant, interpreting the main effects (the individual effects of factor $A$ and factor $B$) becomes highly nuanced, as the main effects no longer fully describe the relationship.

Sums of Squares for Two-Way ANOVA

In a balanced design (equal sample sizes in all cells), the total variance is partitioned into four orthogonal components:

$SST = SSA + SSB + SSAB + SSE$

Where:

• SSA: Sum of Squares for Factor A
• SSB: Sum of Squares for Factor B
• SSAB: Sum of Squares for the Interaction
• SSE: Sum of Squares for Error (Within)

Degrees of freedom are similarly partitioned: Let $a$ be the number of levels of Factor A, $b$ be the number of levels of Factor B, and $n$ the number of replicates per cell. Total observations $N = a \times b \times n$.

• $df_A = a - 1$
• $df_B = b - 1$
• $df_{AB} = (a - 1)(b - 1)$
• $df_E = ab(n - 1)$

Three distinct $F$-tests are performed by dividing the corresponding Mean Square ($MSA, MSB, MSAB$) by the Mean Square Error ($MSE$):

$F_A = \frac{MSA}{MSE}, \quad F_B = \frac{MSB}{MSE}, \quad F_{AB} = \frac{MSAB}{MSE}$

Post-Hoc Tests

A significant ANOVA only tells you that at least two means differ, not which means differ. To identify specific pairwise differences, post-hoc tests are required. Conducting multiple standard $t$-tests inflates the family-wise error rate $\alpha_{FW}$ (the probability of making at least one Type I error across all tests).

$\alpha_{FW} = 1 - (1 - \alpha)^c$ where $c$ is the number of comparisons. For 5 groups, there are $c = \frac{5(4)}{2} = 10$ comparisons. If $\alpha=0.05$ per test, the family-wise error rate jumps to $1 - (0.95)^{10} \approx 0.40$ (assuming independence, which is an oversimplification but illustrates the inflation).

Common Post-Hoc Adjustments

1. Tukey’s Honestly Significant Difference (HSD): Compares all possible pairs of means. It is based on the studentized range distribution ($q$) and provides tight control over the family-wise error rate when sample sizes are equal.
2. Bonferroni Correction: The most conservative method. It simply divides the desired family-wise alpha level by the number of comparisons: $\alpha_{corrected} = \frac{\alpha}{c}$. While it strictly prevents Type I errors, it severely impacts statistical power (increasing Type II errors).
3. Scheffé’s Method: Used for all possible linear contrasts, not just pairwise comparisons. It is the most conservative post-hoc test when performing purely pairwise comparisons, but is highly flexible.

Effect Size

The $p$-value from an $F$-test indicates statistical significance but not practical significance. Effect size metrics quantify the magnitude of the differences between groups.

Eta-Squared ($\eta^2$)

Eta-squared represents the proportion of total variance in the dependent variable that is associated with membership in the different groups defined by the independent variable.

$\eta^2 = \frac{SSB}{SST}$

While intuitive, $\eta^2$ is an upwardly biased estimator of the population effect size (it tends to overestimate).

Partial Eta-Squared ($\eta_p^2$)

In multi-factor designs (like Two-Way ANOVA), $\eta^2$ can be misleading because the effects of one factor reduce the variance available to be explained by another. Partial eta-squared isolates the variance explained by a specific factor relative to the unexplained variance (error) and the variance of that specific factor.

$\eta_p^2 = \frac{SS_{effect}}{SS_{effect} + SSE}$

Omega-Squared ($\omega^2$)

Omega-squared is a more complex but unbiased estimator of the population variance explained. It corrects for the bias present in $\eta^2$ by incorporating degrees of freedom and Mean Square terms.

$\omega^2 = \frac{SSB - df_{Between} \times MSW}{SST + MSW}$

Repeated Measures ANOVA

Repeated Measures ANOVA is the equivalent of the one-way ANOVA, but for related, not independent groups. It is the extension of the dependent (paired) $t$-test. Examples include measuring the same participants across multiple time points (e.g., Blood pressure at baseline, week 1, and week 2) or exposing the same participants to all conditions in an experiment.

The key advantage of Repeated Measures ANOVA is that it removes variance attributable to individual differences from the Error Sum of Squares. This typically makes the analysis much more powerful (higher probability of detecting a true effect) than a standard independent-samples ANOVA.

$SST = SS_{Between Subjects} + SS_{Within Subjects}$ $SS_{Within Subjects} = SS_{Treatment} + SS_{Error}$

The Assumption of Sphericity

Repeated measures designs require the assumption of Sphericity. Sphericity requires that the variances of the differences between all pairs of related groups are equal. It is evaluated using Mauchly’s Test of Sphericity.

If the assumption of sphericity is violated (Mauchly’s Test $p < 0.05$), the Type I error rate inflates. To correct this, the degrees of freedom are adjusted downwards. Common corrections include:

1. Greenhouse-Geisser Correction: The most conservative correction. Used when sphericity is severely violated (epsilon $\epsilon < 0.75$).
2. Huynh-Feldt Correction: Less conservative, used when sphericity violation is mild (epsilon $\epsilon > 0.75$).

If $\epsilon$ is close to 1, the sphericity assumption holds perfectly. The corrections effectively increase the critical $F$-value required for significance by artificially reducing the degrees of freedom.