Hypothesis Testing

Hypothesis testing is a formal mathematical framework for making inferential decisions about population parameters based on sample data. It provides a structured methodology to evaluate whether observed data yields sufficient evidence to reject a predefined baseline assumption.

The Null and Alternative Hypotheses

The foundation of any statistical test consists of two mutually exclusive statements about a population parameter: the null hypothesis ($H_0$) and the alternative hypothesis ($H_a$ or $H_1$).

The null hypothesis ($H_0$) typically represents a state of no effect, no difference, or the historical baseline. It is the hypothesis that is assumed true until statistical evidence indicates otherwise.

The alternative hypothesis ($H_a$) represents the claim or theory that the researcher asserts is true, provided the sample data provides sufficient evidence to reject $H_0$.

For a population mean $\mu$ evaluated against a hypothesized value $\mu_0$, tests are formulated in one of three ways:

1. Two-tailed test: $H_0: \mu = \mu_0 \quad \text{vs.} \quad H_a: \mu \neq \mu_0$
2. Right-tailed test (Upper-tailed): $H_0: \mu \le \mu_0 \quad \text{vs.} \quad H_a: \mu > \mu_0$
3. Left-tailed test (Lower-tailed): $H_0: \mu \ge \mu_0 \quad \text{vs.} \quad H_a: \mu < \mu_0$

The objective of the testing procedure is not to computationally “prove” $H_0$, but rather to determine if there is enough evidence to reject it in favor of $H_a$.

Decision Errors in Inference

Because hypothesis testing relies on sample data rather than an exhaustive population census, inferential decisions are subject to probabilistic errors.

Type I Error ($\alpha$)

A Type I Error occurs when the null hypothesis is rejected when it is, in fact, true in the population. This is equivalent to a false positive. The probability of committing a Type I error is denoted by $\alpha$, which is also strictly defined as the significance level of the test.

$\alpha = P(\text{Reject } H_0 \mid H_0 \text{ is true})$

Type II Error ($\beta$)

A Type II Error occurs when the null hypothesis is not rejected when the alternative hypothesis is true. This is a false negative. The probability of a Type II error is denoted by $\beta$.

$\beta = P(\text{Fail to reject } H_0 \mid H_a \text{ is true})$

Statistical Power

The power of a statistical test is the probability of correctly rejecting a false null hypothesis. It is the compliment of the Type II error rate.

$\text{Power} = 1 - \beta = P(\text{Reject } H_0 \mid H_a \text{ is true})$

Power depends on several factors: the significance level $\alpha$, the sample size $n$, the true effect size (the magnitude of the difference between the true parameter and $\mu_0$), and the population variance $\sigma^2$. Increasing sample size generally increases the power of a test.

Test Statistics and the Z-Test

A test statistic is a standardized value calculated from sample data during a hypothesis test. It measures the degree of agreement between the sample data and the null hypothesis.

Consider testing the mean of a normally distributed population with a known variance $\sigma^2$. Let $X_1, X_2, \dots, X_n$ be an independent and identically distributed (i.i.d.) random sample from $N(\mu, \sigma^2)$. The sample mean $\bar{X}$ follows a normal distribution:

$\bar{X} \sim N\left(\mu, \frac{\sigma^2}{n}\right)$

Under the null hypothesis $H_0: \mu = \mu_0$, the test statistic $Z$ is constructed by standardizing $\bar{X}$:

$Z = \frac{\bar{X} - \mu_0}{\frac{\sigma}{\sqrt{n}}}$

If $H_0$ is true, the test statistic $Z$ follows a standard normal distribution, $Z \sim N(0, 1)$. This distribution governs the probability of observing the test statistic.

The Rejection Region (Critical Value Approach)

The rejection region is the set of values for the test statistic that leads to the rejection of $H_0$. Its boundaries are determined by the critical values, which depend on the pre-specified significance level $\alpha$ and the directionality of the test.

For a two-tailed test at significance level $\alpha$, the critical values are $\pm z_{\alpha/2}$. The decision rule is: Reject $H_0$ if $|Z| > z_{\alpha/2}$.

For instance, when $\alpha = 0.05$, $z_{0.025} \approx 1.96$. Therefore, if the calculated $Z$ falls outside the interval $[-1.96, 1.96]$, $H_0$ is rejected.

The P-Value Approach

Modern statistical software generally reports the p-value, an alternative to the critical value approach that provides more granular information regarding the strength of the evidence against $H_0$.

The p-value is defined as the probability, calculated under the assumption that the null hypothesis is true, of obtaining a test statistic at least as extreme as the one actually observed.

For the standard normal test statistic $Z_{obs}$:

• Two-tailed test: $p = 2 \cdot P(Z \ge |Z_{obs}|)$
• Right-tailed test: $p = P(Z \ge Z_{obs})$
• Left-tailed test: $p = P(Z \le Z_{obs})$

Decision Rule:

• If $p \leq \alpha$, reject $H_0$.
• If $p > \alpha$, fail to reject $H_0$.

A smaller p-value constitutes stronger evidence against the null hypothesis. It is crucial to note that the p-value is not the probability that the null hypothesis is true ($P(H_0 \mid \text{data})$). It is the probability of the data given the null hypothesis ($P(\text{data} \mid H_0)$).

The Student’s t-Test

In practical applications, the population variance $\sigma^2$ is almost always unknown. Replacing the population standard deviation $\sigma$ with the sample standard deviation $s$ changes the distribution of the test statistic.

When $X_1, \dots, X_n \sim N(\mu, \sigma^2)$ but $\sigma$ is unknown, the test statistic follows a Student’s t-distribution with $n - 1$ degrees of freedom ($df$):

$t = \frac{\bar{X} - \mu_0}{\frac{s}{\sqrt{n}}} \sim t_{n-1}$

The t-distribution is symmetric and bell-shaped like the standard normal distribution but possesses heavier tails. These heavier tails artificially introduce more probability in the extremes, accounting for the additional uncertainty incurred by estimating continuous variance from a finite sample. As $n \to \infty$, the t-distribution converges to the standard normal distribution $N(0,1)$.

Multiple Hypothesis Testing

When conducting multiple hypothesis tests simultaneously on a single dataset, the probability of committing at least one Type I error compounds. If a researcher conducts $m$ independent tests each at significance level $\alpha$, the family-wise error rate (FWER)—the probability of making one or more false discoveries—is given by:

$\text{FWER} = 1 - (1 - \alpha)^m$

For example, performing 20 tests at $\alpha = 0.05$ yields an FWER of $\approx 0.64$. Without correction, false positives are extremely likely.

The Bonferroni Correction

The most conservative method to control the FWER is the Bonferroni correction. To maintain a given family-wise $\alpha_{FWER}$, each individual test is evaluated at a newly adjusted significance level:

$\alpha_{individual} = \frac{\alpha_{FWER}}{m}$

If 20 tests are conducted and the desired global false positive rate is 5%, each individual p-value must be compared against $\alpha_{individual} = 0.05 / 20 = 0.0025$.

While mathematically rigorous and guaranteed to bound the FWER under all forms of dependence among tests, the Bonferroni strictly reduces statistical power, exponentially increasing Type II error rates when the number of tests ($m$) is massive, as is common in genomics and machine learning algorithms.