Regression analysis is a statistical method for estimating the relationships among variables. It focuses primarily on the relationship between a dependent variable (often called the response or outcome variable) and one or more independent variables (often called predictors, covariates, or explanatory variables). The objective is to understand how the typical value of the dependent variable changes when any one of the independent variables is varied, while the other independent variables are held fixed.

Simple Linear Regression

The most fundamental form of regression analysis is simple linear regression, which models the relationship between a single independent variable $X$ and a dependent variable $Y$. The true relationship is postulated to be a linear function of $X$ plus a stochastic error term.

The population model is defined as: $Y_i = \beta_0 + \beta_1 X_i + \epsilon_i$

where:

• $Y_i$ is the $i$-th observation of the dependent variable.
• $X_i$ is the $i$-th observation of the independent variable.
• $\beta_0$ is the $y$-intercept (the expected value of $Y$ when $X = 0$).
• $\beta_1$ is the slope coefficient (the expected change in $Y$ for a one-unit change in $X$).
• $\epsilon_i$ is the unobserved random error or disturbance term.

Assumptions of Simple Linear Regression

For the standard estimation techniques to be valid and possess desirable statistical properties, certain assumptions regarding the error term $\epsilon_i$ must hold:

1. Linearity: The expected value of the response variable is a linear function of the explanatory variables. $\mathbb{E}[Y | X] = \beta_0 + \beta_1 X$, meaning $\mathbb{E}[\epsilon | X] = 0$.
2. Independence: The errors are independent of each other. $\text{Cov}(\epsilon_i, \epsilon_j) = 0$ for all $i \neq j$.
3. Homoscedasticity (Constant Variance): The errors have a constant variance across all levels of the independent variable. $\text{Var}(\epsilon_i | X_i) = \sigma^2$ for all $i$.
4. Normality (Optional for estimation, required for inference): The errors are normally distributed. $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$.

Ordinary Least Squares (OLS) Estimation

The most common method for estimating the unknown parameters $\beta_0$ and $\beta_1$ is Ordinary Least Squares (OLS). The OLS method chooses the estimates $\hat\beta_0$ and $\hat\beta_1$ that minimize the sum of the squared residuals (SSR).

The residual $e_i$ for the $i$-th observation is the difference between the observed $Y_i$ and the predicted value $\hat{Y}_i$: $e_i = Y_i - \hat{Y}_i = Y_i - (\hat\beta_0 + \hat\beta_1 X_i)$

The Sum of Squared Residuals (SSR) is: $S(\hat\beta_0, \hat\beta_1) = \sum_{i=1}^n e_i^2 = \sum_{i=1}^n (Y_i - \hat\beta_0 - \hat\beta_1 X_i)^2$

To minimize $S$, we take the partial derivatives with respect to $\hat\beta_0$ and $\hat\beta_1$ and set them to zero: $\frac{\partial S}{\partial \hat\beta_0} = -2 \sum_{i=1}^n (Y_i - \hat\beta_0 - \hat\beta_1 X_i) = 0$ $\frac{\partial S}{\partial \hat\beta_1} = -2 \sum_{i=1}^n X_i (Y_i - \hat\beta_0 - \hat\beta_1 X_i) = 0$

Solving these normal equations yields the OLS estimators: $\hat\beta_1 = \frac{\sum_{i=1}^n (X_i - \bar{X})(Y_i - \bar{Y})}{\sum_{i=1}^n (X_i - \bar{X})^2} = \frac{\text{Cov}(X, Y)}{\text{Var}(X)}$ $\hat\beta_0 = \bar{Y} - \hat\beta_1 \bar{X}$

Where $\bar{X}$ and $\bar{Y}$ are the sample means of $X$ and $Y$, respectively.

Multiple Linear Regression

Multiple linear regression extends the simple linear model to include two or more independent variables. The model with $k$ predictors is written as: $Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \dots + \beta_k X_{ik} + \epsilon_i$

Because writing out summations becomes unwieldy, multiple regression is almost universally represented using matrix algebra.

Let $Y$ be an $n \times 1$ vector of observations of the dependent variable, $X$ be an $n \times (k+1)$ matrix (the design matrix) where the first column is typically all 1s (for the intercept), $\beta$ be a $(k+1) \times 1$ vector of parameters, and $\epsilon$ be an $n \times 1$ vector of errors.

$Y = X\beta + \epsilon$

The OLS estimator vector $\hat\beta$ minimizes $(Y - X\hat\beta)^T (Y - X\hat\beta)$. Expanding this and taking the derivative with respect to the vector $\hat\beta$ yields the matrix formulation of the normal equations: $X^T X \hat\beta = X^T Y$

Assuming $X^T X$ is invertible (which requires no perfect multicollinearity among the predictors), the OLS estimator is: $\hat\beta = (X^T X)^{-1} X^T Y$

The Gauss-Markov Theorem

The Gauss-Markov theorem justifies the use of the OLS estimator. It states that under the classical linear regression model assumptions (linearity, strict exogeneity/independence, no perfect multicollinearity, and homoscedasticity), the OLS estimator $\hat\beta$ is the Best Linear Unbiased Estimator (BLUE).

1. Linear: $\hat\beta$ is a linear function of the observed random variables $Y$. We can write $\hat\beta = AY$ where $A = (X^T X)^{-1} X^T$.
2. Unbiased: The expected value of the estimator is the true parameter. $\mathbb{E}[\hat\beta] = \beta$. $\mathbb{E}[\hat\beta] = \mathbb{E}[(X^T X)^{-1} X^T (X\beta + \epsilon)] = \beta + (X^T X)^{-1} X^T \mathbb{E}[\epsilon] = \beta + 0 = \beta$
3. Best: It has the minimum variance among all linear unbiased estimators. $\text{Var}(\hat\beta_{OLS}) \leq \text{Var}(\tilde{\beta})$ for any other linear unbiased estimator $\tilde{\beta}$.

The variance-covariance matrix of the OLS estimator is: $\text{Var}(\hat\beta) = \sigma^2 (X^T X)^{-1}$ Where $\sigma^2$ is the variance of the error term, typically estimated by $s^2 = \frac{e^T e}{n - k - 1}$, with $e$ being the vector of residuals.

Goodness of Fit and Inference

To assess how well the model fits the data, we decompose the total variation in the dependent variable into explained and unexplained components.

• Total Sum of Squares (SST): Measures the total variation in $Y$ around its mean. $\text{SST} = \sum_{i=1}^n (Y_i - \bar{Y})^2$
• Model/Explained Sum of Squares (SSM): Measures the variation in $Y$ explained by the regression model. $\text{SSM} = \sum_{i=1}^n (\hat{Y}_i - \bar{Y})^2$
• Residual/Error Sum of Squares (SSR): Measures the variation in $Y$ not explained by the model. $\text{SSR} = \sum_{i=1}^n (Y_i - \hat{Y}_i)^2 = \sum_{i=1}^n e_i^2$

The relationship is $\text{SST} = \text{SSM} + \text{SSR}$.

Coefficient of Determination ($R^2$)

The $R^2$ statistic represents the proportion of variance in the dependent variable explained by the independent variables in the model. $R^2 = \frac{\text{SSM}}{\text{SST}} = 1 - \frac{\text{SSR}}{\text{SST}}$

While $0 \leq R^2 \leq 1$, adding more predictors to a model will mechanically never decrease $R^2$, even if the predictors are irrelevant. To account for this, the Adjusted $R^2$ penalizes models for adding variables that do not significantly improve the fit: $\bar{R}^2 = 1 - \left( \frac{\text{SSR} / (n - k - 1)}{\text{SST} / (n - 1)} \right) = 1 - (1 - R^2)\frac{n-1}{n-k-1}$

Hypothesis Testing

Under the assumption that $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$, the OLS estimators are normally distributed: $\hat\beta \sim \mathcal{N}(\beta, \sigma^2 (X^T X)^{-1})$

Test of Individual Significance (t-test)

To test the hypothesis that a single independent variable $X_j$ has no effect on $Y$ (i.e., $H_0: \beta_j = 0$), a t-statistic is used: $t = \frac{\hat\beta_j - 0}{\text{SE}(\hat\beta_j)}$ where $\text{SE}(\hat\beta_j)$ is the standard error of the estimate, found directly from the square root of the $j$-th diagonal element of the estimated variance-covariance matrix $s^2(X^T X)^{-1}$. Under the null hypothesis, this statistic follows a Student’s t-distribution with $n - k - 1$ degrees of freedom.

Test of Overall Significance (F-test)

To test the joint hypothesis that all slope coefficients (excluding the intercept) are simultaneously zero ($H_0: \beta_1 = \beta_2 = \dots = \beta_k = 0$), an F-statistic is constructed from the sums of squares: $F = \frac{\text{SSM} / k}{\text{SSR} / (n - k - 1)}$ Under the null hypothesis, this follows an F-distribution with $(k, n - k - 1)$ degrees of freedom. A large F-statistic provides evidence against the null hypothesis, indicating that at least one predictor variable is significantly related to the response variable.

Residual Diagnostics

Estimation is only part of the process; structural validation ensures the model assumptions hold. Analyzing the residuals ($e_i$) is the primary tool for diagnostics.

• Non-linearity: Plotting residuals against predicted values ($\hat{Y}_i$) or individual predictors ($X_i$). A non-random U-shape or pattern suggests the relationship is non-linear, perhaps requiring polynomial terms or transformations.
• Heteroscedasticity: If the spread of the residuals increases or decreases with $\hat{Y}_i$ (often forming a “funnel” shape in a residual plot), the constant variance assumption is violated. This makes OLS standard errors incorrect, invalidating hypothesis tests. Robust standard errors or Weighted Least Squares (WLS) can address this.
• Non-normality: A Normal Q-Q (quantile-quantile) plot compares the distribution of the residuals to a theoretical normal distribution. Significant deviations from the straight line, particularly at the tails, imply non-normal errors.
• Outliers and Leverage: Observations with extreme $Y$ values given their $X$ values are outliers. Observations with extreme $X$ values have high leverage. Points with both high leverage and large residuals exert undue influence on the regression line. Cook’s Distance is a metric used to quantify the overall influence of an observation on the estimated coefficients.

$D_i = \frac{\sum_{j=1}^n (\hat{Y}_j - \hat{Y}_{j(i)})^2}{(k+1)s^2}$ where $\hat{Y}_{j(i)}$ is the predicted value of the $j$-th observation when the model is refitted without the $i$-th observation. A high Cook’s distance indicates a highly influential data point.

Regression analysis serves as the foundational mathematical bedrock for predictive modeling and causal inference, bridging classical statistics to modern machine learning applications.