Chapter 6

Regression Analysis

Linear regression is an approach for modeling the linear relationship between two variables.

Ordinary Least Squares

The ordinary least squares (OLS) approach to regression allows us to estimate the parameters of a linear model. The goal of this method is to determine the linear model that minimizes the sum of the squared errors between the observations in a dataset and those predicted by the model. Explore the OLS method through the four infamous datasets contained in Anscombe's Quartet.

Choose one of the quartets to investigate.

Drag and drop data points to explore how this affects the OLS line.

Click on a column of the regression table to learn more about this parameter.

	$\displaystyle{n}$	$\displaystyle{\bar{\cssId{xMEAN}{x}}}$	$\displaystyle{\bar{\cssId{yMEAN}{y}}}$	$\displaystyle{\hat{\cssId{BETA0}{B_{0}}}}$	$\displaystyle{\hat{\cssId{BETA1}{B_{1}}}}$	$\displaystyle{SSE}$
Model

$n$ is the sample size. This is simply the number of data points in the given dataset.

$\bar{x}$ is the mean value of the x data. It is defined mathematically as the following:

$$\bar{x} = \sum_{i=1}^{n}\dfrac{x_{i}}{n}$$

$\bar{y}$ is the mean value of the y data. It is defined mathematically as the following:

$$\bar{y} = \sum_{i=1}^{n}\dfrac{y_{i}}{n}$$

$\hat{B_{0}}$ is the y intercept of the regression line. The current estimate has a variance of __. It is defined mathematically as the following:

$$\hat{B_{0}} = \bar{y} - \hat{B_{1}}\bar{x}$$

$\hat{B_{1}}$ is the slope of the regression line. The current estimate has a variance of __. It is defined mathematically as the following:

$$\hat{B_{1}} = \dfrac{S_{xy}}{S_{xx}}$$

$SSE$ is an abbreviation for the sum of squared errors which is the sum of the square residuals. It is defined mathematically as the following:

$$SSE = \sum_{i=1}^{n}\Big{(}y_{i} - \big{(}\hat{B_{0}} + \hat{B_{1}}x_{i}\big{)}\Big{)}^{2}$$

Correlation

Correlation is a measure of the linear relationship between two variables. It is defined for a sample as the following and takes value between +1 and −1 inclusive:

$$r = \dfrac{s_{xy}}{\sqrt{s_{xx}}\sqrt{s_{yy}}}$$

$s_{xy},s_{xx},s_{yy}$ are defined as:

$$\begin{align*} s_{xy} &=\sum^n_{i=1} (x_i-\bar{x})(y_i-\bar{y})\\ s_{xx} &=\sum^n_{i=1} (x_i-\bar{x})^2\\ s_{yy} &=\sum^n_{i=1} (y_i-\bar{y})^2 \end{align*}$$

It can also be understood as the cosine of the angle formed by the ordinary least square line determined in both variable dimensions. Explore this concept through Edgar Anderson's famous Iris flower dataset.

Check which species to investigate.

Click on a cell of the correlation matrix to visualize the relationship between these traits.

	Sepal Length	Sepal Width	Petal Length	Petal Width
Sepal Length
Sepal Width
Petal Length
Petal Width

Analysis of Variance

Analysis of Variance (ANOVA) is a statistical method for testing whether groups of data have the same mean. ANOVA generalizes the t-test to two or more groups by comparing the sum of square error within and between groups.

Choose one of the following datasets to investigate.

Drag and drop data points to explore how this affects the result of the ANOVA test.

Click on a column of the ANOVA table to learn more about this parameter.

	$\displaystyle{SSE}$	$\displaystyle{df}$	$\displaystyle{MS}$	$\displaystyle{F}$	$\displaystyle{p}$
Treatment
Error
Total

$SSE$ is an abbreviation for the sum of squared errors which is the sum of the square residuals. SSE is defined mathematically as the following:

$$SSE = \sum_{i=1}^{n} \big{(}y_{i} - \bar{y}\big{)}^{2}$$

$df$ is an abbreviation for degrees of freedom which is related to the number of data points. It is defined mathematically as the following:

$$df = n - 1$$

$MSE$ is an abbreviation for mean square error which is the quotient of the sum of square errors and the degrees of freedom. It is defined mathematically as the following:

$$MSE = \dfrac{SSE}{df}$$

$F$ is a test statistic which is defined as the following:

$$F = \dfrac{SST/(k-1)}{SSE/(n-k)} \sim f_{k-1,n-k}$$

$p$ is the p value which is determined from the F statistic.

	\(\displaystyle{n}\)	\(\displaystyle{\bar{\cssId{xMEAN}{x}}}\)	\(\displaystyle{\bar{\cssId{yMEAN}{y}}}\)	\(\displaystyle{\hat{\cssId{BETA0}{B_{0}}}}\)	\(\displaystyle{\hat{\cssId{BETA1}{B_{1}}}}\)	\(\displaystyle{SSE}\)
Model

	\(\displaystyle{SSE}\)	\(\displaystyle{df}\)	\(\displaystyle{MS}\)	\(\displaystyle{F}\)	\(\displaystyle{p}\)
Treatment
Error
Total