Linear Regression

Fitting a straight line to data to model and predict the relationship between two variables.

Linear regression chooses the line with the smallest squared residuals

Current line

y = 0.80x + 1.43

SSR = 0.86

minimum SSR = 0.86

slope 0.80

flatbest 0.80steep

Each red segment is a residual. Least squares chooses the line that makes the squared residual total as small as possible.

Definition

Linear regression fits a straight line through a scatter of data points to model the relationship between two variables.

Given data on $x$ and $y$ , we find the line $\hat{y} = a + bx$ that best summarises the pattern — "best" meaning the line minimises the total squared distance between the actual $y$ values and the line's predictions.

$b$ is the slope: how much $y$ changes for each unit increase in $x$
$a$ is the intercept: the predicted value of $y$ when $x = 0$

Predicting exam scores from study time

Data from 8 students:

Hours studied	Exam score
1	45
2	52
3	58
4	64
5	70
6	75
7	79
8	83

The regression line turns out to be approximately $\hat{y} = 38 + 5.9x$ .

Prediction: a student who studies 5 hours is predicted to score $38 + 5.9 \times 5 = 67.5$ points.

Try it

Using the regression equation $\hat{y} = 38 + 5.9x$ , predict the score for a student who studies 10 hours. Should you trust this prediction? Why or why not?

Solution

$\hat{y} = 38 + 5.9 \times 10 = 97$ . The prediction is 97 points.

Be cautious: this is extrapolation — predicting outside the range of the data (1–8 hours). The relationship may not continue to be linear beyond 8 hours. Also, a score above 100 may not be possible if the test is capped. Regression predictions are most reliable within the range of observed data.

Related concepts

Needs first

Mean

Generalized Linear Models Logistic Regression Pseudoinverse Regularization Standard Deviation

View in full concept graph →