Lines of Best Fit, Residuals and Correlation Coefficient
A line of best fit is the straight line that describes the trend of a scatter plot, a residual is the vertical distance between a data point and the line, and the correlation coefficient r measures how strongly the points follow a line. In brief: r lies between −1 and 1. A value near 1 means a strong positive linear relationship, near −1 a strong negative one and near 0 no linear relationship. The least squares line makes the sum of the squared residuals as small as possible.
Once a scatter plot shows a linear pattern, the next step is to describe it with an equation, to use it for predictions and to judge how well it fits. This guide shows how to compute the least squares line and the correlation coefficient for a small data set step by step, how to interpret the slope and r and how to read residuals.
What is the line of best fit?
The line of best fit, or regression line, has the form ŷ = a + bx, where ŷ (“y-hat”) is the predicted value of y for a given x, b is the slope and a is the intercept. Drawing a line by eye is possible, but the standard method is least squares: choose a and b so that the sum of the squares of the residuals is minimal. The line always passes through the point of means (x̄, ȳ).
The slope tells how much y changes on average when x increases by one unit, and the intercept is the predicted y at x = 0, which may not have a practical meaning.
How do you calculate the least squares line?
The formulas are b = Sxy/Sxx and a = ȳ − b x̄, where Sxx = Σ(x − x̄)2 and Sxy = Σ(x − x̄)(y − ȳ).
- Compute the means x̄ and ȳ.
- For each pair, compute x − x̄, y − ȳ, their product and (x − x̄)2.
- Add the columns to get Sxy and Sxx (and Syy for r).
- Compute b and then a.
Example: five points
Data: (1, 2), (2, 4), (3, 5), (4, 4), (5, 5). The means are x̄ = 3 and ȳ = 4.
| x | y | x − 3 | y − 4 | (x − 3)(y − 4) | (x − 3)2 |
|---|---|---|---|---|---|
| 1 | 2 | −2 | −2 | 4 | 4 |
| 2 | 4 | −1 | 0 | 0 | 1 |
| 3 | 5 | 0 | 1 | 0 | 0 |
| 4 | 4 | 1 | 0 | 0 | 1 |
| 5 | 5 | 2 | 1 | 2 | 4 |
| Sum | 20 | 0 | 0 | 6 | 10 |
Sxy = 6 and Sxx = 10, so b = 0.6. Then a = 4 − 0.6 · 3 = 2.2. The line is ŷ = 2.2 + 0.6x. Interpretation: each additional unit of x is associated with an average increase of 0.6 in y.
What are residuals?
The residual of a data point is e = y − ŷ: the observed value minus the predicted value. A positive residual means that the point lies above the line. For least squares, the residuals add up to zero. In the example:
- x = 1: ŷ = 2.8, e = 2 − 2.8 = −0.8
- x = 2: ŷ = 3.4, e = 4 − 3.4 = 0.6
- x = 3: ŷ = 4.0, e = 5 − 4 = 1.0
- x = 4: ŷ = 4.6, e = 4 − 4.6 = −0.6
- x = 5: ŷ = 5.2, e = 5 − 5.2 = −0.2
The sum is −0.8 + 0.6 + 1.0 − 0.6 − 0.2 = 0 ✓. The sum of squares is 0.64 + 0.36 + 1 + 0.36 + 0.04 = 2.4. No other line gives a smaller sum. A residual plot (residuals against x) should show a random scatter around zero. A curved pattern means that a line is not appropriate.
What does the correlation coefficient tell you?
Pearson’s correlation coefficient is r = Sxy/√(Sxx · Syy). For the example, Syy = 4 + 0 + 1 + 0 + 1 = 6, so r = 6/√(10 · 6) = 6/7.746 ≈ 0.775. It is a moderately strong positive linear association. The square r2 ≈ 0.6 means that about 60% of the variation in y is accounted for by the linear relationship with x.
Properties: −1 ≤ r ≤ 1; r has no units; it does not depend on which variable is x; the sign of r equals the sign of the slope; and r = ±1 only if all points lie exactly on a line. Because r measures only linear association, a strong curved pattern can have r near 0.
Interpreting values of r
| Value of r | Description |
|---|---|
| 0.9 to 1 | very strong positive |
| 0.7 to 0.9 | strong positive |
| 0.4 to 0.7 | moderate positive |
| −0.4 to 0.4 | weak or no linear relationship |
| −0.7 to −0.4 | moderate negative |
| −1 to −0.7 | strong to very strong negative |
These bands are conventions that differ between fields. Always look at the scatter plot as well.
How do you use the line for predictions?
Substitute a value of x into the line. For x = 3.5, the prediction is 2.2 + 0.6 · 3.5 = 4.3. This is interpolation, as 3.5 lies within the range of the data, and it is usually reliable. Prediction outside the range, such as x = 20, gives 14.2. This is extrapolation and is risky, since the pattern may not continue. A strong correlation does not prove that x causes y.
Where do these ideas come from?
The method of least squares was published by Adrien-Marie Legendre in 1805, and Carl Friedrich Gauss claimed to have used it earlier. The term regression comes from Francis Galton, who noticed that tall parents tend to have children closer to the average height, which he called regression toward the mean. Karl Pearson defined the correlation coefficient that carries his name in the 1890s.
Where are they used today?
Regression lines are used to forecast sales, to calibrate instruments, in medicine to relate dose and response and in sport to model performance. Spreadsheets and statistical programs compute them instantly, and the mathematics behind them, linear algebra and minimisation of squares, is the starting point of machine learning.
Which mistakes are common?
Interpreting r as a percentage
r = 0.8 does not mean 80% of anything. The value r2 = 0.64 gives the share of variation explained.
Ignoring outliers
One far point can change the slope and r a lot. Look at the plot first.
Extrapolating too far
Predictions outside the observed range are speculation.
Assuming correlation means causation
Correlation shows association only, as discussed in the article on scatter plots.