Blek NgossangaSoftware Engineering · AI Vision · Research

FR, Lire cette page en français

English interface, partly translated: navigation, lab, projects, series, about and contact are; the home page sections, the site search and the content itself (articles, tutorials, research) are still in French.

Interactive lab

Linear regression

Move the points and watch the least squares line follow. Then learn the same line by gradient descent and compare.

Level
Beginner
Topics
Moindres carrés · MSE · R²
0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10
Figure 1 — Drag a point to move it, click on empty space to add one. Orange segments are the residuals of the learned line.
  • Least squares (exact solution)
  • Gradient descent (displayed step)
0 10 20
Figure 2 — Mean squared error of the learned line; the horizontal line is the minimum, reached by the exact solution.
Data
18
0.8
Gradient descent
0.01
200
30
Lines compared: slope, intercept, error and R²
methodslope aintercept berror (MSE)R²
Least squares0.7451.20650.43780.8903
Gradient, step 00028.6683−6.1859

Fitting a line

We look for the line y^=ax+b\hat y = a x + b that passes "as close as possible" to nn points (xi,yi)(x_i, y_i). "As close as possible" is measured by the mean squared error:

MSE(a,b)=1n∑i=1n(axi+b−yi)2\mathrm{MSE}(a, b) = \frac{1}{n} \sum_{i=1}^{n} (a x_i + b - y_i)^2

Each term is the square of a residual, the vertical gap between a point and the line.

The exact solution

The MSE is a convex function of (a,b)(a, b): its minimum is found by setting its derivatives to zero, which gives

a=∑(xi−xˉ)(yi−yˉ)∑(xi−xˉ)2b=yˉ−a xˉa = \frac{\sum (x_i - \bar x)(y_i - \bar y)}{\sum (x_i - \bar x)^2} \qquad b = \bar y - a\,\bar x

The coefficient R2=1−∑(y^i−yi)2∑(yi−yˉ)2R^2 = 1 - \frac{\sum (\hat y_i - y_i)^2}{\sum (y_i - \bar y)^2} tells what share of the variance of yy the line explains.

The same line, learned

Gradient descent starts from an arbitrary line and corrects it at each step in the direction that lowers the error:

a←a−η⋅2n∑(axi+b−yi) xib←b−η⋅2n∑(axi+b−yi)a \leftarrow a - \eta \cdot \frac{2}{n} \sum (a x_i + b - y_i)\,x_i \qquad b \leftarrow b - \eta \cdot \frac{2}{n} \sum (a x_i + b - y_i)

It converges to the exact solution. This is the method that generalises to models with no closed form: a neural network is trained exactly this way.

What to observe

  • The slope is corrected within a few steps, the intercept much more slowly: xx ranges from 0 to 10, so the loss is a narrow valley. Standardising the data would make descent far quicker.
  • An outlier (drag a point away from the others) pulls the line strongly: the squared error gives large gaps a large weight.
  • With a lot of noise, R2R^2 drops but the line remains the best in the least squares sense.

Other demos

  • Convolution and filters

    Computer vision, Beginner

    Slide a 3 × 3 kernel over an image: blur, sharpen, edge detection. Edit the weights and follow the computation of one pixel, term by term.

  • Haar wavelets (DWT)

    Computer vision, Intermediate

    Decompose an image into LL, HL, LH and HH sub-bands over several levels, drop the small coefficients and measure what the reconstruction loses.

  • Pathfinding: BFS, DFS, Dijkstra, A*

    Algorithms, Beginner

    Four algorithms explore the same grid. Draw walls, add costly terrain and compare the visited cells and the cost of the path.