The general idea of regression is to fit a model to input-output data, where the output data is numeric (and not categorical).
The idea of linear regression is to use a linear model, i.e. to fit a line (in the 2-dimensional case) to the data. The mathematical form of the model looks as follows for the 2-dimensional case:
$y=ax+c,$
Or more generally for the n-dimensional case:
$y=a_1 x_1 + a_2 x_2 + … + a_n x_n + c.$
Linear regression: fitting a line to the data.
To make the line pass through the points, we need to select the ai parameters so as to minimize the distance between the line and all the points in some sense. This is typically done by iterating over all the x coordinates (the inputs) and measuring the mean squared error between the real y and what the linear model tells us.
The advantage of linear models is that if we want to minimize this mean squared error, there is a simple way to do that, which only involves a few matrix multiplications and a matrix inversion. For non-linear models, such optimization generally needs to be done iteratively, using more sophisticated optimization methods.