You are about to erase your work on this activity. Are you sure you want to do this?
Updated Version Available
There is an updated version of this activity. If you update to the most recent version of this activity, then your current progress on this activity will be erased. Regardless, your record of completion will remain. How would you like to proceed?
Mathematical Expression Editor
We suppose given points in the plane , with distinct -coordinates (in practice, such
sets of points can arise as data based on the measurement of some quantity -
recorded as the -coordinate - as a function of some parameter recorded as the
-coordinate). Then we would like to find the equation of the line that best fits these
points (by exactly what measurement the line represents a best possible fit is
explained below). if we write the equation of the line as for indeterminants , then
what we are looking for is a least-squares solution to the system of equations
Note that, in this system, the and are constants, and we are trying to solve for
and . For there will be a solution, but in the overdetermined case there
almost always fails to be one. Hence the need to work in the least-squares
setting.
We wish to find the least-squares fit by a linear equation to the set of points .
This problem can be represented by the matrix equation Where , , and .
We note that this matrix is full rank. Therefore least-squares solution is
unique and given by Thus the desired equation is given by We can also
measure the degree to which this comes close to being an actual solution
(which would only exist if the points were colinear). Given , the vector
is (by the above) the least-squares approximation to by a vector in the
column space of (accurate to 4 decimal places). The accuracy can then
be estimated by the distance of this approximation to the original vector :
The last computation in this example indicates what is being minimized when one
fits data points in this way.
Using least-squares linear approximation techniques to find the best linear fit to a set
of data points results in the equation of a line which minimizes the sum of the
squares of the vertical distances from the given points to the line: Note that, unless
the line is horizontal, the vertical distance will be slightly larger than the actual
distance, which is measured in the direction orthogonal to the line, and minimizing
the sum of squares of those distances would correspond geometrically to what one
might normally think of as constituting a least-squares fit. However, the computation
needed to find the best fit with respect to this sum is quite a bit more involved,. This
linear algebraic approach provides a simple and efficient method for finding a
good approximation by a line which will be exact whenever the points are
colinear.
The setup above provides a method for finding not just linear approximations, but
higher order ones as well. The linear algebra is essentially the same. To
illustrate,
Suppose instead we were asked to find the least-squares fit by a quadratic equation to
the same set set of points . As before, this problem can be represented by the
matrix equation Where , , and . We note that the matrix is again full
rank (it has rank 3). Therefore least-squares solution is unique and given
by Thus the desired equation is given by Measuring the degree to which
this comes close to being an actual solution (which would only exist if the
points all lay on the same quadratic graph), we compute is (by the above)
the least-squares approximation to by a vector in the column space of
(accurate to 4 decimal places). The accuracy can then be estimated by the
distance of this approximation to the original vector : As with the linear fit,
the quantity being minimized is the sum of squares of vertical distances
of the original points to the graph of this quadratic function. Notice the
modest improvement; from to . Because the column space of contains the
columns space of , the least-squares approximation has to be at least as good
as the linear one , and almost always will be closer to the original vector
.
We will illustrate our final point by looking at what happens if we go one degree
higher.
We will find the least-squares fit by a cubic equation to the same set set of points .
As before, this problem can be represented by the matrix equation Where , , and .
The matrix is still full rank (it has rank 4). Therefore least-squares solution
is unique and given by Thus the desired equation is given by However,
now when we compute the least-squares approximation we get which is
not just an approximation but rather the vector on the nose; .. In other
words, given these four points, there is a unique cubic equation which fits
the points exactly. Inspecting the computation more carefully, we see why:
the matrix is both full rank and square. In other words, non-singular. In
this case the system of equations is no longer over determined but rather
balanced. And with a non-singular coefficient matrix, we get a unique solution.
Symbolically, this can be seen by noting that the non-singularity of results
in a simplified expression for , confirming it is indeed an exact solution:
This set of examples, in which we compute successively higher order approximations
to a set of data points until we finally arrive at an exact fit, is part of a
more general phenomenon, which we record without proof by the following
theorem.
Given points in with distinct -coordinates , the least-squares fit by a polynomial of
degree is computed by finding the least-squares solution to the matrix equation
where and is the matrix with . The matrix will have full column rank for all , and
so the least-squares solution is unique and given by with degree polynomial
least-squares fit given by Because is non-singular, there will be a polynomial of
degree at most which fits the points exactly. Moreover, the polynomial of degree at
most which accomplishes this will be unique.