Orthogonality and Projections

Orthogonal and Orthonormal Sets

In this section, we examine what it means for vectors (and sets of vectors) to be orthogonal and orthonormal. Recall that two non-zero vectors are orthogonal if their dot product is zero. A collection of non-zero vectors in \(\RR ^n\) is called orthogonal if the vectors are pair-wise orthogonal. The diagram below shows two orthogonal vectors in \(\RR ^2\) and three orthogonal vectors in \(\RR ^3\).

[Picture] [Picture]

If every vector in an orthogonal set of vectors is also a unit vector, then we say that the given set of vectors is orthonormal.

[Picture]

Formally, we can define orthogonal and orthonormal vectors as follows.

An orthogonal set of vectors may not be orthonormal. To convert an orthogonal set to an orthonormal set, we need to divide each vector by its own length.

We illustrate this concept in the following example.

Orthogonal and Orthonormal Bases

Recall that every basis of \(\RR ^n\) (or a subspace \(W\) of \(\RR ^n\)) imposes a coordinate system on \(\RR ^n\) (or \(W\)) that can be used to express any vector of \(\RR ^n\) (or \(W\)) as a linear combination of the elements of the basis. For example, vectors \(\vec {v}_1\) and \(\vec {v}_2\) impose a coordinate system onto the plane, as shown in the figure below. We readily see that \(\vec {x}\), contained in the plane, can be written as \(\vec {x}=\vec {v}_1+2\vec {v}_2\).

[Picture]

Vector \(\vec {x}\) is visually easy to work with. In general, one way to express an arbitrary vector as a linear combination of the basis vectors is to solve a system of linear equations, which can be costly. One reason we like \(\{\vec {i},\vec {j}\}\) as a basis of \(\RR ^2\) is because any vector \(\vec {x}\) of \(\RR ^2\) can be easily expressed as the sum of the orthogonal projections of \(\vec {x}\) onto the basis vectors \(\vec {i}\) and \(\vec {j}\), as shown below.

[Picture]

We can see why an “upright" coordinate system with basis \(\{\vec {i},\vec {j}\}\) works well. What if we tilt this coordinate system while preserving the orthogonal relationship between the basis vectors? The following exploration allows you to investigate the consequences.

In the following GeoGebra interactive, vectors \(\vec {v}_1\) and \(\vec {v}_2\) are orthogonal (slopes of the lines containing them are negative reciprocals of each other). These vectors are clearly linearly independent and span \(\RR ^2\). Therefore \(\{\vec {v}_1,\vec {v}_2\}\) is a basis of \(\RR ^2\).

Let \(\vec {x}\) be an arbitrary vector. Orthogonal projections of \(\vec {x}\) onto \(\vec {v}_1\) and \(\vec {v}_2\) are depicted in light grey.

  • Use the tip of vector \(\vec {x}\) to manipulate the vector and convince yourself that \(\vec {x}\) is always the diagonal of the parallelogram (a rectangle!) determined by the projections.
  • Use the tips of \(\vec {v}_1\) and \(\vec {v}_2\) to change the basis vectors. What happens when \(\vec {v}_1\) and \(\vec {v}_2\) are no longer orthogonal?
  • Pick another pair of orthogonal vectors \(\vec {v}_1\) and \(\vec {v}_2\). Verify that \(\vec {x}\) is the sum of its projections.

As you have just discovered in Exploration , we can express an arbitrary vector of \(\RR ^2\) as the sum of its projections onto the basis vectors, provided that the basis is orthogonal. It turns out that this result holds for any subspace of \(\RR ^n\), making a basis consisting of orthogonal vectors especially useful.

If an orthogonal set is a basis, we call it an orthogonal basis. Similarly, if an orthonormal set is a basis, we call it an orthonormal basis.

The following theorem generalizes our observation in Exploration . As you read the statement of the theorem, it will be helpful to recall that the orthogonal projection of vector \(\vec {x}\) onto a non-zero vector \(\vec {d}\) is given by

\begin{equation}\label {eq:orthProj} \mbox {proj}_{\vec {d}}\vec {x}=\left (\frac {\vec {x}\cdot \vec {d}}{\norm {\vec {d}}^2}\right )\vec {d} \end{equation}

(See Orthogonal Projections.)

We may express \(\vec {x}\) as a linear combination of the basis elements:

\[ \vec {x} = c_1 \vec {f}_1 + c_2 \vec {f}_2 + \cdots + c_m \vec {f}_m. \]
We claim that \(c_i = \frac {\vec {x}\dotp \vec {f}_i}{\norm {\vec {f}_i}^2}\) for \(i=1,\ldots ,m\). To see this, we take the dot product of each side with the vector \(\vec {f}_i\) and obtain the following.
\begin{equation*} \vec {x} \dotp \vec {f}_i = \left (c_1\vec {f}_1 + c_2\vec {f}_2 + \cdots + c_m\vec {f}_m\right ) \dotp \vec {f}_i \end{equation*}
Our basis is orthogonal, so \(\vec {f}_j \dotp \vec {f}_i = 0\) for all \(j \neq i\), which means after we distribute the dot product, only one term will remain on the right-hand side. We have
\begin{equation*} \vec {x} \dotp \vec {f}_i = c_i\vec {f}_i \dotp \vec {f}_i \end{equation*}

We now divide both sides by \(\vec {f}_i \dotp \vec {f}_i = \norm {\vec {f}_i}^2\), and since our claim holds for \(i=1,\ldots ,m\), the proof is complete.

Theorem 5 shows one important benefit of a basis being orthogonal. With an orthogonal basis it is easy to represent any vector in terms of the basis vectors.

The formula from Theorem 5 is easy to use, and it becomes even easier when our basis is orthonormal.

This is a special case of Theorem 5. Because \(\norm {\vec {u_i}} = 1\) for \(i=1,\ldots ,m\), the terms are given by

\[\left (\frac {\vec {x}\cdot \vec {q}_i}{\norm {\vec {q}_i}^2}\right )\vec {q}_i=\left (\vec {x}\dotp \vec {q}_i\right ) \vec {q}_i.\]

Orthogonal Projection onto a Subspace

In the previous section we found that given a subspace \(W\) of \(\RR ^n\) with an orthogonal basis \(\mathcal {B}\), every vector \(\vec {x}\) in \(W\) can be expressed as the sum of the orthogonal projections of \(\vec {x}\) onto the elements of \(\mathcal {B}\). Note that our premise was that \(\vec {x}\) is in \(W\). In this section, we look into the meaning of the sum of orthogonal projections of \(\vec {x}\) onto the elements of an orthogonal basis of \(W\) for those vectors \(\vec {x}\) of \(\RR ^n\) that are not in \(W\).

In the GeoGebra interactive below, \(W\) is a plane spanned by \(\vec {v}_1\) and \(\vec {v}_2\), in \(\RR ^3\). \(W\) is subspace of \(\RR ^3\). In the initial set up, \(\vec {v}_1\) and \(\vec {v}_2\) are orthogonal. Vector \(\vec {x}\) is not in \(W\).

Use check-boxes to construct the sum of orthogonal projections of \(\vec {x}\) onto \(\vec {v}_1\) and \(\vec {v}_2\). RIGHT-CLICK and DRAG to rotate the image.

If moved, return the basis vectors \(\vec {v}_1\) and \(\vec {v}_2\) to their default position (set \(s_1=s_2=0\)) to ensure that they are orthogonal.
  • Rotate the image to convince yourself that the perpendiculars dropped from the tip of \(\vec {x}\) to \(\vec {v}_1\) and \(\vec {v}_2\) are indeed perpendicular to \(\vec {v}_1\) and \(\vec {v}_2\) in the diagram. (You’ll have to look at it just right to convince yourself of this.) Are both of these perpendiculars also necessarily perpendicular to the plane? Yes, No

  • Use sliders \(x_1, x_2\) and \(x_3\) to manipulate \(\vec {x}\). Rotate the figure for a better view. What is true about about vector \(\vec {p}\)?

    \(\vec {p}=\vec {x}-(\mbox {proj}_{\vec {v}_1}\vec {x}+\mbox {proj}_{\vec {v}_2}\vec {x})\). Vector \(\vec {p}\) is orthogonal to \(W\). All of the above.
  • Rotate the figure so that you’re looking directly down at the plane. If you’re looking at it correctly, you will notice that (1) the parallelogram determined by the projections of \(\vec {x}\) onto \(\vec {v}_1\) and \(\vec {v}_2\) is a rectangle; (2) the sum of projections, \(\mbox {proj}_{\vec {v}_1}\vec {x}+\mbox {proj}_{\vec {v}_2}\vec {x}\), is located directly underneath \(\vec {x}\), like a shadow at midday.
Use sliders \(s_1\) and \(s_2\) to manipulate the basis vectors \(\vec {v}_1\) and \(\vec {v}_2\) so that they are no longer orthogonal.

  • Rotate the figure for a better view. Which of the following is true?

    \(\vec {p}=\vec {x}-(\mbox {proj}_{\vec {v}_1}\vec {x}+\mbox {proj}_{\vec {v}_2}\vec {x})\). Vector \(\vec {p}\) is orthogonal to \(W\). All of the above.

  • Rotate your figure so that you’re looking directly down at the plane. Which of the following is true?

    Parallelogram determined by \(\vec {v}_1\) and \(\vec {v}_2\) is a rectangle. \(\mbox {proj}_{\vec {v}_1}\vec {x}+\mbox {proj}_{\vec {v}_2}\vec {x}\) is located directly underneath \(\vec {x}\). None of the above.

In Exploration , you discovered that given a plane, spanned by orthogonal vectors \(\vec {v}_1,\vec {v}_2\), in \(\RR ^3\), and a vector \(\vec {x}\), not in the plane, we can interpret the sum of orthogonal projections of \(\vec {x}\) onto \(\vec {v}_1\) and \(\vec {v}_2\) as a “shadow" of \(\vec {x}\) that lies in the plane directly underneath the vector \(\vec {x}\). We say that this “shadow" is an orthogonal projection of \(\vec {x}\) onto \(W\). You have also found that if \(\vec {v}_1,\vec {v}_2\) are not orthogonal, the parallelogram representing the sum of the orthogonal projections of \(\vec {x}\) onto \(\vec {v}_1\) and \(\vec {v}_2\) will not be a rectangle. In this case, \(\vec {x}\) minus this sum will NOT be orthogonal to the plane. It is essential that \(\vec {v}_1,\vec {v}_2\) are orthogonal for \(\mbox {proj}_{\vec {v}_1}\vec {x}+\mbox {proj}_{\vec {v}_2}\vec {x}\) to be considered an orthogonal projection.

In general, we can define an orthogonal projection of \(\vec {x}\) in \(\RR ^n\) onto a subspace \(W\) of \(\RR ^n\) as the sum of the orthogonal projections of \(\vec {x}\) onto the elements of an orthogonal basis of \(W\). Definition 12 and the subsequent diagram summarize this discussion.

An illustration of Definition 12 for a two-dimensional subspace \(W\) with orthogonal basis \(\{\vec {f}_1,\vec {f}_2\}\) is shown below.

[Picture]

Using equation (1) multiple times, we can also express \(\vec {w}\) in Definition 12 using the following formula.

Orthogonal Decomposition of \(\vec {x}\)

Definition 12 allows us to express \(\vec {x}\) as the sum of its orthogonal projection, \(\vec {w}=\mbox {proj}_W\vec {x}\), located in \(W\), and a vector we will call \(\vec {w}^\perp \) (pronounced “W-perp"), given by \(\vec {w}^\perp =\vec {x}-\vec {w}\). This decomposition of \(\vec {x}\) is shown in the diagram below.

[Picture]

You have already met \(\vec {w}^\perp \), under the name of \(\vec {p}\) in Exploration , and observed that this vector is orthogonal to \(W\). We will now prove that \(\vec {w}^\perp \) is orthogonal to every vector in \(W\). This will be accomplished in two steps. First, in Theorem 14 we will prove that \(\vec {w}^\perp \) is orthogonal to all of the basis elements of \(W\). Next, you will use this result to demonstrate that \(\vec {w}^\perp \) is orthogonal to every vector in \(W\).

We will use Formula 13 to show that \(\vec {w}^\perp \cdot \vec {f}_i\)=0. Recall that \(\{\vec {f}_{1}, \vec {f}_{2}, \dots , \vec {f}_{m}\}\) is an orthogonal basis. Therefore \(\vec {f}_j\dotp \vec {f}_i=0\) for \(i\neq j\). This observation enables us to compute as follows.

\begin{eqnarray*} \vec {w}^\perp \cdot \vec {f}_i&=&\left [\vec {x}-\left (\frac {\vec {x} \dotp \vec {f}_{1}}{\norm {\vec {f}_{1}}^2}\vec {f}_{1} +\dots + \frac {\vec {x} \dotp \vec {f}_{i}}{\norm {\vec {f}_{i}}^2}\vec {f}_{i}+ \dots +\frac {\vec {x} \dotp \vec {f}_{m}}{\norm {\vec {f}_{m}}^2}\vec {f}_{m}\right )\right ]\cdot \vec {f}_i\\ & =& \vec {x}\dotp \vec {f}_i- \frac {\vec {x} \dotp \vec {f}_{i}}{\norm {\vec {f}_{i}}^2}(\vec {f}_{i}\dotp \vec {f}_i)\\ &=& \vec {x}\dotp \vec {f}_i- \frac {\vec {x} \dotp \vec {f}_{i}}{\norm {\vec {f}_{i}}^2}\norm {\vec {f}_{i}}^2=\vec {x}\dotp \vec {f}_i-\vec {x}\dotp \vec {f}_i=0 \end{eqnarray*}

We leave the proof of the following Corollary as Practice Problem

The fact that the decomposition of \(\vec {x}\) into the sum of \(\vec {w}\) and \(\vec {w}^\perp \) is unique is the subject of the Orthogonal Decomposition Theorem which we will prove in Orthogonal Complements and Decompositions.

Throughout this section we have worked with orthogonal bases of subspaces. Does every subspace of \(\RR ^n\) have an orthogonal basis? If so, how do we find one? These questions will be addressed in Gram-Schmidt Orthogonalization.

Practice Problems

Retry Example 7 using Gaussian elimination.
Let \(\vec {x}_1, \vec {x}_2, \ldots , \vec {x}_k\in \RR ^n\) and suppose \(\mbox {span}\{\vec {x}_1, \vec {x}_2, \ldots , \vec {x}_k\}=\RR ^n\). Furthermore, suppose that there exists a vector \(\vec {v}\in \RR ^n\) for which \(\vec {v}\dotp \vec {x}_j=0\) for all \(j\), \(1\leq j\leq k\). Show that \(\vec {v}=\vec {0}\).
Let \(\vec {x} = \begin{bmatrix}1\\ -2\\ 1\\ 6\end{bmatrix}\) in \(\RR ^4\), and let \(W = \mbox {span}\left (\begin{bmatrix}2\\ 1\\ 3\\ -4\end{bmatrix}, \begin{bmatrix}1\\ 2\\ 0\\ 1\end{bmatrix}\right )\).
Compute \(\mbox {proj}_W(\vec {x})\).

Answer:

\[\frac {1}{10}\begin{bmatrix}\answer {-9}\\\answer {3}\\\answer {-21}\\\answer {33}\end{bmatrix}\]
Show that \(\left \{\begin{bmatrix}1\\ 0\\ 2\\ -3\end{bmatrix}, \begin{bmatrix}4\\ 7\\ 1\\ 2\end{bmatrix}\right \}\) is another orthogonal basis of \(W\).
Use the basis you found in the previous part to compute \(\mbox {proj}_W(\vec {x})\).

Answer:

\[\frac {1}{70}\begin{bmatrix}\answer {-63}\\\answer {21}\\\answer {-147}\\\answer {231}\end{bmatrix}\]
Prove Corollary 16

Text Source

A portion of the text in this section is an adaptation of Section 4.11.1 of Ken Kuttler’s A First Course in Linear Algebra. (CC-BY)

Ken Kuttler, A First Course in Linear Algebra, Lyryx 2017, Open Edition, p. 233-238.