Orthogonal Matrices and Symmetric Matrices

Recall that an matrix is diagonalizable if and only if it has linearly independent eigenvectors. (see Diagonalizable Matrices and Multiplicity) Moreover, the matrix with these eigenvectors as columns is a diagonalizing matrix for , that is \begin{equation*} P^{-1}AP \mbox{ is diagonal.} \end{equation*} As we have seen, the nice bases of are the orthogonal ones, so a natural question is: which matrices have orthogonal eigenvectors, so that columns of form an orthogonal basis for ? These turn out to be precisely the symmetric matrices (matrices for which ), and this is the main result of this section.

Orthogonal Matrices

Recall that an orthogonal set of vectors is called orthonormal if for each vector in the set, and that any orthogonal set can be “normalized”, i.e. converted into an orthonormal set . In particular, if a matrix has orthogonal eigenvectors, they can (by normalizing) be taken to be orthonormal. The corresponding diagonalizing matrix (we will use instead of ) has orthonormal columns, and such matrices are very easy to invert.

Proof
First note that condition th:orthogonal_matrices_a is equivalent to . Let denote the columns of . Then is the th row of , so the -entry of is . Thus means that if and if . Hence condition th:orthogonal_matrices_a is equivalent to th:orthogonal_matrices_c. The proof of the equivalence of th:orthogonal_matrices_a and th:orthogonal_matrices_b is similar.

It is not enough that the rows of a matrix are merely orthogonal for to be an orthogonal matrix. Here is an example.

Let
(a)
Check that matrix has rows that are orthogonal.
(b)
Check that matrix has columns that are NOT orthogonal.
(c)
Check that matrix has rows that are NOT orthonormal.
(d)
Create a matrix by normalizing each of the rows of .
(e)
Check that is an orthogonal matrix.

Click the arrow to see the answer.

You should get , and one can check that this is orthogonal in a number of ways.

This exploration can certainly be done by hand (although it takes some time), but it also makes for a very nice Octave exercise. 

                                                                  

                                                                  
    %Exploration from Section 9.4 Orthogonal Matrices and Symmetric Matrices
 
    A=[2 1 1; -1 1 1; 0 -1 1]
 
    %Check that matrix A has rows that are orthogonal.
 
    A(1,:)*transpose(A(2,:))
 
    A(2,:)*transpose(A(3,:))
 
    A(1,:)*transpose(A(3,:))
 
    %Check that matrix A has columns that are NOT orthogonal.
 
    transpose(A(:,1))*A(:,2)
 
    %(This is 1 of 3 calculations to do.)
 
    %Check that matrix A in the Octave window has rows that are NOT orthonormal.
 
    %(See the results from the first question.)
 
    %Create a matrix Q by normalizing each of the rows of A.
 
    q1=A(1,:)/norm(A(1,:));
 
    q2=A(2,:)/norm(A(2,:));
 
    q3=A(3,:)/norm(A(3,:));
 
    Q = [q1;q2;q3]
 
    %Check that Q is an orthogonal matrix.
 
    Q*transpose(Q)
 
    %(You may get numbers close to zero in some places you expect
 
    %to get zero due to rounding error)

Link to code.

For more information about using Octave, please see Octave for Chapter 9.

We studied the idea of closure when we studied Subspaces of . The next theorem tells us that orthogonal matrices are closed under matrix multiplication.

Symmetric Matrices

We now shift our focus from orthogonal matrices to another important class of matrices called symmetric matrices. A symmetric matrix is a matrix which is equal to its transpose. We saw a few examples of such matrices in Transpose of a Matrix.

When we began our study of eigenvalues and eigenvectors, we saw numerous examples of matrices with entries that were real numbers with eigenvalues that were complex numbers. It can be shown that symmetric matrices only have real eigenvalues. We also learned that some matrices are diagonalizable while other matrices are not. It turns out that every symmetric matrix is diagonalizable. In fact, we can say more, but first we need the following definition.

We have learned earlier that when we diagonalize a matrix , we write for some matrix where is diagonal, and the diagonal entries are the eigenvalues of . We have also learned that the columns of the matrix are the corresponding eigenvectors of . So when a matrix is orthogonally diagonalizable, we are able to accomplish the diagonalization using a matrix consisting of eigenvectors that form an orthonormal basis for . The following remarkable theorem shows that the matrices that have this property are precisely the symmetric matrices.

Proof
If is orthogonally diagonalizable, then it is an easy exercise to prove that it is symmetric. You are asked to do this in Practice Problem prob:ortho_diag_implies_symmetric.

To prove the “only if” part of this theorem, we assume is symmetric, and we need to show it is orthogonally diagonalizable. We proceed by induction on , the size of the symmetric matrix. If , is already diagonal. If , assume that we know the “only if” statement holds for symmetric matrices. Let be an eigenvalue of , and let , where . Next, set , and use the Gram-Schmidt algorithm to find an orthonormal basis for . Let , so that is an orthogonal matrix. We have \begin{align*} Q_{1}^TAQ_{1} &= \begin{bmatrix} -- & \vec{q}_{1}^T & -- \\ -- & \vec{q}_{2}^T & -- \\ & \vdots & \\ -- & \vec{q}_{n}^T & -- \end{bmatrix} A \begin{bmatrix} | & | & & | \\ \vec{q}_1 & \vec{q}_2 & \cdots & \vec{q}_n \\ | & | & & | \end{bmatrix} \\ &= \begin{bmatrix} -- & \vec{q}_{1}^T & -- \\ -- & \vec{q}_{2}^T & -- \\ & \vdots & \\ -- & \vec{q}_{n}^T & -- \end{bmatrix} \begin{bmatrix} | & | & & | \\ A\vec{q}_1 & A\vec{q}_2 & \cdots & A\vec{q}_n \\ | & | & & | \end{bmatrix} \\ &= \begin{bmatrix} \lambda _{1} & B \\ \vec{0} & A_{1} \end{bmatrix},\\ \end{align*}

where the block has dimensions , and the block under is a zero matrix, because of the orthogonality of the basis vectors.

Next, using the fact that is symmetric, we notice that so is symmetric. It follows that is also a zero matrix and that is symmetric. Since is an symmetric matrix, we may apply the inductive hypothesis, so there exists an orthogonal matrix such that is diagonal. We observe that is orthogonal, and we compute: \begin{align*} (Q_{1}Q_{2})^TA(Q_{1}Q_{2}) &= Q_{2}^T(Q_{1}^TAQ_{1})Q_{2} \\ &= \begin{bmatrix} 1 & 0 \\ 0 & Q^T \end{bmatrix} \begin{bmatrix} \lambda _{1} & 0 \\ 0 & A_{1} \end{bmatrix}\begin{bmatrix} 1 & 0 \\ 0 & Q \end{bmatrix}\\ &= \begin{bmatrix} \lambda _{1} & 0 \\ 0 & D_{1} \end{bmatrix} \end{align*}

is diagonal. Because is orthogonal by Theorem th:orthogonal_product_inverseth:orthogonal_product, this completes the proof.

Because the eigenvalues of a real symmetric matrix are real, Theorem th:PrinAxes is also called the Real Spectral Theorem, and the set of distinct eigenvalues is called the spectrum of the matrix. A similar result holds for matrices with complex entries (Theorem th:025890).

Actually, the fact that the eigenvectors in Example ex:DiagonalizeSymmetricMatrix are orthogonal is no coincidence. These vectors certainly must be linearly independent (they correspond to distinct eigenvalues). We will see that the fact that the matrix is symmetric implies that the eigenvectors are orthogonal. To prove this we need the following useful fact about symmetric matrices.

Proof
Recall that for all columns and . Because , we get \begin{equation*} (A\vec{x}) \dotp \vec{y} = (A\vec{x})^T\vec{y} = \vec{x}^TA^T\vec{y} = \vec{x}^TA\vec{y} = \vec{x} \dotp (A\vec{y}) \end{equation*}

Proof
Let and , where . We compute \begin{equation*} \lambda (\vec{x} \dotp \vec{y}) = (\lambda \vec{x}) \dotp \vec{y} = (A\vec{x}) \dotp \vec{y} = \vec{x} \dotp (A\vec{y}) = \vec{x} \dotp (\mu \vec{y}) = \mu (\vec{x} \dotp \vec{y}) \end{equation*} Hence , and so because .

Now the procedure for diagonalizing a symmetric matrix is clear. Find the distinct eigenvalues and find orthonormal bases for each eigenspace (the Gram-Schmidt algorithm may be needed when there is a repeated eigenvalue). Then the set of all these basis vectors is orthonormal (by Theorem th:symmetric_has_ortho_ev) and contains vectors. Here is an example.

If we are willing to replace “diagonal” by “upper triangular” in the real spectral theorem, we can weaken the requirement that is symmetric to insisting only that has real eigenvalues.

Proof
See Practice Problem prob:SchurChallenge

The eigenvalues of an upper triangular matrix are displayed along the main diagonal. Because and have the same determinant and trace whenever is orthogonal (for they are similar matrices), Theorem th:Schur gives:

This corollary remains true even if the eigenvalues are not real.

Practice Problems

Suppose is orthogonally diagonalizable. Prove that is symmetric. (This is the easy direction of the ”if and only if” in Theorem th:PrinAxes.)

Problems prob:make_ortho_matrix3-prob:make_ortho_matrix7

Normalize the rows to make each of the following matrices orthogonal.

For each matrix , find an orthogonal matrix such that is diagonal.
(a)
(b)
(c)
(d)
(challenging problem)
Prove the converse of Theorem th:dotpSymmetric:

If for all -columns and , then is symmetric.

Let be an orthogonal matrix.
(a)
Show that or .
(b)
Give examples of such that and .
(c)
If , show that has no inverse.
.
(d)
If is and , show that has no inverse.
A matrix that we obtain from the identity matrix by writing its rows in a different order is called a permutation matrix (see Theorem th:LUPA). Show that every permutation matrix is orthogonal.
Show that the following are equivalent for an matrix .
(a)
is orthogonal.
(b)
for all .
(c)
for all , .
(d)
for all columns , .
For (d) (a), show that column of equals , where is column of the identity matrix.
Modify the proof of Theorem th:PrinAxes to prove Theorem th:Schur.

Text Source

This section was adapted from Section 8.2 of Keith Nicholson’s Linear Algebra with Applications. (CC-BY-NC-SA)

W. Keith Nicholson, Linear Algebra with Applications, Lyryx 2018, Open Edition, p. 424

2024-09-26 22:12:51