Home

PDF letter size

Contents

1 Similarity transformation

A similarity transformation is$$B=M^{-1}AM$$ Where $B,A,M$ are square matrices. The goal of similarity transformation is to find a $B$ matrix which has a simpler form than $A$ so that we can use $B$ in place of $A$ to ease some computational work. Lets set our goal in having $B$ be a diagonal matrix (a general diagonal form is called block diagonal or Jordan form, but here we are just looking at the case of $B$ being a diagonal matrix).

The question becomes: Given $A$, can we find $M$ such that $M^{-1}AM$ is diagonal?

The standard method to show the above is via an algebraic method, which we show first. Next we show a geometric method that explains similarity transformation geometrically.

1.1 Derivation of similarity transformation based on algebraic method

Starting with$$B=M^{-1}AM$$ Our goal is to find a real matrix $B$ such that it is diagonal.  From the above, by pre multiplying each side by $M$ we obtain$$AM=MB$$ Now, since our goal is to make $B$ diagonal, let us select the eigenvalues of $A$ to be the diagonal of $B$. Now we write the above in expanded form as follows$$\left[\begin{array}{ccc}{a}_{11}& \cdots & a_{1n}\\ ⋮& \ddots & ⋮\\ a_{n1}& \cdots & a_{nn}\end{array}\right]\left[\begin{array}{ccc}{m}_{11}& \cdots & m_{1n}\\ ⋮& \ddots & ⋮\\ m_{n1}& \cdots & m_{nn}\end{array}\right]=\left[\begin{array}{ccc}{m}_{11}& \cdots & m_{1n}\\ ⋮& \ddots & ⋮\\ m_{n1}& \cdots & m_{nn}\end{array}\right]\left[\begin{array}{ccc}{\lambda }_1& 0& 0\\ 0& \ddots & 0\\ 0& 0& {\lambda }_n\end{array}\right]$$ The above can be written as $n$ separate equations by looking at the column view of matrix multiplication$$\left[\begin{array}{ccc}{a}_{11}& \cdots & a_{1n}\\ ⋮& \ddots & ⋮\\ a_{n1}& \cdots & a_{nn}\end{array}\right]\left[\begin{array}{c}{m}_{11}\\ ⋮\\ m_{n1}\end{array}\right]={\lambda }_1\left[\begin{array}{c}{m}_{11}\\ ⋮\\ m_{n1}\end{array}\right]$$ And$$\left[\begin{array}{ccc}{a}_{11}& \cdots & a_{1n}\\ ⋮& \ddots & ⋮\\ a_{n1}& \cdots & a_{nn}\end{array}\right]\left[\begin{array}{c}{m}_{12}\\ ⋮\\ m_{n2}\end{array}\right]={\lambda }_2\left[\begin{array}{c}{m}_{12}\\ ⋮\\ m_{n2}\end{array}\right]$$ All the way to the n${}^{th}$ column of $M$$$\left[\begin{array}{ccc}{a}_{11}& \cdots & a_{1n}\\ ⋮& \ddots & ⋮\\ a_{n1}& \cdots & a_{nn}\end{array}\right]\left[\begin{array}{c}{m}_{1n}\\ ⋮\\ m_{nn}\end{array}\right]={\lambda }_n\left[\begin{array}{c}{m}_{1n}\\ ⋮\\ m_{nn}\end{array}\right]$$ Each of the above $n$ equations is in the form$$A\mathbf{x}\mathbf{=}\lambda \mathbf{x}$$ and this is the key idea.

The above shows that if we take the diagonal elements of $B$ to be the eigenvalues of $A$ then the $M$ matrix is the (right) eigenvectors of $A.$

Hence, if we want the $B$ matrix to be diagonal and real, this means the $A$ matrix itself must have some conditions put on it. Specifically, $A$ eigenvalues must be all distinct and real. This happens when $A$ is real and symmetric. Therefore, we have the general result that given a matrix $A$ which has distinct and real eigenvalues, only then can we find a similar matrix to it which is real and diagonal, and these diagonal values are the eigenvalues of $A$.

1.2 Derivation of similarity transformation based on geometric method

It turns out that this derivation is more complicated than the algebraic approach. It involves a change of basis matrix (which will be our $M$ matrix) and the representation of linear transformation $T$ as a matrix under some basis (which will be our $A$ matrix). Let us start from the beginning.

Given the vector $\mathbf{x}$ in ${\mathbb{R}}^n$, it can have many different representations (or coordinates) depending on which basis are used. There are an infinite number of basis that span ${\mathbb{R}}^n$, hence there are infinite representations for the same vector. Consider for example ${\mathbb{R}}^2$. The standard basis vectors are $\{\left[\begin{array}{c}1\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 1\end{array}\right]\}$, but another basis are $\{\left[\begin{array}{c}1\\ 1\end{array}\right],\left[\begin{array}{c}0\\ 1\end{array}\right]\}$, and yet another are $\{\left[\begin{array}{c}1\\ 1\end{array}\right],\left[\begin{array}{c}0\\ -1\end{array}\right]\}$, and so on.

In ${\mathbb{R}}^n$ any $n$ linearly independent vectors can be used as basis. Let ${\mathbf{x}}_v$ be the vector representation in w.r.t. basis $v$, and let ${\mathbf{x}}_{v^{\prime }}$ be the vector representation w.r.t. basis $v^{\prime }$.

An important problem in linear algebra is to obtain ${\mathbf{x}}_v$ given ${\mathbf{x}}_{v^{\prime }}$ and the change of basis matrix ${\left[M\right]}_{v^{\prime }\to v}$.

This requires finding a matrix representation for the change of basis. Once the ${\left[M\right]}_{v^{\prime }\to v}$ matrix is found, we can write $$\begin{array}{}\text{(1)}& {\mathbf{x}}_v={\left[M\right]}_{v^{\prime }\to v}{\mathbf{x}}_{v^{\prime }}\end{array}$$ Where ${\left[M\right]}_{v^{\prime }\to v}$ is the change of basis matrix which when applied to ${\mathbf{x}}_{v^{\prime }}$ returns the coordinates of the vector w.r.t. basis $v.$

From (1) we see that given ${\mathbf{x}}_v$, then to obtain ${\mathbf{x}}_{v^{\prime }}$ we write$$\begin{array}{}\text{(1A)}& {\mathbf{x}}_{v^{\prime }}=[M{]}_{v^{\prime }\to v}^{-1}{\mathbf{x}}_v\end{array}$$ Another important problem is that of linear transformation $T$, where now we apply some transformation on the whole space and we want to find what happens to the space coordinates (all the position vectors) due to this transformation.

Consider for example $T$ which is a rotation in ${\mathbb{R}}^2$ by some angle $\theta$. Here, every position vector in the space is affected by this rotation but the basis remain fixed, and we want to find the new coordinates of each point in space.  

Let ${\mathbf{x}}_v$ be the position vector before applying the linear transformation, and let ${\mathbf{y}}_v$ be the new position vector. Notice that both vectors are written with respect to the same basis $v$. Similar to change basis, we want a way to find ${\mathbf{y}}_v$ from ${\mathbf{x}}_v$, hence we need a way to represent this linear transformation by a matrix, say ${\left[T\right]}_v$ with respect to the basis $v$ and so we could write the following$$\begin{array}{}\text{(2)}& {\mathbf{y}}_v={\left[T\right]}_v{\mathbf{x}}_v\end{array}$$

Assume the basis is changed to $v^{\prime }$. Let the representation of the same linear transformation $T$ w.r.t. basis $v^{\prime }$ be called ${\left[T\right]}_{v^{\prime }}$, so we write$$\begin{array}{}\text{(2A)}& {\mathbf{y}}_{v^{\prime }}={\left[T\right]}_{v^{\prime }}{\mathbf{x}}_{v^{\prime }}\end{array}$$

Hence when the basis changes from $v$ to $v^{\prime }$, $T$ representation changes from ${\left[T\right]}_v$ to ${\left[T\right]}_{v^{\prime }}.$

Now assume we are given some linear transformation $T$ and its representation ${\left[T\right]}_v$ w.r.t. basis $v$, how could we find $T$ representation ${\left[T\right]}_{v^{\prime }}$ w.r.t. to new basis $v^{\prime }$?

From (2) we have$${\mathbf{y}}_v={\left[T\right]}_v{\mathbf{x}}_v$$ But from (1) ${\mathbf{x}}_v={\left[M\right]}_{v^{\prime }\to v}{\mathbf{x}}_{v^{\prime }}$, hence the above becomes$${\mathbf{y}}_v={\left[T\right]}_v\left({\left[M\right]}_{v^{\prime }\to v}{\mathbf{x}}_{v^{\prime }}\right)$$ Pre-multiplying from left the above equation by $[M{]}_{v^{\prime }\to v}^{-1}$ we obtain$$[M{]}_{v^{\prime }\to v}^{-1}{\mathbf{y}}_v=[M{]}_{v^{\prime }\to v}^{-1}[T{]}_v[M{]}_{v^{\prime }\to v}{\mathbf{x}}_{v^{\prime }}$$ But since $[M{]}_{v^{\prime }\to v}^{-1}{\mathbf{y}}_v={\mathbf{y}}_{v^{\prime }}$, then above reduces to$${\mathbf{y}}_{v^{\prime }}=[M{]}_{v^{\prime }\to v}^{-1}[T{]}_v[M{]}_{v^{\prime }\to v}{\mathbf{x}}_{v^{\prime }}$$ But from (2A) ${\mathbf{y}}_{v^{\prime }}={\left[T\right]}_{v^{\prime }}{\mathbf{x}}_{v^{\prime }}$, hence the above becomes$$[T{]}_{v^{\prime }}{\mathbf{x}}_{v^{\prime }}=[M{]}_{v^{\prime }\to v}^{-1}[T{]}_v[M{]}_{v^{\prime }\to v}{\mathbf{x}}_{v^{\prime }}$$ Therefore$$\begin{array}{}\text{(3)}& [T{]}_{v^{\prime }}=[M{]}_{v^{\prime }\to v}^{-1}[T{]}_v[M{]}_{v^{\prime }\to v}\end{array}$$ Notice the difference between change of basis and linear transformation. In change of basis, the vector $\mathbf{x}$ remained fixed in space, but the basis changed, and we want to find the coordinates of the vector w.r.t the new basis. With linear transformation, the vector itself is changed from ${\mathbf{x}}_v$ to ${\mathbf{y}}_v$, but both vectors are expressed w.r.t. the same basis.

Equation (3) allows us to obtain a new matrix representation of the linear transformation $T$ by expressing the same $T$ w.r.t. to different basis. Hence if we are given a representation of $T$ which is not the most optimal representation, we can, by change of basis, obtain a different representation of the same $T$ by using (3). The most optimal matrix representation for linear transformation is a diagonal matrix.

Equation (3) is called a similarity transformation.  To make it easier to compare (3) above with what we wrote in the previous section when we derived the above using an algebraic approach, we let ${\left[T\right]}_{v^{\prime }}=B,{\left[T\right]}_v=A$, hence (3) is $$B=M^{-1}AM$$

The matrix ${\left[T\right]}_{v^{\prime }}$ is similar to ${\left[T\right]}_v$. (i.e. $B$ is similar to $A)$. Both matrices represent the same linear transformation applied on the space. We will show below some examples of how to find ${\left[T\right]}_{v^{\prime }}$ given ${\left[T\right]}_v$ and $\left[M\right]$. But first we show how to obtain matrix representation of $T.$

1.2.1 Finding matrix representation of linear transformation$T$

Given a vector $\mathbf{x}\in {\mathbb{R}}^n$ and some linear transformation (or a linear operator) $T$ that acts on the vector $\mathbf{x}$ transforming this vector into another vector $\mathbf{y}\in {\mathbb{R}}^n$ according to some prescribed manner $T:\mathbf{x}\to \mathbf{y}$. Examples of such linear transformation are rotation, elongation, compression, and reflection, and any combination of these operations, but it can not include the translation of the vector, since translation is not linear.

The question to answer here is how to write down a representation of this linear transformation? It turns out that $T$ can be represented by a matrix of size $n\times n$, and the actual numbers that go into the matrix, or the shape of the matrix, will depend on which basis in ${\mathbb{R}}^n$ we choose to represent $\mathbf{x}$.

We would like to pick some basis so that the final shape of the matrix is the most simple shape.

Let us pick a set of basis $v=\{{\overrightarrow{e}}_1,{\overrightarrow{e}}_2,\cdots ,{\overrightarrow{e}}_n\}$ that span ${\mathbb{R}}^n$ hence $${\mathbf{x}}_v=a_1{\overrightarrow{e}}_1+a_2{\overrightarrow{e}}_2+\cdots +a_n{\overrightarrow{e}}_n$$ Now $$T{\mathbf{x}}_v=T(a_1{\overrightarrow{e}}_1+a_2{\overrightarrow{e}}_2+\cdots +a_n{\overrightarrow{e}}_n)$$ And since $T$ is linear we can rewrite the above as$$\begin{array}{}\text{(1)}& T{\mathbf{x}}_v=a_{1}T{\overrightarrow{e}}_1+a_{2}T{\overrightarrow{e}}_2+\cdots +a_{n}T{\overrightarrow{e}}_n\end{array}$$ We see from above that the new transformed vector $T{\mathbf{x}}_v$ has the same coordinates as ${\mathbf{x}}_v$ if we view $T{\overrightarrow{e}}_i$ as the new basis.

Now $T{\overrightarrow{e}}_i$ itself is an application of the linear transformation but now it is being done on each basis vector ${\overrightarrow{e}}_i$. Hence it will cause each specific basis vector to transform in some manner to a new vector. Whatever this new vector is, we must be able to represent it in terms of the same basis vectors $\{{\overrightarrow{e}}_1,{\overrightarrow{e}}_2,\cdots ,{\overrightarrow{e}}_n\},$ therefore, we write$$T{\overrightarrow{e}}_1={\xi }_{11}{\overrightarrow{e}}_1+{\xi }_{21}{\overrightarrow{e}}_2+\cdots +{\xi }_{n1}{\overrightarrow{e}}_n$$ Where ${\xi }_{ij}$ is the $i^{th}$ coordinate of the vector $T{\overrightarrow{e}}_j$. And we do the same for $T{\overrightarrow{e}}_2$$$T{\overrightarrow{e}}_2={\xi }_{12}{\overrightarrow{e}}_1+{\xi }_{22}{\overrightarrow{e}}_2+\cdots +{\xi }_{n2}{\overrightarrow{e}}_n$$ And so on until$$T{\overrightarrow{e}}_n={\xi }_{1n}{\overrightarrow{e}}_1+{\xi }_{2n}{\overrightarrow{e}}_2+\cdots +{\xi }_{nn}{\overrightarrow{e}}_n$$ Or$$T{\overrightarrow{e}}_j={\xi }_{ij}{\overrightarrow{e}}_i$$ Now plug the above into (1) and obtain$$\begin{array}{rl}T{\mathbf{x}}_v& =a_1({\xi }_{11}{\overrightarrow{e}}_1+{\xi }_{21}{\overrightarrow{e}}_2+\cdots +{\xi }_{n1}{\overrightarrow{e}}_n)\\ & +a_2({\xi }_{12}{\overrightarrow{e}}_1+{\xi }_{22}{\overrightarrow{e}}_2+\cdots +{\xi }_{n2}{\overrightarrow{e}}_n)\\ & +\cdots \\ & +a_n({\xi }_{1n}{\overrightarrow{e}}_1+{\xi }_{2n}{\overrightarrow{e}}_2+\cdots +{\xi }_{nn}{\overrightarrow{e}}_n)\end{array}$$

Hence, if we take the ${\overrightarrow{e}}_i$ as common factors, we have$$\begin{array}{rl}T\mathbf{x}& ={\overrightarrow{e}}_1(a_1{\xi }_{11}+a_2{\xi }_{12}+\cdots +a_n{\xi }_{1n})\\ & +{\overrightarrow{e}}_2(a_1{\xi }_{21}+a_2{\xi }_{22}+\cdots +a_n{\xi }_{2n})\\ & +\cdots \\ & +{\overrightarrow{e}}_n(a_1{\xi }_{n1}+a_2{\xi }_{n2}+\cdots +a_n{\xi }_{nn})\end{array}$$

Hence, since $T{\mathbf{x}}_v=a_{1}T{\overrightarrow{e}}_1+a_{2}T{\overrightarrow{e}}_2+\cdots +a_{n}T{\overrightarrow{e}}_n$ from (1), then by comparing coefficients of each basis vector ${\overrightarrow{e}}_i$ we obtain$$\begin{array}{rl}{a}_{1}T& =a_1{\xi }_{11}+a_2{\xi }_{12}+\cdots +a_n{\xi }_{1n}\\ a_{2}T& =a_1{\xi }_{21}+a_2{\xi }_{22}+\cdots +a_n{\xi }_{2n}\\ & \cdots \\ a_{n}T& =a_1{\xi }_{n1}+a_2{\xi }_{n2}+\cdots +a_n{\xi }_{nn}\end{array}$$

Or in matrix form$$T\left(\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_n\end{array}\right)=\left(\begin{array}{cccc}{\xi }_{11}& {\xi }_{12}& \cdots & {\xi }_{1n}\\ {\xi }_{21}& {\xi }_{22}& \cdots & {\xi }_{2n}\\ ⋮& ⋮& \cdots & ⋮\\ {\xi }_{n1}& {\xi }_{n2}& \cdots & {\xi }_{nn}\end{array}\right)\left(\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_n\end{array}\right)$$ Hence we finally obtain that$$T\Rightarrow {\left[T\right]}_v=\left(\begin{array}{cccc}{\xi }_{11}& {\xi }_{12}& \cdots & {\xi }_{1n}\\ {\xi }_{21}& {\xi }_{22}& \cdots & {\xi }_{2n}\\ ⋮& ⋮& \cdots & ⋮\\ {\xi }_{n1}& {\xi }_{n2}& \cdots & {\xi }_{nn}\end{array}\right)$$ Let us call the matrix that represents $T$ under basis $v$ as ${\left[T\right]}_v.$

We see from the above that the $j^{th}$ column of ${\left[T\right]}_v$ contain the coordinates of the vector $T{\overrightarrow{e}}_j$. This gives us a quick way to construct ${\left[T\right]}_v$: Apply $T$ to each basis vector ${\overrightarrow{e}}_j$, and take the resulting vector and place it in the $j^{th}$ column of ${\left[T\right]}_v$.

We now see that ${\left[T\right]}_v$ will have a different numerical values if the basis used to span ${\mathbb{R}}^n$ are different from the ones used to obtain the above ${\left[T\right]}_v.$

Let use pick some new basis, say $v^{\prime }=\{{\overrightarrow{e}}_1^{\prime },{\overrightarrow{e}}_2^{\prime },\cdots ,{\overrightarrow{e}}_n^{\prime }\}$. Let the new representation of $T$ now be the matrix ${\left[T\right]}_{v^{\prime }}$, then the $j^{th}$ column of ${\left[T\right]}_{v^{\prime }}$ is found from applying $T$ on the new basis ${\overrightarrow{e}}_j^{\prime }$ $$T{\overrightarrow{e}}_j^{\prime }={\zeta }_{ij}{\overrightarrow{e}}_i^{\prime }$$ Where now ${\zeta }_{ij}$ is the $i^{th}$ coordinates of the the vector $T{\overrightarrow{e}}_j^{\prime }$ which will be different from ${\xi }_{ij}$ since in one case we used the basis set $v^{\prime }=\left\{{\overrightarrow{e}}_i^{\prime }\right\}$ and in the other case we used the basis set $v=\left\{{\overrightarrow{e}}_i\right\}$. Hence we see that ${\left[T\right]}_{v^{\prime }}$ will numerically be different depending on the basis used to span the space even though the linear transformation $T$ itself did not change.

1.2.2 Finding matrix representation of change of basis

Now we show how to determine ${\left[M\right]}_{v\to v^{\prime }}$, the matrix which when applied to a vector ${\mathbf{x}}_v$ will result in the vector ${\mathbf{x}}_{v^{\prime }}.$

Given a vector $\mathbf{x}$, it is represented w.r.t. basis $v=\{{\overrightarrow{e}}_1,{\overrightarrow{e}}_2,\cdots ,{\overrightarrow{e}}_n\}$ as $${\mathbf{x}}_v=a_1{\overrightarrow{e}}_1+a_2{\overrightarrow{e}}_2+\cdots +a_n{\overrightarrow{e}}_n$$ and w.r.t. basis $v^{\prime }=\{{\overrightarrow{e}}_1^{\prime },{\overrightarrow{e}}_2^{\prime },\cdots ,{\overrightarrow{e}}_n^{\prime }\}$ as$${\mathbf{x}}_{v^{\prime }}=b_1{\overrightarrow{e}}_1^{\prime }+b_2{\overrightarrow{e}}_2^{\prime }+\cdots +b_n{\overrightarrow{e}}_n^{\prime }$$

But the vector itself is invariant under any change of basis, hence ${\mathbf{x}}_v={\mathbf{x}}_{v^{\prime }}$$$\begin{array}{}\text{(1)}& a_1{\overrightarrow{e}}_1+a_2{\overrightarrow{e}}_2+\cdots +a_n{\overrightarrow{e}}_n=b_1{\overrightarrow{e}}_1^{\prime }+b_2{\overrightarrow{e}}_2^{\prime }+\cdots +b_n{\overrightarrow{e}}_n^{\prime }\end{array}$$ Now write each basis ${\overrightarrow{e}}_i^{\prime }$ in terms of the basis ${\overrightarrow{e}}_j$, hence we have$$\begin{array}{rl}{\overrightarrow{e}}_1^{\prime }& =c_{11}{\overrightarrow{e}}_1+c_{12}{\overrightarrow{e}}_2+\cdots +c_{1n}{\overrightarrow{e}}_n\\ {\overrightarrow{e}}_2^{\prime }& =c_{21}{\overrightarrow{e}}_1+c_{22}{\overrightarrow{e}}_2+\cdots +c_{2n}{\overrightarrow{e}}_n\\ & ⋮\\ \text{(2)}& {\overrightarrow{e}}_n^{\prime }& =c_{n1}{\overrightarrow{e}}_1+c_{n2}{\overrightarrow{e}}_2+\cdots +c_{nn}{\overrightarrow{e}}_n\end{array}$$

Substitute (2) into RHS of (1) we obtain$$\begin{array}{rl}{a}_1{\overrightarrow{e}}_1+a_2{\overrightarrow{e}}_2+\cdots +a_n{\overrightarrow{e}}_n& =b_1(c_{11}{\overrightarrow{e}}_1+c_{12}{\overrightarrow{e}}_2+\cdots +c_{1n}{\overrightarrow{e}}_n)+\\ & b_2(c_{21}{\overrightarrow{e}}_1+c_{22}{\overrightarrow{e}}_2+\cdots +c_{2n}{\overrightarrow{e}}_n)+\\ & \cdots \\ & +b_n(c_{n1}{\overrightarrow{e}}_1+c_{n2}{\overrightarrow{e}}_2+\cdots +c_{nn}{\overrightarrow{e}}_n)\end{array}$$

Factor out the basis ${\overrightarrow{e}}_i$ from the RHS, we obtain$$\begin{array}{rl}{a}_1{\overrightarrow{e}}_1+a_2{\overrightarrow{e}}_2+\cdots +a_n{\overrightarrow{e}}_n& =(b_1{c}_{11}{\overrightarrow{e}}_1+b_1{c}_{12}{\overrightarrow{e}}_2+\cdots +b_1{c}_{1n}{\overrightarrow{e}}_n)+\\ & (b_2{c}_{21}{\overrightarrow{e}}_1+b_2{c}_{22}{\overrightarrow{e}}_2+\cdots +b_2{c}_{2n}{\overrightarrow{e}}_n)+\\ & \cdots \\ & +(b_n{c}_{n1}{\overrightarrow{e}}_1+b_n{c}_{n2}{\overrightarrow{e}}_2+\cdots +b_n{c}_{nn}{\overrightarrow{e}}_n)\end{array}$$

Hence$$\begin{array}{rl}{a}_1{\overrightarrow{e}}_1+a_2{\overrightarrow{e}}_2+\cdots +a_n{\overrightarrow{e}}_n& ={\overrightarrow{e}}_1(b_1{c}_{11}+b_2{c}_{21}+\cdots +b_n{c}_{n1})+\\ & {\overrightarrow{e}}_2(b_1{c}_{12}+b_2{c}_{22}+\cdots +b_n{c}_{n2})+\\ & \cdots \\ & +{\overrightarrow{e}}_n(b_1{c}_{1n}+b_2{c}_{2n}+\cdots +b_n{c}_{nn})\end{array}$$

Now comparing coefficients of each of the basis ${\overrightarrow{e}}_i$ we obtain the following result$$\begin{array}{rl}{a}_1& =b_1{c}_{11}+b_2{c}_{21}+\cdots +b_n{c}_{n1}\\ a_2& =b_1{c}_{12}+b_2{c}_{22}+\cdots +b_n{c}_{n2}\\ & \cdots \\ a_n& =b_1{c}_{1n}+b_2{c}_{2n}+\cdots +b_n{c}_{nn}\end{array}$$

Or in Matrix form, we write$$\begin{array}{}\text{(3)}& \left[\begin{array}{c}{a}_1\\ a_2\\ ⋮\\ a_n\end{array}\right]=\left[\begin{array}{ccccc}{c}_{11}& c_{21}& c_{31}& \cdots & c_{n1}\\ c_{12}& c_{22}& c_{32}& \cdots & c_{n2}\\ ⋮& \cdots & \cdots & \ddots & ⋮\\ c_{1n}& c_{2n}& c_{3n}& \cdots & c_{nn}\end{array}\right]\left[\begin{array}{c}{b}_1\\ b_2\\ ⋮\\ b_n\end{array}\right]\end{array}$$ The above gives a relation between the coordinates of the vector $\mathbf{x}$ w.r.t. basis $v$ (these are the $a_i$ coordinates) to the coordinates of the same vector w.r.t. to basis $v^{\prime }$ (these are the coordinates $b_j$). The mapping between these coordinates is the matrix shown above which we call the $\left[M\right]$ matrix. Since the above matrix returns the coordinates of the vector w.r.t. $v$ when it is multiplied by the coordinates of the vector w.r.t. basis $v^{\prime }$, we write it as ${\left[M\right]}_{v^{\prime }\to v}$ to make it clear from which basis to which basis is the conversion taking place.

Looking at (2) and (3) above, we see that column $j$ in ${\left[M\right]}_{v^{\prime }\to v}$ is the coordinates of the ${\overrightarrow{e}}_j^{\prime }$ w.r.t. basis $v$.

Hence the above gives a method to generate the change of basis matrix ${\left[M\right]}_{v^{\prime }\to v}$. Simply represent each basis in $v^{\prime }$ w.r.t. to basis $v$. Take the result of each such representation and write it as a column in ${\left[M\right]}_{v^{\prime }\to v}$. We now show few examples to illustrate this.

Example showing how to find change of basis matrix

Given the ${\mathbb{R}}^2$ basis $v=\{{\overrightarrow{e}}_1,{\overrightarrow{e}}_2\}=\{\left[\begin{array}{c}1\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 1\end{array}\right]\}$, and new basis $v^{\prime }=\{{\overrightarrow{e}}_1^{\prime },{\overrightarrow{e}}_2^{\prime }\}=\{\left[\begin{array}{c}1\\ 1\end{array}\right],\left[\begin{array}{c}-1\\ 1\end{array}\right]\}$, find the change of basis matrix ${\left[M\right]}_{v\to v^{\prime }}$

Column 1 of ${\left[M\right]}_{v\to v^{\prime }}$ is the coordinates of ${\overrightarrow{e}}_1$ w.r.t. basis $v^{\prime }$. i.e. $$\begin{array}{}\text{(4)}& {\overrightarrow{e}}_1=c_{11}{\overrightarrow{e}}_1^{\prime }+c_{21}{\overrightarrow{e}}_2^{\prime }\end{array}$$ and column 2 of ${\left[M\right]}_{v\to v^{\prime }}$ is the coordinates of ${\overrightarrow{e}}_2$ w.r.t. basis $v^{\prime }$. i.e. $$\begin{array}{}\text{(5)}& {\overrightarrow{e}}_2=c_{12}{\overrightarrow{e}}_1^{\prime }+c_{22}{\overrightarrow{e}}_2^{\prime }\end{array}$$ But (4) is$$\left[\begin{array}{c}1\\ 0\end{array}\right]=c_{11}\left[\begin{array}{c}1\\ 1\end{array}\right]+c_{21}\left[\begin{array}{c}-1\\ 1\end{array}\right]$$ Hence $1=c_{11}-c_{21}$ and $0=c_{11}+c_{21}$, solving, we obtain $c_{11}=\frac{1}{2},c_{21}=\frac{-1}{2}$ and (5) is$$\left[\begin{array}{c}0\\ 1\end{array}\right]=c_{12}\left[\begin{array}{c}1\\ 1\end{array}\right]+c_{22}\left[\begin{array}{c}-1\\ 1\end{array}\right]$$ Hence $0=c_{12}-c_{22}$ and $1=c_{12}+c_{22}$, solving we obtain $c_{12}=\frac{1}{2}$, $c_{22}=\frac{1}{2}$, hence$${\left[M\right]}_{v\to v^{\prime }}=\left[\begin{array}{cc}\frac{1}{2}& \frac{1}{2}\\ \frac{-1}{2}& \frac{1}{2}\end{array}\right]$$ Now we can use the above change of basis matrix to find coordinates of the vector under $v^{\prime }$ given its coordinates under $v$. For example, given ${\mathbf{x}}_v=\left[\begin{array}{c}10\\ 5\end{array}\right]$, then $${\mathbf{x}}_{v^{\prime }}=\left[\begin{array}{cc}\frac{1}{2}& \frac{1}{2}\\ \frac{-1}{2}& \frac{1}{2}\end{array}\right]\left[\begin{array}{c}10\\ 5\end{array}\right]=\left[\begin{array}{c}7.5\\ -2.5\end{array}\right]$$

1.2.3 Examples of similarity transformation

Example 1 This example from Strang book (Linear Algebra and its applications, 4th edition), but shown here with little more details.

Consider ${\mathbb{R}}^2$, and let $T$ be the projection onto a line at angle $\theta ={135}^0$. Let the first basis $v=\{\left[\begin{array}{c}1\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 1\end{array}\right]\}$ and let the second basis $v^{\prime }=\{\left[\begin{array}{c}1\\ -1\end{array}\right],\left[\begin{array}{c}1\\ 1\end{array}\right]\}$.

The first step is to find ${\left[M\right]}_{v^{\prime }\to v}$. The first column of ${\left[M\right]}_{v^{\prime }\to v}$ is found by writing $e_1^{\prime }$w.r.t. basis $v$, and the second column of ${\left[M\right]}_{v^{\prime }\to v}$ is found by writing $e_2^{\prime }$w.r.t. basis $v$, Hence $$\begin{array}{rl}{e}_1^{\prime }& =c_{11}{e}_1+c_{21}{e}_2\\ \left[\begin{array}{c}1\\ -1\end{array}\right]& =c_{11}\left[\begin{array}{c}1\\ 0\end{array}\right]+c_{21}\left[\begin{array}{c}0\\ 1\end{array}\right]\end{array}$$

Or $1=c_{11}$ and $c_{21}=-1$. And$$\begin{array}{rl}{e}_2^{\prime }& =c_{12}{e}_1+c_{22}{e}_2\\ \left[\begin{array}{c}1\\ 1\end{array}\right]& =c_{12}\left[\begin{array}{c}1\\ 0\end{array}\right]+c_{22}\left[\begin{array}{c}0\\ 1\end{array}\right]\end{array}$$

Hence $c_{12}=1$ and $c_{22}=1$. Hence $${\left[M\right]}_{v^{\prime }\to v}=\left[\begin{array}{cc}1& 1\\ -1& 1\end{array}\right]$$ Now we need to find ${\left[T\right]}_v$ the representation of $T$ w.r.t. basis $v$. The first column of ${\left[T\right]}_v$ is the new coordinates of the basis $v_1$ after applying $T$ onto it. but $$\begin{array}{rl}T{e}_1& =T\left[\begin{array}{c}0\\ 1\end{array}\right]\\ & =\left[\begin{array}{c}0.5\\ -0.5\end{array}\right]\end{array}$$

and the second column of ${\left[T\right]}_v$ is the new coordinates of the basis $v_2$ after applying $T$ onto it. but $$\begin{array}{rl}T{e}_2& =T\left[\begin{array}{c}1\\ 0\end{array}\right]\\ & =\left[\begin{array}{c}-0.5\\ 0.5\end{array}\right]\end{array}$$

Hence $${\left[T\right]}_v=\left[\begin{array}{cc}0.5& -0.5\\ -0.5& 0.5\end{array}\right]$$ Therefore$$\begin{array}{rl}[T{]}_{v^{\prime }}& =[M{]}_{v^{\prime }\to v}^{-1}[T{]}_v[M{]}_{v^{\prime }\to v}\\ & ={\left[\begin{array}{cc}1& 1\\ -1& 1\end{array}\right]}^{-1}\left[\begin{array}{cc}0.5& -0.5\\ -0.5& 0.5\end{array}\right]\left[\begin{array}{cc}1& 1\\ -1& 1\end{array}\right]\\ {\left[T\right]}_{v^{\prime }}& =\left[\begin{array}{cc}1& 0\\ 0& 0\end{array}\right]\end{array}$$

Hence we see that the linear transformation $T$ is represented as $\left[\begin{array}{cc}1& 0\\ 0& 0\end{array}\right]$ w.r.t. basis $v^{\prime }$, while it is represented as $\left[\begin{array}{cc}0.5& -0.5\\ -0.5& 0.5\end{array}\right]$w.r.t. basis $v^{\prime }$. Therefore, it will be better to perform all calculations involving this linear transformation under basis $v^{\prime }$ instead of basis $v$

Example 2

Consider ${\mathbb{R}}^2$, and let $T$ be a rotation of space by $\theta =\frac{\pi }{4}$. Let the first basis $v=\{\left[\begin{array}{c}1\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 1\end{array}\right]\}$ and let the second basis be $v^{\prime }=\{\left[\begin{array}{c}1\\ 1\end{array}\right],\left[\begin{array}{c}-1\\ 1\end{array}\right]\}$.

The first step is to find ${\left[M\right]}_{v^{\prime }\to v}$. The first column of ${\left[M\right]}_{v^{\prime }\to v}$ is found by writing $e_1^{\prime }$w.r.t. basis $v$, and the second column of ${\left[M\right]}_{v^{\prime }\to v}$ is found by writing $e_2^{\prime }$w.r.t. basis $v$, Hence $$\begin{array}{rl}{e}_1^{\prime }& =c_{11}{e}_1+c_{21}{e}_2\\ \left[\begin{array}{c}1\\ 1\end{array}\right]& =c_{11}\left[\begin{array}{c}1\\ 0\end{array}\right]+c_{21}\left[\begin{array}{c}0\\ 1\end{array}\right]\end{array}$$

Or $1=c_{11}$ and $c_{21}=1$ and$$\begin{array}{rl}{e}_2^{\prime }& =c_{12}{e}_1+c_{22}{e}_2\\ \left[\begin{array}{c}-1\\ 1\end{array}\right]& =c_{12}\left[\begin{array}{c}1\\ 0\end{array}\right]+c_{22}\left[\begin{array}{c}0\\ 1\end{array}\right]\end{array}$$

Hence $c_{12}=-1$ and $c_{22}=1$. Hence $${\left[M\right]}_{v^{\prime }\to v}=\left[\begin{array}{cc}1& -1\\ 1& 1\end{array}\right]$$ Now we need to find ${\left[T\right]}_v$ the representation of $T$ w.r.t. basis $v$. The first column of $[T{]}_v$ is the new coordinates of the basis $v_1$ after applying $T$ onto it. but $$\begin{array}{rl}T{e}_1& =T\left[\begin{array}{c}0\\ 1\end{array}\right]\\ & =\left[\begin{array}{c}\cos \theta \\ \sin \theta \end{array}\right]\end{array}$$

and the second column of ${\left[T\right]}_v$ is the new coordinates of the basis $v_2$ after applying $T$ onto it. but $$\begin{array}{rl}T{e}_2& =T\left[\begin{array}{c}1\\ 0\end{array}\right]\\ T{e}_2& =\left[\begin{array}{c}-\sin \theta \\ \cos \theta \end{array}\right]\end{array}$$

Hence $$\begin{array}{rl}{\left[T\right]}_v& =\left[\begin{array}{cc}\cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{array}\right]\\ & =\left[\begin{array}{cc}\cos \frac{\pi }{4}& -\sin \frac{\pi }{4}\\ \sin \frac{\pi }{4}& \cos \frac{\pi }{4}\end{array}\right]\\ {\left[T\right]}_v& =\left[\begin{array}{cc}\frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}\end{array}\right]\end{array}$$

Therefore$$\begin{array}{rl}[T{]}_{v^{\prime }}& =[M{]}_{v^{\prime }\to v}^{-1}[T{]}_v[M{]}_{v^{\prime }\to v}\\ & ={\left[\begin{array}{cc}1& -1\\ 1& 1\end{array}\right]}^{-1}\left[\begin{array}{cc}\frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}\end{array}\right]\left[\begin{array}{cc}1& -1\\ 1& 1\end{array}\right]\\ & =\left[\begin{array}{cc}\frac{1}{2}& \frac{1}{2}\\ -\frac{1}{2}& \frac{1}{2}\end{array}\right]\left[\begin{array}{cc}0& -\sqrt{2}\\ \sqrt{2}& 0\end{array}\right]\\ & =\left[\begin{array}{cc}\frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}\end{array}\right]\end{array}$$

Hence we see that the linear transformation $T$ is represented as $\left[\begin{array}{cc}\frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}\end{array}\right]$ w.r.t. basis $v^{\prime }$, while it is represented as $\left[\begin{array}{cc}\frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}\end{array}\right]$w.r.t. basis $v^{\prime }$. Therefore the change of basis selected above did not result in making the representation of $T$ any simpler (in fact there was no change in the representation).  This means we need to find a more direct method of finding the basis under which $T$ has the simplest representation. Clearly we can’t just keep trying different basis to find if $T$ has simpler representation under the new basis.

1.2.4 How to find the best basis to simplify the representation of $T$?

Our goal of simpler representation of $T$ is that of a diagonal matrix or as close as possible to being diagonal matrix (i.e. Jordan form). Given ${\left[T\right]}_v$ and an infinite number of basis $v^{\prime }$ that we could select to represent $T$ under in the hope we can find a new representation ${\left[T\right]}_{v^{\prime }}$ such that it is simpler than ${\left[T\right]}_v$, we now ask, how does one go about finding such basis?

It turns out the if we select the eigenvectors of ${\left[T\right]}_v$ as the columns of ${\left[M\right]}_{v^{\prime }\to v}$, this will result in ${\left[T\right]}_{v^{\prime }}$ being diagonal or as close to being diagonal as possible (block diagonal or Jordan form).

Let us apply this to the second example in the previous section. In that example, we had$${\left[T\right]}_v=\left[\begin{array}{cc}\frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}\end{array}\right]$$ The eigenvectors of ${\left[T\right]}_v$ are $\left[\begin{array}{c}-i\\ 1\end{array}\right]$ and $\left[\begin{array}{c}i\\ 1\end{array}\right]$, ${\left[M\right]}_{v^{\prime }\to v}=\left[\begin{array}{cc}-i& i\\ 1& 1\end{array}\right]$. Therefore$$\begin{array}{rl}[T{]}_{v^{\prime }}& =[M{]}_{v^{\prime }\to v}^{-1}[T{]}_v[M{]}_{v^{\prime }\to v}\\ & ={\left[\begin{array}{cc}-i& i\\ 1& 1\end{array}\right]}^{-1}\left[\begin{array}{cc}\frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}\end{array}\right]\left[\begin{array}{cc}-i& i\\ 1& 1\end{array}\right]\\ {\left[T\right]}_{v^{\prime }}& =\left[\begin{array}{cc}(\frac{1}{2}-\frac{1}{2}i)\sqrt{2}& 0\\ 0& (\frac{1}{2}+\frac{1}{2}i)\sqrt{2}\end{array}\right]\end{array}$$

Which is diagonal representation. Hence we see that the linear transformation $T$ is represented as $\left[\begin{array}{cc}(\frac{1}{2}-\frac{1}{2}i)\sqrt{2}& 0\\ 0& (\frac{1}{2}+\frac{1}{2}i)\sqrt{2}\end{array}\right]$ w.r.t. basis $v^{\prime }$

1.3 Summary of similarity transformation

To obtain $B=M^{-1}AM$ such that $B$ is real and diagonal requires that $A$ be real and symmetric. The eigenvalues of $A$ goes into the diagonal of $B$ and the eigenvectors of $A$ go into the columns of $M.$ This is an algebraic view.

Geometrically, $A$ is viewed as the matrix representation under some basis $v$ of a linear transformation $T$. And $B$ is the matrix representation of the same linear transformation $T$ but now under a new basis $v^{\prime }$ and $M$ is the matrix that represents the change of basis from $v^{\prime }$ to $v$.

The question then immediately arise: If $A$ must be real and symmetric (for $B$ to be real and diagonal), what does this mean in terms of the linear transformation $T$ and change of basis matrix $M$? This clearly mean that, under some basis $v$, not every linear transformation represented by $A$ can have a similar matrix $B$ which is real and diagonal. Only those linear transformations which result in real and symmetric representation can have a similar matrix which is real and diagonal. This is shown in the previous examples, where we saw that $T$ defined as rotation by ${45}^0$ under the standard basis $v=\{\left[\begin{array}{c}1\\ 0\end{array}\right],\left[\begin{array}{c}0\\ 1\end{array}\right]\}$ resulted in $A=\left[\begin{array}{cc}\frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}\end{array}\right]$ and since this is not symmetric, hence we will not be able to factor this matrix using similarity transformation to a diagonal matrix, no matter which change of basis we try to represent $T$ under. The question then, under a given basis $v$ what is the class of linear transformation which leads to a matrix representation that is symmetric? One such example was shown above which is the projection into the line at ${135}^0$. This question needs more time to look into.

We now write $\mathrm{\Lambda }$ to represent a diagonal matrix, hence the similarity transformation above can be written as$$\mathrm{\Lambda }=M^{-1}AM$$ Or$$A=M\mathrm{\Lambda }{M}^{-1}$$