3 Rank and inverse

3.1 Linear combination

Let \(\boldsymbol x_1, \ldots, \boldsymbol x_n\) be vectors of the same order, and let \(\lambda_1, \ldots, \lambda_n\) be scalars. The vector \[ \lambda_1 \boldsymbol x_1 + \lambda_2 \boldsymbol x_2 + \ldots + \lambda_n \boldsymbol x_n \]

is called linear combination of \(\boldsymbol x_1, \ldots, \boldsymbol x_n\). A linear combination can also be written as a matrix-vector product. Let \(\boldsymbol X =\begin{pmatrix} \boldsymbol x_1 & \ldots & \boldsymbol x_n \end{pmatrix}\) be the matrix with columns \(\boldsymbol x_1, \ldots, \boldsymbol x_n\), and let \(\boldsymbol \lambda = (\lambda_1, \ldots, \lambda_n)'\). Then, \[ \lambda_1 \boldsymbol x_1 + \lambda_2 \boldsymbol x_2 + \ldots + \lambda_n \boldsymbol x_n = \boldsymbol X \boldsymbol \lambda.\] The vectors \(\boldsymbol x_1, \ldots, \boldsymbol x_n\) are called linearly dependent if at least one can be written as a linear combination of the others. That is, there exists a nonzero vector \(\boldsymbol \lambda\) with \[ \boldsymbol X \boldsymbol \lambda = \lambda_1 \boldsymbol x_1 + \ldots + \lambda_n \boldsymbol x_n = \boldsymbol 0. \] The vectors \(\boldsymbol x_1, \ldots, \boldsymbol x_n\) are called linearly independent if \[ \boldsymbol X \boldsymbol \lambda = \lambda_1 \boldsymbol x_1 + \ldots + \lambda_n \boldsymbol x_n \neq \boldsymbol 0 \] for all nonzero vectors \(\boldsymbol \lambda\).

To check whether the vectors are linearly independent, we can solve the system of equations \(\boldsymbol X \boldsymbol \lambda = \boldsymbol 0\) by Gaussian elimination. If \(\boldsymbol \lambda = \boldsymbol 0\) is the only solution, then the columns of \(\boldsymbol X\) are linearly independent. If there is a solution \(\boldsymbol \lambda\) with \(\boldsymbol \lambda \neq \boldsymbol 0\), then the columns of \(\boldsymbol X\) are linearly dependent.

3.2 Column rank

The rank of a \(k \times m\) matrix \(\boldsymbol A = \begin{pmatrix} \boldsymbol a_1 & \ldots & \boldsymbol a_m \end{pmatrix}\), written as \(\mathop{\mathrm{rank}}(\boldsymbol A)\), is the number of linearly independent columns \(\boldsymbol a_i\). We say that \(\boldsymbol A\) has full column rank if \(\mathop{\mathrm{rank}}(\boldsymbol X) = m\).

The identity matrix \(\boldsymbol I_k\) has full column rank (i.e., \(\mathop{\mathrm{rank}}(\boldsymbol I_n) = k\)). As another example, consider \[ \boldsymbol X = \begin{pmatrix} 2 & 1 & 4 \\ 0 & 1 & 2 \end{pmatrix}, \] which has linearly dependent columns since the third column is a linear combination of the first two columns: \[ \begin{pmatrix} 4 \\ 2 \end{pmatrix} = 1 \begin{pmatrix} 2 \\ 0 \end{pmatrix} + 2 \begin{pmatrix} 1 \\ 1 \end{pmatrix}. \] The first two columns are linearly independent since \(\lambda_1 = 0\) and \(\lambda_2 = 0\) are the only solutions to the equation \[ \lambda_1 \begin{pmatrix} 2 \\ 0 \end{pmatrix} + \lambda_2 \begin{pmatrix} 1 \\ 1 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix}. \] Therefore, we have \(\mathop{\mathrm{rank}}(\boldsymbol X) = 2\), i.e., \(\boldsymbol X\) does not have a full column rank.

Some useful properties are

\(\mathop{\mathrm{rank}}(\boldsymbol A) \leq \min(k,m)\)
\(\mathop{\mathrm{rank}}(\boldsymbol A) = \mathop{\mathrm{rank}}(\boldsymbol A')\)
\(\mathop{\mathrm{rank}}(\boldsymbol A \boldsymbol B) = \min(\mathop{\mathrm{rank}}(\boldsymbol A), \mathop{\mathrm{rank}}(\boldsymbol B))\)
\(\mathop{\mathrm{rank}}(\boldsymbol A) = \mathop{\mathrm{rank}}(\boldsymbol A' \boldsymbol A) = \mathop{\mathrm{rank}}(\boldsymbol A \boldsymbol A')\).

We can use the qr() function to extract the rank in R. Let’s compute the rank of the matrices \[ \boldsymbol A = \begin{pmatrix} 1 & 2 & 3 \\ 3 & 9 & 1 \\ 0 & 11 & 5 \end{pmatrix}, \] \(\boldsymbol B = \boldsymbol I_3\), and \(\boldsymbol X\) from the example above:

A = matrix(c(1,3,0,2,9,11,3,1,5), nrow=3)
qr(A)$rank

[1] 3

B = matrix(c(1,1,1,1,1,1,1,1,1), nrow=3)
qr(B)$rank

[1] 1

X = matrix(c(2,0,1,1,4,2), ncol=3)
qr(X)$rank

[1] 2

3.3 Nonsingular matrix

A square \(k \times k\) matrix \(\boldsymbol A\) is called nonsingular if it has full rank, i.e., \(\mathop{\mathrm{rank}}(\boldsymbol A) = k\). Conversely, \(\boldsymbol A\) is called singular if it does not have full rank, i.e., \(\mathop{\mathrm{rank}}(\boldsymbol A) < k\).

3.4 Determinant

Consider a square \(k \times k\) matrix \(\boldsymbol A\). The determinant \(\det(\boldsymbol A)\) is a measure of the volume of the geometric object formed by the columns of \(\boldsymbol A\) (a parallelogram for \(k=2\), a parallelepiped for \(k=3\), a hyper-parallelepiped for \(k>3\)). For \(2 \times 2\) matrices, the determinant is easy to calculate: \[ \boldsymbol A = \begin{pmatrix} a & b \\ c & d \end{pmatrix}, \quad \det(\boldsymbol A) = ad - bc. \]

If \(\boldsymbol A\) is triangular (upper or lower), the determinant is the product of the diagonal entries, i.e., \(\det(\boldsymbol A) = \prod_{i=1}^k a_{ii}\). Hence, Gaussian elimination can be used to compute the determinant by transforming the matrix to a triangular one. The exact definition of the determinant is technical and of little importance to us. A useful relation is the following: \[\begin{align*} \det(\boldsymbol A) = 0 \quad &\Leftrightarrow \quad \boldsymbol A \ \text{is singular} \\ \det(\boldsymbol A) \neq 0 \quad &\Leftrightarrow \quad \boldsymbol A \ \text{is nonsingular}. \end{align*}\]

In R, we have the det() function to compute the determinant:

det(A)

[1] 103

det(B)

[1] 0

Since \(\det(\boldsymbol A) \neq 0\) and \(\det(\boldsymbol B) = 0\), we conclude that \(\boldsymbol A\) is nonsingular and \(\boldsymbol B\) is singular.

3.5 Inverse matrix

The inverse \(\boldsymbol{A}^{-1}\) of a square \(k \times k\) matrix \(\boldsymbol A\) is defined by the property \[\boldsymbol{A} \boldsymbol{A}^{-1} = \boldsymbol{A}^{-1} \boldsymbol{A} =\boldsymbol{I}_k.\] When multiplied from the left or the right, the inverse matrix produces the identity matrix. The inverse exists if and only if \(\boldsymbol{A}\) is nonsingular, i.e., \(\det(\boldsymbol A) \neq 0\). Therefore, a nonsingular matrix is also called invertible matrix. Note that only square matrices can be inverted.

For \(2 \times 2\) matrices, there exists a simple formula: \[\boldsymbol{A}^{-1} = \frac{1}{\det(\boldsymbol{A})} \begin{pmatrix}d&-b\\-c&a\end{pmatrix}\,,\] where \(\det(\boldsymbol{A}) = ad - bc\). We swap the main diagonal elements, reverse the sign of the off-diagonal elements, and divide all entries by the determinant. Example: \[\displaystyle\boldsymbol{A}=\begin{pmatrix}5&6\\1&2\end{pmatrix}\] We have \(\det(\boldsymbol{A}) = ad-bc=5\cdot2-6\cdot1=4\), and \[\boldsymbol{A}^{-1}= \frac{1}{4} \cdot \begin{pmatrix}2&-6\\-1&5\end{pmatrix}.\] Indeed, \(\boldsymbol A^{-1}\) is the inverse of \(\boldsymbol A\) since \[\boldsymbol{A}\boldsymbol{A}^{-1} =\begin{pmatrix}5&6\\1&2\end{pmatrix} \cdot \frac{1}{4} \cdot \begin{pmatrix}2&-6\\-1&5\end{pmatrix} =\frac{1}{4} \cdot \begin{pmatrix}4&0\\0&4\end{pmatrix} = \left(\begin{matrix}1 & 0 \\ 0 & 1 \end{matrix}\right) = \boldsymbol{I}_2.\]

One way to calculate the inverse of higher order square matrices is to solve equation \(\boldsymbol A \boldsymbol A^{-1} = \boldsymbol I\) with Gaussian elimination. R can compute the inverse matrix quickly using the function solve():

solve(A) #inverse if A

           [,1]        [,2]        [,3]
[1,]  0.3300971  0.22330097 -0.24271845
[2,] -0.1456311  0.04854369  0.07766990
[3,]  0.3203883 -0.10679612  0.02912621

We have the following relationship between invertibility, rank, and determinant of a square matrix \(\boldsymbol A\):

\[\begin{align*} &\boldsymbol A \ \text{is nonsingular} \\ \Leftrightarrow \quad &\text{all columns of} \ \boldsymbol A \ \text{are linearly independent} \\ \Leftrightarrow \quad &\boldsymbol A \ \text{has full column rank} \\ \Leftrightarrow \quad &\text{the determinant is nonzero} \ (\det(\boldsymbol A) \neq 0). \end{align*}\]

Similarly, \[\begin{align*} &\boldsymbol A \ \text{is singular} \\ \Leftrightarrow \quad &\boldsymbol A \ \text{has linearly dependent columns} \\ \Leftrightarrow \quad &\boldsymbol A \ \text{does not have full rank} \\ \Leftrightarrow \quad &\text{the determinant is zero} \ (\det(\boldsymbol A) = 0). \end{align*}\]

Below you will find some important properties for nonsingular matrices:

\((\boldsymbol{A}^{-1})^{-1} = \boldsymbol{A}\)
\((\boldsymbol{A}')^{-1} = (\boldsymbol{A}^{-1})'\)
\((\lambda\boldsymbol{A})^{-1} = \frac{1}{\lambda}\boldsymbol{A}^{-1}\) for any \(\lambda \neq 0\)
\(\det(\boldsymbol A^{-1}) = \frac{1}{\det(\boldsymbol A)}\)
\((\boldsymbol{A} \boldsymbol{B})^{-1} = \boldsymbol{B}^{-1} \boldsymbol{A}^{-1}\)
\((\boldsymbol A \boldsymbol B \boldsymbol C)^{-1} = \boldsymbol C^{-1} \boldsymbol B^{-1} \boldsymbol A^{-1}\)
If \(\boldsymbol{A}\) is symmetric, then \(\boldsymbol{A}^{-1}\) is symmetric.