Matrix Multiplication - The Inner and Outer Products

The Inner and Outer Products

Given two column vectors a and b, the Euclidean inner product and outer product are the simplest special cases of the matrix product, by transposing the column vectors into row vectors.

The inner product

is a column vector multiplied on the left by a row vector:

More explicitly,

$\mathbf{a}\cdot \mathbf{b} = \begin{pmatrix}a_1 & a_2 & \cdots & a_n\end{pmatrix} \begin{pmatrix}b_1 \\ b_2 \\ \vdots \\ b_n\end{pmatrix} = a_1b_1+a_2b_2+\cdots+a_nb_n = \sum_{i=1}^n a_ib_i$

The outer product

is a row vector multiplied on the left by a column vector:

where

$\mathbf{a}\mathbf{b}^\mathrm{T} = \begin{pmatrix}a_1 \\ a_2 \\ \vdots \\ a_n\end{pmatrix} \begin{pmatrix}b_1 & b_2 & \cdots & b_n\end{pmatrix} = \begin{pmatrix} a_1 b_1 & a_1 b_2 & \cdots & a_1 b_n \\ a_2 b_1 & a_2 b_2 & \cdots & a_2 b_n \\ \vdots & \vdots & \ddots & \vdots \\ a_n b_1 & a_n b_2 & \cdots & a_n b_n \\ \end{pmatrix}.$

Matrix product (in terms of inner product)

Suppose that the first n×m matrix A is decomposed into its row vectors a_i, and the second m×p matrix B into its column vectors b_i:

$\mathbf{A} = \begin{pmatrix} A_{1 1} & A_{1 2} & \cdots & A_{1 m} \\ A_{2 1} & A_{2 2} & \cdots & A_{2 m} \\ \vdots & \vdots & \ddots & \vdots \\ A_{n 1} & A_{n 2} & \cdots & A_{n m} \end{pmatrix} = \begin{pmatrix} \mathbf{a}_1 \\ \mathbf{a}_2 \\ \vdots \\ \mathbf{a}_n \end{pmatrix},\quad \mathbf{B} = \begin{pmatrix} B_{1 1} & B_{1 2} & \cdots & B_{1 p} \\ B_{2 1} & B_{2 2} & \cdots & B_{2 p} \\ \vdots & \vdots & \ddots & \vdots \\ B_{m 1} & B_{m 2} & \cdots & B_{m p} \end{pmatrix} = \begin{pmatrix} \mathbf{b}_1 & \mathbf{b}_2 & \cdots & \mathbf{b}_p \end{pmatrix}$

where

The entries in the introduction were given by:

$\mathbf{AB} = \begin{pmatrix} \mathbf{a}_1 \\ \mathbf{a}_2 \\ \vdots \\ \mathbf{a}_n \end{pmatrix} \begin{pmatrix} \mathbf{b}_1 & \mathbf{b}_2 & \dots & \mathbf{b}_p \end{pmatrix} = \begin{pmatrix} (\mathbf{a}_1 \cdot \mathbf{b}_1) & (\mathbf{a}_1 \cdot \mathbf{b}_2) & \dots & (\mathbf{a}_1 \cdot \mathbf{b}_p) \\ (\mathbf{a}_2 \cdot \mathbf{b}_1) & (\mathbf{a}_2 \cdot \mathbf{b}_2) & \dots & (\mathbf{a}_2 \cdot \mathbf{b}_p) \\ \vdots & \vdots & \ddots & \vdots \\ (\mathbf{a}_n \cdot \mathbf{b}_1) & (\mathbf{a}_n \cdot \mathbf{b}_2) & \dots & (\mathbf{a}_n \cdot \mathbf{b}_p) \end{pmatrix}$

It is also possible to express a matrix product in terms of concatenations of products of matrices and row or column vectors:

These decompositions are particularly useful for matrices that are envisioned as concatenations of particular types of row vectors or column vectors, e.g. orthogonal matrices (whose rows and columns are unit vectors orthogonal to each other) and Markov matrices (whose rows or columns sum to 1).

Matrix product (in terms of outer product)

An alternative method results when the decomposition is done the other way around, i.e. the first matrix A is decomposed into column vectors and the second matrix B into row vectors :

$\mathbf{AB} = \begin{pmatrix} \mathbf{\bar a}_1 & \mathbf{\bar a}_2 & \cdots & \mathbf{\bar a}_m \end{pmatrix} \begin{pmatrix} \mathbf{\bar b}_1 \\ \mathbf{\bar b}_2 \\ \vdots \\ \mathbf{\bar b}_m \end{pmatrix} = \mathbf{\bar a}_1 \otimes \mathbf{\bar b}_1 + \mathbf{\bar a}_2 \otimes \mathbf{\bar b}_2 + \cdots + \mathbf{\bar a}_m \otimes \mathbf{\bar b}_m = \sum_{i=1}^m \mathbf{\bar a}_i \otimes \mathbf{\bar b}_i$

where this time

This method emphasizes the effect of individual column/row pairs on the result, which is a useful point of view with e.g. covariance matrices, where each such pair corresponds to the effect of a single sample point.

Read more about this topic: Matrix Multiplication