INNER PRODUCTS, LENGTHS and ORTHOGONALITY

Inner or Dot Product:

Assume ${\bf u}, {\bf v}, {\bf w}\in R^n$, $s \in R$.

• Dot product ${\bf u}\cdot {\bf v}= {\bf u}^T{\bf v}= u_1v_1 + \cdots + u_nv_n$.
• Properties:

  1.

  ${\bf u}\cdot {\bf v}= {\bf v}\cdot {\bf u}$.

  2.

  $({\bf u}+ {\bf v})\cdot {\bf w}= {\bf u}\cdot {\bf w}+ {\bf v}\cdot {\bf w}$.

  3.

  $(s{\bf u}) \cdot {\bf v}= s({\bf u}\cdot {\bf v}) = {\bf u}\cdot (s{\bf v})$.

  4.

  ${\bf u}\cdot {\bf u}\geq 0$, with ${\bf u}\cdot {\bf u}= 0$ iff ${\bf u}= {\bf0}$.

Length or Norm:

Assume ${\bf u}, {\bf v}\in R^n$ and $s \in R$

• Norm of ${\bf u}$: $\vert\vert{\bf u}\vert\vert = \sqrt{{\bf u}\cdot {\bf u}}$; note: $\vert\vert s{\bf u}\vert\vert = \vert s\vert \vert\vert{\bf u}\vert\vert$.
• Distance between ${\bf u}$ and ${\bf v}$: $dist({\bf u},{\bf v}) = \vert\vert{\bf u}- {\bf v}\vert\vert$.

Orthogonality:

If ${\bf u}, {\bf v}\in R^n$ and $s \in R$,

• Vectors ${\bf u}$ and ${\bf v}$ are orthogonal if ${\bf u}\cdot {\bf v}= 0$.
• Pythagorean Theorem:
  ${\bf u}$ is orthogonal to ${\bf v}$ iff $\vert\vert{\bf u}+ {\bf v}\vert\vert^2 = \vert\vert{\bf u}\vert\vert^2 + \vert\vert{\bf v}\vert\vert^2$.
• Orthogonal Complement of a subspace W, $W^{\perp}$:
  $W^{\perp}= \{ {\bf u}: {\bf u}\mbox{ is orthogonal to all} {\bf w}\in W\}$.

  1.

  ${\bf u}\in W^{\perp}$ iff ${\bf u}$ is $\perp$ to all ${\bf w}$ in any W spanning set.

  2.

  $W^{\perp}$ is a subspace of Rⁿ.

  3.

  Matrices: if $A \in R^{m \times n}$,
  $(Row\ A)^{\perp} = Nul\ A$ and $(Col\ A)^{\perp} = Nul\ A^T$.

Angle:

The angle $\theta$ between ${\bf u}$ and ${\bf v}$ is defined by
${\bf u}\cdot {\bf v}= \vert\vert{\bf u}\vert\vert \vert\vert{\bf v}\vert\vert \cos{\theta}$.

ORTHOGONAL SETS

Definition:

A set of vectors $\{ {\bf u}_1, {\bf u}_2, \dots, {\bf u}_p \}$is an
orthogonal set if ${\bf u}_i \cdot {\bf u}_j = 0$ for all $i \neq j$.

Orthogonal Basis:

If $S = \{ {\bf u}_1, \dots, {\bf u}_p \}$ is orthogonal,

• S is linearly independent and forms a basis for Span(S).
• S is an orthogonal basis.
• If ${\bf y}\in Span(S)$ then
  ${\bf y}= c_1{\bf u}_1 + c_2{\bf u}_2 + \cdots + c_p{\bf u}_p$, with $c_i = \frac{{\bf y}\cdot {\bf u}_i }{{\bf u}_i \cdot {\bf u}_i}$.

Orthonormal Sets:

• An orthogonal set S of unit vectors is an orthonormal set.
• An orthonormal S is an orthonormal basis for Span(S).
• A matrix U has orthonormal columns iff U^TU = I.
• If U has orthonormal columns, then $\vert\vert U{\bf x}\vert\vert = \vert\vert{\bf x}\vert\vert$,
  $U{\bf x}\cdot U{\bf y}= {\bf x}\cdot {\bf y}$ and $U{\bf x}\cdot U{\bf y}= 0$ iff ${\bf x}\cdot {\bf y}= 0$.
• An orthogonal matrix is a square matrix U
  with U^TU = UU^T = I and U^-1 = U^T.

Orthogonal Projections:

• $\hat{ {\bf y}} = \frac{{\bf y}\cdot {\bf u}}{{\bf u}\cdot {\bf u}} {\bf u}$ is orthogonal projection of ${\bf y}$ onto ${\bf u}$.
• ${\bf z}= {\bf y}- \frac{{\bf y}\cdot {\bf u}}{{\bf u}\cdot {\bf u}}{\bf u}$ is component of ${\bf y}$ orthogonal to ${\bf u}$.

GRAM-SCHMIDT ORTHOGONALIZATION

Gram-Schmidt Process:

Assume $\{ {\bf x}_1, \dots, {\bf x}_p \}$ is a basis for some subspace W.

• Set ${\bf v}_1 = {\bf x}_1$.
• Set ${\bf v}_2 = {\bf x}_2 - \frac{{\bf x}_2 \cdot {\bf v}_1 }{{\bf v}_1 \cdot {\bf v}_1} {\bf v}_1$.
• Set ${\bf v}_3 = {\bf x}_3 - \frac{{\bf x}_3 \cdot {\bf v}_1 }{{\bf v}_1 \cdot {\bf ... ... v}_1 - \frac{{\bf x}_3 \cdot {\bf v}_2 }{{\bf v}_2 \cdot {\bf v}_2} {\bf v}_2$.
  $\vdots$
• Set ${\bf v}_p = {\bf x}_p - \frac{{\bf x}_p \cdot {\bf v}_1 }{{\bf v}_1 \cdot {\bf ... ...\bf x}_p \cdot {\bf v}_{p-1} }{{\bf v}_{p-1} \cdot {\bf v}_{p-1}} {\bf v}_{p-1}$.

Gram-Schmidt Result:

If $\{ {\bf v}_1, \dots, {\bf v}_p \}$ is determined from $\{ {\bf x}_1, \dots, {\bf x}_p \}$ using the Gram-Schmidt process,
then $\{ {\bf v}_1, \dots, {\bf v}_p \}$ is an orthogonal basis for W.

Orthonormal Basis:

Use ${\bf u}_i={\bf v}_i/\vert\vert{\bf v}_i\vert\vert$, for i = 1, 2, ..., p.