January 29, 2013
Table of contents Metrics
Metric Given a space X, then d : X X R + 0 and z in X if: d(x, y) = 0 is equivalent to x = y d(x, y) = d(y, x) d(x, y) d(x, z) + d(z, y) is a metric is for all x, y
Example of a metric Euclidean Distance: Given X = R n, d(x, y) := ( n (x i y i ) 2 ) 1 2 i=1 d(a, b) = 0 is equivalent to a = b d(a, b) = d(b, a) d(a, b) d(a, c) + d(c, b) (this is the triangle inequality)
Vector Space A vector space is a space X such that for all x, y X and for all α R: x + y X αx X
Examples of vector spaces Real Numbers: given x, y R, and α R: x + y R αx R R n : given x, y R n, and α R: x + y R n αx R n
Examples of vector spaces Polynomials: given f (x) = n a i x i and g(x) = n b i x i, and α R: i=0 i=0 f (x) + g(x) = n (a i + b i )x i, i.e. polynomial of order n αf (x) = n i=0 i=0 αa i x i, i.e. polynomial of order n
Cauchy Series Given a space X, a Cauchy series is a series x i X for which for every ɛ > 0 there exist an n 0 such that for all m, n n 0, d(x m, x n ) ɛ
Completeness A space X is complete if the limit of every Cauchy series X. For example, (0, 1) is not complete but [0, 1] is. The set Q of rational numbers is not complete: you can construct a sequence that converges to 2 but 2 is not in Q.
Norm Given a vector space X, a norm is a mapping. : X R + 0 satisfies, for all x, y X and for all α R: x = 0 if and only if x = 0 αx = α x x + y x + y (triangle inequality) that A norm is also a metric: d(x, y) := x y
Banach Space A Banach Space is a complete vector space X together with a norm.. ( m ) 1 l m p Spaces: R m with the norm x := x i p p i=1 ( ) 1 l p Spaces: These are subspaces of R N with x := x i p p i=1 Function Spaces L p (X ): Over X, f := ( X f (x) p dx ) 1 p.
Dot Product Given a vector space X, a dot product is a mapping.,. : X X R that satisfies, for all x, y and z X and for all α R: Symmetry: x, y = y, x Linearity: x, αy = α x, y Additivity: x, y + z = x, y + x, z
A is a complete vector space X together with a dot product.,.. The dot product automatically generates a norm: x := x, x. Hilbert spaces are special cases of Banach spaces.
Examples of s Euclidean spaces and the standard dot product for x, y R m : x, y = m x i y i i=1 Function spaces (L 2 (X )): functions on X with f : X C for all f, g F, with the dot product: f, g = X f (x)g(x)dx l 2 series of real numbers (infinite), R N : x, y = x i y i i=1
A matrix M R m n corresponds to a linear map from R m to R n. A symmetric matrix M R m m satisfies M ij = M ji. An anti-symmetric matrix M R m m satisfies M ij = M ji. Rank: Denote by I the image of R m under M. rank(m) is the smallest number of vectors that span I.
: orthogonality A matrix M R m m is orthogonal if M T M = I. This means M T = M 1. An orthogonal matrix consists of mutually orthogonal rows and columns.
Matrix Norms The norm of a linear operator between two Banach spaces X and Y: Ax A := max x X x αa = max x X αax x A + B = max x X A + B A = 0 implies max A = 0. = α A (A+B)x x x X Ax x max x X Ax x + max x X Bx x = and thus Ax = 0 for all x, i.e.
Matrix Norms Frobenius norm: (in analogy with vector norm) M 2 Frob = m m Mij 2 i=1 j=1
Eigen Systems Given M in R m m, then λ R is an eigenvalue and x R m is an eigenvector if: Mx = λx
Eigen Systems, symmetric matrices For symmetric matrices all eigenvalues are real and the matrix is fully diagonalizable (i.e. m eigenvectors). All eigenvectors with different eigenvalues are mutually orthogonal: Proof, for two eigenvectors x and x with respective eigenvalues λ and λ : λx T x = (Mx) T x = x T (M T x ) = x T (Mx ) = λ x T x so λ = λ or x T x = 0. We can decompose M = O T ΛO.
Eigen Systems, symmetric matrices We also have the operator norm: M 2 Mx 2 = max x R m x 2 = max x R m and x =1 Mx 2 = max x R m and x =1 xt M T Mx = max x R m and x =1 xt OΛO T OΛO T x = max x R m and x =1 x T Λ 2 x = max i [m] λ2 i
Eigen Systems, symmetric matrices Frobenius norm: M 2 Frob = tr(mm T ) = tr(oλo T OΛO T ) m = tr(λo T OΛO T O) = tr(λ 2 ) = i=1 λ 2 i
: Invariants Trace: tr(m) = m M ii. i=1 tr(ab) = tr(ba). For symmetric matrices: tr(m) = tr(o T ΛO) = tr(λoo T ) = tr(λ) = m Determinant: det(m) = m λ i i=1 λ i i=1
Positive A Positive Definite Matrix is a matrix M R m m for which for all x R m : x T Mx > 0 if x 0 This matrix has only positive eigenvalues: x T Mx = λx T x = λ x > 0 Induced norm: x 2 M = xt Mx
Singular Value Decomposition Want to find similar thing for arbitrary matrix M R m n where m n: M = UΛO U R m n, U T U = I O R n n, O T O = I Λ = diag(λ 1, λ 2,...λ n )