Vector spaces. DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis.

Transcription

1 Vector spaces DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis Carlos Fernandez-Granda

2

3 Vector space Consists of: A set V A scalar field (usually R or C) Two operations + and

4 Properties For any x, y V, x + y belongs to V For any x V and any scalar α, α x V There exists a zero vector 0 such that x + 0 = x for any x V For any x V there exists an additive inverse y such that x + y = 0, usually denoted by x

5 Properties The vector sum is commutative and associative, i.e. for all x, y, z V x + y = y + x, ( x + y) + z = x + ( y + z) Scalar multiplication is associative, for any scalars α and β and any x V α (β x) = (α β) x Scalar and vector sums are both distributive, i.e. for any scalars α and β and any x, y V (α + β) x = α x + β x, α ( x + y) = α x + α y

6 Subspaces A subspace of a vector space V is any subset of V that is also itself a vector space

7 Linear dependence/independence A set of m vectors x 1, x 2,..., x m is linearly dependent if there exist m scalar coefficients α 1, α 2,..., α m which are not all equal to zero and m α i x i = 0 i=1 Equivalently, any vector in a linearly dependent set can be expressed as a linear combination of the rest

8 Span The span of { x 1,..., x m } is the set of all possible linear combinations span ( x 1,..., x m ) := { y y = } m α i x i for some scalars α 1, α 2,..., α m i=1 The span of any set of vectors in V is a subspace of V

9 Basis and dimension A basis of a vector space V is a set of independent vectors { x 1,..., x m } such that V = span ( x 1,..., x m ) If V has a basis with finite cardinality then every basis contains the same number of vectors The dimension dim (V) of V is the cardinality of any of its bases Equivalently, the dimension is the number of linearly independent vectors that span V

10 Standard basis e 1 = , e 2 = ,..., e n = The dimension of R n is n

11

12 Inner product Operation, that maps a pair of vectors to a scalar

13 Properties If the scalar field is R, it is symmetric. For any x, y V x, y = y, x If the scalar field is C, then for any x, y V x, y = y, x, where for any α C α is the complex conjugate of α

14 Properties It is linear in the first argument, i.e. for any α R and any x, y, z V α x, y = α x, y, x + y, z = x, z + y, z. If the scalar field is R, it is also linear in the second argument It is positive definite: x, x is nonnegative for all x V and if x, x = 0 then x = 0

15 Dot product Inner product between x, y R n x y := i x [i] y [i] R n endowed with the dot product is usually called a Euclidean space of dimension n If x, y C n x y := i x [i] y [i]

16 Sample covariance Quantifies joint fluctuations of two quantities or features For a data set (x 1, y 1 ), (x 2, y 2 ),..., (x n, y n ) cov ((x 1, y 1),..., (x n, y n)) := 1 n 1 where the average or sample mean is defined by n (x i av (x 1,..., x n)) (y i av (y 1,..., y n)) i=1 av (a 1,..., a n) := 1 n If (x 1, y 1 ), (x 2, y 2 ),..., (x n, y n ) are iid samples from x and y E (cov ((x 1, y 1 ),..., (x n, y n ))) = Cov (x, y) := E ((x E (x)) (y E (y))) n i=1 a i

17 Matrix inner product The inner product between two m n matrices A and B is ( ) A, B := tr A T B m n = A ij B ij i=1 j=1 where the trace of an n n matrix is defined as the sum of its diagonal tr (M) := n i=1 M ii For any pair of m n matrices A and B ( ) tr B T A := tr (AB ) T

18 Function inner product The inner product between two complex-valued square-integrable functions f, g defined in an interval [a, b] of the real line is f g := b a f (x) g (x) dx

19

20 Norm Let V be a vector space, a norm is a function from V to R with the following properties It is homogeneous. For any scalar α and any x V α x = α x. It satisfies the triangle inequality x + y x + y. In particular, x 0 x = 0 implies x = 0

21 Inner-product norm Square root of inner product of vector with itself x, := x, x

22 Inner-product norm Vectors in R n or C n : l 2 norm x 2 := x x = n x[i] 2 i=1 Matrices in R m n or C m n : Frobenius norm m A F := tr (A T A) = n i=1 j=1 Square-integrable complex-valued functions: L 2 norm b A 2 ij f L2 := f, f = a f (x) 2 dx

23 Cauchy-Schwarz inequality For any two vectors x and y in an inner-product space Assume x, 0, then x, y x, y, x, y = x, y, y = y, x, x x, y = x, y, y = y, x, x

24 Sample variance and standard deviation The sample variance quantifies fluctuations around the average var (x 1, x 2,..., x n ) := 1 n 1 n (x i av (x 1, x 2,..., x n )) 2 If x 1, x 2,..., x n are iid samples from x E (var (x 1, x 2,..., x n )) = Var (x) := E ((x E (x)) 2) i=1 The sample standard deviation is std (x 1, x 2,..., x n ) := var (x 1, x 2,..., x n )

25 Correlation coefficient Normalized covariance ρ (x1,y 1 ),...,(x n,y n) := cov ((x 1, y 1 ),..., (x n, y n )) std (x 1,..., x n ) std (y 1,..., y n ) Corollary of Cauchy-Schwarz and 1 ρ (x1,y 1 ),...,(x n,y n) 1 ρ x, y = 1 y i = av (y 1,..., y n ) std (y 1,..., y n ) std (x 1,..., x n ) (x i av (x 1,..., x n )) ρ x, y = 1 y i = av (y 1,..., y n ) + std (y 1,..., y n ) std (x 1,..., x n ) (x i av (x 1,..., x n ))

26 Correlation coefficient ρ x, y ρ x, y

27 Temperature data Temperature in Oxford over 150 years Feature 1: Temperature in January Feature 1: Temperature in August ρ = April August

28 Temperature data Temperature in Oxford over 150 years (monthly) Feature 1: Maximum temperature Feature 1: Minimum temperature 20 ρ = Minimum temperature Maximum temperature

29 Parallelogram law A norm on a vector space V is an inner-product norm if and only if for any x, y V 2 x y 2 = x y 2 + x + y 2

30 l 1 and l norms Norms in R n or C n not induced by an inner product x 1 := n x[i] i=1 x := max x[i] i Hölder s inequality x, y x 1 y

31 Norm balls l 1 l 2 l

32 Distance The distance between two vectors x and y induced by a norm is d ( x, y) := x y

33 Classification Aim: Assign a signal to one of k predefined classes Training data: n pairs of signals (represented as vectors) and labels: { x 1, l 1 },..., { x n, l n }

34 Nearest-neighbor classification nearest neighbor

35 Face recognition Training set: images from 40 different subjects (9 each) Test set: 1 new image from each subject We model each image as a vector in R 4096 and use the l 2 -norm distance

36 Face recognition Training set

37 Nearest-neighbor classification Errors: 4 / 40 Test image Closest image

38

39 Orthogonality Two vectors x and y are orthogonal if and only if x, y = 0 A vector x is orthogonal to a set S, if x, s = 0, for all s S Two sets of S 1, S 2 are orthogonal if for any x S 1, y S 2 x, y = 0 The orthogonal complement of a subspace S is S := { x x, y = 0 for all y S}

40 Pythagorean theorem If x and y are orthogonal x + y 2, = x 2, + y 2,

41 Orthonormal basis Basis of mutually orthogonal vectors with inner-product norm equal to one If { u 1,..., u n } is an orthonormal basis of a vector space V, for any x V x = n u i, x u i i=1

42 Gram-Schmidt Builds orthonormal basis from a set of linearly independent vectors x 1,..., x m in R n 1. Set u 1 := x 1 / x For i = 1,..., m, compute i 1 v i := x i u j, x i u j and set u i := v i / v i 2 j=1

43

44 Direct sum For any subspaces S 1, S 2 such that S 1 S 2 = {0} the direct sum is defined as S 1 S 2 := { x x = s 1 + s 2 s 1 S 1, s 2 S 2 } Any vector x S 1 S 2 has a unique representation x = s 1 + s 2 s 1 S 1, s 2 S 2

45 Orthogonal projection The orthogonal projection of x onto a subspace S is a vector denoted by P S x such that x P S x S The orthogonal projection is unique

46 Orthogonal projection

47 Orthogonal projection Any vector x can be decomposed into x = P S x + P S x. For any orthonormal basis b 1,..., b m of S, P S x = m i=1 x, b i bi The orthogonal projection is a linear operation. For x and y P S ( x + y) = P S x + P S y

48 Dimension of orthogonal complement Let V be a finite-dimensional vector space, for any subspace S V ( dim (S) + dim S ) = dim (V)

49 Orthogonal projection is closest The orthogonal projection P S x of a vector x onto a subspace S is the solution to the optimization problem minimize u subject to x u, u S

50 Proof Take any point s S such that s P S x x s 2,

51 Proof Take any point s S such that s P S x x s 2, = x P S x + P S x s 2,

52 Proof Take any point s S such that s P S x x s 2, = x P S x + P S x s 2, = x P S x 2, + P S x s 2,

53 Proof Take any point s S such that s P S x x s 2, = x P S x + P S x s 2, = x P S x 2, + P S x s 2, > x P S x 2, if s P S x

54

55 Denoising Aim: Estimating a signal from perturbed measurements If the noise is additive, the data are modeled as the sum of the signal x and a perturbation z y := x + z The goal is to estimate x from y Assumptions about the signal and noise structure are necessary

56 Denoising via orthogonal projection Assumption: Signal is well approximated as belonging to a predefined subspace S Estimate: P S y, orthogonal projection of the noisy data onto S Error: x P S y 2 2 = P S x P S z 2 2

57 Proof x P S y

58 Proof x P S y = x P S x P S z

59 Proof x P S y = x P S x P S z = P S x P S z

60 Error P S z P S y S P S x error 0 x z y

61 Face denoising Training set: images from 40 different subjects (9 each) Noise: iid Gaussian noise SNR := x 2 z 2 = 6.67 We model each image as a vector in R 4096

62 Face denoising We denoise by projecting onto: S 1 : the span of the 9 images from the same subject S 2 : the span of the 360 images in the training set Test error: x P S1 y 2 x 2 = x P S2 y 2 x 2 = 0.078

63 S 1 S 1 := span ( )

64 Denoising via projection onto S 1 Projection onto S 1 Projection onto S 1 Signal x = Noise z = = Data y = + Estimate

65 S 2 S 2 := span ( )

66 Denoising via projection onto S 2 Projection onto S 2 Projection onto S 2 Signal x = Noise z = = Data y = + Estimate

67 P S1 z and P S2 z P S1 z P S2 z = P S 1 z 2 x 2 < P S 2 z 2 x 2 = = 6.14 dim (S 2 ) dim (S 1 ) (not a coincidence)

68 P S 1 x and P S 2 x P S 1 x P S 2 x = P S x 2 P S x 2 1 < 2 = x 2 x 2

69 P S1 y and P S2 y x P S1 y P S2 y