2. Norm, distance, angle

Transcription

1 L. Vandenberghe EE133A (Spring 2017) 2. Norm, distance, angle norm distance k-means algorithm angle hyperplanes complex vectors 2-1

2 Euclidean norm (Euclidean) norm of vector a R n : a = a a a2 n = a T a if n = 1, a reduces to absolute value a measures the magnitude of a sometimes written as a 2 to distinguish from other norms, e.g., a 1 = a 1 + a a n Norm, distance, angle 2-2

3 Properties Positive definiteness a 0 for all a, a = 0 only if a = 0 Homogeneity βa = β a for all vectors a and scalars β Triangle inequality a + b a + b for all vectors a and b of equal length (proof on page 2-7) Norm, distance, angle 2-3

4 Cauchy-Schwarz inequality a T b a b for all a, b R n moreover, equality a T b = a b holds if: a = 0 or b = 0; in this case a T b = 0 = a b a 0 and b 0, and b = γa for some γ > 0; in this case 0 < a T b = γ a 2 = a b a 0 and b 0, and b = γa for some γ > 0; in this case 0 > a T b = γ a 2 = a b Norm, distance, angle 2-4

5 Proof of Cauchy-Schwarz inequality 1. trivial if a = 0 or b = 0 2. assume a = b = 1; we show that 1 a T b 1 0 a b 2 = (a b) T (a b) = a 2 2a T b + b 2 = 2(1 a T b) with equality only if a = b 0 a + b 2 = (a + b) T (a + b) = a 2 + 2a T b + b 2 = 2(1 + a T b) with equality only if a = b 3. for general nonzero a, b, apply case 2 to the unit-norm vectors 1 a a, 1 b b Norm, distance, angle 2-5

6 Average and RMS value let a be a real n-vector the average of the elements of a is avg(a) = a 1 + a a n n = 1T a n the root-mean-square value is the root of the average squared entry rms(a) = a a a2 n n = a n Exercises show that avg(a) rms(a) show that average of b = ( a 1, a 2,..., a n ) satisfies avg(b) rms(a) Norm, distance, angle 2-6

7 Triangle inequality from Cauchy-Schwarz inequality for vectors a, b of equal size a + b 2 = (a + b) T (a + b) = a T a + b T a + a T b + b T b = a 2 + 2a T b + b 2 a a b + b 2 (by Cauchy-Schwarz) = ( a + b ) 2 taking squareroots gives the triangle inequality triangle inequality is an equality if and only if a T b = a b (see page 2-4) also note from line 3 that a + b 2 = a 2 + b 2 if a T b = 0 Norm, distance, angle 2-7

8 Outline norm distance k-means algorithm angle hyperplanes complex vectors

9 Distance the (Euclidean) distance between vectors a and b is defined as a b a b 0 for all a, b and a b = 0 only if a = b triangle inequality a c a b + b c for all a, b, c c a c b c a a b b RMS deviation between n-vectors a and b is rms(a b) = a b n Norm, distance, angle 2-8

10 Standard deviation let a be a real n-vector the de-meaned vector is the vector of deviations from the average a avg(a)1 = a 1 avg(a) a 2 avg(a). a n avg(a) = a 1 (1 T a)/n a 2 (1 T a)/n.. a n (1 T a)/n the standard deviation is the RMS deviation from the average std(a) = rms(a avg(a)1) = a ((1 T a)/n)1 n the de-meaned vector in standard units is 1 (a avg(a)1) std(a) Norm, distance, angle 2-9

11 Mean return and risk of investment vectors represent time series of returns on an investment (as a percentage) average value is (mean) return of the investment standard deviation measures variation around the mean, i.e., risk ak 0 bk 0 ck 0 dk k k k k (mean) return a b c d risk Norm, distance, angle 2-10

12 Exercise show that avg(a) 2 + std(a) 2 = rms(a) 2 Solution std(a) 2 a avg(a)1 2 = n = 1 ( ) T ( ) a 1T a n n 1 a 1T a n 1 ( = 1 a T a (1T a) 2 (1T a) 2 ( 1 T ) 2 a + n) n n n n = 1 (a T a (1T a) 2 ) n n = rms(a) 2 avg(a) 2 Norm, distance, angle 2-11

13 Exercise: nearest scalar multiple given two vectors a, b R n, with a 0, find scalar multiple ta closes to b b ˆta line {ta t R} Solution squared distance between ta and b is ta b 2 = (ta b) T (ta b) = t 2 a T a 2ta T b + b T b a quadratic function of t with positive leading coefficient a T a derivative with respect to t is zero for ˆt = at b a T a = at b a 2 Norm, distance, angle 2-12

14 Exercise: average of collection of vectors given N vectors x 1,..., x N R n, find the n-vector z that minimizes z x z x z x N 2 x 4 x 3 x 5 z x 1 x 2 z is also known as the centroid of the points x 1,..., x N Norm, distance, angle 2-13

15 Solution: sum of squared distances is z x z x z x N 2 n ( = (zi (x 1 ) i ) 2 + (z i (x 2 ) i ) (z i (x N ) i ) 2) = i=1 n ( Nz 2 i 2z i ((x 1 ) i + (x 2 ) i + + (x N ) i ) + (x 1 ) 2 i + + (x N ) 2 ) i i=1 here (x j ) i is ith element of the vector x j term i in the sum is minimized by z i = 1 N ((x 1) i + (x 2 ) i + + (x N ) i ) solution z is component-wise average of the points x 1,..., x N : z = 1 N (x 1 + x x N ) Norm, distance, angle 2-14

16 Outline norm distance k-means algorithm angle hyperplanes complex vectors

17 k-means clustering a popular iterative algorithm for partitioning N vectors x 1,..., x N in k clusters Norm, distance, angle 2-15

18 Algorithm choose initial representatives z 1,..., z k for the k groups and repeat: 1. assign each vector x i to the nearest group representative z j 2. set the representative z j to the mean of the vectors assigned to it as a variation, choose a random initial partition and start with step 2 initial representatives are often chosen randomly solution depends on choice of initial representatives or partition can be shown to converge in a finite number of iterations in practice, often restarted a few times, with different starting points Norm, distance, angle 2-16

19 Example: first iteration assignment to groups updated representatives Norm, distance, angle 2-17

20 Example: iteration 2 assignment to groups updated representatives Norm, distance, angle 2-18

21 Example: iteration 3 assignment to groups updated representatives Norm, distance, angle 2-19

22 Example: iteration 9 assignment to groups updated representatives Norm, distance, angle 2-20

23 Example: iteration 10 assignment to groups updated representatives Norm, distance, angle 2-21

24 Example: iteration 11 assignment to groups updated representatives Norm, distance, angle 2-22

25 Example: iteration 12 assignment to groups updated representatives Norm, distance, angle 2-23

26 Image clustering MNIST dataset of handwritten digits N = 60, 000 grayscale images of size (vectors x i of size 28 2 = 784) 25 examples: Norm, distance, angle 2-24

27 Group representatives (k = 20) k-means algorithm, with k = 20 and randomly chosen initial partition 20 group representatives Norm, distance, angle 2-25

28 Group representatives (k = 20) result for another initial partition Norm, distance, angle 2-26

29 Document topic discovery N = 500 Wikipedia articles, from weekly most popular lists (9/2015 6/2016) dictionary of 4423 words each article represented by a word histogram vector of size 4423 result of k-means algorithm with k = 9 and randomly chosen initial partition Cluster 1 largest coefficients in cluster representative z 1 word fight win event champion fighter... coefficient documents in cluster 1 closest to representative Floyd Mayweather, Jr, Kimbo Slice, Ronda Rousey, José Aldo, Joe Frazier,... Norm, distance, angle 2-27

30 Cluster 2 largest coefficients in cluster representative z 2 word holiday celebrate festival celebration calendar... coefficient documents in cluster 2 closest to representative Halloween, Guy Fawkes Night, Diwali, Hannukah, Groundhog Day,... Cluster 3 largest coefficients in cluster representative z 3 word united family party president government... coefficient documents in cluster 3 closest to representative Mahatma Gandhi, Sigmund Freund, Carly Fiorina, Frederick Douglass, Marco Rubio,... Norm, distance, angle 2-28

31 Cluster 4 largest coefficients in cluster representative z 4 word album release song music single... coefficient documents in cluster 4 closest to representative David Bowie, Kanye West, Celine Dion, Kesha, Ariana Grande,... Cluster 5 largest coefficients in cluster representative z 5 word game season team win player... coefficient documents in cluster 5 closest to representative Kobe Bryant, Lamar Odom, Johan Cruyff, Yogi Berra, José Mourinho,... Norm, distance, angle 2-29

32 Cluster 6 largest coefficients in representative z 6 word series season episode character film... coefficient documents in cluster 6 closest to cluster representative The X-Files, Game of Thrones, House of Cards, Daredevil, Supergirl,... Cluster 7 largest coefficients in representative z 7 word match win championship team event... coefficient documents in cluster 7 closest to cluster representative Wrestlemania 32, Payback (2016), Survivor Series (2015), Royal Rumble (2016), Night of Champions (2015),... Norm, distance, angle 2-30

33 Cluster 8 largest coefficients in representative z 8 word film star role play series... coefficient documents in cluster 8 closest to cluster representative Ben Affleck, Johnny Depp, Maureen O Hara, Kate Beckinsale, Leonardo DiCaprio,... Cluster 9 largest coefficients in representative z 9 word film million release star character... coefficient documents in cluster 9 closest to cluster representative Star Wars: The Force Awakens, Star Wars Episode I: The Phantom Menace, The Martian (film), The Revenant (2015 film), The Hateful Eight,... Norm, distance, angle 2-31

34 Outline norm distance k-means algorithm angle hyperplanes complex vectors

35 Angle between vectors the angle between nonzero real vectors a, b is defined as arccos ( a T b ) a b this is the unique value of θ [0, π] that satisfies a T b = a b cos θ b θ a Cauchy-Schwarz inequality guarantees that 1 at b a b 1 Norm, distance, angle 2-32

36 Terminology θ = 0 a T b = a b vectors are aligned or parallel 0 θ < π/2 a T b > 0 vectors make an acute angle θ = π/2 a T b = 0 vectors are orthogonal (a b) π/2 < θ π a T b < 0 vectors make an obtuse angle θ = π a T b = a b vectors are anti-aligned or opposed Norm, distance, angle 2-33

37 Orthogonal decomposition given a nonzero a R n, every n-vector x can be decomposed as x = ta + y with y a y x ta t = at x a 2, y = x at x a 2a proof is by inspection decomposition (i.e., t and y) exists and is unique for every x ta is projection of x on the line through a and the origin (see page 2-12) since y a, we have x 2 = ta 2 + y 2 Norm, distance, angle 2-34

38 Correlation coefficient the correlation coefficient between non-constant vectors a, b is ρ ab = ãt b ã b where ã = a avg(a)1 and b = b avg(b)1 are the de-meaned vectors only defined when a and b are not constant (ã 0 and b 0) ρ ab is the cosine of the angle between the de-meaned vectors a number between 1 and 1 ρ ab is the average product of the deviations from the mean in standard units ρ ab = 1 n n i=1 (a i avg(a)) std(a) (b i avg(b)) std(b) Norm, distance, angle 2-35

39 Examples a k b k b k ρ ab = k k a k a k b k b k ρ ab = k k a k a k b k b k ρ ab = k k a k Norm, distance, angle 2-36

40 Regression line scatter plot shows two n-vectors a, b as n points (a k, b k ) straight line shows affine function f(x) = c 1 + c 2 x with f(a k ) b k, k = 1,..., n f(x) x Norm, distance, angle 2-37

41 Least squares regression use coefficients c 1, c 2 that minimize J = 1 n n (f(a k ) b k ) 2 k=1 J is a quadratic function of c 1 and c 2 : J = 1 n n (c 1 + c 2 a k b k ) 2 k=1 = ( nc n avg(a)c 1 c 2 + a 2 c 2 2 2n avg(b)c 1 2a T bc 2 + b 2) /n to minimize J, set derivatives with respect to c 1, c 2 to zero: c 1 + avg(a)c 2 = avg(b), n avg(a)c 1 + a 2 c 2 = a T b solution is c 2 = at b n avg(a) avg(b) a 2 n avg(a) 2, c 1 = avg(b) avg(a)c 2 Norm, distance, angle 2-38

42 Interpretation slope c 2 can be written in terms of correlation coefficient of a and b: c 2 = (a avg(a)1)t (b avg(b)1) a avg(a)1 2 = ρ ab std(b) std(a) hence, expression for regression line can be written as f(x) = avg(b) + ρ ab std(b) (x avg(a)) std(a) correlation coefficient ρ ab is the slope after converting to standard units: f(x) avg(b) std(b) = ρ ab x avg(a) std(a) Norm, distance, angle 2-39

43 Examples ρ ab = 0.91 ρ ab = 0.89 ρ ab = 0.25 dashed lines in top row show average ± standard deviation bottom row shows scatter plots of top row in standard units Norm, distance, angle 2-40

44 Outline norm distance k-means algorithm angle hyperplanes complex vectors

45 Hyperplane one linear equation in n variables x 1, x 2,..., x n : in vector notation: a T x = b a 1 x 1 + a 2 x a n x n = b let H be the set of solutions: H = {x R n a T x = b} H is empty if a 1 = a 2 = = a n = 0 and b 0 H = R n if a 1 = a 2 = = a n = 0 and b = 0 H is called a hyperplane if a = (a 1, a 2,..., a n ) 0 for n = 2, a straight line in a plane; for n = 3, a plane in 3-D space,... Norm, distance, angle 2-41

46 Example b = 5 x 2 b = 10 a = (2, 1) b = 15 x 1 b = 15 b = 10 b = 0 b = 5 Norm, distance, angle 2-42

47 Geometric interpretation of hyperplane recall formula for orthogonal decomposition of x with respect to a (page 2-34): x = at x a 2a + y with y a H x satisfies a T x = b if and only if y x x = b a 2a + y with y a (b/ a 2 )a point (b/ a 2 )a is the intersection of hyperplane with line through a add arbitrary vectors y a to get all other points in hyperplane Norm, distance, angle 2-43

48 Exercise: projection on hyperplane show that the point in H = {x a T x = b} closest to c R n is ˆx = c + b at c a 2 a show that the distance of c to the hyperplane H = {x a T x = b} is a T c b a Norm, distance, angle 2-44

49 Solution a T c a 2a c H = {x a T x = b} b a 2a y ˆx line through a and origin ˆx = c + b at c a 2 a Norm, distance, angle 2-45

50 Solution general point x in H is x = b a 2a + y, y a decomposition of c with respect to a is c = at c a 2a + d with d = c at c a 2a squared distance between x and c is c x 2 = a T 2 c b a T a a + d y = (at c b) 2 a 2 + d y 2 (2nd step because d y a); distance is minimized by choosing y = d Norm, distance, angle 2-46

51 Kaczmarz algorithm Problem: find (one) solution of set of linear equations a T 1 x = b 1, a T 2 x = b 2,..., a T mx = b m here a 1, a 2,..., a m are nonzero n-vectors we assume the equations are solvable (have at least one solution) n is huge, so we can only use simple vector operations Algorithm: start at some initial x and repeat the following steps pick an index i {1,..., m}, for example, cyclically or randomly replace x with projection on hyperplane H i = { x a T i x = b i} x := x + b i a T i x a i 2 a i Norm, distance, angle 2-47

52 Tomography reconstruct unknown image from line integrals ray i a ij pixel j x represents unknown image with n pixels a ij is length of intersection of ray i and pixel j n b i is a measurement of the line integral a ij x j along ray i j=1 Kaczmarz alg. is also known as Algebraic Reconstruction Technique (ART) Norm, distance, angle 2-48

53 Outline norm distance k-means algorithm angle hyperplanes complex vectors

54 Norm norm of vector a C n : a = a a a n 2 = a H a positive definite: a 0 for all a, a = 0 only if a = 0 homogeneous: βa = β a for all vectors a, complex scalars β triangle inequality: a + b a + b for all vectors a, b of equal size Norm, distance, angle 2-49

55 Cauchy-Schwarz inequality for complex vectors a H b a b for all a, b C n moreover, equality a H b = a b holds if: a = 0 or b = 0 a 0 and b 0, and b = γa for some (complex) scalar γ exercise: generalize proof for real vectors on page 2-4 we say a and b are orthogonal if a H b = 0 we will not need definition of angle, correlation coefficient,... in C n Norm, distance, angle 2-50