Finite Dimensional Space Orthogonal Projection Operators

Transcription

1 Finite Dimensional Space Orthogonal Projection Operators Robert M. Haralick Computer Science, Graduate Center City University of New York

2 Spaces Spaces have points called vectors Spaces have sets of points Some sets are called subspaces Spaces have directions Spaces have sets of directions Spaces have a language of representing points in terms of traveling different lengths in different directions

3 Vector Spaces Definition A space V is a vector space over a field of scalars S if and only if x V and y V implies x + y V x + y = y + x x + (y + z) = (x + y) + z There exists 0 V satisfying x + 0 = x for every x V x V implies there exists a unique y V such that x + y = 0 If x V and α S, then αx V If α, β S and x V, then α(βx) = (αβ)x There exists a scalar 1 S x V implies 1x = x α(x + y) = αx + αy (α + β)x = αx + βx

4 Language of Spaces The words of the language are the basis elements b 1, b 2,..., b N, b n = 1, n = 1,..., N The basis elements specify independent directions The sentence takes the form N n=1 α nb n The meaning of the sentence is Begin at the origin Go α 1 in direction b 1 Go α 2 in direction b 2... Go α N in direction b N And you arrive at the point represented by (α 1,..., α N ) The set of all places that can be reached by such a sentence is called the space spanned by the directions b 1,..., b N The interesting sentences are the minimal ones

5 Linear Independence Minimal sentence means using independent directions. Definition b 1,..., b N are independent directions (linearly independent) when α n b n = 0 if and only if α n = 0, n = 1,..., N n=1 If you travel α 1 in direction b 1, then travel α 2 in direction b 2,..., then travel α N in direction b N and you return to the origin, then the directions are dependent.

6 Linear Dependence Definition b 1,..., b N are linearly dependent if and only if for some α 1,..., α N, not all 0 α n b n = 0 n=1 If you travel α 1 in direction b 1, then travel α 2 in direction b 2,..., then travel α N in direction b N and you return to the origin, then the directions are dependent.

7 Finite Dimensional Vectors and Transpose x = x 1 x 2. x N y = y 1 y 2. y N x y = xy = x n y n n=1 x 1 y 1 x 1 y 2... x 1 y N x 2 y 1 x 2 y 2... x 2 y N. x N y 1 x N y 2... x N y N

8 Angles b i b j cosine of the angle between directions b i and b j b i b j = b j b i b i b i = squared length b i b i = b i 2 b (c + d) = b c + b d (αb) c = α(b c) b i b j means geometrically orthogonal b i b j if and only if b i b j = 0 b 1,..., b N is orthonormal if and only if b i b j = 0 when i j b i = 1

9 Lengths Definition The length of a vector x is its distance from the origin. x = x x Let x = N n=1 α nb n and b 1,..., b N be orthonormal x 2 = = = α i b i 2 = ( α i b i ) i=1 i=1 N α i b i ( α j b j ) = i=1 j=1 αi 2 b i b i + i=1 i=1 N j=1 i=1 j=1 α j b j α i α j b i b j α i α j b i b j = j=1 j i i=1 α 2 i

10 Coordinate Representation Let b 1,..., b N be an orthonormal basis Let x be a vector Find α 1,..., α N such that x = N n=1 α nb n Suppose x = N n=1 α nb n. N b i x = b i ( α n b n ) = n=1 α n b i b n n=1 = α i b i b i = α i x = (α 1,..., α N ) with respect to basis b 1,..., b n Change the basis and you change the coordinate representation.

11 Inner Product Let x = (α 1,..., α N ), y = (β 1,..., β N ), be coordinates with respect to orthonormal basis b 1,..., b N x y = ( α i b i ) ( β j b j ) = = = i=1 j=1 N α i b i ( β j b j ) = i=1 j=1 α i β i b i b i + i=1 α i β i i=1 i=1 i=1 j=1 α i β j b i b j j=1 j i α i β j b i b j

12 Dimensionality N: Dimension of Space Number of directions required in a minimal sentence to specify (reach) any point in the space Number of degrees of freedom needed to represent a point in the space {x x = N n=1 α nb n } M: Dimension of Subspace Number of directions required in a minimal sentence to specify (reach) any point in the subspace Number of degrees of freedom needed to represent a point in the subspace {x x = M β mb m } {x x = M β mb m + N m=m+1 0b m} M degrees of freedom; N M degrees of constraint

13 Constraints M: Dimension of Subspace Number of directions required in a minimal sentence to specify (reach) any point in the subspace Number of degrees of freedom needed to represent a point in the subspace {x x = M β mb m } {x x = M β mb m + N m=m+1 0b m} M degrees of freedom; N M degrees of constraint Let i {M + 1,..., N} and b 1,..., b N be orthonormal. Consider b i x b i x = b i = M β m b m = M β m 0 = 0 M b i β mb m = M β m b i b m

14 Co-Dimension M: Dimension of Subspace Number of directions required in a minimal sentence to specify (reach) any point in the subspace Number of degrees of freedom needed to represent a point in the subspace {x x = M β mb m } {x x = M β mb m + N m=m+1 0b m} M degrees of freedom N M degrees of constraint N M Co-dimension N M Constraints Let i {M + 1,..., N} and b 1,..., b N be orthonormal. b i x = 0, i {M + 1,..., N}

15 Basis Vectors Let b 1,..., b N be an orthonormal basis for a space S. Each b n is a direction The length of each b n is one A direction and a length represents a point or vector in the space b i b j, i j b n is a point or vector in the space

16 Representing Subspaces 1-Dimensional Space b 2 b 1 2-Dimensional Space {x for some α 1, x = α 1 b 1 } {x b 2 x = 0}

17 Representing Subspaces N Dimensional Space S M Dimensional Subspace T b 1,..., b N orthonormal basis S = {x for some α 1,..., α N, x = N n=1 α nb n } M degrees of freedom T = {x for some α 1,..., α M, x = M α mb m } N M degrees of constraint T = {x b i x = 0, i {M + 1,..., N}}

18 Orthogonal Subspaces Definition A subspace T is orthogonal to a subspace U if and only if t T and u U implies t u = 0 Definition Let T be a subspace of S. The orthogonal complement of T, denoted by T, is defined by T = {x S for every t T, x t = 0}

19 Orthogonal Subspaces Proposition Let b 1,..., b N be an orthogonal basis of S. Let V be a subspace of S spanned by b 1,..., b M. Then V is the subspace spanned by b M+1..., b N Proof. V = {x S v V implies x v = 0} M x v = x α mb m = ( β nb n) = n=1 β n M n=1 α mb nb m = M M α mβ m α mb m M αmβm = 0 for all α 1,..., α M implies β 1 = 0,..., β M = 0 Therefore, V = {x x = β i b i } i=m+1

20 Orthogonal Representations Proposition Let V be a subspace of S and let x S. Then there exists a v V and w V such that x = v + w Proof. Let b 1,..., b N be an orthonormal basis for S such that b 1,..., b M is an orthonormal basis for V and b M+1,... b N is an orthonormal basis for V Then for some α 1,..., α N, x = α n b n = n=1 M α n b n + n=1 i=m+1 α i b i But v = M n=1 α nb n V and w = N i=m+1 α ib i V. Therefore x = v + w for v V and w V.

21 Orthogonal Projection Definition Let V be a subspace of S. Let x S and x = v + w where v V and w V. Then v is called the orthogonal projection of x onto V.

22 Orthogonal Projections are Unique Proposition Let V be a subspace of S. Let x S and x = v 1 + w 1 = v 2 + w 2 where v 1, v 2 V and w 1, w 2 V. Then v 1 = v 2. Proof. Let b 1,..., b M be an orthonormal basis for V. Then v 1 = M α mb m and v 2 = M β mb m. b i x = b i (v 1 + w 1 ) = b i = b i (v 2 + w 2 ) = b i Therefore, α i = β i, i = 1,..., M M α m b m = α i M β m b m = β i

23 Orthogonal Projection Operator Proposition Let V be an M dimensional subspace of S. Let x S and x = v + w where v V and w V. Let b 1,..., b N be an orthonormal basis of S and b 1,..., b M be an orthonormal basis of V. Then v = Px where P = M b mb m. Proof. x S implies x = N n=1 β nb n = M β mb m + N n=m+1 β nb n. Then Now, v = b mx = b m β n b n = n=1 M β m b m = = Px β n b mb n = β m n=1 M M (b mx)b m = ( b m b m)x

24 Orthogonal Projection Operator Proposition Let b 1,..., b N be an orthonormal basis S and b 1,..., b M an orthonormal basis for the subspace V of S. Then P = M b mb m is the orthogonal projection operator to V.

25 Orthogonal Projection Operators Proposition If P is an orthogonal projection operator to the subspace V of S, then P 2 = P P = P Proof. Let b 1,..., b M be an orthonormal basis for V. Then, P 2 = = M M b mb m b i b i M b m M i=1 i=1 (b mb i )b i = M b mb m = P M M P = ( b mb m) = (b mb m) M = b mb m = P

26 Uniqueness Proposition Suppose P = P 2, P = P, Q = Q 2, and Q = Q. If PQ = Q and QP = P, then Q = P. Proof. Q = PQ = (PQ) = Q P = QP = P

27 Orthogonal Projection Operators are Unique Proposition Let V be a M dimensional subspace of S. Let b 1,..., b M be one orthonormal basis for V and let c 1,..., c M be another orthonormal basis for V. Define P = M b mb m and Q = M c mc m. Then Q = P. Proof. By the definition of orthogonal projection operators, both P and Q are orthogonal projection operators onto V. Hence, P = P 2 and P = P. Likewise, Q = Q 2 and Q = Q. Since the columns of P and Q are in V, PQ = Q and QP = P By the uniqueness proposition, Q = P.

28 Orthogonal Projection Operator Characterization Theorem Theorem If P = P 2 and P = P, then P is the orthogonal projection operator onto Col(P). Proof. Let b 1,..., b M be an orthonormal basis for Col(P). Define Q = M b mb m. Then Q = Q 2 and Q = Q. Clearly, Col(Q) = Col(P) so that QP = P and PQ = Q. By the uniqueness proposition, P = Q. And since Q is the orthogonal projection operator onto Col(P), P must also be the orthogonal projection operator onto Col(P).

29 Projection Operators Definition P is called a projection operator if and only if P 2 = P ( ) ( ) ( ) = ( ) ( ) ( ) = ( ) ( ) ( ) =

30 Trace Definition Let A = (a ij ) be a square N N matrix. Proposition Trace(A) = n=1 a nn Trace( N n=1 α na n ) = N n=1 α ntrace(a n )

31 Trace Proposition Proof. Trace(AB) = Trace(BA) Let C N N = (c ij ) = A N K B K N and D K K = (d mn) = B K N A N K. c ij = d mn = Trace(C) = = K a ik b kj k=1 b mi a in i=1 c ii = i=1 K a ik b ki i=1 k=1 K K b ki a ik = d kk = Trace(D) = Trace(BA) k=1 i=1 k=1

32 Trace Corollary x Ax = Trace(Axx ) Proof. x Ax = Trace(x Ax) = Trace(x (Ax)) = Trace((Ax)x ) = Trace(Axx )

33 Trace Proposition Let A = (a ij ) be a M N matrix. Then M amn 2 = Trace(AA ) n=1 Proof. Let B = (b ij ) = AA. Then b ij = N n=1 a ina jn. Hence, b ii = N n=1 a ina in = N n=1 a2 in. Therefore Trace(B) = Trace(AA ) = M b mm = M N n=1 a2 mn

34 Trace Proposition Let P be an orthogonal projection operator to the M dimensional subspace V. Then Trace(P) = M Proof. Let b 1,..., b M be an orthonormal basis for V. Then P = M b mb m Trace(P) = Trace( = = M b m b m) M Trace(b m b m) = M Trace(1) = M Trace(b mb m ) M 1 = M

35 Trace Proposition Let P be an orthogonal projection operator onto a M dimensional subspace. Then i=1 j=1 pij 2 = M Proof. pij 2 = Trace(PP ) = Trace(PP) = Trace(P) = M i=1 j=1

36 Kernel and Range Definition The Kernel of a matrix operator A is Kernel(A) = {x Ax = 0} The Range of a matrix operator A is Range(A) = {y for some x, y = Ax}

37 Kernel and Range Proposition Let P be a projection operator onto subspace V of S. Then Range(P) + Ker(P) = S Proof. Let x S. Px + (I P)x = Px + x Px = x. Certainly Px Range(P). Consider (I P)x. P[(I P)x] = Px PPx = Px Px = 0 Therefore, by definition of Kernel(P), (I P)x Kernel(P).

38 Kernel and Range Proposition Let P be an orthogonal projection operator. Then Range(P) Kernel(P) Proof. Let x Range(P) and y Kernel(P). Then for some u, x = Pu. Consider x y. x y = (Pu) y = u P y = u Py But y Kernel(P) so that Py=0. Therefore x y = 0.

39 Projecting Kernel(P) Px Range(P) x b 2 b 1 P = ( )

40 Proposition Let P be the orthogonal projection operator onto the subspace V. Then I P is the orthogonal projection operator onto the subspace V. Proof. (I P)(I P) = I P P + P 2 = I 2P + P = I P (I P) = I P = I P V = Kernel(P). Let x V. Then Px = 0. Consider (I P)x = x Px = x

41 Orthogonal Projection Minimizes Error Theorem Let V be a subspace of S. Let f : S V and x S. min(x f (x)) (x f (x)) f is achieved when f is the orthogonal projection operator from S to V Proof. Let x S. Then there exists v V and w V such that x = v + w. Consider ɛ 2 = (x f (x)) (x f (x)) = x x (v + w) f (x) f (x) (v + w) + f (x) f (x) = x x v f (x) f (x) v f (x) f (x) = (v + w) (v + w) v f (x) f (x) v f (x) f (x) = v v v f (x) f (x) v f (x) f (x) + w w = (v f (x)) (v f (x)) + w w ɛ 2 is minimized by making f (x) = v, the orthogonal projection of x onto V.

42 Constructing an Orthogonal Projection Operator Proposition Let X Z N be of full rank. Then the orthogonal projection operator onto the subspace col(x) is given by X(X X) 1 X. Proof. By definition col(x) is spanned by the columns of X. Notice that X(X X) 1 X operating on X produces X. X(X X) 1 X X = X[(X X) 1 (X X)] = X Hence anything in col(x) will be a fixed point of X(X X) 1 X. Furthermore, for any z R N, X(X X) 1 X z = X[(X X) 1 X z] col(x). Therefore, X(X X) 1 X maps onto col(x). Now we check that X(X X) 1 X satisfies the definition of orthogonal projection operator. [X(X X) 1 X ][X(X X) 1 X ] = X[(X X) 1 X X](X X) 1 X = X(X X) 1 X [X(X X) 1 X ] = X(X X) 1 X