Geometric interpretation of signals: background

Transcription

1 Geometric interpretation of signals: background David G. Messerschmitt Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No. UCB/EECS June 7, 006

2 Copyright 006, by the author(s). All rights reserved. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission.

3 Geometric interpretation of signals: background by David G. Messerschmitt University of California at Berkeley Version.3, February 9, 005 Copyright David G. Messerschmitt, 004. This work is licensed under the following Creative Commons license: Attribution-NonCommercial-ShareAlike The opportunity A number of problems in signal processing can be formulated within a common geometric framework. This offers several related opportunities: to observe commonalities among seemingly distinct contexts, to contribute to intuition through geometric reasoning, and to quickly identify solutions to common problems. Vector spaces Let I be a field. For our purposes, there are two fields of interest, R, the field of real numbers, and, the field of complex numbers. In the following we will consistently assume that I ; that is, complex-valued scalar fields. A vector space Õ is a set upon which two binary operations are defined, addition of two vectors ( + ) and multiplication of a vector by a scalar ( ). Specifically, +:ÕäÕ Õ and : äõ Õ must satisfy, for all U, V, W Õ and for all α, β : ( U + V ) + W U + ( V + W ) U + V V + U There exists a 0 Õ such that U + 0 U There exists a ( U ) Õ such that U + ( U ) 0 α ( β U ) ( αβ ) U U U α ( U + V ) α U + α V ( α + β ) U α U + β U Example. Let n be the space of n-dimensional complex-valued vectors under the normal rules of linear algebra. That is, each column vector of dimension n is associated with a vector,

4 z z Z... z n and multiplication by a scalar can be defined as α z α z α Z... α z n A finite-time discrete-time complex-valued signal can thus be modeled as a vector in space n, which is a linear space. It is important to note the notation, in which Z is a vector, and the operator associates that vector with a mathematical object, such as a column vector or discretetime signal. Subspaces A subspace Ì of a vector space Õ is a subset Ì Õ which is itself a vector space, and hence is closed with respect to all vector space operations. Example: The easiest way to define a subspace is to choose a set of linearlyindependent basis vectors, and then define the subspace as all linear combinations of those vectors. Consider 3 and define two vectors 0 Z 0 and Z. 0 0 Then Ì { Z : Z Z + β Z } subspace of 3. Metric, sequences, and convergence α for any scalars α, β is a two-dimensional

5 A metric d( U, V ) has the interpretation of a distance between vectors U and V ; thus, it adds a geometric interpretation to a vector space. Specifically, for vector space Õ, a metric d (, ) : ÕµÕ R (note that a metric is real-valued) must satisfy, for all U,V Õ, d ( U, V ) 0 iff U V d ( U, V ) d( V, U ) Triangle inequality: d( U, V ) + d( V, W ) d( U, W ) A metric d (, ) imposes a set of topological properties, such as open and closed sets and the convergence of sequences of vectors. A sequence { U k },k,,..., is a Cauchy sequence when for everyε > 0 there exists an N such that d ( U m, U n) < ε for all m, n > N. A sequence { U k },k,,..., is convergent to vector U when for every ε > 0 there exists an N such that d( U m, U ) < ε for all m > N. Every convergent sequence is a Cauchy sequence, but not the reverse. Example: Consider a space x (0,] under the Euclidean metric (this is a metric space, but not a vector space) and the sequence x n. This is a Cauchy n sequence which does not converge to an element within the space, because the vector{ 0} is missing. A metric space is said to be complete when every Cauchy sequence of vectors in that space is convergent. Normed spaces 3

6 Let Í be a vector space. A norm has the interpretation as the length of a vector. Specifically, : Í R (note that a norm is real-valued) has the properties, for all U, V Í and all α : V 0 with equality iff V 0 α V α V Triangle inequality: U + V U + V A normed space is the pair (Í, ), a linear space plus a norm defined on that space. Example. Consider n as defined earlier. If Z Z for a column vector Z, then a valid norm is T Z Z Z where T Z is the matrix transpose of Z. A norm induces a metric through the relation d( U, V ) U V. (This is easily verified from the definitions.) Thus, any normed space possesses all the topological properties of a metric space (including Cauchy and convergent sequences). A normed space that is complete (with respect to its induced metric) is called a Banach space. Inner-product spaces 4

7 Let È be a vector space. An inner product is a type of multiplication operator on two vectors. Specifically, : È µ È (note that an inner product is complex-valued) has the properties, for all U, V È and all α, U V + W U V + α U V U V α U ( V U ) U W U U 0 with equality iff U 0 V From these properties, it is easily inferred that U + W V U V + W V and U α V α U V. U U is real valued (this follows from the fourth property, since ( U U ) U U ). An inner product space is a pair (È, defined on that space. ), a vector space together with an inner product Example. Let È n, the space of complex-valued column vectors of dimension n, and let H be an n n positive definite Hermitian matrix ( H T H ). Then for any U, V Õ, define an inner product U V H u It is easy to verify that this satisfies all the conditions to be an inner product, so (È, ) is an inner product space. (The subscript H can be used to eliminate any confusion over which inner product is in play.) The following result has wide applicability to optimization over inner product spaces. T Hv Theorem (Schwarz inequality): For an inner product space (È, arbitrary vectors X, Y È, X Y X Y ) and with equality iff X α Y for some scalar α. Proof: X α Y X + α α Y α X Y α ( X Y ) 0 5

8 with equality iff X α Y. This inequality is true for any scalarα, but it conveys the most information if we choose α to minimize the left side. Differentiating this expression w.r.t. α and setting to zero, a stationary point is at X Y α. Y Substituting this α into the inequality, we get X Y X 0. Y Theorem. An inner product induces a norm through the relation U U U. Proof: That the triangle inequality is satisfied is the only non-trivial step in establishing that U defined in this way is a norm. Note that for every X, Y È, { X Y } X + Y + X Y X + Y X + Y + Re The inequality follows from an inequality for complex variables, + i y x + y x or x i y x x x +. Invoking the Schwarz inequality, to further upper bound the third term, X + Y X + Y + X Y X + Y. ( ) Thus, every inner product space is also a normed space and a metric space. An inner product space that is complete under the induced metric is called a Hilbert space. Every Hilbert space is implicitly a Banach space. In signal processing, we call a Hilbert space of possible signals a signal space. The following theorem has numerous applications to signal processing. 6

9 Projection theorem: Let Hilbert space Ç have a closed subspace M and let X Ç but X M. Then there exists a unique P P( X : M ) M with the following two equivalent properties: Closeness: X P X Y for everyy M Orthogonality principle: X P Y 0 for every Y M In words, P ( X : M ) is called the projection of X on M, and that projection is the vector within M that is closest to X. Also, the vector X P is orthogonal to M (meaning that it is orthogonal to every vector in M, including P ). In signal processing, P is often interpreted as an best estimate or approximation to X in subspace M with respect to metric, and thus X P is an error vector and the magnitude of the error is X P. The orthogonality principle restated: for the optimum estimate of X based upon a vector in M, the error vector is orthogonal to the subspace M (orthogonal to every vector in M). A convenient relation for the norm of this error vector follows from the Pythagorean theorem, or X X P + P X P + P (because X P P 0 ) X P X P. Partial proof of projection theorem: Existence: This can be done by constructing a Cauchy sequence that converges to the projection and invoking completeness (details omitted). Uniqueness: Assume that two vectors Y M and Y M both have orthogonal errors, X Y Y X Y Y 0 for all Y M. By linearity of the inner product, it follows that X Y X + Y Y Y Y Y 0. Since ( Y Y ) M, this must be true for Y ( Y Y ) Y ( Y Y ) 0, or Y 0. Closeness property orthogonality principle: Assume that X P X Y for every Y M. 7

10 Lettingα be a scalar, it follows, since or X P X P P M that for every α and Y M X P X P α Y { X P Y } + α Y Re α. Letting α β X P Y for any real-valued β, this becomes 0 β Y β X P Y Since for β the first term is negative, it must be that Y X P Y 0. Orthogonality principle closeness property: Suppose X P Y 0 for every Y M. Then X Y X P + P Y Expanding this term by term, X Y Since by assumption must be zero. Thus End of proof. X Y X P Given N linearly independent vectors + P Y P M and M X P X P + P Y + Re X P + P Y { X P P Y } Y and hence ( P Y ) M, the third term + P Y X P., X X N, an N -dimensional subspace M is X,..., defined by all linear combinations of these vectors. Use the notation M { X i, i N} to denote this subspace. It is more convenient to have a set of N orthogonal vectors, Y Y N that also spans this subspace; that is, M { Y i i N} Y,...,, and Y 0 for i j. (A set of orthogonal vectors is necessarily linearly independent.) The Y, Y,..., YN is called an orthogonal basis for M, and it is easily generated by a Gram-Schmidt orthogonalization procedure. Actually, this is a straightforward application of the projection theorem as follows. Define Y X and let Y : ) X Y { Y } X X P( X Y. Y. i Y j 8

11 which we know () has a component in the direction of X and () by the orthogonality principle we know that this Y is orthogonal to Y. Now we can proceed by induction, defining Y n in terms of Y Y,..., Y n,, n X n Y k { Y, Y,..., Y n } ) X n Y k Y n X n P X n ( :. k Y k The coefficients in this expansion have been determined by the orthogonality principle, n k X n α Y k Y j 0, j n and exploiting the orthogonality of Y Y,...,. k, Y n The following are important Hilbert spaces in signal processing applications. Details such as the exact meaning of integrals and proofs of completeness are left to the references. Example. Ç n is a Hilbert space with the appropriate definition of inner T product. Let H be an n n positive-definite Hermitian matrix ( H H ), and for H a vector U u let u denote the conjugate transpose of u. Then two (equivalent possibilities) for an inner product are to let vectors correspond to n (row) H matrices with U V uhv and to let vectors correspond to n (column) matrices with U V u T Hv. When I complex-valued Euclidean space. H (identity matrix), this is ordinary Example. Let å be the space of all double-infinite complex-valued time sequences of the form Z where the sequence has finite energy, {..., z( ), z(0), z(),... } n and with the inner product defined as U V z( n) < u( n) v ( n). Then å is a Hilbert space. When the limits of summation are restricted to a finite interval, this is reverts to the earlier example. Example. Let Ë be the space of all complex-valued continuous-time signals of the form Z ), where each signal has finite energy, { z( t < t < } 9

12 z( t) and with the inner product defined as dt < U V u( t) v ( t) dt. Then Ë is a Hilbert space. When the integrals are restricted to a finite or semiinfinite interval, this remains a Hilbert space. Ω be a probability space ( Ω is the sample space, I is the set of all events defined on that sample space, and 0 P is a probability measure defined over all events), and let Ë ( Ω) be the space of all complex-valued random variables Z with zero mean and finite second moments, E [ Z ] 0 and E [ Z ] < with a vector associated with each such random variable, Z Z and the inner product defined as U V E[ U V ]. Example. Let (,I, P) Then Ë ( Ω ) is a Hilbert space. References [] Luenberger, D. G. Optimization by Vector Space Methods. Wiley, 997. [] A. W. Naylor and G. R. Sell, Linear Operator theory in Engineering and Science, Springer Verlag, 98. 0