Ecole Polytechnique Federale de Lausanne (EPFL) Audio-Visual Communications Laboratory (LCAV) Fourier and Wavelet Signal Processing Martin Vetterli Amina Chebira, Ali Hormati Spring 2011 2/25/2011 1 Outline Vector spaces, Hilbert spaces Linear operators, Projection operators Orthogonal and biorthogonal bases, frames Matrix representations, expansions Goal : Set the basics in a Hilbert space setup through geometric intuition Readings: Chapter 1,, of Fourier and Wavelet Signal Processing 2/25/2011 2
For a vector space, we need: A set of vectors V These can be vectors in, functions, etc Think of geometry in or, we will use pictures! A field of scalars F Real or complex numbers Vector addition + Scalar multiplication Vector Spaces Easy case: N finite, linear algebra, matrices Beware: N goes to infinity convergence! 2/25/2011 3 Vector Spaces: Axioms A vector space is defined over a field (think or ) as a set with two operations Vector addition: Scalar multiplication: That satisfies the following axioms 2/25/2011 4
Subspace Span is a subspace when it is closed under vector addition and scalar multiplication: Vector Spaces: Key notions : set of vectors (could be infinite) = set of all finite linear combinations of vectors in is always a subspace 2/25/2011 5 Linear independence is linearly independent when: If then If is infinite, we need every finite subset to be linearly independent Dimension Vector Spaces: Key notions Dim(V) = N if V contains a linearly independent set with N vectors and every set with N+1 or more vectors is linearly dependent V is infinite dimensional if no such finite N exists 2/25/2011 6
Inner Products Formalize the geometric notions of orientation and orthogonality Measure similarity between vectors An inner product for V is a function satisfying 2/25/2011 7 Inner Products: Examples 2/25/2011 8
Geometry in Inner Product Spaces In : (true in general) 2/25/2011 9 Orthogonality and are orthogonal when,, written is orthogonal when for all we have is orthonormal when it is orthogonal and for all is orthogonal to when for all,written and are orthogonal when every is orthogonal to, written 2/25/2011 10
Orthogonal complement If is a subspace of, the orthogonal complement of is the set If is closed (contains all limits) then given, there exists s.t. 2/25/2011 11 Norm Measure length, size of vectors A norm on is a function satisfying 2/25/2011 12
Norms: Examples 2/25/2011 13 Distances, norms and inner products A norm induces a distance An inner product induces a norm Not all norms are induced by an inner product 2/25/2011 14
Norms induced by inner products: Properties Cauchy-Schwarz inequality Parallelogram Law 2/25/2011 15 Norms induced by inner products: Properties Pythagorean theorem 2/25/2011 16
Normed vector spaces: Standard spaces : : square-summable sequences ( finite energy sequences ) : square-integrable functions ( finite energy functions ) 2/25/2011 17 Normed vector spaces: Standard spaces spaces : The norm :, for spaces :, for Extend norm to norm as : the only norm induced by an inner product 2/25/2011 18
Normed vector spaces: Standard spaces spaces : Extend to : norm : only norm induced by an inner product 2/25/2011 19 Geometry of spaces 2/25/2011 20
Space of random variables Random variables with finite second moment Inner product and norm Apply all the abstract theorems to random variables. : inner product space of complex, continuous functions over interval : inner product space of complex, continuous functions with p-continuous derivatives over interval Usual inner product, usual norm Spaces Example: set of polynomial functions over an interval forms a subspace of, for any in and p in. Why: closed under vector space operations, and polynomials are indefinitely differentiable 2/25/2011 22
Hilbert spaces: Completeness A sequence is a Cauchy sequence in a normed space when for any, there exists for all A normed vector space V is complete if every Cauchy sequence converges in V A complete inner product space = Hilbert space A complete normed vector space = Banach space 2/25/2011 23 Hilbert spaces: Examples is not a Hilbert space because it is not complete is a Hilbert space 2/25/2011 24
Hilbert spaces: Examples All finite dimensional spaces are complete and are complete and are Hilbert spaces are not complete except under norm Vector space of random variables as already defined is a Hilbert space. 2/25/2011 25 Summary on spaces 2/25/2011 26
Linear operators is a linear operator when for all 1. Additivity: 2. Scalability: Null space (subspace of ): Range space (subspace of ): Operator norm: A is bounded when: Inverse: 2/25/2011 27 Adjoint Operators Generalizing Hermitian transpose of matrices is the adjoint of when If is self-adjoint or Hermitian Intuitively, if has an effect, preserves the geometry of that effect while acting on reversed domain. 2/25/2011 28
Adjoint operators: local averaging 2/25/2011 29 Adjoint operators: Properties Let be a bounded linear operator 1. exists and is unique 2. 3. and are self-adjoint 4. 5. If invertible, 6. bounded, 7. bounded, 2/25/2011 30
Unitary Operators bounded linear operator is unitary when: 1. is invertible 2. preserves inner products: for every If is unitary, then is unitary iff 2/25/2011 31 Best approximation closed subspace of a Hilbert space Best approximation problem 2/25/2011 32
Orthogonal projection Existence of Orthogonality: Uniqueness of Linearity: where is a linear operator Idempotency: Self-adjointness: Example Consider: Find the closest degree 1 polynomial Solution: use orthogonality
Projection operators is idempotent when A projection operator is a bounded linear operator that is idempotent An orthogonal projection operator is a self-adjoint projection operator An oblique projection operator is not self adjoint 2/25/2011 35 Orthogonal projection operators A bounded linear operator or satisfies iff is an orthogonal projection operator Example: Orthogonal projection onto 1D subspace If bounded and is a left inverse of, then is a projection operator. If then, is an orthogonal projection 2/25/2011 36
Projection operators: Example Local averaging example is identity, is orthogonal projection Projections and direct sums projection on H, then If closed subspaces s.t then there exists projection on H s.t and 2/25/2011 38
Summary Set the basics in a Hilbert space setup through geometric intuition Vector spaces, subspaces Norms, inner products Hilbert spaces Linear operators Projections