2. Review of Linear Algebra

Transcription

1 2. Review of Linear Algebra ECE 83, Spring 217 In this course we will represent signals as vectors and operators (e.g., filters, transforms, etc) as matrices. This lecture reviews basic concepts from linear algebra that will be useful. 1 / 27

2 Signal vectors Linear vector space A linear vector space X is a collection of elements satisfying the following properties: addition: x, y, z X 1. x + y X 2. x + y = y + x 3. (x + y) + z = x + (y + z) 4. X, such that x + = x 5. x X, x X such that x + ( x) = multiplication: x, y X and a, b R 1. ax X 2. a(bx) = (ab) x 3. 1x = x, x = 4. a(x + y) = ax + ay 2 / 27

3 Example: R n R n, the n-dimensional Euclidean space, is a linear vector space. Example: C n C n, the n-dimensional complex space, is a linear vector space. Example: L 2 ([, T ]) The space of finite energy signals on the interval [, T ] L 2 ([, T ]) := is a linear vector space. { x : T } x 2 (t) dt < 3 / 27

4 Inner Products Definition: Inner product An inner product is a mapping from X X to R. The inner product between any x, y X is denoted by x, y and it satisfies the following properties for all x, y, z X : 1. x, y = y, x 2. ax, y = a x, y for all scalars a 3. x + y, z = x, z + y, z 4. x, x and x, x = = x = A space X equipped with an inner product is called an inner product space. Roughly speaking, x, y measures the alignment or similarity of x and y. 4 / 27

5 Definition: Orthogonal vectors x and y are orthogonal vectors if x, y = Example: Euclidean space Let X = R n. Then x, y := x T y = n i=1 x iy i. Example: Complex space Let X = C n. Then x, y := x H y = n i=1 x i y i. Example: L 2 Let X = L 2 ([, 1]). Then x, y := 1 x(t)y(t) dt. 5 / 27

6 Norms Definition: Norm The inner product induces a norm defined as x := x, x. The norm measures the length/size of x. The inner product x, y = x y cos(θ), where θ is the angle between x and y. Cauchy-Schwarz Inequality For every x, y X we have x, y x y with equality if and only if x and y are linearly dependent or parallel ; i.e., θ =. Triangle inequality x y x z + z y 6 / 27

7 Homogeneity For x X and a a scalar ax = a x. Pythagorean theorem If x, y X and x, y =, then x 2 + y 2 = x + y 2 Parallelogram law If x, y X, x + y 2 + x y 2 = 2 x y 2 7 / 27

8 Hilbert spaces Definition: Hilbert space An inner product space that contains all its limits is called a Hilbert Space and in this case we often denote the space by H; i.e., if x 1, x 2,... are in H and lim n x n exists, then the limit is also in H. It is easy to verify that R n, L 2 ([, T ]), and l 2 (Z), the set of all finite energy sequences (e.g., discrete-time signals), are all Hilbert spaces. 8 / 27

9 Linear Independence Definition: Linear independence Consider a set of vectors x 1, x 2,..., x p X If there exists a set of numbers a 1, a 2,..., a p R such that not all are zero and a 1 x 1 + a 2 x a p x p = (1) then we say that the vectors are linearly dependent. If Eq. 1 only holds for the case a 1 = a 2 = = a p =, then the vectors are said to be linearly independent. Note that if Eq. 1 holds and a k then x k = 1 a i x i a k i k and x k can be expressed as a linear combination of the other vectors (hence the term dependent). 9 / 27

10 Bases Definition: Basis A set of vectors φ 1,..., φ n is a basis for X if every vector x X can be represented as a linear combination of {φ k } n k=1. That is, there exist numbers θ 1,..., θ n so that x = Definition: Orthonormal basis A basis {φ i } is orthonormal if n θ k φ k k=1 φ i φ j = δ i,j = { 1, i = j, i j 1 / 27

11 Orthobasis of Hilbert space Every x H can be represented in terms of an orthonormal basis {φ i } i 1 (or orthobasis for short) according to: x = i 1 x, φ i φ i This is easy to see as follows. Suppose x has a representation i θ iφ i. Then x, φ j = i = i θ i φ i, φ j θ i φ i, φ j = i θ i δ i,j =θ j. 11 / 27

12 Example: Sample space orthonormal basis. φ k = 1. Then {φ k } k is a basis for R n. i.e. φ k,i = { i k 1 i = k (2) Example: Orthobasis of L 2 ([, 1]) For i = 1, 2,... φ 2i 1 (t) := 2 cos(2π(i 1)t) φ 2i (t) := 2 sin(2πit) 12 / 27

13 Gram-Schmidt Orthogonalization Any basis can be converted into an orthonormal basis using Gram-Schmidt Orthogonalization. Start with an arbitrary (non-orthogonal) basis {φ i }. 1. ψ 1 := φ 1 / φ for k = 3,..., n, ψ 2 :=φ 2 ψ 1, φ 2 ψ 1 ψ 2 := ψ 2 / ψ 2 k 1 ψ k := φ k ψ i, φ k ψ i i=1 ψ k := ψ k / ψ k 13 / 27

14 Subspaces Consider a set of vectors x 1, x 2,..., x p X. The span of these vectors is the set of all vectors x X that can be generated from linear combinations of the set } p span ({x i } p i=1 {x ) := : x = a i x i, a 1,..., a p R i=1 This set is also called a signal subspace of X. Definition: subspace A subset M X is a subspace if x, y M = ax + by M 14 / 27

15 If φ 1,..., φ p is an orthonormal basis for M R n, then every x M can be written as x = p θ i φ i. i=1 Hence, even though the signal x is a length-n vector, the fact that it lies in the subspace M means that it is actually a function of only p n free parameters or degrees of freedom. We say that M is a p-dimensional subspace of R n (and it is isometric to R r ). 15 / 27

16 Example: n = 3 Example: n = 3 φ 1 = 1 φ 2 = M = span (φ 1, φ 2 ) = φ 1 = 1/ 2 1/ 2 M = span (φ 1, φ 2 ) = a b 1 φ 2 = a a b : a, b R 1 : a, b R 16 / 27

17 Orthogonal Projections Let H be a Hilbert space and let M H be a subspace. Every x H can be written as x = y + z where y M and z M, which is shorthand for z orthogonal to M; that is v M, v, z =. The vector y is the optimal approximation to x in terms of vectors in M in the following sense: x y = min x v v M The vector y is called the projection of x onto M. 17 / 27

18 Orthogonal subspace projection Let M H and let {φ i } r i=1 be an orthobasis for M. For any x H, the projection of x onto M is given by y = r φ i, x φ i i=1 and this projection can be viewed as a sort of filter that removes all components of the signal x that are orthogonal to M. 18 / 27

19 Example: [ ] 1 Let H = R 2. Consider the canonical coordinate system φ 1 = [ ] and φ 2 =. Let M be the subspace spanned by φ 1 1. The projection of any x = [x 1 x 2 ] T R 2 onto M is [ ] [ ] ( [ ]) [ ] [ ] x1 P 1 x = x, = [x 1 x 2 ] = The projection operator P 1 is just a matrix and it is given by [ ] [ ] P 1 := φ 1 φ T = [1 ] = [ ] [ ] 1/ 2 It is also easy to check that φ 1 = 1/ 1/ 2 and φ 2 = 2 1/ 2 is an orthobasis for R 2. What is the projection operator onto the span of φ 1 in this case? 19 / 27

20 Orthogonal projections in Euclidean subspaces More generally suppose we are considering R n and we have a orthonormal basis {φ i } r i=1 for some r-dimensional, r < n, subspace M of R n. Then the projection matrix is given by P M = r φ i φ T i. i=1 Moreover, if {φ i } r i=1 is a basis for M, but not necessarily orthonormal, then P M = Φ(Φ T Φ) 1 Φ T where Φ = [φ 1,..., φ r ], a matrix whose columns are the basis vectors. 2 / 27

21 Example: Let H = L 2 ([, 1]) and let M = {linear functions on [, 1]}. Since all linear functions have the form y(t) = at + b, for t [, 1], here is a basis for M: φ 1 (t) =1, φ 2 (t) =t. Note that this means that M is two-dimensional. That makes sense since every line is defined by its slope and intercept (two real numbers). Using the Gram-Schmidt procedure we can construct the orthobasis ψ 1 (t) = 1, ψ 2 (t) = t 1/2.fix this normalize! Now, consider any signal/function x L 2 ([, 1]). The projection of x onto M is [P M x](t) = x, 1 + x, t 1/2 (t 1/2) = 1 x(τ) dτ + (t 1/2) 1 (τ 1/2)x(τ) dτ 21 / 27

22 Eigendecomposition of a Symmetric Matrix Let C be a real, symmetric matrix (C = C). v R n is an eigenvector of C if Cv = }{{} λ v eigenvalue If C is n n, then there are n orthonormal eigenvectors {v 1,..., v n } such that Let V = [v 1,..., v n ]. Then v i, v j = δ i,j. C = V ΛV where λ 1 Λ =..... λ n 22 / 27

23 Singular Value Decomposition The SVD of an n p matrix H is written as H = }{{} U n p }{{} Σ p p V }{{} p p U = [u 1 u p ] and {u i } p i=1 are n 1 vectors called the left singular vectors of H. U U = I σ 1 Σ =..... (a diagonal p p matrix) and {σ i } p i=1 σ p are the singular values of H, sorted so σ 1 σ 2 σ p. V = [v 1 v p ] and {v i } p i=1 are p 1 vectors called the right singular vectors of H. V V = I 23 / 27

24 Also note that HH = UΣV V Σ U = U(ΣΣ )U H H = V Σ U UΣV = V (Σ Σ)V So, {σ 2 1,..., σ2 p} are the eigenvalues of H H and {v 1,..., v p } are the corresponding eigenvectors. Also, {σ 2 1,..., σ2 p} are the first p eigenvalues of HH (n p remaining eigenvalues are identically zero) and {u 1,..., u p } are the associated eigenvectors. 24 / 27

25 Application of SVD Suppose we want to solve the following overdetermined set of linear equations: x }{{} n 1 = H }{{} n p }{{} θ p 1, p < n where x is the observation, H is a known matrix, and θ is unknown. If H were square (and non-singular), then θ = H 1 x, but here H is not square. Notice that x = UΣV θ If the p columns of H are linearly independent, then σ 1 σ 2 σ p >. So, we can proceed in the following manner: U x =ΣV θ Σ 1 U x =V θ V Σ 1 U x =θ, since U U = I p p, since σ i >, i = 1,..., p, since V V = I p p 25 / 27

26 Definition: pseudoinverse is called the pseudoinverse of H. H # V Σ 1 U = (H H) 1 H proof: First, recall that (ABC) 1 = C 1 B 1 A 1 and V 1 = V. (H H) 1 H = = = ( ) 1 (UΣV ) (UΣV ) (UΣV ) 1 V Σ U}{{ U} ΣV I p p ( V Σ 2 V ) 1 V Σ U = V Σ 2 V}{{ V} Σ U I p p = V Σ 1 U V Σ U 26 / 27

27 Also, notice x = Hθ H x = H Hθ θ = (H H) 1 H x }{{} H # Hθ = H(H H) 1 H x }{{} P H where H H is p p and invertible when columns of H are linearly independent. 27 / 27