Review of Matrices and Vectors 1/45

Transcription

1 Reiew of Matrices and Vectors /45

2 /45 Definition of Vector: A collection of comple or real numbers, generally put in a column [ ] T "! Transpose b a b a b b a a " " " b a b a Definition of Vector Addition: Add element-by-element Vectors & Vector Spaces

3 Definition of Scalar: A real or comple number. If the ectors of interest are comple alued then the set of scalars is taen to be comple numbers; if the ectors of interest are real alued then the set of scalars is taen to be real numbers. Multiplying a Vector by a Scalar : a a " a αa αa " α a changes the ector s length if α reerses its direction if α < 3/45

4 Arithmetic Properties of Vectors: ector addition and scalar multiplication ehibit the following properties pretty much lie the real numbers do Let, y, and z be ectors of the same dimension and let α and β be scalars; then the following properties hold:. Commutatiity + y y α α +. Associatiity 3. Distributiity ( + y) + z y + ( + z) α( β) α( + y) ( αβ) α + αy ( α + β) α + β 4. Scalar Unity & Scalar Zero, where is the zero ector of all zeros 4/45

5 Definition of a Vector Space: A set V of -dimensional ectors (with a corresponding set of scalars) such that the set of ectors is: (i) closed under ector addition (ii) closed under scalar multiplication In other words: addition of ectors gies another ector in the set multiplying a ector by a scalar gies another ector in the set ote: this means that AY linear combination of ectors in the space results in a ector in the space If,, and 3 are all ectors in a gien ector space V, then α + α is also in the ector space V. + α i α i i 5/45

6 Aioms of Vector Space: If V is a set of ectors satisfying the aboe definition of a ector space then it satisfies the following aioms:. Commutatiity (see aboe). Associatiity (see aboe) 3. Distributiity (see aboe) 4. Unity and Zero Scalar (see aboe) 5. Eistence of an Additie Identity any ector space V must hae a zero ector 6. Eistence of egatie Vector: For eery ector in V its negatie must also be in V So a ector space is nothing more than a set of ectors with an arithmetic structure 6/45

7 Def. of Subspace: Gien a ector space V, a subset of ectors in V that itself is closed under ector addition and scalar multiplication (using the same set of scalars) is called a subspace of V. Eamples:. The space R is a subspace of R 3.. Any plane in R 3 that passes through the origin is a subspace 3. Any line passing through the origin in R is a subspace of R 4. The set R is OT a subspace of C because R isn t closed under comple scalars (a subspace must retain the original space s set of scalars) 7/45

8 Geometric Structure of Vector Space Length of a Vector (Vector orm): For any ector in C we define its length (or norm ) to be i Properties of Vector orm: i i i α α α + + β α β < C iff 8/45

9 Distance Between Vectors: the distance between two ectors in a ector space with the two norm is defined by: d(, ) ote that: d(, ) iff 9/45

10 Angle Between Vectors & Inner Product: Motiate the idea in R : A θ Acosθ Asin θ u u ote that: i ui i Acosθ + Asin θ Acosθ Clearly we see that This gies a measure of the angle between the ectors. ow we generalize this idea! /45

11 Inner Product Between Vectors : Define the inner product between two comple ectors in C by: < u, > u i i Properties of Inner Products:. Impact of Scalar Multiplication: * i < αu, > α < u, > < u, β > * β < u, >. Impact of Vector Addition: < < u, + z >< u, > + < u, z > u + w, >< u, > + < w, > 3. Lining Inner Product to orm: <, > 4. Schwarz Inequality: < u, > u 5. Inner Product and Angle: (Loo bac on preious page!) < u, > u cos( θ ) /45

12 Inner Product, Angle, and Orthogonality : < u u, > cos( θ ) (i) This lies between and ; (ii) It measures directional alieness of u and + when u and point in the same direction when u and are a right angle when u and point in opposite directions Two ectors u and are said to be orthogonal when <u,> If in addition, they each hae unit length they are orthonormal /45

13 Building Vectors From Other Vectors Can we find a set of prototype ectors {,,, M } from which we can build all other ectors in some gien ector space V by using linear combinations of the i? M α u Same Ingredients just different amounts of them!!! M β We want to be able to do is get any ector just by changing the amounts To do this requires that the set of prototype ectors {,,, M } satisfy certain conditions. We d also lie to hae the smallest number of members in the set of prototype ectors. 3/45

14 Span of a Set of Vectors: A set of ectors {,,, M } is said to span the ector space V if it is possible to write each ector in V as a linear combination of ectors from the set: This property establishes if there are enough ectors in the proposed prototype set to build all possible ectors in V. M α It is clear that: Eamples in R Does not Span R Spans R. We need at least ectors to span C or R but not just any ectors.. Any set of mutually orthogonal ectors spans C or R (a set of ectors is mutually orthogonal if all pairs are orthogonal). 4/45

15 Linear Independence: A set of ectors {,,, M } is said to be linearly independent if none of the ectors in it can be written as a linear combination of the others. If a set of ectors is linearly dependent then there is redundancy in the set it has more ectors than needed to be a prototype set! For eample, say that we hae a set of four ectors {,, 3, 4 } and lets say that we now that we can build from and 3 then eery ector we can build from {,, 3, 4 } can also be built from only {, 3, 4 }. It is clear that: Eamples in R Linearly Independent ot Linearly Independent. In C or R we can hae no more than linear independent ectors.. Any set of mutually orthogonal ectors is linear independent (a set of ectors is mutually orthogonal if all pairs are orthogonal). 5/45

16 Basis of a Vector Space: A basis of a ector space is a set of linear independent ectors that span the space. Span says there are enough ectors to build eerything Linear Indep says that there are not more than needed Orthonormal (O) Basis: If a basis of a ector space contains ectors that are orthonormal to each other (all pairs of basis ectors are orthogonal and each basis ector has unit norm). Fact: Any set of linearly independent ectors in C (R ) is a basis of C (R ). Dimension of a Vector Space: The number of ectors in any basis for a ector space is said to be the dimension of the space. Thus, C and R each hae dimension of. 6/45

17 Epansion and Transformation Fact: For a gien basis {,,, }, the epansion of a ector in V is unique. That is, for each there is only one, unique set of coefficients {α, α,, α } such that α In other words, this epansion or decomposition is unique. Thus, for a gien basis we can mae a -to- correspondence between ector and the coefficients {α, α,, α }. We can write the coefficients as a ector, too: α [ ] T α! α " to α α " α Epansion can be iewed as a mapping (or transformation) from ector to ector α. We can iew this transform as taing us from the original ector space into a new ector space made from the coefficient ectors of all the original ectors. 7/45

18 Fact: For any gien ector space there are an infinite number of possible basis sets. The coefficients with respect to any of them proides complete information about a ector some of them proide more insight into the ector and are therefore more useful for certain signal processing tass than others. Often the ey to soling a signal processing problem lies in finding the correct basis to use for epanding this is equialent to finding the right transform. See discussion coming net lining DFT to these ideas!!!! 8/45

19 9/45 DFT from Basis Viewpoint: If we hae a discrete-time signal [n] for n,, - Define ector: Define a orthogonal basis from the eponentials used in the IDFT: [ ] T ] [ [] ] [! " d π π j j e e / ) ( / " d π π j j e e / ) ( / " d π π j j e e / ) )( ( ) / ( " d Then the IDFT equation can be iewed as an epansion of the signal ector in terms of this comple sinusoid basis: ] [ X d #%#$ & α th coefficient T X X X ] [ [] []! α coefficient ector

20 /45 What s So Good About an O Basis?: Gien any basis {,,, } we can write any in V as α Gien the ector how do we find the α s? In general hard! But for O basis easy!! If {,,, } is an O basis then i i j i j j j i j j j i α α α δ #%#$ & ] [,,, i i, α i th coefficient inner product with i th O basis ector Usefulness of an O Basis

21 Another Good Thing About an O Basis: They presere inner products and norms (called isometric ): If {,,, } is an O basis and u and are ectors epanded as Then. α. <,u > < α, β > (Preseres Inner Prod.). α and u β (Preseres orms) u β So using an O basis proides: Easy computation ia inner products Preseration of geometry (closeness, size, orientation, etc. /45

22 /45 Eample: DFT Coefficients as Inner Products: Recall: -pt. IDFT is an epansion of the signal ector in terms of Orthogonal ectors. Thus / * ] [ ] [ ] [ ] [ n n j n e n n d n X π,d See reading notes for some details about normalization issues in this case

23 Matri: Is an array of (real or comple) numbers organized in rows and columns. a a a3 a4 Here is a 34 eample: We ll sometimes iew a matri as being built from its columns; The 34 eample aboe could be written as: [ a a a a ] A a [ a a a3 ] T 3 We ll tae two iews of a matri: 4 Matrices A a a. Storage for a bunch of related numbers (e.g., Co. Matri) 3 a a 3 a a 3 33 a a A transform (or mapping, or operator) acting on a ector (e.g., DFT, obseration matri, etc. as we ll see) 3/45

24 Matri as Transform: Our main iew of matrices will be as operators that transform one ector into another ector. Consider the 34 eample matri aboe. We could use that matri to transform the 4-dimensional ector into a 3-dimensional ector u: u A [ a a a3 a4 ] a + a + 3a3 + 4a4 3 4 Clearly u is built from the columns of matri A; therefore, it must lie in the span of the set of ectors that mae up the columns of A. ote that the columns of A are 3-dimensional ectors so is u. 4/45

25 Transforming a Vector Space: If we apply A to all the ectors in a ector space V we get a collection of ectors that are in a new space called U. In the 34 eample matri aboe we transformed a 4-dimensional ector space V into a 3-dimensional ector space U A 3 real matri A would transform R 3 into R : Facts: If the mapping matri A is square and its columns are linearly independent then (i) the space that ectors in V get mapped to (i.e., U) has the same dimension as V (due to square part) (ii) this mapping is reersible (i.e., inertible); there is an inerse matri A - such that A - u (due to square & LI part) A 5/45

26 Transform Matri Vector: a VERY useful iewpoint for all sorts of signal processing scenarios. In general we can iew many linear transforms (e.g., DFT, etc.) in terms of some inertible matri A operating on a signal ector to gie another ector y: y i A i i A y i A, A y y A, A We can thin of A and A - as mapping bac and forth between two ector spaces 6/45

27 Matri View & Basis View Basis Matri & Coefficient Vector: Suppose we hae a basis {,,, } for a ector space V. Then a ector in space V can be written as: α Another iew of this: α α [ ] &# #! %## $ matri " α Vα The Basis Matri V transforms the coefficient ector into the original ector 7/45

28 Three Views of Basis Matri & Coefficient Vector: View # Vector is a linear combination of the columns of basis matri V. View # Matri V maps ector α into ector. View #3 There is a matri, V -, that maps ector into ector α. α Vα α V Aside: If a matri A is square and has linearly independent columns, then A is inertible and A - eists such that A A - A - A I where I is the identity matri haing s on the diagonal and zeros elsewhere. ow hae a way to go bac-andforth between ector and its coefficient ector α 8/45

29 9/45 Basis Matri for O Basis: We get a special structure!!! Result: For an O basis matri V V - V H (the superscript H denotes hermitian transpose, which consists of transposing the matri and conjugating the elements) To see this: I VV > < > < > < > < > < > < > < > < > <,,,,,,,,,! " ' "!!! " ' "!! H Inner products are or because this is an O basis

30 Unitary and Orthogonal Matrices A unitary matri is a comple matri A whose inerse is A - A H For the real-alued matri case we get a special case of unitary the idea of unitary matri becomes orthogonal matri for which A - A T Two Properties of Unitary Matrices: Let U be a unitary matri and let y U and y U. They presere norms: y i i.. They presere inner products: < y, y > <, > That is the geometry of the old space is presered by the unitary matri as it transforms into the new space. (These are the same as the preseration properties of O basis.) 3/45

31 3/45 DFT from Unitary Matri Viewpoint: Consider a discrete-time signal [n] for n,, -. We e already seen the DFT in a basis iewpoint: ow we can iew the DFT as a transform from the Unitary matri iewpoint: ] [ X d #%#$ & α π π π π π π j j j j j j e e e e e e / ) )( ( ) / ( / ) ( / / ) ( / ] [ "!!! " " " d d d D D H ~ D ~ DFT IDFT (Acutally D is not unitary but -/ D is unitary see reading notes)

32 Geometry Preseration of Unitary Matri Mappings Recall unitary matrices map in such a way that the sizes of ectors and the orientation between ectors is not changed. A, A y y A, A Unitary mappings just rigidly rotate the space. 3/45

33 Effect of on-unitary Matri Mappings A, A y y A, A 33/45

34 More on Matrices as Transforms We ll limit ourseles here to real-alued ectors and matrices y A m m n n A maps any ector in R n into some ector y in R m Mostly interested in two cases:. Tall Matri m > n. Square Matri m n R n A y R m ector y weighted sum of columns of A may only be able to reach certain y s Range(A): Range Space of A set of all ectors in R m that can be reached by mapping 34/45

35 Range of a Tall Matri (m > n) R n The range(a) R m R m A y Proof : Since y is built from the n columns of A there are not enough to form a basis for R m (they don t span R m ) Range of a Square Matri (m n) If the columns of A are linearly indep.the range(a) R m because the columns form a basis for R m Otherwise.The range(a) R m because the columns don t span R m 35/45

36 36/45 Ran of a Matri: ran(a) largest # of linearly independent columns (or rows) of matri A For an m n matri we hae that ran(a) min(m,n) An m n matri A has full ran when ran(a) min(m,n) Eample: This matri has ran of 3 because the 4 th column cam be written as a combination of the first 3 columns A

37 Characterizing Tall Matri Mappings We are interested in answering: Gien a ector y, what ector mapped into it ia matri A? Tall Matri (m > n) Case If y does not lie in range(a), then there is o Solution If y lies in range(a), then there is a solution (but not necessarily just one unique solution) y A y range(a) y range(a) o Solution A full ran A not full ran One Solution Many Solutions 37/45

38 Full-Ran Tall Matri (m > n) Case R n A y Range(A) y A R m For a gien y range(a) there is only one that maps to it. This is because the columns of A are linearly independent and we now from our studies of ector spaces that the coefficient ector of y is unique is that coefficient ector By looing at y we can determine which gae rise to it 38/45

39 onfull-ran Tall Matri (m > n) Case R n A A y A y R m Range(A) For a gien y range(a) there is more than one that maps to it This is because the columns of A are linearly dependent and that redundancy proides seeral ways to combine them to create y By looing at y we can not determine which gae rise to it 39/45

40 Characterizing Square Matri Mappings Q: Gien any y R n can we find an R n that maps to it? A: ot always!!! A full ran y A A not full ran Careful!!! This is quite a different flow diagram here!!! One Solution y range(a) y range(a) o Solution Many Solutions When a square A is full ran then its range coers the complete new space then, y must be in range(a) and because the columns of A are a basis there is a way to build y 4/45

41 A Full-Ran Square Matri is Inertible A square matri that has full ran is said to be. nonsingular, inertible Then we can find the that mapped to y using A - y Seeral ways to chec if n n A is inertible:. A is inertible if and only if (iff) its columns (or rows) are linearly independent (i.e., if it is full ran). A is inertible iff det(a) 3. A is inertible if (but not only if) it is positie definite (see later) 4. A is inertible if (but not only if) all its eigenalues are nonzero Pos. Def. Matrices Inertible Matrices 4/45

42 Eigenalues and Eigenectors of Square Matrices If matri A is n n, then A maps R n R n Q: For a gien n n matri A, which ectors get mapped into being almost themseles??? More precisely Which ectors get mapped to a scalar multiple of themseles??? Een more precisely which ectors satisfy the following: A λ Input Output These ectors are special and are called the eigenectors of A. The scalar λ is that e-ector s corresponding eigenalue. A 4/45

43 Eigen-Facts for Symmetric Matrices If n n real matri A is symmetric, then e-ectors corresponding to distinct e-alues are orthonormal e-alues are real alued can decompose A as T A VΛV V [! ] n VV T I Λ diag If, further, A is pos. def. (semi-def.), then e-alues are positie (non-negatie) ran(a) # of non-zero e-alues { λ λ, λ },, Pos. Def. Full Ran (and therefore inertible) Pos. Semi-Def. ot Full Ran (and therefore not inertible) When A is P. D., then we can write For P.D. A, A - has the same e-ectors and has reciprocal e-alues A Λ n VΛ T V { },,, diag λ λ λ n 43/45

44 Other Matri Issues We ll limit our discussion to real-alued matrices and ectors Quadratic Forms and Positie-(Semi)Definite Matrices Quadratic Form Matri form for a nd -order multiariate polynomial Eample: a A a a a ariable fied The quadratic form of matri A is: Q A (, ) T A ( ) ( ) ( ) ( ) scalar scalar i j a ij i j a + a + ( a + a ) 44/45

45 Values of the elements of matri A determine the characteristics of the quadratic form Q A () If Q A () then say that Q A () is positie semi-definite If Q A () > then say that Q A () is positie definite Otherwise say that Q A () is non-definite These terms carry oer to the matri that defines the Quad Form If Q A () then say that A is positie semi-definite If Q A () > then say that A is positie definite 45/45