Sample ECE275A Midterm Exam Questions

Transcription

1 Sample ECE275A Midterm Exam Questions The questions given below are actual problems taken from exams given in in the past few years. Solutions to these problems will NOT be provided. These problems and old homework problems have been recyled on midterm and final exams in the recent past, so you should study the questions given below as well as all past homework solutions carefully. You should also study the solutions given to the sample undergraduate midterm. 1. Geometry Induced by a Linear Operator A and the Pseudoinverse. Let A be an m n complex matrix and consider the linear inverse problem y = Ax. Let Ω and W be positive-definite weighting matrices which define inner products on the domain and codomain of A respectively. Let A + denote the pseudoinverse of A. (a) Prove that N (A )=R(A) and R(A )=N(A). (b) State the four Moore-Penrose Pseudoinverse Conditions and prove that as a consequence of these conditions, P = AA +, I P, Q = A + A, and I Q are orthogonal projection operators. Give the domain and range for each of these operators. (c) Derive an expression for the adjoint operator, A, in terms of Ω, W, and A. (d) Derive closed-form expressions for A + (in terms of Ω, W, and A) valid for the two distinct cases when A is 1-to-1 and when it is onto respectively. Justify every key step in your derivations. 2. The Singular Value Decomposition (SVD). Perform the following numerical computations. Assume that A maps between real spaces using the standard (unweighted) inner product. (a) Determine the SVD for the matrix, A = ( ). (b) Give the dimension and an orthonormal basis for each of the four fundamental subspaces of the real matrix A. (c) Construct the orthogonal projections operators onto each of the four fundamental subspaces (with respect to the standard inner product). (d) Construct the pseudoinverse, A + (with respect to the standard inner product). 3. Constructing the Pseudoinverse. Let the matrix A shown below denote a linear mapping between two complex weighted inner product spaces as follows. Let the domain be C 3 with inner product weighting matrix, j j Ω= 1 2 j j j 2 + j

2 Let the codomain be C 4 with inner product weighting matrix, W = Let the mapping between the complex spaces C 3 and C 4 be, A = Construct the pseudoinverse, A +, fully showing all of its components. Justify every step. 4. Orthogonal Projection Operator onto the Space of Symmetric Matrices Let X = C n n be the Hilbert space of n n complex matrices, X, with Frobenius inner product X 1,X 2 = tr X H 1 X 2. Let V be the set of symmetric (not hermitian) n n complex matrices V = V T, V X = C n n. Finally, define the mapping P ( ) :X X by P (X) X + XT 2. (a) Prove that V is a (Hilbert) subspace of X and give its dimension. Prove that the set of hermitian matrices is not a subspace. (b) Prove that P ( ) is the orthogonal projection of X onto the subspace V. Please note that at the outset we know nothing about P ( ) other than its definition. 5. Geometric Approach to Solving Linear Inverse Problems I Let X = Sym(C,n) C n n be the vector space of symmetric n n complex matrices 1 with Frobenious inner product X 1,X 2 = tr X1 H X 2. Let Y = C n m be the Hilbert space of n m complex matrices with inner product Y 1,Y 2 = tr Y H 1 Y 2. Finally for a given full row-rank n m matrix A, definite the mapping A( ) :X Y by A(X) XA, rank(a) =n. 1 In the previous problem, the set of symmetric, complex matrices was proven to be a Hilbert subspace. Here, we treat this set as a Hilbert space in its own right. 2

3 (a) Prove that the least-squares solution to the inverse problem Y = A(X) is the necessarily unique solution to the (matrix) Lyapunov equation XM + M T X =Λ, where (1) M AA H and Λ YA H +(YA H ) T. (b) The Matlab Controls Toolbox provides a numerical solver for the Lyapunov Equation (1). Mathematically, it can be readily shown that a unique solution to the Lyapunov equation is theoretically given by X = e M T t Λ e Mt dt (2) 0 provided that the real parts of the eigenvalues of M are all strictly greater than zero. Given this fact, justify the claim that the unique solution to (1) is given by equation (2). 6. Geometric Approach to Solving Linear Inverse Problems II Let X X belong to the space, X,ofcomplex m n matrices and Y Y to the space, Y,ofcomplex p q matrices. On each of these two spaces define the (unweighted) Frobenius inner product, X 1,X 2 = trace ( X H 1 X 2 ), with the associated (induced) Frobenius norm of a matrix, X F = trace (X H X). We want to find a minimum-norm least-squares solution to the linear inverse problem, Y = A(X), where A(X) =CXB, for a fixed complex p m matrix C and a fixed complex n q matrix B. It is a fact that the operator A is 1-to-1 iff both rank(c) =m and rank(b) =n. Alternatively, A is onto iff both rank(c) =p and rank(b) =q. In general, of course, A might be rank-deficient, and hence neither 1-to-1 nor onto. (a) Determine the adjoint operator, A (Y ). (b) Derive the normal equations (i.e., the algebraic condition that a least squares solution, X, must satisfy) and the algebraic condition for a solution, X, to be minimum norm. (c) For the operator A 1-to-1 derive a closed-form expression for the solution and for the pseudoinverse, A + (Y ). Justify every key step in your derivation (d) For A 1-to-1 give a closed-form expression for the orthogonal projector onto R(A). (e) for A onto derive a closed-form expression for the solution and for the pseudoinverse, A + (Y ). Justify every key step in your solution. (f) For A onto give a closed-form expression for the orthogonal projector onto R(A ). 3

4 (g) Express the operator pseudoinverse expressions derived in parts (c) and (d) in terms of the matrix pseudoinverses C + and B + (assuming the standard inner products on the domains and codomains of the matrices C and B). Based on these expressions conjecture the form of the operator pseudoinverse, A +, for the case when A is neither 1-to-1 nor onto, and then describe in words (a proof is not necessary) how you would prove that your conjectured form is indeed the correct form for the pseudoinverse. 7. Geometric Approach to Solving Linear Inverse Problems III. Let X = R m n be the space of real m n matrices X, m and n arbitrary, with inner product X 1,X 2 = tr X T 1 X 2. Let Y = RV m be the space of finite-variance real random vectors y(ω) R m with inner product Let A : X Ybe the linear mapping y 1,y 2 = E { y T 1 y 2 }. A(X) =Xb(ω) where b is a specified real n-dimensional random vector with covariance E { bb T } = I. (a) Show that A is one-to-one. (b) Find the least squares solution to the inverse problem and give the least-squares estimate of y. y = A(X) 8. Geometric Approach to Solving Linear Inverse Problems IV. Let Y belong to the Hilbert space of complex p q matrices, Y = C p q, with weighted Frobenius inner product, Y 1,Y 2 = trace (Y1 H WY 2 ), where W is a q q positive-definite hermitian weighting matrix. The space Y is pq dimensional. Let the complex vector c =(c 1,,c n ) T belong to the Hilbert space X = C n with the standard inner product, c 1, c 2 = c H 1 c 2 and dimension n>pq. Consider the following linear mapping between X and Y which is onto but NOT one-to-one, with X i Yfor i =1,,n. Y = c 1 X c n X n, (3) (a) (i) Is the system (3) solvable for all Y? (ii) What is the dimension of the range of the mapping? (iii) What is the dimension of the null space? (iv) Is the null space trivial or nontrivial? (v) What are the solution possibilities of the inverse problem given by (3). (b) Determine the adjoint operator for the mapping shown in (3). 4

5 (c) Determine the pseudoinverse solution to the inverse problem (3). (If you are unable to solve part (b), show in detail how you would use the result of part (b) to solve part (c)). 9. Real Vector Derivatives and Coordinate Transformations. Let x =(x, y) T be the standard cartesian coordinates in the plane R 2. An alternative curvilinear coordinate system in R 2 is given by parabolic coordinates, ξ =(ξ,η) T. The relationship between these two coordinate representations is x = ξη and y = 1 ( η 2 ξ 2). 2 (a) Determine the jacobian J xξ = x ξ, the metric tensor Ω ξ, the cogradient ξ, and the gradient ξ. Are the parabolic coordinates orthogonal? Are they orthonormal? (Explain your answers.) (b) Let ˆξ denote the coordinate system obtained by normalizing the parabolic-coordinates canonical basis vectors e 1 =(1, 0) T and e 2 =(0, 1) T to produce unit vectors ɛ 1, ɛ 2 for the new ˆξ-system. Find the cogradient ˆξ and the gradient ˆξ in the ˆξ-system Vector Derivatives and Regularized Least-Squares. One way to regularize an ill posed linear linear problem, y = Ax, is to find a solution which minimizes the regularized least-squares cost function, for a given regularization parameter, γ>0. 3 l(x) = y Ax 2 W + γ x 2 Ω, (4) (a) Assuming that all the quantities in (4) are real, minimize the cost function (4) by finding a stationary point, ˆx, ofl(x) and showing that the corresponding hessian is positive definite. Use the chain-rule in your derivation and make it clear where it was used. (b) Assuming that all the quantities in (4) are complex, minimize the cost function (4) by finding a stationary point, ˆx, ofl(x) and showing that the corresponding hessian is positive definite. Did you need to use the chain-rule in your derivation? (c) Assume that A is one-to-one. Show that in the limit γ 0 we obtain ˆx = A + y for the solution to part b above. (d) Assume that A is onto. Show that in the limit γ 0 we obtain ˆx = A + y for the solution to part b above. (As suggested by parts (c) and (d), the solutions to the regularized least-squares minimization problem form a family of solutions which include the pseudoinverse as a special case (this is true even when A is rank deficient). Thus regularized least-squares provides a generalization of the pseudoinverse approach.) 2 Hint: Directly exploit the fact that in the new system Ωˆξ = I. 3 This problem also arises in other contexts. For instance, this is the loss function which is solved to obtain the so-called Leaky LMS adaptive filtering algorithm, where x denotes the vector of tap-filter weights. 5

6 11. Complex Vector Derivatives and Constrained Optimization I A so-called Minimum Variance Distortionless Response (MVDR) filter is an FIR filter, h l 0 x k y k = h j x k j = h H X k, h =., X k =., (5) j=0 h l x k l designed to pass without distortion a desired signal at a known frequency ω 0, H(e jω 0 )=1, H(Z) = h 0 + h 1 Z h l Z l, (6) while blocking noise and jamming signals located at other frequencies. Such filters can be designed for signals distributed in time or in space, where in the latter case the procedure is known as the MVDR beamformer. 4 It is assumed that x k is a zero-mean, finite-variance stationary random sequence. The l +1optimal filter coefficients h are found from minimizing the variance (signal power) of the filter output E { y k 2} subject to the constraint (6). This forces the filter to attenuate signals at all frequencies other than ω 0 while passing signals at ω 0 without distortion. (a) Show that the MVDR optimization problem can be stated as min h h H Ωh subject to g(h) =1 φ(ω 0 ) H h =0 (7) clearly stating the definition of the (l +1) (l +1)hermitian matrix Ω and the (l +1)- dimensional vector function φ(ω 0 ). (b) The theory of lagrange multipliers is well-posed when the objective function and constraints are real-valued functions of real unknown variables. 5 Note that a vector of p complex equality constraint conditions, g(h) =0 C p, corresponds to 2p real equality constraints corresponding to Re g(h) =0 and Im g(h) =0. Thus, a well-defined lagrangian is given by L = h H Ωh + λ T R Re g(h)+λ T I Im g(h), for real-valued p-dimensional lagrange multiplier vectors λ R and λ I. Define the complex lagrange multiplier vector λ by λ = λ R + jλ I and show that the lagrangian given above can be equivalently written as L = h H Ωh + Re λ H g(h). (8) 4 See pages of S. Haykin, Adaptive Filter Theory, 3rd Edition, Note that signals and filter coefficients are assumed to be complex. 5 The unknown variables here can be taken to be the 2(l +1)real-valued real and imaginary parts of the l +1filter coefficients h k, k = 0,,l. The real-valued constraints obviously correspond to the two scalar (p = 1) conditions Re g(h) = 0 and Im g(h) =0. 6

7 (c) Using the properties of complex vector derivatives, determine the MVDR filter coefficients by finding a stationary point of the lagrangian (8). Assume that Ω is full rank. (d) Again determine the MVDR filter coefficients, this time without taking derivatives, by solving problem (7) using the geometric approach. Assume that Ω is full rank. 12. Complex Vector Derivatives and Constrained Optimization II Let ẑ C n be the pseudoinverse solution to the linear inverse problem s A z C m (9) where all spaces are complex Hilbert spaces with the standard inner product (identity metric tensor). Assume that A has full column rank and Let z denote the complex conjugate of z. (a) Let z = x + jy C n for x, y R n, and let l(z) =l(x, y) R be a real-valued loss-function of z which is to be minimized subject to a vector of constraints on the values of z: g(z) =g(x, y) =0 C p. The theory of lagrange multipliers is well-posed when the objective function and constraints are real-valued functions of real unknown variables. Thus a well-defined lagrangian is given by L = l(x, y)+λ T R Re g(x, y)+λ T I Im g(x, y), for real-valued p-dimensional lagrange multiplier vectors λ R and λ I. Define the complex lagrange multiplier vector λ by λ = λ R + jλ I and show that the lagrangian given above can be equivalently written as L = l(z)+re λ H g(z). (10) (b) Find the constrained least-squares solution, ẑ c, which provides a least-squares solution to the inverse problem (9) subject to the constraint B z =0 C p, assuming that B has full row rank. Justify the existence of every inverse you take. (c) i) Show that the constrained least-squares solution ẑ c is the projection of the unconstrained leastsquares solution ẑ onto the null space of B by specifically determining the projection operator. (Be sure to prove that it is indeed a projection operator onto the nullspace of B.) ii) Is this projection an orthogonal projection? (Explain your answer.) 7

8 13. Complex Vector Derivatives, Regularized Optimization, and the Leaky LMS Algorithm. In the lecture supplement on complex derivatives we adaptively learned a complex parameter-vector of FIR filter coefficients c C n by attempting to minimizing the instantaneous quadratic error ˆl(c) = e k 2, e k = y k c H x k, (11) at each sample instant via gradient descent on ˆl c ĉ k+1 = ĉ k α cˆl(ĉk ) (12) where y k C, x k C n, and α>0. 6 This results in the LMS Algorithm. Unfortunately, the basic LMS Algorithm can perform poorly in nonstationary environments because it tends to persistently remember learned values of c. (This can be seen by noting that if e k 0 for k j we have ĉ k ĉ j for all k j.) As a consequence ĉ k might not be able to adapt quickly enough to accommodate changes in the statistical behavior of the signals y k and x k. The Leaky LMS Algorithm attempts to rectify this behavior by penalizing the size of c k by the addition of the weighting term γ c 2 to the right-hand-side (RHS) of equation (11) for γ>0. ˆl(c) = e k 2 + γ c 2 (13) (a) Derive the Leaky LMS Algorithm from equations (12) and (13). Write the RHS of the algorithm in two ways. First in terms of ĉ k and e k and Then in terms of ĉ k, x k, and y k. (b) Show that the resulting algorithm forgets the value of ĉ k for k j when e k =0for k j and α and γ are chosen appropriately. 7 What happens to the leakiness if we set γ =0? What happens if we set the product αγ =1? 6 For simplicity, it is assumed that the parameter space is Euclidean so that Ω c = I and that α is constant. 7 The parameter γ is called a forgetting factor and allows learned values of c to leak to zero when filtering error is zero. 8