Detection and Estimation Theory

Transcription

1 ESE 54 Detection and Estiation Theory Joseph A. O Sullivan Sauel C. Sachs Professor Electronic Systes and Signals Research Laboratory Electrical and Systes Engineering Washington University 11 Urbauer Hall Lynda answers jao@wustl.edu J. A. O'S. ESE 54, Lecture 17, 03/19/09 1

2 Outline: Expectation-axiization E Algorith Review of E algorith Alternate Derivation Using the Convex Decoposition o Lea Exaples J. A. O'S. ESE 54, Lecture 17, 03/19/09

3 axiu Likelihood Estiation axiu likelihood estiation often involves axiizing a coplicated nonlinear function Use standard optiization algoriths or the E algorith E algorith is atched to probles that can be odeled with hidden data Depster, Laird, Rubin, J. Royal Statistical Society, I. iller and D. L. Snyder, Proc. IEEE, DLS Tutorial axiu likelihood Estiation θl = arg ax p r θ = arg ax ln p r θ θ View as function of θ θl = arg ax l θ, l θ = ln p r θ θ θ s Source p s θ p r s J. A. O'S. ESE 54, Lecture 17, 03/19/09 θ r 3

4 E Algorith Start with a reasonable guess as the initial iti estiate t Copute the expected value of the coplete data loglikelihood function given the incoplete or observed data and the current estiate axiize this function over the paraeters Iteratet E Algorith k l lik lih d f ti i 0 1. Initialization step: Select θ, let k = 0.. E-step: Copute the expected value k Q θθ = E ln p s θ r, θ ln, k = p s θ p s r θ ds 3. -step: axiize this function k 1 θ + = arg ax Q k θθ θ 4. Iteration step: stop if converged; else k = k+ 1, go to step. θ s Source p s θ p r s r J. A. O'S. ESE 54, Lecture 17, 03/19/09 4

5 k+ 1 k ln ln Properties of the E k+ 1 k k+ 1 k k Q θ θ H θ θ C θ Algorith k k k k k Q θ θ H θ θ + C θ The E algorith onotonically increases the loglikelihood function at every iteration. i Equality if and only if Current estiate is a axiu, and Posterior density reains unchanged Applicable to AP probles by odifying the -step p r θ p r θ = + k+ 1 k k k θ θ Q θ θ = Q k k k+ 1 k + H θ θ H θ θ k+ 1 k k k Q θ θ Q θ θ by -step H k k k+ 1 k θ θ H θ θ k p s r, θ, k = p s r θ ln ds 0 k 1 p s r, θ + θ θ k+ 1 k ln ln p r axiu A Posteriori AP Estiation θ = arg ax ln θ + ln θ L θ p r [ p r + p ] J. A. O'S. ESE 54, Lecture 17, 03/19/09 k+ 1 k 5 New -Step: θ = arg ax Q θ θ + ln p θ θ

6 Alternate Derivation of E Algorith Via Convex Optiization L Estiation Proble θ = arg ax ln p r θ L θ Hidden variables odel p r θ = p r s p s θ ds ln p r θ = ln p r s p s θ ds Define the set of probability density functions P { : s s 0, s ds 1} = Φ Φ Φ = Fix θ= θ k, then ln k s p p θ d Φ =in Φ s ln ds r s s s Φ P k p r s p s θ Double axiization for L Estiation p r s p s θ ax ln p r θ = ax ax Φ s ln ds θ θ Φ P Φ s J. A. O'S. ESE 54, Lecture 17, 03/19/09 6

7 E Algorith as an Alternating axiization Algorith E Algorith Alternately axiize over the posterior density and the paraeter vector Based on a variational representation of the loglikelihood function fro the convex decoposition lea We lift the original proble to a higher diensional proble where the optiization is easier. θ k 1. Select initial guess. Set 0. k. E-step: axiize over with fixed. k Φ s = k Φ P k p r s p s θ k p r s' p s' θ ds' k 3. -step. axiize i over θ with Φ s fixed k 1 θ + k k = arg ax Φ s ln p s θ ds= arg ax Q θ θ θ 4. Check for convergence; else k = k + 1, go to step. Double axiization for L Estiation p r s p s θ ax ln p r θ = ax ax Φ s ln ds θ θ Φ P Φ s P = { Φ: s Φ s 0, Φ s ds =1} J. A. O'S. ESE 54, Lecture 17, 03/19/09 = θ θ 7

8 Convex Decoposition Lea Lea: Suppose q <, q 0, at least one q > 0. Then i i i i pi ln qi = in piln, where i p P i q i P = p: pi 0, pi = 1 i Proof: pi L p = piln ν pi 1 i qi i pl L p = ln + 1 ν = 0 p q p l q = qe = q * ν 1 l l l * p p * ln i i i q = ln qi i i i l i J. A. O'S. ESE 54, Lecture 17, 03/19/09 8

9 Exaple of Convex Decoposition Lea: Poisson Data odel Change to a double axiization i j yi, jln λ, i j λ, i j = K/ L/ K/ L/ yi, jln hklci, k, jl hklci, k, jl i j k= K/ l= L/ k= K/ l=l/ hklci, k, j l = ax Φ kl, i, jy i, j ln h k, l c i k l Φ k l Φ k, l i, j, j i j k l [ ] J. A. O'S. ESE 54, Lecture 17, 03/19/09 9

10 E Algorith E-step is a weighted Poisson ean -step sets next value to the ean s c y c y c E ln P, = E s i, j, ln c i, j c i, j i j + 1 c k, l = E s k, l yc, c k, l y i, j = h i k, j l hi k, j l h ik', j l' c k', l' i, j i, j k', l' This algorith is widely used in astronoical iaging called the Lucy-Richardson algorith; see also D. L. Snyder and T. Schulz, positron eission toography PET; any other situations EL expectation axiization axiu likelhood lh J. A. O'S. ESE 54, Lecture 17, 03/19/09 10

11 Exaple: Poisson Plus Gaussian Signal odel is a Poisson rando variable plus additive discrete-tie tie white Gaussian noise. The ean of the Poisson represents the activity of interest. This is a odel for the data available at the readout of any charge- coupled devices. The charges are shifted fro one well to another, then read out serially at one location using an aplifier. The aplifier noise is well odeled as white Gaussian. The charge is odeled as resulting fro counting gphotons and is odeled as Poisson. J. A. O'S. ESE 54, Lecture 17, 03/19/09 y = n+ w k λ λ n e, k 0 k! w N0, σ p Y y k λ = k! k = 0 Yk e λ 1 σ e πσ λ = arg ax ln p Y L λ y 7

12 Exaple: Poisson Plus Gaussian Hidden data: Poisson rando variable Coplete data loglikelihood is Poisson Expected value of the coplete data loglikelihood given the easured data and the current estiate depends only on the posterior ean of the data. The next estiate of the paraeter equals the posterior ean. k λ 1 λ py Y = e e k 0 k! = πσ λ = arg ax ln p Y L λ Q λ λ = Enln λλ Y, λ k+ 1 k λ = arg ax Q λ λ λ λ + = E n Y, λ k+ 1 k y Yk σ J. A. O'S. ESE 54, Lecture 17, 03/19/09 8

13 Exaple: Poisson Plus Gaussian Yk k Iterations involve a λ 1 λ σ p y Y = e e nonlinear function in k 0 k! = πσ this case. k+ 1 k λ = En Y, λ Nuerical, analytical, or lookup k Y λ k approxiation ay be 1 λ σ e e required. k 0! = En Y, λ πσ = k Y λ k 1 λ σ e e = 0! πσ 1 k λ Y σ e k k = 1 1! En Y, λ = λ k Y λ σ e = 0! J. A. O'S. ESE 54, Lecture 17, 03/19/09 9

14 Estiate Variance of Gaussian in White Gaussian Noise Suppose that N i.i.d. easureents of zero ean Gaussian rando variables s n are ade in additive white Gaussian noise w n of fknown variance. Find the axiu likelihood estiate t of the unknown variance. Two approaches: Analytical E algorith r = s, 1,,..., + w = s i.i.d. N 0, P, w i.i.d. N 0, N, s wk,, k r iid i.i.d. N 0, P + N 1 1 r l r P = ln P+ N = 1 P + N First order necessary condition r = 1 l 1 1 = + = 0 P P+ N P + N 1 PL = r N ax,0 = 1 J. A. O'S. ESE 54, Lecture 17, 03/19/09 30

15 Gaussian Variance in AWGN Analytical: can find the solution directly E algorith Coplete data coprise the pairs of rando variables s n,w n r = s + w, = 1,,..., s i.i.d. N 0, P, w i.i.d. N 0, N, s w,, k k r i.i.d. N 0, P+ N 1 1 l r P = ln P+ N = 1 r P+ N Coplete e data a loglikelihood ood function s lcd s P = ln P P = 1 k cd s k Q P P = E l P r, P Q k Q PP 1 = ln P E s r, P 1 P = k 1 Q P P = ln P Es, r P = P 1 k+ 1 1 k P = Es r, P J. A. O'S. ESE 54, Lecture 17, 03/19/09 = 1 k k 31

16 Estiate Variance of Gaussian in White Gaussian Noise r = s + w, = 1,,..., Coplete data loglikelihood function 1 Q P P P E s r P k k = ln, = 1 P 1 P E s r P k+ 1 k =, = 1 E s r P Es r P E s E s r P r P k 1 k P k k Es = r, P r, var, = s r k P P k P + N 1+ P N k k k k,,,, = + k k+ 1 k P 1 k = + k P N = 1 P = P + r P N + J. A. O'S. ESE 54, Lecture 17, 03/19/09 3

17 Estiate Variance of Gaussian in White Gaussian Noise Fixed point at axiu likelihood solution Region of convergence? All positive starting ti points. Rate of convergence? Linear. See below, where the equality holds for positive estiate. If the estiate is 0, then the convergence is sublinear k 1 k k P + 1 k P = P + r k P N P N + = 1 k * k+ 1 * k P 1 * * k = k P N + = 1 P P P P r P N P P + k * k P = P P 1 k P + N * k P P 1 P + N * P * k P P J. A. O'S. * ESE 54, Lecture 17, 03/19/09 SNR = 1 1+ SNR 33

18 Estiation of a Source Distribution fro Sensor Array Data Suppose that an array of sensors is distributed over soe area on a two-diensional plane. The location of each array eleent is denoted x k,y k. A coplex-valued signal is incident upon the array fro angle θ,φ relative to the x-y axes θ is pitch, φ is yaw. For radar, sonar, or radio counications, the coplex values can represent the in phase and quadrature coponents of the signal. A narrowband approxiation is usually ade. The signal is assued to coe fro the far field. The sensors saple the incoing waves at their respective locations. There is assued to be additive white Gaussian noise at the sensors. Far field assuption: The curvature of the wavefront relative to the extent of the array is negligible. Thus the phase shift relative to the center of the array at the center frequency is deterined only by the direction cosines. Narrowband assuption: The bandwidth of the signal relative to the center frequency is sall often taken at less than 10%. The sapling rate of the sensors satisfies the Nyquist criterion at least twice the bandwidth. Sapling is often done at an interediate frequency or at baseband. The change in the signal across the sensor array is negligible. That is, relative to the axiu J. A. O'S. ESE delay 54, across Lecture 17, the 03/19/09 array, the signal does not change substantially. 34

19 Estiation of a Source Distribution fro Sensor Array Data The signal can coe fro a source or fro reflections of a transitted wave. For reflections, we assue diffuse and incoherent scatterers, so the saples of the reflectivity are independent and identically distributed rando variables. If they coe fro a collection of saller scatterers, then a coplex Gaussian odel is appropriate. For a source, we assue a distributed incoherent source, whose saples are well odeled by saples of a coplex Gaussian distribution. For coplex Gaussian distributions, the real and iaginary parts are independent Gaussian rando variables with zero ean and equal variance. This is also referred to as Goodan class. The AWGN is coplex Gaussian, independent of the signal. J. A. O'S. ESE 54, Lecture 17, 03/19/09 35

20 Estiation of a Source Distribution fro Sensor Array Data Under the assuptions, a signal vector received by the array equals a linear cobination of direction vectors ties coplex Gaussian rando variables. The data vector that is available equals this signal vector plus a noise vector. If all sensors are identical, and the noise is independent d fro sensor to sensor, the covariance atrix for the noise is a constant ties an identity atrix. Direction vector is deterined by the tie delay a θ, φ = cosθ cos φ, a θ, φ = sinφ x d θφ, = x a θφ, + y a θφ, k k x k y y dk θφ, τk θ, φ = c Phase shift is deterined by the tie delay and the center frequency j π f 0τk θ, φ d, e exp k θφ = j π λ0 Signal equals a linear cobination dk θ, φ skn = cn expjπ = 1 λ0 Data equals signal plus noise rkn = s, kn + wkn k = 1,,.., K, n= 1,,..., N J. A. O'S. ESE 54, Lecture 17, 03/19/09 36

21 Estiation of a Source Distribution fro Sensor Array Data Signal equals a linear cobination d, k θ φ skn = cn expjπ = 1 λ0 Data equals signal plus noise rkn = skn + wkn, k = 1 1,,.., K, n= 1 1,,..., N rn = sn + wn, wn CN 0, N0I sn = Acn d1 θ1, φ1 d1 θ, φ d1 θ, φ expjπ expjπ exp jπ λ0 λ0 λ0 d θ1, φ1 d θ, φ d θ, φ expjπ expjπ expjπ A = λ0 λ0 λ0 d θ1, φ1 d θ, φ d θ, φ expjπ K expjπ K expjπ K λ0 J. A. O'S. ESE 54, Lecture λ0 17, 03/19/09 λ0 37

22 Estiation of a Source Distribution fro Sensor Array Data Data equals signal plus noise r s w, w 0, I, i.i.d., 1,,..., n = n + n n CN N0 n= N s = Ac, c CN 0, Σ, i.i.d., iid n = 1, 1,..., N n n n n 0, + 0, i.i.d., = 1,,..., r CN AΣA N I n N A is the coplex conjugate transpose. ore on coplex Gaussian: Let x and y be independent Gaussian rando variables with zero ean and variances equal to σ. Then the joint probability density function is x + y x + y 1 x + y 1 1 σ σ N0 e = e = e. πσ πσ π N0 That is, the joint pdf is paraeterized by the total variance. J. A. O'S. ESE 54, Lecture 17, 03/19/09 38