Effi x. X da Yi. Yi Xt Zi MSEIE Reading. and MA is detection rules What if we instead wish to estate. App

Transcription

1 Reading Estimation of Rado Variables In lecture 5 we saw how the likelihood and posterior distributions can be used to decide between two hypotheses resulting in ML and MA is detection rules What if we instead wish to estate an actual parameter regression instead of deciding among a few classification possible options E Yi Xt Zi possibly sub Gaussian noise where X is a parmeter of interestand Zi is zero areas X da Yi ly Zi App Miriam Mean Squared Error MUSE One approach is to minize the rear squared error fresz between the estimate and the true value i e MSEIE Effi x

2 Caseli No data suppose we w t to estate X without obtaining any observations What is the best choice In this case the expectation is only over X foot I so Elle XY E If ELITE x ELIE E Ex ELLEN x 2 Eth EET EQ XI Note that Had EEx are both deterministic so the last tern above becomes 2 I EEx Effect X 2 I EEx EES EE o Hence MSEII X EEx Elf X EEx bias E ar X We can't ar change x so the X that minimizes the above is I T MASE no data

3 Case2 ha e observed data Now we observe suppose Y and Unt to incorporate this information To do this we first establish the orthogonality principle Thy orthogonalityprinciple Let f g Y be an estimator of X If E HH Xgmt o for all functions h then E Ix gndyee x hhdy i e GH is the tense estimator Intuition Think of X and Y as vectors in 1122 Then we're trying to find the closest vector X i i Icy gas that lies in the direction of Y x gri Intuitively we want to project X onto the span of Y so the residual X g Y should be orthogonal to Y and any faction of Y

4 Proof Use the add and subtract trick E X KYD Ex X glyttgh he 11 Ex.cl x glxdyte lghl hlyl5 2Exy lx gh h1yi g1y 5141 The cross term is zero by assumption of the theorem Thus E.nl x hhdy te lxglyj1yte lghl hlyl5 a Ex high as desired We now present a Action gly that satisfies the orthogonality principle The Let gly TEETH Then for all fuctions h E hey Xgmt o

5 Boot Use the law of total probability E hlyllx gl.it Ey Em 441 x gi 11 E htt lx gly f ly dy hly EXIT g gigi Sylgldy o for gly ELIN g Co bury the theorems above we see that EE Muse with data App Linear MMSE Wienerfilter So ties finely E MY is too difficult so we neg restrict ourselves to sifter functions One such restriction is to require I to be hear really affine is Y i e I a Ytb

6 Let µ EEK ad my EFI Then E Ix E5 Elli la Hb Ef X Mx alt re c hex au b El X err ah my Lux au 21in any b E EX Mx a Ym b Note that E X nx attend Effy Mx a E Y nd o S E Ix E 2 E x err ah n Lux au Minimizing b is easy since we can rake the second term Zero by taking b b ax am To find a let I X ax ad I Y My We wish to minimize over a

7 EINE att Elf tart 2 IT We can ignore the first term so we wish to solve man a EET 2A EEE'T Differentiate and setting to zero we see that be careful when E E'T we generalize a Cov Xi nearly to vectors Hence the lie MMSE estate is i Femme Y m µ hear Mrs E App s Minimum Absolute Error MAE Another possible cost function is the absolute error A Elf II x As with homework 3 problem 4 the optimal MAE estrater is the medias of the resulting distribution minimum absolute Iµ me imf iyk error MAE

8 Maximum Likelihood Approacht ML The ML estimator is probably the most used aed has some nice that are beyond lie scope of this course It is properties defined as Eu aefyµ1 mi fitalihood Approaches Maxim a Posteriori MAP Just as with detection we can define the MAP estimator which is the same as ML if we have uniform proors TIA P ayirfyµ gh l

9 Dobby et X and Y be jointly Gaussian RVs with near Zero variances p z and Ty and ca i ee T y We say their co crime reef.ir is then Note X ad 1 Jx t have zero new Elegant ELI II i Fnl EEXIT its PDF the linear MUSE estate at themap estate of X Solution Fu pg 313 at the book we have that f Hy N Foray or'll 41 where fry try is the correlation coefficient Therefore EExt D ftp.n.y Fact For Z NIM re we here aznn am at Hence EIGHT Nco oxtail

10 The linear MUSE estate is calx 1 F ruse Y my thx Txt j l Hence AYER q y ftp.iy the LMMSE estate is the Muse estate Finally the ML estimate is Inc as faxgh og Nl any aili p.nl og expf IT x a g 2 The above is maximized for X estate is the So I t Px J so the ML

11 Probley Let Y Y EE NIde T Fire the classical ML estates of µ ad T2 Solution Since the RVs are ied we here f a.ua g ynl fii e ad the likelihood Lation B La nu y ynju.r J II e Taking the log again we get elu.at i.gl e 3 E glitter Hi FI Iz log12am II Ji

12 First differentiate 2 Tullar'S with respect to re 2 En bi i Elyin E o Now use this rt to find T 2 else a E ft II I z o N f II giant fz f lyi

13 Problemy Let Y i Yz Yu Poisson T Find the ML estate of the fer ha para Note This is an exople of classical ter estates where we want para to find the parameters of a distribution instead of the estate of Ru Solution are ied their joint PDF is the product of the Sincethe Yi's y individual f C AT e Further sire 1 is fired y y and therefore y denotes thatthis f ly 1 e it isn't really a fi ae yi y 1 fi E conditional distribution We want to optimize this over 1 so it will be easier to take the leg first We call this the log lihelihood l.ci UH t.gl f e E HI e'i yi log 111 loglyi A

14 Note that yi doesit depend on 1 So we can ignore it I ay E yi log 11 a FI lgtd.e.ge na Now differentiate and set to zero eat fi.es w I toeiji This is the Saple mean which makes sense because EEE I f r all i