Estimation Theory Chapter 3

Transcription

1 stimatio Theory Chater 3

2 Likelihood Fuctio Higher deedece of data PDF o ukow arameter results i higher estimatio accuracy amle : If ˆ If large, W, Choose  P  small,  W POOR GOOD i Oly data samle Data PDFas a fuctio of ukow aramter likelihood fuctio i e i :

3 Likelihood Fuctio P 3 Small oise ose variace a More deedece o P 3 3 Large oise variace 3

4 Likelihood Fuctio P i =3 vs is the robability likelihood of observig =3 whe is the true value If σ =/3 => robability ~ for >4 sice most likely values of are 3 3σ =,4 Cocetratio of P i =3 restricts ossible values of PDF Cocetratio Parameter ccuracy

5 PDF Curvature To measure the sharess of the PDF, use curvature l P defied d by for Gaussia df s σ ote:, curvature icreases. Var ˆ Var l P I geeral, - - is a l P ote: tosimlify otatio, we deotep by P Curvature radom variable

6 Therefore, to measure l P Scalar CRLB Theorem PDF Curvature PDF curvature we use l P If PDF P satisfies mild regularity coditios, the the variace of ay ubiased lso, If Var ˆ ad oly if l l P I g estimator ˆ satisfies l P true value of ote : eectatio is w.r.t. where is determiistic

7 is satisfied for some fuctios g ad I, a estimator ca be foud that attais CRLB. That estimator is ˆ g ad the miimum variace is / I amle: Problem : Fid MVU W WG with Variace roach : Comute CRLB, e estimator If satisfyig CRLB boud,,,..., - of if estimator it eists ca be foud the we have MVU!

8 amle Cot d e / P / l P l  Var Var

9 but amle Cot d is Ubiased d has variace is MVU! What if we did ot recall that had these roerties? From CRLB Theorem, but lp lp equality hld holds iff I g

10 CRLB satisfies boud Var ˆ I o variac e ad as eected. Summary: CRLB Theorem ot oly gives us a lower boud o variace of ay ubiased estimator but also fids the estimator if it eisits that attais the lower boud

11 fficiet stimator stimator that attais Cramer Rao lower boud o variace termed as FFICIT it makes efficiet use of data VR VR ˆ ˆ ˆ efficiet & MVU ˆ MVU but ot efficiet

12 amle Phase stimatio estimate kow, f,,,.. - : f Cos W e f Cos cos l f P si cos f f si f

13 amle Cot d si4 si f f cos l f P cos4 cos f f cos l - f P f 4 cos f

14 amle Cot d cos4 cos4 f f But cos4 f cos4 f for f var ˆ Boud decreases as SR See roblem 3.7 i SK ot ear or / ercise : Comute CRLB for estimatig, fo, or oise variace for this eamle icreases

15 Fisher Iformatio CRLBdecreasesasSR icreases ad/or icreasesas eected. Doesefficiet estimator eist? MVU? Wedefie Do t Kow l P I - ote: I see remark o, ca t factorize l P Whe CRLB is attaied, estimator variace is recirocal of Fisher iformatio as thefisher Iformatio below dditivefor ideedet observatios whe Iformatio icreases variace ucertaity decreases

16 Fisher Iformatio Matri l l P P l, l P P l P l l P P I the each for df are also idetically distributed same observatios If i I the for each, df i I

17 s Fisher Iformatio Matri l P i -, for IID case CRLB What haes for erfectly ad artially- correlated data samles? See Problem 3.9 i SK Remark : l P I Usually easier to use the origial roof soo squarig icreases owers while differetiatio decreases owers form

18 Geeral CRLB for Sigals i WG Geeral CRLB for Sigals i WG W S,,,... - WG o Deeds W S P e S l S S P S The oly deedece of o the ukow arameter is through S

19 Geeral CRLB for Sigals i WG l S S S P l S P ˆ Var Var S chage result i more accurate estimators for their ukow their ukow arameters that chage more raidly as Sigals aramters

20 amle : If S S Var Var Try also for the hase eamle ote that the imortat asect of a sigal its rate of chage with here is

21 Trasformatio of Parameters CRLB to estimate g where df is arameteri zed by ad g. is some trasforma tio o liear i geeral g Var ˆ l quality is achieved iff l where I I θ ˆ g l g Proof soo

22 l DC L l i WG amle: DC Level i WG W 4 ˆ for CRLB g g deeds o! 4 l Var g  for.is efficiet was ˆ from revious slide? o rove usig theorem for efficiet, is distributed as

23 Var ot eve ubiased! I geeralefficiecy destroyed by o liear trasformatio. Maitaiedover liear affie trasformatios

24 ffie Trasformatios ˆ efficiet a ˆ b efficiet for sice a ˆ b a ˆ Var a ˆ b a b Var ˆ g a l a b CRLB b ow, we are ready to rove the CRLB Theorem

25 Proof of Scalar Cramer-Rao Lower B d CRLB Th Boud CRLB Theorem ˆ ˆ : stimator Ubiased give : g g stimate Objetive d l l : Coditio Regularity d ˆ g l d d d d key ste. itegratio ad atio differeti of order e iterchag ca we as log as hold always will coditio regularity Hece,

26 Proof cot., w.r.t. of sides both atig by differeti ow g d ˆ g l g d l ˆ d l g d l ˆ Sice l d, Sice Re Coditio gularity

27 Proof cot. ow let us cosider Cauchy - Schwarz iequality y fuctio form : w ghd w g d. for w with equality iff w h g c h for costat 'c'. Comare with vector form :.y d y Let, w g ˆ l h

28 Proof cot. iequality to, Schwarz Cauchy ly d l d. ˆ g ˆ var 3 l g ˆ var l regularity coditio l ow, w.r.t. By differetiatig d l d l

29 Proof cot. l l - from 3 ad 4 varˆ g l - l l l d l 4 d lterative form of Fisher Iformatio

30 Proof cot. ˆ l g h iff occurs quality 5 ˆ l g if, I articular c g c. w.r.t. 5 ate differeti c determie To 5 c l l ˆ c c l c c l - I l c

31 Proof for Geeral g quality i Cauchy-Schwartz iff l l Iθ ˆ g Multily 6 by ˆ g & take eectatios l ˆ g Usig g Iθ var ˆ Iθ ˆ g

32 From 6 g Proof for Geeral l I θ g I θ ˆ g l g I θ ---8 From 7 ad 8 varˆ g l

33 of MVU estimator Fid -,,, ~ G, w : with varies MVU estimator variace where amle e. MVU estimator of Fid / What if we are tryig to estimate the stadard deviatio of the oise? l l l l deviatio of the oise Is there a efficiet estimator i this case?? l 4 MVU & fficiet 4 ^ ^ 4 g I o deeds which var ˆ var ad ˆ 4

34 CRLB for Vector Parameter ssume Var ˆ i where I I is ˆ T roof i SK aedi 3B UBISD I I ii ii ij l l i i geeral j symmetricad ositive - defiite Fisher Iformatio Matri hece ivertible

35 Fisher Iformatio Matri for Sigal i WG g W S. -,,,... WG Deeds S e P WG o Deeds S P e l P i i S S P l l l j i j i j i S S P S S S S P j i j i

36 amle, ~ where g.4 : SK 3.6 G w w amle / T e -,,, l l l l l l l radom! 4 l &

37 amle cot l I 4 but ot i geeral case i this diagoal 4 4 ^ ˆ var CRLB for joit estimatio of the arameters is same as CRLB for searate estimatio assumig the other arameter is kow 4 var

38 Fittig Lie amle: had B. DC Level W B g B WG first I To Fid CRLB comute B l l I B l l I B B

39 amle Cot d amle Cot d B e l B l B B l B

40 amle Cot d l amle Cot d l l B l B B always, why? l B I

41 amle Cot d 6 But I 6 why? - symmetric lways I - 6

42 amle Cot d Var ˆ I Var Bˆ I Observatios : Var Var ow, Var Recall for W for Questio : Comute CRLB for data samles k, k+,... k+ where k is a iteger See Prob. 3. for geeral roof

43 more estimate we as decrease ot ca CRLB ˆ task difficult more arameters more estimate we as decrease ot ca CRLB 3 6 Bˆ CRLB Â CRLB lso, or S i chage large B causes i chage slight is Reaso estimate. easier to B is S or i S chage large i B causes B B S B B B

44 more sesitive to chages i B tha to o oise =,B= =B= =,B= =B= 3 4

45 amle.57 amle :SK 3.4 & where, -,,, cos o o f w f ca be show that It o T o o f f f cos cos It ca be show that, o o I do calculatio ad submit et class si cos, o o I si

46 amle cot. By usig i It i ca I be the i var ˆ where, i followig idetities si 4f o cos 4 f o show that show for i them!,, ad f ot ear or /. fter iversio, we get varfˆ SR o Blows u for = why? - var ˆ Questio : how do these bouds comare to those Of sigle-arameter ad two arameter estimatio?

47 CRLB Theorem- Vector Parameter C ˆ as ˆ covariace matri of Deote Z Y 3 If T Z ˆ ˆ ˆ ˆ ˆ C T ˆ ˆ Z Y Z Y ˆ ˆ ˆ ˆ Vector Row Colum Vector

48 Var ˆ Covˆ Cov, Ŷ Y Covˆ Cov, Ẑ Z Covˆ,Ŷ VarŶ CovŶ, Ẑ ˆ ˆ ˆ ˆ ˆ Cov, Z CovY, Z VarZ If PDF satisfies regularity coditios, the C ˆ I is POSITIV SMIDFIIT. valuate I at true value of. ubiased estimator attais CRLB iff l I g ote : regularity coditio imlies, from quatio *, that efficiet estimator is ubiased *

49 for some g ad I. That efficiet i adhece MVU estimator is ˆ g ad its covariace PLTIO is a ad matri is C ˆ I ositive semidefiite matri if T for all T C I articular, ˆ I let... th i PLC it is symmetric T e i

50 But e T i ei ii e T i C C - I θ e θˆ θˆ ii Var ˆ i I i θ I θ as before ii ii ow, cosider l GRDIT θ

51 l l For equality i CRLB to hold, we require l θ I g θˆ ˆ is efficiet θ

52 Remarks CRLB theorem ca be alied wheever is available. I articular, it is directly alicable to all additive-oise models where it is easy to write oise eed ot be i.i.d. or zero-mea Gaussia Key ste is to write dow the eressio for y geeral -dimesioal oise df

53 Revisitig MPL o Vector CRLB l Lie Fittig l l l B B ˆ B θˆ-θ Iθ ˆ 6 B B θ θ Iθ 6 ˆ Verify it is bi d ubiased

54 MPL Cot d Bˆ 6 Verify it is ubiased ot obvious! This eamle is a secial case of Liear Model Chater 4

55 oucemets HW : Chater 3 : from SK 3., 3.5,3.7,3.,3. Review roblems : 3.4, 3.9, 3.3 Sectios ot covered i Chater 3 : 3.8,3.9,3.,3. ecet amle 3.4