Chper Probbiliy, Esimors, nd Sionriy Consider signl genered by dynmicl process, R, R. Considering s funcion of ime, we re opering in he ime domin. A fundmenl wy o chrcerize he dynmics using he ime domin d is in erms of probbiliy. Here, we ssume he eisence of probbiliy densiy p() such h Prob{ b} = wih consrin h p() d = 1 (refer o Fig..1 for illusrion). b p() d, (.1) p() b Figure.1: A generic smple of signl nd corresponding probbiliy densiy funcion p() Remrk: In hese noes probbiliy disribuion will be clled P () = p(s)ds (.) so p() = dp ()/d. However, ofen p() is clled disribuion, oo, in which cse P () will be clled he cumulive disribuion. Thinking of in erms of probbiliy densiy or disribuion is poin of view h considers i s rndom process bu, i does no imply nyhing bou he cul process genering i, which my indeed be deerminisic..1 Descripion of Probbiliy Given p(), we define he epeced vlue of ny funcion of, f(), s E [f()] 11 f() p() d. (.3)
1 CHAPTER. PROBABILITY, ESTIMATORS, AD STATIOARITY.1.1 Men or Firs Momen of p() In he bsence of oher informion, E[f()] is good guess he vlue of f(). A spcil cse is when f(), so for i we define he men vlue of or he firs momen of p(): µ = E[] = Remrk 1: Oher possible good mesures for f() = migh be he medin = µ 1/, defined by µ 1/ he mode = µ m, defined by p(µ m ) p() p() d. (.4) p() d = µ 1/ p() d, or. Remrk : For he Gussin (orml) densiy funcion: [ p() = 1 ep 1 ( ) ] µ πσ σ (.5) µ = µ 1/ = µ m. In generl his is no rue, unless p() is symmeric wih respec o = µ..1. Vrince or Second Cenrl Momen of p() Anoher imporn epeced vlue is for f() = ( µ), which is clled vrince of or second cenrl momenum of p(): σ E [ ( µ) ] = ( µ) p() d. (.6) The squre roo of vrince, or σ, is clled sndrd deviion. σ mesures he spred of he rndom process bou is men, µ. 1 Plese lso noe h: σ = ( µ + µ ) p() d (.7) = E [ µ + µ ] (.8) = E [ ] µe[] + µ E[1] (.9) = E [ ] µ + µ (.1) = E [ ] (E[]), (.11) where E[1] nd E [ ] re he zeroh nd he second momens of p(), respecively..1.3 Higher Order Epeced Vlues Oher imporn nd frequenly used epeced vlues re skewness: α 3 = γ 1 = E [ ( µ) 3], (.1) (E [( µ) 3/ ]) which is bsed on he hird cenrl momen nd mesures symmery of he densiy funcion, nd kurosis: α 4 = β = E [ ( µ) 4] (E [( µ) ]), (.13) which is bsed on he fourh cenrl momen nd mesures pekedness of he densiy funcion. In ddiion we hve kurosis ecess γ = β 3. For Gussin disribuion/densiy funcions γ 1 = γ =. 1 The vrince is lso he covrince of wih iself (uo-covrince in Chper 3).
.. ESTIMATORS 13. Esimors µ, σ, γ 1, β nd oher momen reled quniies re properies of p() nd, indeed, define p() vi chrcerisic funcion. However, we usully do no know probbiliy disribuion p(). Wh we usully do hve re smples of : {( 1 ), ( ), ( 3 ),...} { 1,, 3,...}. (.14) Thus he quesion is, given he finie sequence { i } described sisics? wh re he bes esimes of he bove..1 Men Esimor Le us begin wih he bes esime of he men. Le µ be he esime of µ. Then he error in he esime is: e = ( i µ), (.15) nd we seek µ h minimizes e: Thus minimizion yields: which gives he following ( e = i i µ + µ ) = i µ i + µ. (.16) de d µ = = i + µ, (.17) µ = 1 where µ is he smple men or n esimor of µ. Remrk 1: An esimor â is consisen if i, (.18) lim â =. (.19) For µ = 1 i µ s by he Lw of Lrge umbers (or Chebishev Inequliy). Remrk : An esimor â is unbised if E [â] =. (.) For µ = 1 i, E[ µ] = 1 E[i ] = 1 (µ) = µ, hus i is unbised... Cenrl Limi Theorem Gussin (orml) densiies, Eq. (.5), ply n imporn role in eperimenl mesuremens becuse of cenrl limi heorem (CLM). CLM: If µ is he smple men of { i } from ny probbiliy densiy wih men µ nd vrince σ, hen he normlized rndom vrible y = i µ σ = µ µ σ/ (.1) refer o he sochsic sysems or rndom processes ebooks
14 CHAPTER. PROBABILITY, ESTIMATORS, AD STATIOARITY hs probbiliy densiy 3 p(y) (, 1) (.) s, where is he orml (i.e., Gussin) densiy wih zero men nd uni vrince. Remrk: p(y) 1 ep ( 1 ) [ 1 π y = ep 1 ( ) ] µ µ π(σ/ ) σ/ (.3) The CLT sys h he disribuion of esimes µ Gussin no mer wh he originl pren disribuion is, nd h he men of he mens E[ µ] = µ, wih σ µ = σ/. Thus, σ µ s, oo...3 Vrince, Skewness, nd Kurosis Esimors The obvious guess for he vrince σ esimor, give he logic of µ, would be: σ = 1 ( i µ) = 1 ( i i µ + µ ) = 1 i µ. (.4) However, E [ σ ] = 1 E [ ] i E [ µ ] = 1 E [ ] E [ µ ] = E [ ] E [ µ ] = E [ ] E[] + E [ µ] E [ µ ] }{{}}{{} σ σ µ (.5) = σ σ µ = σ σ using CLT. where we hve used he fcs h E [ ] [ ] i = E, nd E[] = E [ µ] since µ is unbised. Therefore, E [ σ ] = 1 σ σ nd Eq. (.6) is bised esimor. However, i is cler h he following is unbised smple vrince: σ = 1 ( i µ). (.6) 1 where The following esimors for skewness nd kurosis re lso unbised: k = σ, k 3 = γ 1 = k 3 k 3/ ( 1)( ), β = k 4 k, (.7) ( i µ) 3, k 4 = [( + 1) ( i µ) 4 3( 1) k ]. ( 1)( )( 3) 3 y is rndom vrible since µ is rndom vrible iself, which vries for differen smple ses. (.8)
.3. STATIOARITY 15.3 Sionriy Inuiively, sionriy mens no chnge over ime, or properies re consn over ime. Specificlly: SS: A process is sricly sionry if p(( 1 ), ( ),..., ( )) = p(( 1 + T ), ( + T ),..., ( + T )), where { 1,,..., } is n rbirry se of smple imes nd T is n rbirry ime shif. Here, p is join probbiliy disribuion: Prob { 1 1 b 1, b,..., b } = b1 b 1 b Remrk: SS is very srong ssumpion. momens re consn. WSS: A process is wide sense (wekly) sionry if: p( 1,,..., ) d d d 1. (.9) I implies, mong oher hings, h ll 1. E[( 1 )] = E[( )] = µ 1,. E[( 1 )( )] = E[( 1 )( 1 + τ)] R(τ), where τ = 1, i is 1 independen, nd R(τ) is clled he uocorrelion. 3. R() = E[ ] <, (i.e., is men squre inegrble or hs bounded power ) Remrk 1: WSS conins only he second order sisics nd is deque for liner sisics, i.e. he invesigion of liner relionships beween properies of he sysem wo differen imes. 4 Remrk : Heurisiclly speking, sionry signl comes from sedy se moion. A bi more precisely, i is governed by n invrin mesure on n rcor. The difficuly in geing hndle on he concep of sionriy is h i is hypohesis h is difficul o es. I is lso highly cone dependen nd depends on wh he smple funcions of rndom process re in probbiliy spce. Consider he following emples 1. = sin + n(), where n() is Gussin whie noise. Emine Fig.., where 1 smple funcions re ploed on op of ech oher. The whole funcion is no sionry since he men of chnges wih ime. However, sin T + n() is sionry for ny fied T.. = sin( + ) is sionry for rndom (rbirry) iniil ime or phse (Fig..3). 3. ow les go bck o = sin +n() cse. Here, verges over ime scle T 1 < 1 period will no be sionry. However, verges over T 1 period will be (pproimely) sionry (Fig..4). I is hrd o figure ou if his is he sme ide of sionriy. 4. Usully rnsien processes re no sionry, bu if we le he rnsiens o die ou we ge pproime sionriy of sedy se. For emple, moion of pricle in double-well poenil (refer o Fig..5) wih dissipion cn be described by: ẍ + γẋ + ( 1) =. (.3) ow le = X(, ), where iniil condiion is rndomly seleced from he bsin of rcion B 1 for = 1 equilibrium poin. Two differen smple funcions (soluions o wo differen iniil condiions: (., ) red line nd (, 1) blue line) re shown in Fig..5. Here, i is cler h he process is no sionry. However, we cn wi unil rnsiens die off nd hen we ge pproime sionriy. 4 One sill migh use liner sisics even if he process genering is nonliner (e.g., rndom number generor).
16 CHAPTER. PROBABILITY, ESTIMATORS, AD STATIOARITY Figure.: sin + n() for 1 differen smple funcions ploed ogeher p() Figure.3: sin( + ) for 3 differen rndom iniil ime Simple Sionriy Tes A simple es for he sionriy of sclr ime series {n } n=1 is he eminion of moving verges wih differen window lenghs = w conining M smples, refer o Fig..6. For he d poins in ech sliding ime window esime smple men µi nd is corresponding sndrd deviion σµb,i = σ bi /. If he smple mens µ bi sy minly inside he ± one sndrd deviion σµb,i of ech oher, we cn sy h our ime series is pproimely sionry. If, on he oher hnd, smple mens show significn vriion in ime hn our ime series, i is no sionry. For he illusrion of his concep plese refer o righ plos in Fig..6. Plese noe h our resuls will depend on he choice of ime window w s ws illusred in he hird emple of he previous secion. T ¼ T1 Figure.4: Differen ime scle for = sin + n() signl
17.4. ERGODICITY V().5. 1.5 1 B1.5 no sionry pproimly sionry.5 1 1.5 1 3 4 5 6 ime, 4 Figure.5: Top lef plo represens wo-well poenil V () = + 4, he boom lef plo shows he bsins of rcion for he equilibrium poins = ±1, nd he righ plo shows wo smple funcion from he B1 bsin of rcion for = 1 equilibrium..4 Ergodiciy Generl signl is smple funcion of rndom process. Thus every ril will genere new funcion. In generl, we hve been considering wo ypes of verging: ensemble verging nd ime verging. Ensemble verge is he verge of quniy over ll relizions (or rils) of rndom vrible. For emple, if is Gussin rndom vrible, p() for ech fied is Gussin. If i is sionry rndom process, hen p((1 )) = p(( )). ow le us look he men vlue of our rndom process for some fied ime j : µ bj, T 1 X i (j ), T (.31) where j is fied, nd we hve ol of T rils of our rndom process. In oher words he µ bj is n esime for Z µj = E [(j )] = (j ) p((j )) d(j ). (.3) Approimly Sionry M smples 1. 1.8 5 1 15 15 o Sionry 1.5 1 w.5 5 1 Figure.6: Moving verges wih differen window lengh w (lef plo), nd simple sionriy es (righ plos)
18 CHAPTER. PROBABILITY, ESTIMATORS, AD STATIOARITY Ensemble Averge Time Averge Figure.7: Ensemble nd ime verges Thus, he ensemble verge is he epeced vlue (or n esime of) rndom process for priculr fied ime. ow good quesion is: does µ j = µ k for j k? The nswer is yes for sionry process. Time verge is he verge using single smple funcion over some ime inervl. For emple, given ny funcion f, 1 T/ f = lim f() d (.33) T T T/ In priculr, for he ime men vlue esimes: µ = 1 T T d, or µ = 1 s ( i ), (.34) s where s is he number of ime smples of. Definiion: We sy h rndom process is ergodic if ensemble verges re equivlen o ime verges. More precisely, given f he sionry process is ergodic if µ f = µ f, or 1 T/ f() p() d = lim f() d. (.35) T T T/ Plese noe h in he bove equion on he lef hnd side he is fied nd we hve n ensemble verge, nd on he righ hnd side we hve ime verge of single smple funcion (see Fig..7). Rndom processes cn be ergodic for some f nd no for ohers. For emple, consider = A sin( + B) rndom process, where A nd B re Gussin rndom vribles. Then, his process is ergodic in he men (f() = ), bu no ergodic in he power (f() = ).
.4. ERGODICITY 19 Problems Problem.1 In Mlb, genere M rils of rndom processes wih he forms 1. = cos() + n();. = cos( + θ) + n(); nd 3. = A cos( + θ) + n(), where T for some smple ime T, n() is normlly-disribued (Gussin) rndom process wih zero men nd vrince σ, θ is uniformly-disribued consn rndom vrible on [; π], nd A is normlly-disribued consn wih zero men nd uni vrince. In ech cse, genere uniformly-smpled ime series { 1 ; ;... ; } where i = ( i ). Use he bove-genered ime series o emine he sionriy nd ergodiciy of he differen processes. Eperimen wih differen vlues of M,, T, nd σ o mke sure you ge good resuls. Discuss your findings. oe: Think crefully bou your smpling ime inervl nd he lengh of he ime series (relionship beween nd T ). Problem. Genere one long ime series of he rndom process (1) in Eercise 1. Esime he men of he process mny imes from differen subses of he ime series, nd show h he disribuion of µ is pproimely Gussin wih men nd vrince, s prediced by he Cenrl Limi Theorem. oe: he wy in which you mke he smples used for ech esime cn hve lrge impc on your resuls!