Introducing Entropy Distributions

Transcription

1 Graubner, Schdt & Proske: Proceedngs of the 6 th Internatonal Probablstc Workshop, Darstadt 8 Introducng Entropy Dstrbutons Noel van Erp & Peter van Gelder Structural Hydraulc Engneerng and Probablstc Desgn, TU Delft Delft, The Netherlands Introducton In ths paper we frst ntroduce an algorth presented by Rocknger and Jondeau to estate Entropy dstrbutons. We then proceed to deonstrate the ablty of these Entropy dstrbutons to capture the general characterstcs of a non-gaussan dstrbuton. Fnally we pont out a ltaton of the algorth, whch s essentally a ltaton of the nuercal pleentaton of nzaton algorths n Matlab and Maple. Defnng Entropy Dstrbutons Followng ROCKINGER & JONDEAU (), we wll defne Entropy dstrbutons to be those dstrbutons whch axze the nforaton entropy easure ( ) log ( ) H = p x p x dx () whle at sae te satsfyng the + oent constrants ( ) x p x dx = b,,,, = () where b =. It can be shown (JAYNES 3) that such Entropy dstrbutons always wll be of the followng (Pearsonan) for ( ) = exp( λ λ λ ) p x x x = exp( λ ) exp = λ x (3) 39

2 van Erp & van Gelder: Maxu Entropy Dstrbutons where, for a gven doan of x, { λ,..., λ } s the unque (JAYNES 3) set of Lagrange ultplers whch defnes the probablty dstrbuton whch axzes the nforaton entropy easure () whle at the sae te satsfyng the oent restrctons (). Three very well known Entropy dstrbutons are the Unfor, Exponentonal and Noral Dstrbutons, whch correspond, respectvely, to specal cases of =, = and =. However for 3, there are no explct solutons of the set { λ,, λ } n ters of the correspondng oent restrctons { b,, b }. So for 3 we wll have to fnd the specfc set { λ,, λ } whch axzes () under oent constrants () nuercally. In the followng we wll ntroduce a very clean algorth presented by ROCKINGER & JONDEAU () whch does just that. 3 The Rocknger and Jondeau Algorth Frst defne the doan of all possble values of x, say, D x. Then nuercally nze Q ( λ,, λ ) exp λ ( x b ) dx D (4) x = by soe coputng package lke Maple or Matlab. Now to obtan the Entropy dstrbuton one sply nserts the values of { λ,, λ } nto the equaton below p ( x) = P ( λ,, λ ) exp λ x (5) = where P ( λ,, λ ) exp D λ x dx (6) x = For a further treatent and an pleentaton of ths algorth n Matlab see Appendces A through C. 33

3 Graubner, Schdt & Proske: Proceedngs of the 6 th Internatonal Probablstc Workshop, Darstadt 8 4 Approxatng an Epercal Dstrbuton of a Lt State Equaton wth Entropy Dstrbutons If the dfference of the local water-level wth the water-level n the polder, h, exceeds the crtcal head dfference h p then falure due to ppng occurs, and the lt state equaton of ths falure s gven as: Z = hp h (7) In our proble h s defned to be deternstc wth value h =. 85, and h p s defned as a functon of eght stochastcs: h.8.8 D D = p B B B 3.68 η d 7 κ lnη d 3 7 ρ p ρ w κ B 3 tanθ (8) whth the stochastcs havng the followng dstrbutons: D ~ N (.5,.5), B ~ N(, ), ρ ~ N( 6.5,.5), k d 7 ρ ~ N w ~ LN (,. ), η ~ LN (.397,.49), (.397,.49), κ ~ LN ( 9.54,.94), (9) θ ~ N ( 38,.9) We saple functon (8) one llon tes n a Monte Carlo sulaton, and proceed to construct the followng epercal dstrbuton: 33

4 van Erp & van Gelder: Maxu Entropy Dstrbutons Fgure : epercal dstrbuton of ppng usng one llon Monte Carlo Realsatons The epercal dstrbuton n Fg. has as ts frst (standardzed) cuulants: µ = 4.57, σ = 6. 57, γ =. 69, κ = () ep ep ep ep In Fg. we now plot the frst-, second-, thrd-, fourth-order Entropy dstrbutons together wth the epercal dstrbuton seen n Fg., and we see that the Entropy dstrbutons quckly converge to the shape of ths target dstrbuton epercal dst. frst-order Entropy dst..6.5 epercal dst. second-order Entropy dst epercal dst. thrd-order Entropy dst..6.5 epercal dst. fourth-order Entropy dst

5 Graubner, Schdt & Proske: Proceedngs of the 6 th Internatonal Probablstc Workshop, Darstadt 8 Fgure : plot of the epercal dstrbuton together wth Entropy dstrbutons In order to get a better vew of the convergence of the ft n the tals we now gve the correspondng se-log-y plots of the c.d.f. s, and we see how the ncluson of ore cuulants gves a better ft at the left tal epercal dst. frst-order Entropy dst epercal dst. second-order Entropy dst epercal dst. thrd-order Entropy dst epercal dst. fourth-order Entropy dst Fgure 3: se-log-y plot of the epercal c.d.f. together wth Entropy c.d.f. s Lookng at Fg.3 one would expect that the ncluson of ore cuulants would lead to better fttng Entropy dstrbutons. Ths s probably true, but ths s also where we encounter a dffculty wth the Jondeau-Rocknger algorth, or rather wth the nzaton packages n Matlab and Maple. The proble beng that f we nze over fve or ore varables the nzaton packages of Matlab and Maple break down. We have here just another nstance of the curse of densonalty at work. To end on a ore postve note, f we have better nzaton algorths, then we probably ay construct hgher order Entropy dstrbutons. 333

6 van Erp & van Gelder: Maxu Entropy Dstrbutons 5 Dscusson So, havng that, n the absence of better nzaton algorths, the fourth-order MaxEnt dstrbuton s the hghest order MaxEnt dstrbuton currently avalable, what ght be the added value of these MaxEnt dstrbutons? For anyone who wants to use a dstrbuton that corrects for skewness and kurtoss the fourth-order MaxEnt dstrbuton ght be a welcoe adton over the well-known Gaussan dstrbuton, the latter beng just a specal case of a second-order MaxEnt dstrbuton. Rocknger and Jondeau, for exaple, develop n ther artcle a GARCH odel n whch the resdual s odelled usng a fourth order Entropy dstrbuton nstead of the ore tradtonal Gaussan and generalzed Student-t dstrbutons. References [] Entropy denstes wth an applcaton to autoregressve condtonal skewness and kurtoss, Rockenger & Jondeau (). Journal of Econoetrcs 6 () 9-4. [] Probablty Theory; the logc of scence, Jaynes (3). Cabrdge Unversty Press. 334

7 Graubner, Schdt & Proske: Proceedngs of the 6 th Internatonal Probablstc Workshop, Darstadt 8 6 Appendx A We are lookng for the densty p ( x ) whch axzes the nforaton entropy easure (), whle at sae te satsfyng the + oent constrants (). Step : Frst we agan take a look at (3). = exp ( ) exp( λ ) p x = λ x (A.) We notce that the factor exp( λ ) s ndependent of x and as such s a scalng factor whch guarantees that the th oent constrant s satsfed, p ( x) dx = exp ( λ ) exp λ = ( ) = x dx p x dx. It follows that exp( λ ) exp λ x dx = = (A.) exp = exp λ x = ( λ ) dx And we have that the scalng factor exp( λ ) equals the ntegral functon P( λ λ ),, P ( λ,, λ ) exp λ x dx (A.3) = So (A.) can be rewrtten as p ( x) = P ( λ,, λ ) exp λ x (A.4) = 335

8 van Erp & van Gelder: Maxu Entropy Dstrbutons It follows that the Entropy dstrbuton whch axzes the nforaton entropy easure () whle at the sae te satsfyng the oent restrctons (), s copletely deterned by the saller set { λ,, λ }. Step : We defne Q ( λ λ ),, as Q ( λ,, λ ) exp λ ( x b ) dx (A.5) = The followng relaton holds between P and Q : = = ( λ,, λ ) = exp λ exp λ ( λ,, λ ) P b b P = exp λb exp λb exp λ x dx = = = = exp λb exp λ ( x b ) dx = = (A.6) = exp λb Q λ,, λ = ( ) By substtutng the result of (A.6) nto (A.4), we can rewrte p ( x ) as p ( x) = P ( λ,, λ ) exp λ x = = Q ( λ,, λ ) exp λb exp λ x Step 3 = = (A.7) = Q x b = ( λ,, λ ) exp λ ( ) We now observe that fro () t follows that 336

9 Graubner, Schdt & Proske: Proceedngs of the 6 th Internatonal Probablstc Workshop, Darstadt 8 ( ) = ( ) ( ) x p x dx b x p x dx b p x dx = ( ) ( ) = x b p x dx (A.8) It follows out of equalty (A.8) that the gradent, say, g of Q equals the zero-vector: g Q ( λ,, λ ) δ s = Q( λ,, λ ) Q ( λ,, λ ) exp λs ( x bs ) dx δλ s= ( λ,, λ ) ( ) ( ) = Q x b p x dx (A.9) = The fact that g = ples that we are lookng for soe nu or axu of Q. We then proceed to fnd that the Hessan atrx, say, G j of Q equals a syetrc varance-covarance atrx of powers x,,, = : G j Q λ λ = j ( λ,, λ ) j [ Q( λ,, λ )] ( x b ) ( x b ) p( x) j dx (A.) And snce the powers x,,, =, are lnearly ndependent t follows that ths varancecovarance atrx G j ust be postve defnte and of full rank. It then follows fro the n-ax rule of ultvarate calculus that the set { λ,, λ } for whch Q s at a global nu s also the set for whch the Entropy dstrbuton (A.4) axzes the nforaton entropy easure () whle at the sae te satsfyng the oent restrctons (). Ths concludes the proof. 337

10 van Erp & van Gelder: Maxu Entropy Dstrbutons 7 Appendx B In the followng we gve a standardzaton of the above ROCKINGER & JONDEAU algorth whch akes t easer to pleent ths algorth. Ths standardzaton s also taken fro ROCKINGER & JONDEAU (). Let µ and σ be, respectvely, the ean and standard devaton of an arbtrary densty, then we ay defne the kth standardzed centralzed oent ( k ) µ as µ ( ) ( ) k k x µ (B.) k σ Suppose that we want to fnd the Entropy dstrbuton h( x ) under the followng oent constrants h( x ) : b = µ, b = σ, b =,, 3 3 µ b µ = (B.) We then ay, wthout any loss of generalty, set the frst two oent constrants b and b, respectvely, to zero and one, so as to get g ( y ) : b =, b =, 3 ( 3) b = µ,, ( ) b = µ (B.3) where y s related to x n the followng way x µ y =, σ dx dy = (B.4) σ Out of (B.), (B.3) and (B.4) t then follows that x µ g ( y) dy = g dx σ σ ( ) = h x dx (B.5) So f we evaluate the ROCKINGER & JONDEAU algorth (4) (6) usng oent constrants (B.), then we ay set the doan of y, say, D y, standard to 338

11 Graubner, Schdt & Proske: Proceedngs of the 6 th Internatonal Probablstc Workshop, Darstadt 8 [ ] D : 6, 6 (B.6) y snce ths wll correspond, as can be deduced fro (B.4), to a doan on x of nus and plus sx standard devatons around the ean [ ] D : µ 6 σ, µ + 6σ (B.7) x 8 Appendx C Scrpt : M Functon functon labda = M(a,b,skew,krt) labda = fnsearch(@(l) Q(l,a,b,skew,krt),[,,,]); functon y = Q(l,a,b,skew,krt) y = quad(@functe,a,b,[],[],l,skew,krt); functon y = functe(x,l,skew,krt) y = exp(x*l() + (x.^ - )*l() + (x.^3 - skew)*l(3) + (x.^4 - krt)*l(4)); Scrpt : M Functon functon c = M(a,b,labda) c = quad(@ruw,a,b,[],[],labda); functon y = ruw(x,labda) y = exp(x*labda() + x.^*labda() + x.^3*labda(3) + x.^4*labda(4)); Scrpt 3: MaxEnt Functon functon y = MaxEnt(x,u,sga,labda,c) y = (/c)*(/sga)*ruw((x-u)/sga,labda); functon y = ruw(x,labda) y = exp(x*labda() + x.^*labda() + x.^3*labda(3) + x.^4*labda(4)); Scrpt 4: Algorth 339

12 van Erp & van Gelder: Maxu Entropy Dstrbutons %cuulants u = 3; sga = ; skew = ; krt = 3.5; %ntegraton lts n standard devatons a = -6; b = 6; %algorth (nput voor MaxEnt functon) labda = M(a,b,skew,krt); c = M(a,b,labda); %graph lts A = u + a*sga ; B = u + b*sga; %graph x = A:.:B; y = MaxEnt(x,u,sga,labda,c); plot(x,y) 34