II.D Many Random Variables

Transcription

1 II.D Many Random Varables Wth more than one random varable, the set of outcomes s an -dmensonal space, S x = { < x 1, x 2,, x < }. For example, descrbng the locaton and velocty of a gas partcle requres sx coordnates. The jont PDF p(x), s the probablty densty of an outcome n a volume element d x = dx around the pont x = {x 1, x 2,, x }. The jont PDF s normalzed such that p x (S) = d x p(x) = 1. (II.32) If, and only f, the random varables are ndependent, the jont PDF s the product of ndvdual PDFs, p(x) = p (x ). (II.33) The uncondtonal PDF descrbes the behavor of a subset of random varables, ndependent of the values of the others. For example, f we are nterested only n the locaton of a gas partcle, an uncondtonal PDF can be constructed by ntegratng over all veloctes at a gven locaton, p( x) = d 3 v p( x, v); more generally p(x 1,, x m ) = =m+1 dx p(x 1,, x ). (II.34) The condtonal PDF descrbes the behavor of a subset of random varables, for specfed values of the others. For example, the PDF for the velocty of a partcle at a partcular locaton x, denoted by p( v x), s proportonal to the jont PDF p( v x) = p( x, v)/. The constant of proportonalty, obtaned by normalzng p( v x), s = d 3 v p( x, v) = p( x), (II.35) the uncondtonal PDF for a partcle at x. In general, the uncondtonal PDFs are obtaned from Bayes Theorem as p(x 1,, x m x m+1,, x ) = p(x 1,, x ) p(x m+1,, x ). (II.36) ote that f the random varables are ndependent, the uncondtonal PDF s equal to the condtonal PDF. 32

2 The expectaton value of a functon F(x), s obtaned as before from F(x) = d x p(x)f(x). (II.37) The jont characterstc functon, s obtaned from the -dmensonal Fourer transformaton of the jont PDF, p(k) = exp k j x j j=1. (II.38) The jont moments and jont cumulants are generated by p(k) and ln p(k) respectvely, as [ ] n1 [ ] n2 ] n x n 1 1 xn 2 2 xn = [ p(k = 0), ( k 1 ) ( k 2 ) ( k ) [ ] n1 [ ] n2 ] n x n 1 1 xn 2 2 xn c = [ ln p(k = 0). ( k 1 ) ( k 2 ) ( k ) (II.39) The prevously descrbed graphcal relaton between jont moments (all clusters of labelled ponts), and jont cumulant (connected clusters) s stll applcable. For example x α x β = x α c x β c + x α x β c, and x 2 α x β = xα 2 c x β c + x 2 α c x β c + 2 x α x β c x α c + x 2 αx β c. (II.40) The connected correlaton, x α x β c, s zero f x α and x β are ndependent random varables. The jont Gaussan dstrbuton s the generalzaton of eq.(ii.15) to random varables, as p(x) = [ 1 (2π) det[c] exp 1 2 ( C 1 ) ] (x mn m λ m )(x n λ n ) mn, (II.41) where C s a symmetrc matrx, and C 1 s ts nverse,. The smplest way to get the normalzaton factor s to make a lnear transformaton from the varables y j = x j λ j, usng the untary matrx that dagonalzes C. Ths reduces the normalzaton to that of the product of Gaussans whose varances are determned by the egenvalues of C. The product of the egenvalues s the determnant det[c]. (Ths also ndcates that the matrx C must be postve defnte.) The correspondng jont characterstc functon s obtaned by smlar manpulatons, and s gven by p(k) = exp [ k m λ m 12 ] C mnk m k n, (II.42) 33

3 where the summaton conventon s used. The jont cumulants of the Gaussan are then obtaned from ln p(k) as x m c = λ m, x m x n c = C mn, (II.43) wth all hgher cumulants equal to zero. In the specal case of {λ m } = 0, all odd moments of the dstrbuton are zero, whle the general rules for relatng moments to cumulants ndcate that any even moment s obtaned by summng over all ways of groupng the nvolved random varables nto pars, e.g. x a x b x c x d = C ab C cd + C ac C bd + C ad C bc. (II.44) Ths result s sometmes referred to as Wck s theorem. II.E Sums of Random Varables & the Central Lmt Theorem Consder the sum X = x, where x are random varables wth a jont PDF of p(x). The PDF for X s p X (x) = d x p(x)δ (x ) x = 1 dx p (x 1,, x 1, x x 1 x 1 ), and the correspondng characterstc functon (usng eq.(ii.38)) s gven by p X (k) = exp k j=1 x j Cumulants of the sum are obtaned by expandng ln p X (k), (II.45) = p (k 1 = k 2 = = k = k). (II.46) ln p (k 1 = k 2 = = k = k) = k 1 =1 x 1 c + ( k)2 2 1, 2 x 1 x 2 c +, (II.47) as X c = x c, X 2 = x c x j c,. (II.48) If the random varables are ndependent, p(x) = p (x ), and p X (k) = p (k). The cross cumulants n eq.(ii.48) vansh, and the n th cumulant of X s smply the sum of the ndvdual cumulants, X n c = xn c. When all the random varables 34,j

4 are ndependently taken from the same dstrbuton p(x), ths mples X n c = x n c, generalzng the result obtaned prevously for the bnomal dstrbuton. For large values of, the average value of the sum s proportonal to, whle fluctuatons around the mean, as measured by the standard devaton, grow only as. The random varable y = (X x c )/, has zero mean, and cumulants that scale as y n c 1 m/2. As, only the second cumulant survves, and the PDF for y converges to the normal dstrbuton, lm p ( ) y = x x c = ) 1 exp ( y2. (II.49) 2π x2 c 2 x 2 c (ote that the Gaussan dstrbuton s the only dstrbuton wth only frst and second cumulants.) The convergence of the PDF for the sum of many random varables to a normal dstrbuton s a most mportant result n the context of statstcal mechancs where such sums are frequently encountered. The central lmt theorem states a more general form of ths result: It s not necessary for the random varables to be ndependent, as the condton 1,, m x 1 x m c O( m/2 ), s suffcent for the valdty of eq.(ii.49). II.F Rules for Large umbers To descrbe equlbrum propertes of macroscopc bodes, statstcal mechancs has to deal wth the very large number, of mcroscopc degrees of freedom. Actually, takng the thermodynamc lmt of leads to a number of smplfcatons, some of whch are descrbed n ths secton. There are typcally three types of dependence encountered n the thermodynamc lmt: (a) Intensve quanttes, such as temperature T, and generalzed forces, e.g. pressure P, and magnetc feld B, are ndependent of,.e. O( 0 ). (b) Extensve quanttes, such as energy E, entropy S, and generalzed dsplacements, e.g. volume V, and magnetzaton M, are proportonal to,.e. O( 1 ). (c) Exponental dependence,.e. O ( exp(φ) ), s encountered n enumeratng dscrete mcro-states, or computng avalable volumes n phase space. Other asymptotc dependences are certanly not ruled out a pror. For example, the Coulomb energy of ons at fxed densty scales as Q 2 /R 5/3. Such dependences are rarely encountered n every day physcs. The Coulomb nteracton of ons s quckly 35

5 screened by counter-ons, resultng n an extensve overall energy. (Ths s not the case n astrophyscal problems snce the gravtatonal energy can not be screened. For example the entropy of a black hole s proportonal to the square of ts mass.) In statstcal mechancs we frequently encounter sums or ntegrals of exponental varables. Performng such sums n the thermodynamc lmt s consderably smplfed due to the followng results. (1) Summaton of Exponental Quanttes Consder the sum S = E, (II.50) where each term s postve, wth an exponental dependence on, 0 E O ( exp(φ ) ), (II.51) and the number of terms, s proportonal to some power of. Such a sum can be approxmated by ts largest term E max, n the followng sense. Snce for each term n the sum, 0 E E max, E max S E max. (II.52) An ntensve quantty can be constructed from ln S/, whch s bounded by ln E max ln S ln E max + ln. (II.53) For p, the rato ln / vanshes n the large lmt, and (2) Saddle Pont Integraton ln S lm = ln E max = φ max. (II.54) Smlarly, an ntegral of the form I = dx exp ( φ(x) ), (II.55) can be approxmated by the maxmum value of the ntegrand, obtaned at a pont x max whch maxmzes the exponent φ(x). Expandng around ths pont, I = { [ dx exp φ(x max ) 1 ]} 2 φ (x max ) (x x max ) 2 +. (II.56) 36

6 ote that at the maxmum, the frst dervatve φ (x max ), s zero, whle the second dervatve φ (x max ), s negatve. Termnatng the seres at the quadratc order results n I e φ(x max) dx exp [ 2 ] φ (x max ) (x x max ) 2 2π φ (x max ) eφ(x max), (II.57) where the range of ntegraton has been extended to [, ]. The latter s justfed snce the ntegrand s neglgbly small outsde the neghborhood of x max. There are two types of correcton to the above result. Frstly, there are hgher order terms n the expanson of φ(x) around x max. These correctons can be looked at perturbatvely, and lead to a seres n powers of 1/. Secondly, there may be addtonal local maxma for the functon. A maxmum at x max, leads to a smlar Gaussan ntegral that can be added to eq.(ii.57). Clearly such contrbutons are smaller by O ( exp{ [φ(x max ) φ(x max )]}). Snce all these correctons vansh n the thermodynamc lmt, [ ln I lm = lm φ(x max ) 1 2 ln( φ ] (x max ) ) 1 + O( 2π 2) = φ(x max ). (II.58) The saddle pont method for evaluatng ntegrals s the extenson of the above result to more general ntegrands, and ntegraton paths n the complex plane. (The approprate extremum n the complex plane s a saddle pont.) The smplfed verson presented above s suffcent for the purposes of ths course. Strlng s approxmaton for! at large can be obtaned by saddle pont ntegraton. In order to get an ntegral representaton of!, start wth the result 0 dxe αx = 1 α. (II.59) Repeated dfferentaton of both sdes of the above equaton wth respect to α leads to 0 dxx e αx =! α +1. (II.60) Although the above result only apples to nteger, t s possble to defne by analytcal contnuaton a functon, Γ( + 1)! = 0 dxx e x, (II.61) 37

7 for all. Whle the ntegral n eq.(ii.61) s not exactly n the form of eq.(ii.55), t can stll be evaluated by a smlar method. The ntegrand can be wrtten as exp ( φ(x) ), wth φ(x) = lnx x/. The exponent has a maxmum at x max =, wth φ(x max ) = ln 1, and φ (x max ) = 1/ 2. Expandng the ntegrand n eq.(ii.61) around ths pont yelds,! dx exp ( ln 1 2 (x )2) e 2π, (II.62) where the ntegral s evaluated by extendng ts lmts to [, ]. Strlng s formula s obtaned by takng the logarthm of eq.(ii.62) as, ln! = ln ln(2π) + O( 1 ). (II.63) II.G Informaton, Entropy, and Estmaton Informaton: Consder a random varable wth a dscrete set of outcomes S = {x }, occurrng wth probabltes {p()}, for = 1,, M. In the context of nformaton theory, there s a precse meanng to the nformaton content of a probablty dstrbuton: Let us construct a message from ndependent outcomes of the random varable. Snce there are M possbltes for each character n ths message, t has an apparent nformaton content of ln 2 M bts;.e. ths many bnary bts of nformaton have to be transmtted to convey the message precsely. On the other hand, the probabltes {p()} lmt the types of messages that are lkely. For example, f p 2 p 1, t s very unlkely to construct a message wth more x 1 than x 2. In partcular, n the lmt of large, we expect the message to contan roughly { = p } occurrences of each symbol. The number of typcal messages thus corresponds to the number of ways of rearrangng the { } occurrences of {x }, and s gven by the multnomal coeffcent g =! M!. (II.64) Ths s much smaller than the total number of messages M n. To specfy one out of g possble sequences requres ln 2 g M p ln 2 p (for ), (II.65) More precsely, the probablty of fndng any that s dfferent from p by more than ± becomes exponentally small n, as. 38

8 bts of nformaton. The last result s obtaned by applyng Strlng s approxmaton for ln!. It can also be obtaned by notng that ( ) 1 = p = M p M { }!! g p p, (II.66) where the sum has been replaced by ts largest term, as justfed n the prevous secton. Shannon s Theorem proves more rgorously that the mnmum number of bts necessary to ensure that the percentage of errors n trals vanshes n the lmt, s ln 2 g. For any non-unform dstrbuton, ths s less than the ln 2 M bts needed n the absence of any nformaton on relatve probabltes. The dfference per tral s thus attrbuted to the nformaton content of the probablty dstrbuton, and s gven by I [{p }] = ln 2 M + M p ln 2 p. (II.67) Entropy: Eq.(II.64) s encountered frequently n statstcal mechancs n the context of mxng M dstnct components; ts natural logarthm s related to the entropy of mxng. More generally, we can defne an entropy for any probablty dstrbuton as S = M p() lnp() = lnp(). (II.68) The above entropy takes a mnmum value of zero for the delta functon dstrbuton p() = δ,j, and a maxmum value of lnm for the unform dstrbuton, p() = 1/M. S s thus a measure of dspersty (dsorder) of the dstrbuton, and does not depend on the values of the random varables {x }. A one to one mappng to f = F(x ) leaves the entropy unchanged, whle a many to one mappng makes the dstrbuton more ordered and decrease S. For example, f the two values, x 1 and x 2, are mapped onto the same f, the change n entropy s [ ] p 1 p 2 S(x 1, x 2 f) = p 1 ln + p 2 ln < 0. (II.69) p 1 + p 2 p 1 + p 2 Estmaton: The entropy S, can also be used to quantfy subjectve estmates of probabltes. In the absence of any nformaton, the best unbased estmate s that all M outcomes are equally lkely. Ths s the dstrbuton of maxmum entropy. If addtonal nformaton s avalable, the unbased estmate s obtaned by maxmzng the entropy subject to the 39

9 constrants mposed by ths nformaton. For example, f t s known that F(x) = f, we can maxmze S (α, β, {p }) = ( ) ( ) p() lnp() α p() 1 β p()f(x ) f, (II.70) where the Lagrange multplers α and β are ntroduced to mpose the constrants of normalzaton, and F(x) = f, respectvely. The result of the optmzaton s a dstrbuton p exp ( βf(x ) ), where the value of β s fxed by the constrant. Ths process can be generalzed to an arbtrary number of condtons. It s easy to see that f the frst n moments (and hence n cumulants) of a dstrbuton are specfed, the unbased estmate s the exponental of an n th order polynomal. In analogy wth eq.(ii.68), we can defne an entropy for a contnuous random varable (S x = { < x < }) as S = dx p(x) lnp(x) = lnp(x). (II.71) There are, however, problems wth ths defnton, as for example S s not nvarant under a one to one mappng. (After a change of varable to f = F(x), the entropy s changed by F (x).) Snce the Jacoban of a canoncal transformaton s unty, canoncally conjugate pars offer a sutable choce of coordnates n classcal statstcal mechancs. The ambgutes are also removed f the contnuous varable s dscretzed. Ths happens qute naturally n quantum statstcal mechancs where t s usually possble to work wth a dscrete ladder of states. The approprate volume for dscretzaton of phase space s set by Planck s constant h. 40