04. Random Variables: Concepts

Transcription

1 University of Rhode Island Nonequilibrium Statistical Physics Physics Course Materials Random Variables: Concepts Gerhard Müller University of Rhode Island, Creative Commons License This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 4. License. Follow this and additional works at: nonequilibrium_statistical_physics Part of the Physics Commons Abstract Part four of course materials for Nonequilibrium Statistical Physics (Physics 626), taught by Gerhard Müller at the University of Rhode Island. Entries listed in the table of contents, but not shown in the document, exist only in handwritten form. Documents will be updated periodically as more entries become presentable. Updated with version 2 on 5/3/216. Recommended Citation Müller, Gerhard, "4. Random Variables: Concepts" (215). Nonequilibrium Statistical Physics. Paper 4. This Course Material is brought to you for free and open access by the Physics Course Materials at DigitalCommons@URI. It has been accepted for inclusion in Nonequilibrium Statistical Physics by an authorized administrator of DigitalCommons@URI. For more information, please contact digitalcommons@etal.uri.edu.

2 Contents of this Document [ntc4] 4. Random Variables: Concepts Probability distributions [nln46] Characteristic function, moments, and cumulants [nln47] Cumulants expressed in terms of moments [nex126] Generating function and factorial moments [nln48] Multivariate distributions [nln7] Transformation of random variables [nln49] Sums of independent exponentials [nex127] Propagation of statistical uncertainty [nex24] Chebyshev s inequality [nex6] Law of large numbers [nex7] Binomial, Poisson, and Gaussian distribution [nln8] Binomial to Poisson distribution [nex15] De Moivre - Laplace limit theorem [nex21] Central limit theorem [nln9] Multivariate Gaussian distribution Robust probability distributions [nex19] Stable probability distributions [nex81] Exponential distribution [nln1] Waiting time problem [nln11] Pascal distribution [nex22]

3 Probability Distribution [nln46] Experiment represented by events in a sample space: S = {A 1, A 2,...}. Measurements represented by stochastic variable: X = {x 1, x 2,...}. Maximum amount of information experimentally obtainable is contained in the probability distribution: P X (x i ), P X (x i ) = 1. Partial information is contained in moments, i X n = i x n i P X (x i ), n = 1, 2,..., or cumulants (as defined in [nln47]), X = X (mean value) X 2 = X 2 X 2 (variance) X 3 = X 3 3 X X X 3 The variance is the square of the standard deviation: X 2 = σx 2. For continuous stochastic variables we have P X (x), dxp X (x) = 1, X n = dx x n P (x). In the literature P X (x) is often named probability density and the term distribution is used for F X (x) = P X (x i ) x i <x or F X (x) = in a cumulative sense. x dx P X (x )

4 Characteristic Function [nln47] + Fourier transform: Φ X (k) =. e ikx = dx e ikx P X (x). Attributes: Φ X () = 1, Φ X (k) 1. Moment generating function: [ + ] (ik) n Φ X (k) = dx P X (x) x n (ik) n = X n n! n! X n. = + n= n= dx x n P X (x) = ( i) n dn dk Φ X(k) n. k= Cumulant generating function: ln Φ X (k) =. (ik) n X n n! n=1 X n = ( i) n dn dk ln Φ X(k) n. k= Cumulants in terms of moments (with X =. X X ): [nex126] X = X X 2 = X 2 X 2 = ( X) 2 X 3 = ( X) 3 X 4 = ( X) 4 3 ( X) 2 2 Theorem of Marcienkiewicz: ln Φ X (k) can only be a polynomial if the degree is n 2. n = 1: ln Φ X (k) = ika P X (x) = δ(x a) n = 2: ln Φ X (k) = ika 1 2 bk2 P X (x) = 1 2πb exp ( ) (x a)2 2b Consequence: any probability distribution has either one, two, or infinitely many non-vanishing cumulants.

5 [nex126] Cumulants expressed in terms of moments The characteristic function Φ X (k) of a probability distribution P X (x), obtained via Fourier transform as described in [nln47], can be used to generate the moments X n and the cumulants X n via the expansions (ik) n Φ X (k) = X n (ik) n, ln Φ X (k) = X n. n! n! n= Use these relations to express the first four cumulants in terms of the first four moments. The results are stated in [nln47]. Describe your work in some detail. n=1 Solution:

6 Generating function [nln48] The generating function G X (z) is a representation of the characteristic function Φ X (k) that is most commonly used, along with factorial moments and factorial cumulants, if the stochastic variable X is integer valued. Definition: G X (z). = z x with z = 1. Application to continuous and discrete (integer-valued) stochastic variables: G X (z) = dx z x P X (x), G X (z) = z n P X (n). n Definition of factorial moments: X m f. = X(X 1) (X m + 1), m 1; X f. =. Function generating factorial moments: (z 1) m G X (z) = X m f, X m f = dm m! dz G X(z) m. z=1 m= Function generating factorial cumulants: (z 1) m ln G X (z) = X m f, m! m=1 X m f = dm dz ln G X(z) m. z=1 Applications: Moments and cumulants of the Poisson distribution [nex16] Pascal distribution [nex22] Reconstructing probability distributions [nex14]

7 Multivariate Distributions [nln7] Let X = (X 1,..., X n ) be a random vector variable with n components. Joint probability distribution: P (x 1,..., x n ). Marginal probability distribution: P (x 1,..., x m ) = dx m+1 dx n P (x 1,..., x n ). Conditional probability distribution: P (x 1,..., x m x m+1,..., x n ). P (x 1,..., x n ) = P (x 1,..., x m x m+1,..., x n )P (x m+1,..., x n ). Moments: X m 1 1 X mn n = dx 1 dx n x m 1 1 x mn n P (x 1,..., x n ). Characteristic function: Φ(k) = e ik X. (ik 1 ) m1 (ik n ) mn Moment expansion: Φ(k) = m 1!... m n! X m 1 1 X mn n. (ik1 ) m1 (ik n ) mn Cumulant expansion: ln Φ(k) = X m 1 1 Xn mn. m 1!... m n! (prime indicates absence of term with m 1 = = m n = ). Covariance matrix: X i X j = (X i X i )(X j X j ). (i = j: variances, i j: covariances). Correlations: C(X i, X j ) = X i X j Xi X j. Statistical independence of X 1, X 2 : P (x 1, x 2 ) = P 1 (x 1 )P 2 (x 2 ). Equivalent criteria for statistical independence: all moments factorize: X m 1 1 X m 2 2 = X m 1 1 X m 2 2 ; characteristic function factorizes: Φ(k 1, k 2 ) = Φ 1 (k 1 )Φ 2 (k 2 ); all cumulants X m 1 1 X m 2 2 with m 1 m 2 vanish. If X 1 X 2 = then X 1, X 2 are called uncorrelated. This property does not imply statistical independence.

8 Transformation of Random Variables [nln49] Consider two random variables X and Y that are functionally related: Y = F (X) or X = G(Y ). If the probability distribution for X is known then the probability distribution for Y is determined as follows: P Y (y) y = dxp X (x) P Y (y) = y<f(x)<y+ y dxp X (x)δ ( y f(x) ) = P X ( g(y) ) g (y). Consider two random variables X 1, X 2 with a joint probability distribution P 12 (x 1, x 2 ). The probability distribution of the random variable Y = X 1 + X 2 is then determined as P Y (y) = dx 1 dx 2 P 12 (x 1, x 2 )δ(y x 1 x 2 ) = dx 1 P 12 (x 1, y x 1 ), and the probability distribution of the random variable Z = X 1 X 2 as dx1 P Z (z) = dx 1 dx 2 P 12 (x 1, x 2 )δ(z x 1 x 2 ) = x 1 P 12(x 1, z/x 1 ). If the two random variables X 1, X 2 are statistically independent we can substitute P 12 (x 1, x 2 ) = P 1 (x 1 )P 2 (x 2 ) in the above integrals. Applications: Transformation of statistical uncertainty [nex24] Chebyshev inequality [nex6] Robust probability distributions [nex19] Statistically independent or merely uncorrelated? [nex23] Sum and product of uniform distributions [nex96] Exponential integral distribution [nex79] Generating exponential and Lorentzian random numbers [nex8] From Gaussian to exponential distribution [nex8] Transforming a pair of random variables [nex78]

9 [nex127] Sums of independent exponentials Consider n independent random variable X 1,..., X n with range x i and identical exponential distributions, P 1 (x i ) = 1 ξ e xi/ξ, i = 1,..., n. Use the transformation relation from [nln49], P 2 (x) = dx 1 dx 2 P 1 (x 1 )P 1 (x 2 )δ(x x 1 x 2 ) = dx 1 P 1 (x 1 )P 1 (x x 1 ), inductively to calculate the probability distribution P n (x), n 2 of the stochastic variable X = X X n. Find the mean value X, the variance X 2, and the value x p where P n (x) has its peak value. Solution:

10 [nex24] Transformation of statistical uncertainty. From a given stochastic variable X with probability distribution P X (x) we can calculate the probability distribution of the stochastic variable Y = f(x) via the relation P Y (y) = dx P X (x)δ (y f(x)). Show by systematic expansion that if P X (x) is sufficiently narrow and f(x) sufficiently smooth, then the mean values and the standard deviations of the two stochastic variables are related to each other as follows: Y = f( X ), σ Y = f ( X ) σ X. Solution:

11 [nex6] Chebyshev s inequality Chebyshev s inequality is a rigorous relation between the standard deviation σ X = X 2 X 2 of the random variable X and the probability of deviations from the mean value X greater than a given magnitude a. ( P [(x X ) 2 > a 2 σx ) 2 ] a Prove Chebyshev s inequality starting from the following relation, commonly used for the transformation of stochastic variables (as in [nln49]): P Y (y) = dx δ(y f(x))p X (x) with f(x) = (x X ) 2. Solution:

12 [nex7] Law of large numbers Let X 1,..., X N be N statistically independent random variables described by the same probability distribution P X (x) with mean value X and standard deviation σ X = X 2 X 2. These random variables might represent, for example, a series of measurements under the same (controllable) conditions. The law of large numbers states that the uncertainty (as measured by the standard deviation) of the stochastic variable Y = (X X N )/N is Prove this result. Solution: σ Y = σ X N.

13 Binomial, Poisson, and Gaussian Distributions [nln8] Consider a set of N independent experiments, each having two possible outcomes occurring with given probabilities. events A + B = S probabilities p + q = 1 random variables n + m = N Binomial distribution: P N (n) = Mean value: n = Np. Variance: n 2 = Npq. N! n!(n n)! pn (1 p) N n. [nex15]] In the following we consider two different asymptotic distributions in the limit N. Poisson distribution: Limit #1: N, p such that Np = n = a stays finite [nex15]. Cumulants: n m = a. P (n) = an n! e a. Factorial cumulants: n m f = aδ m,1. Single parameter: n = n 2 = a. [nex16] Gaussian distribution: Limit #2: N 1, p > with Np Npq. ( ) 1 P N (n) = 2π n2 exp (n n )2. 2 n 2 Derivation: DeMoivre-Laplace limit theorem [nex21]. Two parameters: n = Np, n 2 = Npq. Special case of central limit theorem [nln9].

14 [nex15] Binomial to Poisson distribution Consider the binomial distribution for two events A, B that occur with probabilities P (A) p, P (B) = 1 p q, respectively: P N (n) = N! n!(n n)! pn q N n, where N is the number of (independent) experiments performed, and n is the stochastic variable that counts the number of realizations of event A. (a) Find the mean value n and the variance n 2 of the stochastic variable n. (b) Show that for N, p with Np a >, the binomial distribution turns into the Poisson distribution P (n) = an n! e a. Solution:

15 [nex21] De Moivre Laplace limit theorem. Show that for large Np and large Npq the binomial distribution turns into the Gaussian distribution with the same mean value n = Np and variance n 2 = Npq: ( ) N! P N (n) = n!(n n)! pn q N n 1 P N (n) 2π n2 exp (n n )2 2 n 2. Solution:

16 Central Limit Theorem [nln9] The central limit theorem is a major extension of the law of large numbers. It explains the unique role of the Gaussian distribution in statistical physics. Given are a large number of statistically independent random variables X i, i = 1,..., N with equal probability distributions P X (x i ). The only restriction on the shape of P X (x i ) is that the moments X n i = X n are finite for all n. Goal: Find the probability distribution P Y (y) for the random variable Y = (X 1 X + + X N X )/N. ( ) P Y (y) = dx 1 P X (x 1 ) dx N P X (x N )δ y 1 N [x i X ]. N Characteristic function: Φ Y (k) dy e iky P Y (y), P Y (y) = 1 2π Φ Y (k) = ( ) k Φ N = dx 1 P X (x 1 ) = [ Φ (k/n) ] N, = dx N P X (x N ) exp dx e i(k/n)(x X ) P X (x) = exp ( ( ) 2 ( ) k k X O, N N i=1 dk e iky Φ Y (k). ( i k N ) N [x i X ] i=1 ( ) ) 2 k X 2 + N where we have performed a cumulant expansion to leading order. Φ Y (y) = [ ( )] 1 k2 X 2 k 3 N ( ) N + O exp k2 X 2. 2N 2 N 3 2N where we have used lim N (1 + z/n) N = e z. P Y (y) = N 2π X 2 exp with variance Y 2 = X 2 /N ) ( Ny2 = 2 X 2 1 2π Y 2 e y2 /2 Y 2 Note that regardless of the form of P X (x), the average of a large number of (independent) measurements of X will be a Gaussian with standard deviation σ Y = σ X / N.

17 [nex19] Robust probability distributions Consider two independent stochastic variables X 1 and X 2, each specified by the same probability distribution P X (x). Show that if P X (x) is either a Gaussian, a Lorentzian, or a Poisson distribution, (i) P X (x) = 1 2πσ e x2 /2σ 2, (ii) P X (x) = 1 π a x 2 + a 2, (iii) P X(x = n) = an n! e a. then the probability distribution P Y (y) of the stochastic variable Y = X 1 + X 2 is also a Gaussian, a Lorentzian, or a Poisson distribution, respectively. What property of the characteristic function Φ X (k) guarantees the robustness of P X (x)? Solution:

18 [nex81] Stable probability distributions Consider N independent random variables X 1,..., X N, each having the same probability distribution P X (x). If the probability distribution of the random variable Y N = X X N can be written in the form P Y (y) = P X (y/c N +γ N )/c N, then P X (x) is stable. The multiplicative constant must be of the form c N = N 1/α, where α is the index of the stable distribution. P X (x) is strictly stable if γ N =. Use the results of [nex19] to determine the indices α of the Gaussian and Lorentzian distributions, both of which are both strictly stable. Show that the Poisson distribution is not stable in the technical sense used here. Solution:

19 Exponential distribution [nln1] Busses arrive randomly at a bus station. The average interval between successive bus arrivals is τ. f(t)dt: P (t) = probability that the interval is between t and t + dt. t Relation: f(t) = dp dt. dt f(t ): probability that the interval is larger than t. Normalizations: P () = 1, Mean value: t dt tf(t) = τ. dt f(t) = 1. Start the clock when a bus has arrived and consider the events A and B. Event A: the next bus has not arrived by time t. Event B: a bus arrives between times t and t + dt. Assumptions: 1. P (AB) = P (A)P (B) (statistical independence). 2. P (B) = cdt with c to be determined. Consequence: P (t + dt) = P (A B) = P (A)P ( B) = P (t)[1 cdt]. d dt P (t) = cp (t) P (t) = e ct f(t) = ce ct. Adjust mean value: t = τ c = 1/τ. Exponential distribution: P (t) = e t/τ, f(t) = 1 τ e t/τ. Find the probability P n (t) that n busses arrive before time t. First consider the probabilities f(t )dt and P (t t ) of the two statistically independent events that the first bus arrives between t and t + dt and that no futher bus arrives until time t. Probability that exactly one bus arrives until time t: P 1 (t) = t Then calculate P n (t) by induction. Poisson distribution: P n (t) = dt f(t )P (t t ) = t τ e t/τ. t dt f(t )P n 1 (t t ) = (t/τ)n e t/τ. n!

20 Waiting Time Problem [nln11] Busses arrive more or less randomly at a bus station. Given is the probability distribution f(t) for intervals between bus arrivals. Normalization: dt f(t) = 1. Probability that the interval is larger than t: P (t) = Mean time interval between arrivals: τ B = t dt tf(t) = dt f(t ). dtp (t). Find the probability Q (t) that no arrivals occur in a randomly chosen time interval of length t. First consider the probability P (t + t) for this to be the case if the interval starts at time t after the last bus arrival. Then average P (t + t) over the range of elapsed time t. Q (t) = c dt P (t + t) with normalization Q () = 1. Q (t) = 1 τ B t dt P (t ). Passengers come to the station at random times. The probability that a passenger has to wait at least a time t before the next bus is then Q (t): Probabilty distribution of passenger waiting times: Mean passenger waiting time: τ P = g(t) = d dt Q (t) = 1 τ B P (t). dt tg(t) = dtq (t). The relationship between τ B and τ P depends on the distribution f(t). In general, we have τ P τ B. The equality sign holds for the exponential distribution.

21 [nex22] Pascal distribution. Consider the quantum harmonic oscillator in thermal equilibrium at temperature T. The energy levels (relative to the ground state) are E n = n ω, n =, 1, 2,... (a) Show that the system is in level n with probability P (n) = (1 γ)γ n, γ = exp( ω/k B T ). P (n) is called Pascal distribution or geometric distribution. (b) Calculate the factorial moments n m f and the factorial cumulants n m f of this distribution. (c) Show that the Pascal distribution has a larger variance n 2 than the Poisson distribution with the same mean value n. Solution: