Generalized Central Limit Theorem and Extreme Value Statistics

Transcription

1 Generalized Central Limit Theorem and Extreme Value Statistics Ariel Amir, 9/2017 Let us consider some distribution P (x) from which we draw x1, x2,..., xn. If the distribution has a finite standard deviation σ, then we can define: PN ξ (xi i=1 µ) σ N. (1) Where by the Central Limit Theorem P (ξ) approaches a Gaussian with vanishing mean and standard deviation of 1 as N. What happens when P (x) does not have a finite variance? Or a finite mean? It turns out that there is a generalization of the CLT, which will lead to a different form of scaling, which will depend on the power law tail of the distribution. Our analysis will have some rather surprising consequences in the case of heavy-tailed distributions where the mean diverges. In that case the sum of N variables will be dominated by rare events, and we will quantify this with the aid of extreme value distributions (characterizing for example the maximum of N i.i.d variables). Fig. 1 shows one such example, where a running sum of variables drawn from a distribution whose tail falls off as p(x) 1/x1+µ was used running sum N Figure 1: Running sum for heavy tailed distributions (blue: µ = 0.5, red: µ = 1.5, green: µ = 2.5.) mu=0.5 mu = 1.5 mu = 2.5

2 1 Probability distribution of sums An important quantity to define is the characteristic function of the distribution: ˆf(ω) = f(t)e iωt dt. (2) Given two variables x,y with distributions f(x) and g(y), consider their sum z = x + y. The distribution of z is given by: P (z) = f(x)g(z x)dx, (3) i.e., it is the convolution of the two variables. Consider now the Fourier transform of the distribution (i.e., its characteristic function): e iωz P (z)dz = f(x)g(z x)e iωz dxdz. (4) Changing variables to x and z x y, the Jacobian of the transformation is 1, and we find: e iωz P (z)dz = f(x)g(y)e iω(x+y) dxdy. (5) This is the product of the characteristic functions of f and y which will shortly prove very useful. This can be generalized to the case of the sum of n variables. Defining: we can write: ˆp(ω) = X = x 1 + x x n, (6)... p(x 1 )p(x 2 )..p(x n )dx 1 dx 2... dx n δ(x 1 + x x n X)e iωx dx, (7) which upon doing the integration over X leads to the product of the characteristic functions. 2

3 Finally, the same property also holds for the Laplace transform of positive variables, with essentially the same proof. 2 Tauberian theorems Consider n i.i.d variables drawn from a distribution p(x). The characteristic function of the sum is given by: g n (ω) = (f(ω)) n. (8) It is clear that the characteristic function is bounded by 1 in magnitude, and that as ω 0 it tends to 1. The behavior of the characteristic function near the origin is determined by the tail of p(x) for large x, as we shall quantify later, in what s known as a Tauberian theorem (relating large x and small ω behavior). Consider first a variable whose distribution has a finite variance σ 2 (and hence finite mean). Without loss of generality let s assume that the mean is 0 otherwise we can always consider ew variable with the mean subtracted. We can Taylor expand e iωx 1 + iω ω2 2 x2 near the origin, and find: f(ω) 1 + aω + bω , (9) with: a = i p(x)xdx = 0; b = 1 2 x 2 p(x)dx = σ2 2. (10) Therefore: g n (ω) (1 σ2 ω 2 2 )n. (11) Thus: g n (ω) e n log(1 σ2 ω 2 2 ) e n σ2 ω 2 2. (12) 3

4 In the next section we shall see the significance of the correction to the last expansion. Taking the inverse Fourier transform leads to: P n (x) e x2 2σ 2 n, (13) i.e., the sum is approximated by a Gaussian with standard deviation σ n, as we know from the CLT. Note that the finiteness of the variance associated with the properties of the tail of P (x) was essential to establishing the behavior of f(ω) for small frequencies. This is an example of a Tauberian theorem, where large x behavior determines the small ω behavior, and we shall see that this kind of relation holds also for cases when the variance diverges. Before going to these interesting scenarios, let s consider a particular example with finite variance, and find the regime of validity of the central limit theory. 2.1 Example: Cauchy distribution Consider the following distribution, known as the Cauchy distribution: f(x) = 1 γπ(1 + ( x γ )2 ), (14) (Note that this has the same Lorentzian expression as telegraph noise). Its Fourier transform is: ϕ(ω) = e γ ω. (15) Thus the characteristic function of a sum of N such variables is: ϕ(ω) = e Nγ ω, (16) 4

5 and taking the inverse Fourier transform we find that the sum also follows a Cauchy distribution: f(x) = 1 γπ(1 + ( x Nγ )2 ). (17) Thus, the sum does not converge to a Gaussian, but rather retains its Lorentzian form. Moreover, it is interesting to note that the scaling form governing the width of the Lorentzian evolves with N in a different way than the Gaussian scenario: while in the latter the variance increases linearly with N hence the width increases as N, here the scaling factor is linear in N. This remarkable property is in fact useful for certain computer science algoriths [REF]. 3 Generalized CLT We shall now use a different method to generalize this result further, to distributions with arbitrary support. We will look for distributions which are stable: this means that if we add two (or more) variables drawn from this distribution, the distribution of the sum will retain the same shape i.e., it will be identical up to a potential shift and scaling, by some yet undetermined factors. Clearly, if the sum of a large number of variables drawn from any distribution converges to a distribution with a well defined shape, it must be such a stable distribution. The family of such variables is known as Levy stable distributions. Clearly, a Gaussian distribution obeys this property. We also saw that the inverse Laplace transform of e xµ with 0 < µ < 1 obeys it. We shall now use a RG (renormalization group) approach to find the general form of such distributions, which will turn out to have a simple representation in Fourier rather than real space because the characteristic function is the natural object to deal with here. Defining the partial sums by s n, the general scaling one may consider is: ξ n = s n b n. (18) Here, determines the width of the distribution, and b n is a shift. If the distribution has a finite mean it seems plausible that we should center it by choosing b n = x n. We will show that this is indeed the case, and that if its mean is infinite we can set b n = 0. Note that we have seen this previously for the case of positive variables. 5

6 The scaling we are seeking is of the form of Eq. (18), and our hope is that: lim n P (ξ n = ξ) (19) exists, i.e., the scaled sum converges to some distribution P (x), which is not necessarily Gaussian (or symmetric). Let us denote the characteristic function of the scaled variable by ϕ(ω) (assumed to be independent of n as n ). Consider the variable y n = ξ n. We have: P (y n ) = 1 P (y n / ), (20) with P the limiting distribution of Eq. (19). Therefore the characteristic function of the variable y n is: ϕ y (ω) = ϕ( ω). (21) Note that the 1 prefactor canceled, as it should: since the characteristic function must be 1 at ω = 0. Consider next the distribution of the sum s n. We have s n = y n + b n, and: ˆP (s n ) = P (s n b n ). (22) It is straightforward to see that shifting a distribution by b n implies multiplying the characteristic function by e iωbn. Therefore the characteristic function of the sum is: Φ N (ω) = e ib N ω ϕ y (ω) = e ib N ω ϕ(a N ω). (23) This form will be the basis for the rest of the derivation, where we emphasize that the characteristic function ϕ is N-independent. Consider N = n m, where n; m are two large numbers. The important insight is to realize that one may compute s N in two ways: as the sum of N of the original variables, or as the sum of m variables, each one being the sum of n of the original variables. The sum of n variables drawn from the original distribution 6

7 is given by: Φ n (ω) = e ibnω ϕ( ω). (24) If we take a sum of m variables drawn from that distribution (i.e., the one corresponding to the sums of n s), then its characteristic function will be on one hand: f(ω) = e imbnω (ϕ( ω)) m, (25) and on the other hand it is the distribution of n m = N variables drawn from the original distribution, and hence does not depend on n or m separately but only on their product N. Therefore: ei N n bnω+ N n log[ϕ(anω)] = 0. (26) Defining bn n c n, we find: inω c n N n 2 log(ϕ) + N ϕ n ϕ ω = 0. (27) ϕ ( ω)ω ϕ( ω) = log(ϕ(ω)) n an iω c n n. (28) Multiplying both sides by and defining z ω, we find that: ϕ (z)z ϕ(z) log(ϕ(z)) n an + iz c n n = 0. (29) The equation for ϕ(z) has the mathematical structure: with C 1, C 2 constants. ϕ ϕ C 1 log(ϕ(z)) = ic 2, (30) z 7

8 If C 2 = 0, we can write the equation in the form: ϕ ϕ log(ϕ) = log(ϕ) log(ϕ) = [log(log(ϕ))] = C 1 z. (31) log(log(ϕ)) = C 1 log z + D ϕ = e A z C1. (32) This is the general solution to the homogenous equation. Guessing a particular solution to the inhomogeneous equation in the form log(ϕ) = Dz, leads to: As long as C 1 1, we found a solution! D C 1 D = ic 2 D = ic 2 1 C 1. (33) In the case C 1 = 1, we can guess a solution of the form log(ϕ) = Dz log(z), and find: D log(z) + D D log(z) = ic 2, (34) hence we have a solution when D = ic 2. Going back to Eq. (29), we can also get the scaling for the coefficients: n an = C 1 C 1 log( ) = 1/n. (35) This implies that: log( ) = 1 C 1 log(n) + const n 1/C 1. (36) Similarly: c n n = C 2. (37) 8

9 Hence: c n n1/c 1 2. (38) Therefore: c n = C 3 n 1/C C 4 b n = C 3 n 1/C 1 + C 4 n. (39) The first term will become a constant when we divide by the term n 1/C 1 of Eq. (18), leading to a simple shift of the resulting distribution. The second term will vanish for C 1 < 1. We shall soon see that the case C 1 > 1 corresponds to the case of a variable with finite mean, in which case the C 4 n term will be associated with centering of the scaled variable by subtracting their mean, as in the standard CLT. 3.1 General formula for the characteristic function According to Eqs. (32) and (33), the general formula for the characteristic function for C 1 1 is: ϕ(z) = e A z C 1 +Dz. (40) the D term is associated with a trivial shift of the distribution (related to the linear scaling of b n ) and can be eliminated. We will therefore not consider it in the following. The case of C 1 = 1 will be consider in the next section. The requirement that the inverse Fourier transform of ϕ is a probability distribution imposes that ϕ( z) = ϕ (z). Therefore the characteristic function takes the form: ϕ = e AωC 1 e A ω C 1 ω > 0 ω < 0. (41) This may be rewritten as: ϕ = e a ω µ [1 iβsign(ω)tan( πµ 2 )]. (42) The asymmetry of the distribution is determined by β. For this representation of ϕ, we will show that 9

10 1 β 1, and that β = 1 corresponds to a positive distribution, β = 1 to a strictly negative distribution, and β = 0 to a symmetric one. This formula will not be valid for µ = 1, a case that we will discuss in the next section. Consider f(x) which decays, for x > x, as f(x) = A +, (43) x1+µ with 0 < µ < 1. We shall now derive yet another Tauberian relation, finding the form of the Fourier transform of f(x) near the origin in terms of the tail of the distribution. For small, positive ω we find: Φ(ω) x A + x 1+µ eiωx dx = A + x ω e im dmω1+µ m1+µ ω, (44) where we substituted m = ωx. Evaluating the integral on the RHS by parts we obtain: I = [ ω µ m µ µ eim x ω + im µ ωµ x ω µ eim dm. (45) For µ < 1, we may approximate the integral by replacing the lower limit of integration by 0, to find: I x µ µ eix ω + ωµ µ i e im dm, (46) 0 m µ and e im 0 m dm = iγ(1 µ)e i π µ 2 µ (this can easily be evaluated using contour integration). Thus, for small ω we have: Φ(ω) Cω µ, (47) with C e i π 2 µ. It is possible to bound the rest of the integral to be at most linear in ω for small ω (up to a constant). Therefore the characteristic function near the origin is approximated by: ϕ(ω) 1 Cω µ, (48) 10

11 with: Im(C) Re(C) = tan(π 2 µ) C 1 i tan(π µ). (49) 2 This corresponds to β = 1 in our previous representation. If we similarly look at a distribution with a left tail, a similar analysis leads to the same form of Eq. (48) albeit with C 1 + i tan( π 2 µ), corresponding to β = 1 in Eq. (42). In the general case where both A + and A exit, we obtain the expression ϕ(ω) 1 ω µ (A + e iµ π 2 + A e iµ π 2 ). (50) with β defined as: Im(C) Re(C) = sin( π 2 µ) cos( π 2 µ) ( ) A+ A A + + A = tan( π µ)β, (51) 2 What about the case where 1 < µ < 2? β = A + A A + + A. (52) Note that in this case p(x) = g (x) where g decays as a power-law with 0 < µ < 1 but note that g will decay to a finite constant rather than to zero. Its Fourier transform will thus, by the same logic, be approximated by the following form for small ω: ĝ(ω) a 1 /ω + a 2 ω µ, (53) where the 1/ω divergence stems from the constant value of g at infinity. The Fourier transforms of the two functions are related by ˆp = iωĝ. Therefore for small ω we have: ˆp(ω) = 1 ( iω)cω µ + ibω, (54) where the constant B must be equal to the expectation value of x, and Eq. (49) gives the form of C. We 11

12 can simply this using the trigonometric relations tan(π/2µ) = tan(π/2 µ + π/2) = cot(π/2 µ), to find ic i[1 i tan( π 2 µ)] = tan(π µ) i. (55) 2 Up to a real constant, this is proportional to: 1 + i cot( π 2 µ) = 1 i tan(π µ). (56) 2 Thus the form of Eq. (42) (with β 1) is still intact also for 1 < µ < 2. Note that the linear term will drop out due to the shift of Eq. (18). Also, it is worth mentioning that the asymmetry term vanishes as µ 2: in the case of finite variance we always obtain a symmetric Gaussian, i.e., we become insensitive to the asymmetry of the original distribution. Special Cases µ = 1/2, β = 1: Levy distribution Consider the Levy distribution: p(x) = C e C 2x 2π (x) 3/2 (x 0) (57) The Fourier transform of p(x) for ω > 0 is ϕ(ω) = e 2iCω, (58) which indeed correspond to tan( π 2 µ = ) = 1 β = 1. The case µ = 1 and β = 0 corresponds to the previously discussed Cauchy distribution. In the general case µ = 1 and β 0, we have seen that the general form of the characteristic function is, according to Eqs. (32) and (34): ϕ(z) = e A z C 1 +Dz log(z). (59) 12

13 Repeating the logic we used before to establish the coefficients D for z > 0 and z < 0, based on the power-law tails of the distributions (which in this case fall off like 1/x 2 ) leads to: ϕ(ω) = e Cω [1 iβsign(ω)φ] ; φ = 2 log ω. (60) π This is the only exception to the form of Eq. (42). It remains to be shown why β = 1 corresponds to the case of a strictly positive distribution (which would thus justify the 2 π factor in the definition of φ). To see this, note that the logic following up to Eq. (45) is still intact for the case µ = 1. However, we can no longer replace the lower limit of integration by 0. Instead, note that the real part of the integral can be evaluated by parts, and diverges as log(ω), while the imaginary part of the integral does not suffer from such a divergence and we can well approximate it by replacing the lower limit of integration with 0. Therefore we find: I/ω log(ω) + i This leads to the form of Eq. (60). 0 sin(x)/xdx = log(ω) + iπ/2. (61) 4 Exploring the stable distributions numerically Since we defined the stable distributions in terms of their Fourier transforms, it is very easy to find them numerically by calculating the inverse Fourier transform. Here is a short code that finds the stable distribution for µ = 0.5 and β = 0.5 (i.e., corresponding to a distribution with right and left power-law tails with asymmetric magnitudes). It is easy to check that if β exceeds one, the inverse Fourier transform leads to a function with negative values and hence does not correspond to a probability distribution as we anticipated. mu=0.5; B=0.5; dt=0.001; w_vec=-10000:dt:10000; for indx=1:length(w_vec) w=w_vec(indx); f(indx)=exp(-abs(w)^mu/2*(1-i*b*tan(pi*mu/2)*sign(w))); 13

14 end; x_vec=-5:0.1:5; for indx=1:length(x_vec) x=x_vec(indx); y(indx)=0; for tmp=1:length(f) y(indx)=y(indx)+exp(-i*w_vec(tmp)*x)*f(tmp)*dt; end; end; y=y/(2*pi); plot(x_vec,y); Fig. 2 shows the result. There are much better numerical methods to evaluate the Levy stable distribution [1]. Fig. 3 shows examples for Levy distributions obtained using such a method. Note that for µ < 1 and β = 1 the support of the stable distribution is strictly positive, while this is not the case for µ > 1. This is expected, since for µ 1 we have seen that we can set the shift b n arbitrarily to zero and still convergence to the Levy stable distribution. Since the initial distribution is strictly positive, the scaled sum will also be strictly positive, and hence the result for the limiting distribution follows. Note that this arguments fails for µ > 1. 5 RG Approach for Extreme Value Distributions Consider the maximum of n variables drawn from some distribution p(x), characterized by a cumulative distribution C(x). We will now find the behavior of the maximum for large n, that will turn out to also follow universal statistics much like in the case of the generalized CLT that depend on the tails of p(x). This was discovered by Fisher and Tippett, motivated by an attempt to characterize the distribution of strengths of cotton fibers [2]. If we define: X n max(x 1, x 2,..., x n ), (62) 14

15 P(x) x Figure 2: Levy stable distribution for µ = 0.5 and asymmetry β = 1, obtained by numerically inverting Eq. (42). then we have: P rob(x n < c) = P rob(x 1 < C)P rob(x 2 < C)...P rob(x n < C) = C n (x). (63) For this reason it is natural to work with the cumulative distribution when dealing with extreme value statistics, akin to the role which the characteristic function played in the previous chapter. Clearly, it is easy to convert the question of the minimum of n variables to one related to the maximum, if we define p(x) = p( x). 15

16 µ = 0.5,β = 0 µ = 1,β = 0 µ = 1.5,β = 0 µ = 0.5,β = 1 µ = 1,β = 1 µ = 1.5,β = P(x) x Figure 3: Levy stable distributions for µ ranging from 0.5 to 1.5 and asymmetry β ranging from 0 to Example I: the Gumbel distribution Consider the distribution: p(x) = e x. (64) The cumulative is therefore: C(x) = 1 e x. (65) 16

17 The cumulative distribution for the maximum of n variables is therefore: G(x) = (1 e x ) n e ne x = e e (x µ), (66) with µ log(n). This is an example of the Gumbel distribution. Taking the derivative to find the probability distribution for the maximum, we find: p(x) = e e (x µ) e (x µ). (67) Denoting l e (x µ), we have: p(x) = e l l, (68) and we can find the maximum by taking the derivative with respect to l, leading to: e l l = e l l = 1. (69) Hence the distribution is peaked at x = log(n). It is easy to see that its width is of order unity. We can now revisit the approximation we made in Eq. (66), and check its validity. We have seen before that making the approximation: (1 x/n) n e x, (70) is valid under the condition x n. In our case, this implies: e x n n. (71) At the peak of the distribution, we have e x n = 1, and the approximation is valid for n 1. From 17

18 Eq. (71) we see that this would hold in a region of size o(log( n)) around it since the width of the distribution is of order unity, this implies that for large n the Gumbel distribution would approximate the exact solution well, until we are sufficiently far in the tail that the probability distribution is vanishingly small. However, ote of caution is in place: the logarithmic dependence we found signals a very slow converge to the limiting form. This is also true in the case where the distribution p(x) is Gaussian, as was already noted in Fisher and Tippett s original work [2] We will soon show that for the maximum of a large number of variables drawn from any distributions with a sufficiently fast decaying tail, the Gumbel distribution arises. 5.2 Example II: the Weibull distribution Consider the minimum of the same distribution we had in the previous example. The same logic would give us that: P rob[min(x 1,..x n ) > ξ] = P rob[x 1 > ξ]p rob[x 2 > ξ]..p rob[x n > ξ] = e ξn. (72) This is an example of the Weibull distribution, which occurs when the variable is bounded (e.g: in this case the variable is never negative). The general case, for the case of a maximum of n variables with distribution bounded by C, would be: C x e ( b ) α, x C G(x) = 0, x > C (73) 5.3 Example III: the Frechet distribution The final example will belong in the third possible universality class, corresponding to variables with a power-law tail. If at large x we have: p(x) = a, (74) (x b) 1+µ 18

19 Then the cumulative distribution is: Therefore taking it to a large power n we find: C(x) = 1 a µ(x b) µ. (75) an C n (x) e µ(x b) µ. (76) Upon appropriately scaling the variable, we find that: G(x) = e A ( x B n 1/µ ) µ, (77) (where A and B do not depend on n). We shall now show that these 3 cases can be derived in a general framework, using a similar approach to the one we used earlier. 5.4 General form for extreme value distributions We will follow similar logic to the RG used to find the form of the characteristic functions in the generalized CLT. Let us assume that there exists some scaling coefficients, b n such that when we define: The following limit exists: ξ n X b n, (78) lim P rob(ξ n = ξ) = g(ξ). (79) n (note that this limit is not unique: we can always shift and rescale by a constant). This would imply that p(x) a 1 n g logic we used before, we know that: ( ) ( X bn and the cumulative is given by: G X bn ). By the same ( ) X G m bn (80) 19

20 depends only on the quantity N = n m. Therefore we have: ( ( )) X G N/n bn = 0. (81) This implies that: ( ( ( ))) N X n log bn G = 0. (82) From this we can deduce that: N n 2 log G + N n Now, if we fix and b n, we have that: G [ G ( bn ) x a 2 n ] = 0. (83) G(z) (a zb) = c. (84) G(z) log G(z) From this we can see that: log(log(g)) = 1 a 3 log(a 1 z + a 2 ) (85) Now we simply solve for G to recover that G takes the form: G = e (a 1z+a 2 ) 1 a 3. (86) How can the three forms of extreme value distribution be recovered from this form? Frechet Distribution: One choice would yield the Frechet distribution: e a(x b) µ Frechet, (87) where we have µ > 0. Weibull Distribution: 20

21 A different choice of the signs of the coefficients of Eq. (86) yields the Weibull distribution: with µ > 0. Gumbel Distribution: e a(c x)µ Weibull, (88) Finally, if we take the limit as a 3 = 1/µ 0 one can choose an a such that: ( 1 ( x d bµ )) µ ( exp x d ). (89) b This gets us the Gumbel distribution: ( ( exp exp x d )) b Gumbel. (90) Let us now consider the scaling coefficients and b n. For the Frechet distribution, we have: G N/n (X) = e (N/n)a ( X bn an b ) µ. (91) Since this has to be n-independent, it is easy to see that we must have b n = const + b, and: a µ n 1/n n 1/µ. (92) Note that for µ < 1 this implies that the maximum scales in the same way as the sum! This elucidates the behavior of the Levy flights which we have seen earlier. By the same reasoning we find that for the Weibull distribution of Eq.(87) we have: G(X) = e ( bn X an )µ, (93) with b n = const and n 1/µ. Note that in this case because the original distribution is bounded, the distribution of the maximum 21

22 becomes narrower with larger n. In a similar fashion the scaling for the Gumbel distribution can be determined. By the same logic, the following expression should be n-independent: exp ( Nn exp ( X b n Equivalently, we can demand the following expression to be n-independent: )) d. (94) b X b n d + log(n/n). (95) b This implies, remarkably, that b n log(n) while = constant: the distribution retains the same shape and width, but (slowly) shifts towards larger values. Acknowledgments I thank Ori Hirschberg, Felix Wong and Po-Yi Ho for useful comments and the students of AM 203 at Harvard where this material was first tested. References [1] [2] Fisher, R.A. and Tippett, L.H.C., 1928, April. Limiting forms of the frequency distribution of the largest or smallest member of a sample. In Mathematical Proceedings of the Cambridge Philosophical Society (Vol. 24, No. 2, pp ). Cambridge University Press. 22