Topic 10: The Law of Large Numbers

Size: px

Start display at page:

Download "Topic 10: The Law of Large Numbers"

Chrystal Walton
5 years ago
Views:

1 Topic : October 6, 2 If we choose adult Europea males idepedetly ad measure their heights, keepig a ruig average, the at the begiig we might see some larger fluctuatios but as we cotiue to make measuremets, we expect to see this ruig average settle ad coverge to the true mea height of Europea males. This pheomea is iformally kow as the law of averages. I probability theory, we call this the law of large umbers. Example. We ca simulate me s heights with idepedet ormal radom variables, mea 68 ad stadard deviatio 3.5. The folowig R commads perform this simulatio ad computes a ruig average of the heights. The results are displayed i Figure. > <-c(:) > x<-rorm(,68,3.5) > s<-cumsum(x) > plot(s/,xlab="",ylim=c(6,7),type="l") Here, we begi with a sequece X, X 2,... of radom variables havig a commo distributio. Their average, S = (X + X X ), is itself a radom variable. If the commo mea for the X i s is µ, the by the liearity property of expectatio, the mea of the average E[ S ] = (EX + EX EX ) = (µ + µ + + µ) = µ = µ () If, i additio, the X i s are idepedet with commo variace σ 2, the by the Pythagorea theorem for the variace of idepedet radom variables, Var( S ) = 2 (Var(X ) + Var(X 2 ) + + Var(X )) = 2 (σ2 + σ σ 2 ) = 2 σ2 = σ2. (2) So the mea of these ruig averages remais at µ but the variace is decreasig to at a rate iversely proportioal to the umber of terms i the sum. For example, the mea of the average height of Europea males is 68 iches, the stadard deviatio is σ/ = 3.5/ =.35 iches. For, males, the mea remais 68 iches, he stadard deviatio is σ/ = 3.5/ =.35 iches The mathematical result, the law of large umbers, tells us that the results of these simulatio could have bee aticipated. Theorem 2. For a sequece of idepedet radom variables X, X 2,... havig a commo distributio, their ruig average S = (X + + X ) has a limit as if ad oly if this sequece of radom variables has a commo mea µ. I this case the limit is µ. c 2 Joseph C. Watkis 29

2 Itroductio to Statistical Methodology s/ s/ s/ s/ Figure : Four simulatios of the ruig average S /, =, 2,..., for idepedet ormal radom variables, mea 68 ad stadard deviatio 3.5. The theorem also states that if the radom variables do ot have a mea, the as the ext example shows, the limit will fail to exist. We shall show with the followig example. Example 3. The stadard Cauchy radom variable X has desity fuctio Let Y = X. I a attempt to compute EY, ote that b b x f X (x) dx = 2 as b. Thus, Y has ifiite mea. b π f X (x) = π + x 2. x + x 2 dx = π l( + x2 ) b = l( + b 2 ) 3

3 Itroductio to Statistical Methodology We ow simulate idepedet Cauchy radom variables. > <-c(:) > y<-abs(rcauchy()) > s<-cumsum(y) > plot(s/,xlab="",ylim=c(-6,6),type="l") These radom variables do ot have a fiite mea. As you ca see i Figure 2 that their ruig averages do ot seem to be covergig. Thus, if we are usig a simulatio strategy that depeds o the law of large umbers, we eed to check that the radom variables have a mea. Exercise 4. Check the failure of the large umber of Cauchy radom variables via simulatios. Note that the shocks ca jump either up or dow i this case. Mote Carlo Itegratio Mote Carlo methods use stochastic simulatios to approximate solutios to questios too difficult to solve aalytically. This approach has see widespread use i fields as diverse as statistical physics, astroomy, populatio geetics, protei chemistry, ad fiace. Our itroductio will focus o examples havig relatively rapid computatios. However, may research groups routiely use Mote Carlo simulatios that ca take weeks of computer time to perform. For example, let X, X 2,... be idepedet radom variables uiformally distributed o the iterval [a, b] ad write f X for the desity of a uiform radom variable o the iterval [a, b]. The, by the law of large umbers, g(x) = g(x i ) Eg(X ) = i= b lsrge. Let s deote by I(g) the value of this itegral. Recall that i calculus, we defied the average of g to be b a b a a g(x)f X (x) dx = b a g(x) dx. We ca also iterpret this itegral as the expected value of g(x ). Thus, Mote Carlo itegratio leads to a procedure for estimatig itegrals. simulatig uiform radom variables X, X 2,..., X o the iterval [a, b], computig the average of g(x ), g(x 2 ),..., g(x ) to estimate the average of g, ad multiplyig by b a to estimate the itegral. b a g(x) dx Example 5. Let g(x) = + cos 3 (x) for x [, π], to fid π g(x) dx. The three steps above become the followig R code. > x<-ruif(,,pi) > g<-sqrt(+cos(x)ˆ3) > pi*mea(g) []

4 Itroductio to Statistical Methodology s/ Figure 2: Four simulatios of the ruig average S /, =, 2,..., for the absolute value of idepedet Cauchy radom variables. Note that the ruig averate does ot seem to be settlig dow ad is subject to shocks. The error i the estimate of the itegral ca be estimated by the variace as give i equatio (2). Var(g(X) ) = Var(g(X )). where σ 2 = Var(g(X )) = E(g(X ) µ g(x)) 2 = b a (g(x) µ g(x )) 2 f X (x) dx. Typically this itegral is more difficult to estimate tha b g(x) dx, our origial itegral of iterest. However, we ca see that the variace of the a estimator is iversely proportioal to, the umber of radom umbers i the simulatio. Thus, the stadard deviatio is iversely proportioal to. Mote Carlo techiques are rarely the best strategy for estimatig oe or eve very low dimesioal itegrals. R 32

5 Itroductio to Statistical Methodology does itegratio umerically usig the fuctio ad the itegrate commads. For example, > g<-fuctio(x){sqrt(+cos(x)ˆ3)} > itegrate(g,,pi) with absolute error < 3.8e-6 However, with oly a small chage i the algorithm, we ca also use this to evaluate high dimesioal multivariate itegrals. For example, i three dimesios, the itegral I(g) = g(x, y, z) dx dy dz ca be estimated usig Mote Carlo itegratio by geeratig three sequeces of uiform radom variables, The, X, X 2,..., X, Y, Y 2,..., Y, ad Z, Z 2,... Z I(g) g(x i, Y i, Z i ). (3) i= Example 6. To obtai a sese of the distributio of the approximatios to I(g), we perform simulatios usig uiform radom variable for each of the three coordiates to perform the Mote Carlo itegratio. The commad Ig<-rep(,) creates a vector of zeros. This is added so that R creates a place ahead of the simulatios to store the results. > Ig<-rep(,) > for(i i :){x<-ruif();y<-ruif();z<-ruif(); g<-32*xˆ3/(3*(y+zˆ4+)); Ig[i]<-mea(g)} > hist(ig) > summary(ig) Mi. st Qu. Media Mea 3rd Qu. Max > var(ig) [] > sd(ig) [] Exercise 7. Estimate the variace ad stadard deviatio of the Mote Carlo estimator for the itegral (3) based o = 5 ad radom umbers. Exercise 8. How may observatios are eeded i estimatig (3) so that the stadard deviatio of the average is.5? To modify this techique for a regio [a, b ] [a 2, b 2 ] [a 3, b 3 ] use idepeet uiform radom variables X i, Y i, ad Z i o the respective itervals, the b b2 b3 g(x i, Y i, Z i ) Eg(X, Y, Z ) = g(x, y, z) dz dy dx. b a b 2 a 2 b 3 a 3 a a 2 a 3 i= Thus, the estimate for the itegral is (b a )(b 2 a 2 )(b 3 a 3 ) g(x i, Y i, Z i ). i= 33

6 Itroductio to Statistical Methodology Histogram of Ig Frequecy 5 5 Figure 3: Histogram of Mote Carlo estimates for the itegral R R σ = Importace Samplig Ig R 32x3 /(y + z 4 + ) dx dy dz. The sample stadard deviatio I may of the large simulatios, the dimesio of the itegral ca be i the hudreds ad the fuctio g ca be very close to zero for large regios i the domai of g. Simple Mote Carlo simulatio will the frequetly choose values for g that are close to zero. These values cotribute very little to the average. Due to this iefficiecy, a more sophisticated strategy is employed. Importace samplig methods begi with the observatio that a better strategy may be to cocetrate the radom poits where the itegrad g is large i absolute value. For example, for the itegral e x/2 dx, (4) x( x) the itegrad is much bigger for values ear x = or x =. (See Figure 4) Thus, we ca hope to have a more accurate estimate by cocetratig our sample poits i these places. With this i mid, we perform the Mote Carlo itegratio begiig with Y, Y 2,... idepedet radom variables with commo desityf Y. The goal is to fid a desity f Y that is large whe g is large ad small whe g is small. The desity f Y is called the importace samplig fuctio or the proposal desity. With this choice of desity, we defie the importace samplig weights To justify this choice, ote that, the sample mea w(y ) = w(y i ) i= w(y) = g(y) f Y (y). (5) w(x)f Y (x) dy = g(x) f Y (x) f Y (x) dx = I(g). Thus, the average of the importace samplig weights, by the strog law of large umbers, still approximates the itegral of g. This is a improvemet over simple Mote Carlo itegratio if the variace decreases, i.e., Var(w(Y )) = (w(y) I(g)) 2 f Y (y) dy = σ 2 f << σ 2. 34

7 Itroductio to Statistical Methodology As the formula shows, this ca be best achieved by havig the weight w(y) be close to the itegral I(g). Referrig to equatio (5), we ca ow see that we should edeavor to have f Y proportioal to g. Example 9. For the itegral (4) we ca use Mote Carlo simulatio based o uiform radom variables. > Ig<-rep(,) > for(i i :){x<-ruif();g<-exp(-x/2)*/sqrt(x*(-x));ig[i]<-mea(g)} > summary(ig) Mi. st Qu. Media Mea 3rd Qu. Max > sd(ig) [] Based o a simulatios, we fid a sample mea value of ad a sample stadard deviatio of.394. Because the itegrad is very large ear x = ad x =, we choose look for a desity f Y to cocetrate the radom samples ear the eds of the itervals. Our choice for the proposal desity is a Beta(/2, /2), the f Y (y) = π y/2 ( y) /2 o the iterval [, ]. Thus the weight w(y) = πe y/ f Y (x)=//sqrt(x( x)).5 g(x) = e x/2 /sqrt(x( x)) x Figure 4: Importace samplig usig the desity fuctio f Y to estimate R g(x) dx. The weight w(x) = πe x/2 is the ratio g(x)/f Y (x). Agai, we perform the simulatio multiple times. > IS<-rep(,) > for(i i :){y<-rbeta(,/2,/2);w<-pi*exp(-y/2);is[i]<-mea(w)} > summary(is) Mi. st Qu. Media Mea 3rd Qu. Max. 35

8 Itroductio to Statistical Methodology > var(is) [].2595 > sd(is) [] Based o simulatios, we fid a sample mea value of ad a sample stadard deviatio of.44, about /9th the size of the Mote Carlo weright. Part of the gai is illusory. Beta radom variables take loger to simulate. If they require a factor more tha 8 times loger to simulate, the the extra work eeded to create a good importace sample is ot helpful i producig a more accurate estimate for the itegral. Simpso s rule ca also be used i this case ad we obtai the aswer Exercise. Evaluate the itegral e x 3 x dx times usig = 2 sample poits usig directly Mote Carlo itegratio ad usig importace samplig with radom variables havig desity f X (x) = x o the iterval [, ]. For the secod part, you will eed to use the probability trasform. Compare the meas ad stadard deviatios of the estimates for the itegral. The itegral is approximately Aswers to Selected Exercises 4. Here are the R commads: > par(mfrow=c(2,2)) > x<-rcauchy() > s<-cumsum(x) > plot (,s/,type="l") > x<-rcauchy() > s<-cumsum(x) > plot (,s/,type="l") > x<-rcauchy() > s<-cumsum(x) > plot (,s/,type="l") > x<-rcauchy() > s<-cumsum(x) > plot (,s/,type="l") This produces the plots below. Notice the differeces for the values o the x-axis 7. The stadard deviatio for the average of observatios is σ/ where σ is the stadard deviatio for a sigle observatio. From the output > sd(ig) [] We have that σ/ = σ/. Thus, σ Cosequetly, for 5 observatios, σ/ For observatios σ/ For σ/ =.5, we have that = (σ/.5) So we eed approximately 4 observatios.. For the direct Mote Carlo simulatio, we have 36

9 Itroductio to Statistical Methodology s/ -2 - s/ s/ s/ > Ig<-rep(,) > for (i i :){x<-ruif(2);g<-exp(-x)/xˆ(/3);ig[i]<-mea(g)} > mea(ig) [] > sd(ig) [] For the importace sampler, the itegral is 3 2 e x f X (x) dx. 37

10 Itroductio to Statistical Methodology To simulate idepedet radom variables with desity f X, we first eed the cumulative distributio fuctio for X, F X (x) = The, to fid the probability trasform, ote that x t dt = x t2/3 = x 2/3. u = F X (x) = x 2/3 ad x = F X (u) = u3/2. Thus, to simulate X, we simulate a uiform radom variable U o the iterval [, ] ad evaluate U 3/2. This leads to the followig R commads for both the simple Mote Carlo ad the importace sample simulatio: > Ig<-rep(,) > for (i i :){u<-ruif(2);g<-exp(-u)/uˆ(/3);ig[i]<-mea(g)} > mea(ig) [].4976 > sd(ig) [] > ISg<-rep(,) > for (i i :){u<-ruif(2);x<-uˆ(3/2); w<-3*exp(-x)/2;isg[i]<-mea(w)} > mea(isg) [].4845 > sd(isg) [].232 Thus, the stadard deviatio usig importace samplig is about 2/7-ths the stadard deviatio usig simple Mote Carlo simulatio. Cosequetly, we will ca decrease the umber of samples usig importace samplig by a factor of (2/7)

Topic 8: Expected Values

Topic 8: Expected Values Topic 8: Jue 6, 20 The simplest summary of quatitative data is the sample mea. Give a radom variable, the correspodig cocept is called the distributioal mea, the epectatio or the epected value. We begi