Statistics 3657 : Moment Approximations

Transcription

1 Statistics 3657 : Moment Approximations Preliminaries Suppose that we have a r.v. and that we wish to calculate the expectation of g) for some function g. Of course we could calculate it as Eg)) by the appropriate integral continuous r.v. ) or sum discrete r.v. ). This may not always be easy, for example when itself is a function of several other r.v. s. An approximation can be constructed using Taylor s series. In practice usually we are only interested in first or second order approximations. Below we assume that g has enough derivatives for the expressions to be valid. A first order Taylor s series of a function g, about x 0 is where R x) is the remainder term. gx) = gx 0 ) + g x 0 )x x 0 ) + R x) A second order Taylor s series of a function g, about x 0 is where R 2 x) is the remainder term. gx) = gx 0 ) + g x 0 )x x 0 ) + 2 g x 0 )x x 0 ) 2 + R 2 x) The remainder terms R and R 2 are of the forms R x) = 2 g x )x x 0 ) 2 R 2 x) = 3! g3) x )x x 0 ) 3 where x is some number between x 0 and x. More specifically there is a number α [0, ] such that x = αx 0 + α)x. Typically it is not easy to calculate x, but there are often nice upper bounds on the remainder terms. However in this course we will not be concerned with these beyond noting that they exist. In further probability courses these are studied.

2 Moment Approximations 2 Now consider the Taylor approximation obtained by taking the polynomial part of the Taylor s series, but ignoring the remainder. That is we consider g x) = gx 0 ) + g x 0 )x x 0 ) g 2 x) = gx 0 ) + g x 0 )x x 0 ) + 2 g x 0 )x x 0 ) 2 The function g is a polynomial of degree, and the function g 2 is a polynomial of degree 2. We also call g a first order Taylor s approximation of the function g, and call g 2 a second order Taylor s approximation of the function g. 2 Moment Approximation Let be a random variable with enough finite moments so that the calculations below are well defined. They will be given in more detail below as needed. Suppose that E) = µ is finite, and that σ 2 = Var) is also finite. Let g be the first order Taylor s approximation of g about the x 0 = µ, that is g x) = gµ) + g µ)x µ). Consider the random variable g ). It is an approximation to the random variable g). We can use moments of g ) to approximate moments of g). Using this first Taylor s approximation we can easily calculate and Eg )) = E gµ) + g µ) µ)) = gµ) + g µ)e µ) = gµ) Varg )) = Var gµ) + g µ) µ)) = g µ)) 2 Var µ) = g µ)) 2 σ 2. The idea of the moment approximation is to approximate Eg)) and Varg)) by moments of the Taylor functions used to approximate g.

3 Moment Approximations 3 Example : Suppose Unif, 3) and Y =. Find the first order Taylor approximation to the mean and variance of Y. Also find the variance of, as Var) = 3. µ = E) = 3 x 2 dx = 2 Consider gx) = x as a mapping from [,3] to the reals. g x) = 2 x /2. Thus the first order Taylor s approximation about µ = 2 is Then g x) = g2) + 2 x 2). 2 Eg )) = 2 ) 2 Varg )) = = 24 Thus we find the first order Taylor s approximation for the mean and variance of Y as If we approximate Eg)) by EY ) Eg )) = 2 VarY ) Varg )) = 24 Eg )) = Egµ) + g µ) µ)) = gµ) we do not have a good approximation. Thus we often use the second order Taylor series to approximate Eg)), that is Eg 2 )) = Egµ) + g µ) µ) + 2 g µ) µ) 2 ) = gµ) + g µ) σ 2. 2 To use the second order approximation for the variance we obtain Varg 2 )) = Var gµ) + g µ) µ) + 2 ) g µ) µ) 2 = g µ) 2 Var) + 2g µ) 2 g µ)cov µ, µ) 2 ) + 4 g µ) 2 Var µ) 2 )

4 Moment Approximations 4 Aside : The student should find the mean and variance of Y =. These are exactly and rounded to 4 decimal places 27 EY ) = = VarY ) = = Aside : In general there is no need to carry more digits than required to get a meaningful answer. Here 3 or 4 decimal digits is reasonable, whereas decimal digit is not as it would give a variance of approximately 0.0, and hence not reasonable. Remark Recall VarY 0. Also Varg )) 0 and Varg 2 )) 0. However it is not always true that Eg 2 ) 2 ) Eg ))) 2 is greater than or equal to 0. Similarly it is also not guaranteed that Eg ) 2 ) Eg 2 ))) 2 0. Thus when approximating variances with this method we should in general not mix the two different degrees of approximation.

5 Moment Approximations 5 Functions of Several Variables and Moment Approximation This idea of moment approximation also extends to functions of several variables. consider two variables. Here we only Suppose, Y ) are bivariate r.v.s and Z = g, Y ). We suppose that g has derivatives and hence Taylor approximations. For ease of writing we use the notation g x x, y) = gx, y) x g y x, y) = gx, y) y g xx x, y) = 2 gx, y) 2 x g yy y, y) = 2 gx, y) 2 y g xy x, y) = 2 gx, y). x y First order Taylor s approximation about x 0, y 0 ) : g x, y) = gx 0, y 0 ) + g x x 0, y 0 )x x 0 ) + g y x 0, y 0 )y y 0 ). Second order Taylor s approximation about x 0, y 0 ) : g 2 x, y) = gx 0, y 0 ) + g x x 0, y 0 )x x 0 ) + g y x 0, y 0 )y y 0 ) + 2 g xxx 0, y 0 )x x 0 ) 2 + g xy x 0, y 0 )x x 0 )y y 0 ) + 2 g yyx 0, y 0 )y y 0 ) 2. and Again we approximate Eg, Y )) Eg 2, Y )) Varg, Y )) Varg, Y )) where the Taylor s approximation are obtained about x 0, y 0 ) = µ, µ Y ).

6 Moment Approximations 6 Example Suppose that and Y are independent and we are interested in the ratio g, Y ) = Y. and Then g 2, Y ) = µ Y µ µ Y g, Y ) = µ Y µ Y µ µ 2 µ ) + Y µ Y ) µ 2 µ 2 µ ) + Y µ Y ) + µ 2µ Y µ 3 µ ) µ )Y µ Y ) + 0 Y µ Y ) 2 = µ Y µ Y µ µ 2 µ ) + Y µ Y ) + µ Y µ µ 3 µ ) 2. Using the second order Taylor approximation we obtain Eg, Y )) Eg 2, Y )) = µ Y µ Y µ µ 3 σ 2. Using the first order Taylor approximation we obtain Varg, Y )) Varg, Y )) µy = Var µ Y µ µ 2 µ ) + ) Y µ Y ) µ = µ2 Y µ 4 σ 2 2 µ Y µ 3 Cov, Y ) + µ 2 σy 2 = µ2 Y µ 4 σ 2 + µ 2 σy 2.

7 Moment Approximations 7 QQ Plots Consider the iid random variables, 2,..., n from a continuous cdf F. Next consider the ordered data ), 2),..., n), that is the order statistics from these n r.v.s. Notice that U i = F i ) are iid Uniform0,), and that U i) = F i) ). Thus i) = F U i) ) We can find the marginal distribution of U i) from the methods above. Thus i) = F U i) ) In Chapter 3.7 we found the distribution of i), specifically it has the pdf, say i given by f i x) = i )!n i)! F x)i fx) F x)) n i where F and f are the cdf and pdf of U Uniform0, ). In particular we have In the case of Uniform0,) r.v.s we then obtain the marginal pdf of U i), say g i g i x) = Remark : This is a special case of a Beta distribution. { i )!n i)! xi x) n i if 0 < x < 0 otherwise Remark : Since this is a probability density for any positive integer n and i i n, we also have replacing the integers n, i by m, k) that 0 x k x) m k dx = k )!m k)! m! Thus EU i) ) = = = = = 0 0 xg i x)dx x i )!n i)! xi x) n i dx i )!n i)! i )!n i)! i n + 0 x i x) n+) i+) dx i!n i)! n + )!

8 Moment Approximations 8 Thus to a first order Taylor s approximation we have ) i E i) ) = EF U i) ) = F EU i) )) = F n +. Thus when we order the iid data, 2,..., n, and plot the pairs F ) points are of the form F, F i i n+ n+ i n+ ), i) on average these )), that is they fall on a straight line of slope. This notion corresponds to plotting on one axis the theoretical quantiles and on the other hand the empirical quantiles or order statistics. This type of plot is called a quantile-quantile or QQ plot. Normal QQ Plots In some cases, namely location and scale families of distributions, it is possible to make QQ plots with respect to a particular member of this family. This is what is done for normal QQ plots. First we find the relation between non-standard normal and standard normal quantiles. Recall for a given number 0 < q <, the q-th quantile of a distribution with cdf F is defined as the solution, say x q of the equation F x) = q. Let x q be the q-th quantile for a Nµ, σ 2 ) distribution, and z q be the q-th quantile for a standard normal distribution. Thus, letting Nµ, σ 2 ) and Z N0, ) µ P x q ) = P x ) q µ σ σ = P Z x ) q µ σ Therefore or equivalently x q µ σ = z q F q) = x q = σz q + µ where F is the Nµ, σ 2 ) cdf. Let Φ be the N0, ) cdf. ) Plotting i) against F i n+ will follow approximately a line with intercept 0 and slope. However ) plotting i) against Φ i n+ will follow approximately a straight line of slope σ and intercept µ. In fact this is what is done in software implementations of normal QQ plots, such as qqnorm in the statistical programming language R. Aside : R actually does something slightly different. i ) E i) ) = Φ 2. n

9 Moment Approximations 9 In R the qqnorm function plots the observed data x i) against the normal quantiles above. The student might wish to check this by making the corresponding plots in R. Exponential QQ Plots One may construct other QQ plots for continuous distributions. Consider the exponential distribution, with cdf F x) = e λx if x > 0 and F x) = 0 for x 0. Quantiles are easy to obtain, so that the q-th quantile is the solution of q = F x). Let x q be this solution. Therefore q = e λxq or equivalently x q = λ log q) = log/ q)). λ For data x,..., x n the exponential λ QQ plot would plot the pairs λ log i ) n + ), x i) = λ logn + i n + ), x i) i =,..., n. This plot of course would require knowing λ. Since the exponential λ quantiles are the standard exponential exponential parameter ) quantiles divided by λ, one may also plot log i ) n + ), x i) = log n + i ) n + ), x i), i =,..., n. Notice that if the i are iid exponential, no mater what value for λ, this plot will be approximately a straight line. The departures from the straight line will depend on the sampling variability of the order statistics. The student might wish to write an R, matlab or other program to make these exponential QQ plots and then try it for some simulated exponential data. You will then see the variability is much more than for normal QQ plots, but useful nonetheless. ),