STAT 450: Statistical Theory. Distribution Theory. Reading in Casella and Berger: Ch 2 Sec 1, Ch 4 Sec 1, Ch 4 Sec 6.

Transcription

1 STAT 450: Statistical Theory Distribution Theory Reading in Casella and Berger: Ch 2 Sec 1, Ch 4 Sec 1, Ch 4 Sec 6. Example: Why does t-statistic have t distribution? Ingredients: Sample X 1,...,X n from the N(µ,σ 2 ) distribution. Decide whether µ = µ 0 or µ > µ 0 based on n( X µ T = 0 ) s X is sample mean; s is sample SD. 1

2 Make decision based on P-value: P = P(T > T obs ). Notation: T obs is the value of our statistic we actually observe. Assumption: True value of µ is µ 0 (often µ 0 = 0). So find density, f T (t), of T; compute P = T obs f T (t)dt. This section of notes is about finding f T. 2

3 Univariate Techniques to find density Method 1: compute CDF by integration; differentiate to find f Y. Example: U Uniform[0,1] and Y = logu. 3

4 Example: Z N(0,1), i.e. f Z (z) = 1 2π e z2 /2 What is the distribution of Z 2? 4

5 Indicator notation: So we can write 1(y > 0) = { 1 y > 0 0 y 0 f Y (y) = 1 2πy e y/2 1(y > 0) (changing definition unimportantly at y = 0). Note: Never evaluated F Y before differentiating it. In fact F Y and F Z are integrals I can t do but I can differentiate then anyway. Remember fundamental theorem of calculus: x d f(y)dy = f(x) dx a at any x where f is continuous. 5

6 Summary: for Y = g(x) with X and Y each real valued P(Y y) = P(g(X) y) = P(X g 1 (,y]). Take d/dy to compute the density f Y (y) = d dy {x:g(x) y} f X(x)dx. Often can differentiate without doing integral. 6

7 Method 2: Change of variables. Given X with density f X. New variable is Y = g(x) where g is one to one. Mnemonic: f Y (y)dy = f X (x)dx where y and x are related by y = g(x) and dy = g (x) dx. Absolute value to make sure all terms positive. 7

8 Example: Suppose X has an exponential distribution. f X (x) = e x 1(x > 0). What is the density of e X? Step 1: Give name to new variable. Y = e X. Step 2: Define the function g: g(x) = e x. Step 3: Solve y = g(x) to find x as function of y. 8

9 Step 4: Keep track of those y for which no x exists. For such y f Y (y) = 0. Step 5: Use the memory device. so f Y (y)dy = f X (x)dx f Y (y) = f X(x). dy dx This gives for 0 < y < 1: 9

10 Step 6: VERY IMPORTANT. Get rid of all the xs. Replace by corresponding value of y. Step 7: Write out the full answer and, if possible, recognize the name of the distribution. 10

11 Derivation when g is increasing, differentiable. Interpretation of density (based on f = F ): and P(y Y y +δy) f Y (y) = lim δy 0 δy F = lim Y (y +δy) F Y (y) δy 0 δy P(x X x+δx) f X (x) = lim. δx 0 δx Now assume y = g(x). Define δy by y +δy = g(x+δx). Then P(y Y g(x+δx)) = P(x X x+δx). Get P(y Y y +δy)) δy Take limit to get = P(x X x+δx)/δx {g(x+δx) y}/δx. or f Y (y) = f X (x)/g (x) f Y (g(x))g (x) = f X (x). 11

12 Remark: For g decreasing g < 0. Interval (g(x), g(x+δx)) really (g(x+δx), g(x)). So g(x) g(x+δx) g (x)δx. In both cases this amounts to the formula f X (x) = f Y (g(x)) g (x). 12

13 Example: X Weibull(shape α, scale β) or ( ) α 1 x exp{ (x/β) α }1(x > 0). f X (x) = α β β Let Y = logx or g(x) = log(x). Solve y = logx: x = exp(y) or g 1 (y) = e y. Then g (x) = 1/x and 1/g (g 1 (y)) = 1/(1/e y ) = e y. Hence ( e y ) α 1 exp{ (e y /β) α }1(e y > 0)e y. f Y (y) = α β β For any y, e y > 0 so indicator = 1. So f Y (y) = α β α exp{αy eαy /β α }. Define φ = logβ and θ = 1/α; then, f Y (y) = 1 { { }} y φ y φ θ exp exp. θ θ Extreme Value density with location parameter φ and scale parameter θ. (Note: several distributions are called Extreme Value.) 13

14 Multivariate problems Three basic methods: Compute CDF as integral then differentiate. Change of Variables formula Marginalization. 14

15 Example: Z 1 and Z 2 independent. Density of Z 2 1 +Z2 2? Step 1: Y = Z 2 1 +Z2 2. Step 2: F Y (y) = P(Z 2 1 +Z2 2 y) = f Z1,Z 2 (z 1,z 2 )dz 1 dz 2. What is f Z1,Z 2 (z 1,z 2 )? 15

16 Step 3: Polar co-ordinates: z 1 = rcos(θ), z 2 = rsin(θ). Jacobian dz 1 dz 2 = rdrdθ. Range of integration is a circle in (z 1,z 2 ) plane so 0 r y,0 θ 2π 16

17 Change of variables Example: Z 1, Z 2 independent N(0,1). Sample mean is Sample SD is Simplify and s = Z = Z 1 +Z 2. 2 (Z 1 Z) 2 +(Z 2 Z) 2 Z 1 Z = Z 1 Z 2 2 Z 2 Z = Z 2 Z 1 2 So if Y 1 = (Z 1 +Z 2 )/ 2 and Y 2 = (Z 1 Z 2 )/ 2 then and S = Y 2 2 t = Y 1 S. I find joint density of Y 1 and S in steps. 17

18 Step 1: Find joint density of Z = (Z 1,Z 2 ) using independence. Step 2: Find joint density of Y = (Y 1,Y 2 ) using Change of Variables 18

19 Step 3: Find marginal densities. Step 3a: OR observe independence. 19

20 Step 4: Use previous result to find density of S. Step 5: Find joint density of Y 1 and S. 20

21 Marginalization Simplest multivariate problem: X = (X 1,...,X p ), Y = X 1 (or in general Y is any X j ). Theorem 1 If X has density f(x 1,...,x p ) and q < p then Y = (X 1,...,X q ) has density f Y (x 1,...,x q ) = f(x 1,...,x p )dx q+1...dx p. f X1,...,X q is the marginal density of X 1,...,X q f X the joint density of X but they are both just densities. Marginal just to distinguish from the joint density of X. 21

22 Example: : The function f(x 1,x 2 ) = Kx 1 x 2 1(x 1 > 0,x 2 > 0,x 1 +x 2 < 1) is a density provided P(X R 2 ) = The integral is K 1 1 x1 0 0 f(x 1,x 2 )dx 1 dx 2 = 1. 1 x 1 x 2 dx 1 dx 2 = K x 1(1 x 1 ) 2 dx 1 /2 0 = K(1/2 2/3+1/4)/2 = K/24 so K = 24. The marginal density of x 1 is f X1 (x 1 ) = 24x 1x 2 1(x 1 > 0,x 2 > 0,x 1 +x 2 < 1)dx 2 =24 1 x1 0 x 1 x 2 1(0 < x 1 < 1)dx 2 =12x 1 (1 x 1 ) 2 1(0 < x 1 < 1). This is a Beta(2,3) density. 22

23 General problem has Y = (Y 1,...,Y q ) with Y i = g i (X 1,...,X p ). Case 1: q > p. Y won t have density for smooth g. Y will have a singular or discrete distribution. Problem rarely of real interest. (But, e.g., residuals have singular distribution.) 23

24 Case 2: q = p. We use a change of variables formula Generalizes the one derived above for the case p = q = 1. 24

25 Case 3: q < p. Pad out Y add on p q more variables (carefully chosen) say Y q+1,...,y p. Find functions g q+1,...,g p. Define for q < i p, Y i = g i (X 1,...,X p ) and Z = (Y 1,...,Y p ). Choose g i so that we can use change of variables on g = (g 1,...,g p ) to compute f Z. Find f Y by integration: f Y (y 1,...,y q ) = f Z (y 1,...,y q,z q+1,...,z p )dz q+1...dz p 25

26 Change of Variables: general Suppose Y = g(x) R p with X R p having density f X. Assume g is a one to one ( injective ) map, i.e., g(x 1 ) = g(x 2 ) if and only if x 1 = x 2. Find f Y : Step 1: Solve for x in terms of y: x = g 1 (y). Step 2: Use basic equation: f Y (y)dy = f X (x)dx and rewrite it in the form f Y (y) = f X (g 1 (y)) dx dy. Interpretation of derivative dx dx dy = det ( xi dy ) y j which is the so called Jacobian. when p > 1: 26

27 Equivalent formula inverts the matrix: This notation means dy dx = f Y (y) = f X(g 1 (y)) det dy dx y 1 x 1 y 1 x 2. y p y p x 1 x 2 y 1 x p y p x p but with x replaced by the corresponding value of y, that is, replace x by g 1 (y). 27

28 Example: The density f X (x 1,x 2 ) = 1 2π exp { x2 1 +x2 2 2 is the standard bivariate normal density. } Let Y = (Y 1,Y 2 ) where and Y 1 = X 2 1 +X2 2 0 Y 2 < 2π is angle from the positive x axis to the ray from the origin to the point (X 1,X 2 ). I.e., Y is X in polar co-ordinates. Problem is to find joint density of Y 1 and Y 2. 28

29 Step 1: Solve for x in terms of y: X 1 = Y 1 cos(y 2 ) X 2 = Y 1 sin(y 2 ) Thus g(x 1,x 2 ) = (g 1 (x 1,x 2 ),g 2 (x 1,x 2 )) = ( x 2 1 +x2 2,argument(x 1,x 2 )) g 1 (y 1,y 2 ) = (g1 1 (y 1,y 2 ),g2 1 (y 1,y 2 )) = (y 1 cos(y 2 ),y 1 sin(y 2 )) ( ) dx = dy det cos(y2 ) y 1 sin(y 2 ) sin(y 2 ) y 1 cos(y 2 ) = y 1. 29

30 It follows that f Y (y 1,y 2 ) = 1 2π exp { y2 1 2 } y 1 1(0 y 1 < )1(0 y 2 < 2π). Next: marginal densities of Y 1, Y 2? Factor f Y as f Y (y 1,y 2 ) = h 1 (y 1 )h 2 (y 2 ) where h 1 (y 1 ) = y 1 e y2 1 /2 1(0 y 1 < ) and h 2 (y 2 ) = 1(0 y 2 < 2π)/(2π). 30

31 Then f Y1 (y 1 ) = h 1(y 1 )h 2 (y 2 )dy 2 = h 1 (y 1 ) h 2(y 2 )dy 2 so marginal density of Y 1 is a multiple of h 1. Multiplier makes f Y1 = 1 but in this case h 2(y 2 )dy 2 = so that 2π 0 (2π) 1 dy 2 = 1 f Y1 (y 1 ) = y 1 e y2 1 /2 1(0 y 1 < ). (Special case of Weibull or Rayleigh distribution.) Similarly f Y2 (y 2 ) = 1(0 y 2 < 2π)/(2π) which is the Uniform((0, 2π) density. 31

32 Exercise: W = Y1 2 /2 has standard exponential distribution. Recall: by definition U = Y 2 1 has a χ2 distribution on 2 degrees of freedom. Exercise: find χ 2 2 density. Note: We show below factorization of density is equivalent to independence. 32