Variance Reduction. in Statistics, we deal with estimators or statistics all the time

Transcription

1 Variance Reduction in Statistics, we deal with estiators or statistics all the tie perforance of an estiator is easured by MSE for biased estiators and by variance for unbiased ones hence it s useful to know how to design better estiators, which have saller MSE or variance than existing one(s) May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 1

2 Antithetic Variates I want to estiate paraeter θ let V and W be two negatively correlated unbiased estiators of θ: Cov(V, W ) < 0 then S := 1 2 (V + W ) is also unbiased for θ also V ar(s) = 1 4 V ar(v ) V ar(w ) Cov(V, W ) since Cov(V, W ) < 0, V ar(s) saller than what it would have been if V and W were independent, i.e. if Cov(V, W ) = 0 so negative correlation is better that independence in variance reduction: V and W are called antithetic variates the following theore gives a quick way to cook up antithetic variates, and hence better estiators May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 2

3 Antithetic Variates II Theore 0.1. Let X be a rando variable and for two functions f( ) and g( ), E(f 2 (X)), E(g 2 (X)). If f(x) is increasing and g(x) is decreasing in x, then Cov(f(X), g(x)) 0 If both f(x) and g(x) are increasing or decreasing in x, then the reverse of the above inequality holds. sketch of the proof: note (f(x) f(y))(g(x) g(y)) 0, x, y take X d = Y, where X and Y are independent and consider E {(f(x) f(y ))(g(x) g(y ))} 0 = E(f(X)g(X)) E(f(X)g(Y )) E(f(Y )g(x)) + E(f(Y )g(y )) 0 = 2{E(f(X)g(X)) E(f(X))E(g(X))} = 2Cov(f(X), g(x)) 0 May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 3

4 Antithetic Variates III application of the above theore: let X has c.d.f. F ( ) with inverse F 1 ( ) let θ = E(h(X)), where h( ) is an increasing function then by inversion ethod F 1 (U), F 1 (1 U) d = X, where U Unifor 1 (0, 1) so both V := h(f 1 (U)) and W := h(f 1 (1 U)) are unbiased estiators of θ and have sae variance since h(f 1 (u)) an increasing and h(f 1 (1 u)) is a decreasing functions of u, by the previous theore Cov(V, W ) 0 so for U 1, U 2,..., U n i.i.d. U Unifor 1 (0, 1) all of the following are unbiased estiators of θ, which one s the best? 1 n nx h(f 1 (U i )), 1 n nx h(f 1 (1 U i )), 1 2n nx (h(f 1 (U i ))+h(f 1 (1 U i ))) May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 4

5 Control Variates I want to estiate µ = E(X) where X π( ) we have a control variate C such that we know µ C := E(C), σ CC := V ar(c) and σ CX := Cov(C, X)( 0) consider unbiased estiators of µ of the following for: X(b) = X + b(c µ C ) note V ar(x(b)) = V ar(x) + b 2 σ CC + 2bσ C,X V ar(x(b)) is iniized at b = σ CX σ CC with iniu value V ar(x(b )) = V ar(x) (1 Corr 2 (C, X)) < V ar(x) thus choosing a proper control variate we get a better unbiased estiator X(b ) than X of µ May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 5

6 Rao-Blackwellization I want to estiate I = E(h(X)), where X = (X 1, X 2 ), i.e. X can be divided into two blocks suppose we can copute E(h(X) X 2 ) analytically let we have saples {x (i) i = 1, 2,..., }, fro soe independent Monte Carlo sapler since E[E(h(X) X 2 )] = E(h(X)) = I, we have two copeting unbiased estiators siple average or histogra estiator Rao-Blackwellized average or ixture estiator Î := 1 Ĩ := 1 h(x (i) ) E(h(X) x (i) 2 ) May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 6

7 Rao-Blackwellization II so, which one these Monte Carlo estiators is better? since V ar(h(x)) = V ar[e(h(x) X 2 )] + E[V ar(h(x) X 2 )] we have, V ar(î) = V ar(h(x)) V ar[e(h(x) X 2)] = V ar(ĩ) thus as a rule of thub always perfor analytical siplifications as uch as possible before using any kind of Monte Carlo estiator May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 7

8 Stratified Sapling I let X π( ), µ := E(h(X)) and σ 2 = V ar(h(x)), we want to estiate µ let the saple space X can be divided into k(> 1) sub-regions or strata {D k i = 1, 2,..., k} such that h( ) is relatively hoogeneous in each D i, i let π i := D i π(x) dx be the stratu proportions, µ i := 1 π i D i h(x)π(x) dx be the stratu ean and σ 2 i := 1 π i D i (h(x) µ i ) 2 π(x) dx be the stratu variances i, i = 1, 2,..., k so, we have, µ = = X k h(x)π(x) dx = π i 1 π i k D i h(x)π(x) dx = D i h(x)π(x) dx k π i µ i May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 8

9 Stratified Sapling II also, we have, σ 2 = = = Z D i {(h(x) µ i ) + (µ i µ)} 2 π(x) dx π i 1 π i Z D i {(h(x) µ i ) 2 + (µ i µ) 2 + 2(h(x) µ i )(µ i µ)}π(x) dx π i {σi 2 + (µ i µ) 2 } (1) suppose, we have strata specific rando saples {x (i,j) j = 1, 2,..., i } fro the sub-region D i, i and the strata saples are independent between strata now we could estiate µ i by µ i = 1 i i j=1 h(x(i,j) ) May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 9

10 Stratified Sapling III since µ = k π iµ i, µ = k π i µ i is a candidate estiator of µ since µ i s are within strata saple averages and strata saples are independent between strata, we have V ar( µ) = k πi 2 σ 2 i i (2) one could also consider cobining all the := k i strata saples together: {x (i,j) i = 1, 2,..., k j = 1, 2,..., i } and foring the estiator µ := 1 k i k i h(x (i,j) ) with variance j=1 V ar( µ) = σ 2 k i (3) May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 10

11 Stratified Sapling IV the following will be used later in the discussion: π 2 i i =! i π 2 i i! ( π i i i ) 2 = π i = 1 why? (4) let r := k π 2 i i and by the above r 1 if h( ) is relatively hoogeneous in each D i, i, then σi 2 s are expected to be uch saller than σ 2, say, σi 2 = cσ2, i for soe c > 0 with such a careful stratification can lead to V ar( µ) V ar( µ): V ar(bµ) V ar( µ) = σ 2 (c π 2 i i 1 ) = σ2 ( c π 2 i i 1 = σ2 (cr 1) = σ2 r (c 1/r) 0, c (0, 1/r) (0, 1) May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 11 )

12 Stratified Sapling V but for a bad stratification, e.g. where µ i = µ and σ 2 i = σ2, i = 1, 2,..., k (here each strata is as heterogeneous in h(x) as the whole population, which is no good), µ ay prove to be a bad estiator as well: V ar(bµ) V ar( µ) = π 2 i σ 2 i P k π iσ 2 = σ2 ( π 2 i i 1 ) = σ2 (r 1) 0 thus careful stratification is the key May 2, 2006 c Gopi Goswai (goswai@stat.harvard.edu) Page 12