Idepedet Mote Carlo Iterested Approxmate E X = x dν x = µ µ wth = = x where x, x s sampled rom the probablty, measure ν ( X ). Uder certa regularty codtos, I x,, x x E X = are a d sample rom ν ( X ) x E X = coverges by the law o large umbers, assumg that E ( X) <., the
I addto, the by the CLT ν E X = x d x < = ( x ) s approxmately ormally dstrbuted, wth E X Var X. mea ad varace The usual estmate o Var = σ = X s s = x, ( ) Var X. the usual ubased estmate o Example: Codece terval propertes Wat to look at propertes o the ormal theory 95% CI or µ x ± t 0.05 s m
) Coverage probablty (assumg µ = 0) s C = E I x t 0 x t m x m = E I < t s 0.05 0.05 0.05 ) Mea terval wdth s m s t E w E t0.05 E s m m 0.05 [ ] = = [ ] whe m = 0 or the ollowg dstrbutos ) N ( 0,) ) Cauchy ( 0, ) 3) t 3 4) U (,) 3
For the N ( 0,), t s kow that the true coverage rate s 95% ad the mea terval wdth s t 0.05 Γ Γ ( m ) ( m ) m( m ) For m = 0, the mea wdth s.39597. Also or the Cauchy ( 0, ), the mea terval wdth s. I the other cases, determg the exact values s dcult sce the dstrbutos o x ad s are ot tractable. I each case, m = 000 mputatos wll be geerated Estmates: ) Coverage probablty ˆ x m C = I < t = s 0.05 4
) Mea terval wdth w = = w t 0.05 = m t 0.05 = s m = s Coverage Rate: Dstrbuto Ĉ SE Ĉ N ( 0,) 0.954 0.006645 Cauchy ( 0, ) 0.98 0.004373 t 3 0.958 0.006343 U (,) 0.967 0.0056490 where SE Cˆ = Cˆ ( Cˆ) 5
Mea Iterval Wdth: Dstrbuto w SE w N ( 0,).37 0.00 Cauchy ( 0, ) 34.339 9.63 t 3.49 0.03455 U (,) 0.84 0.00404 The estmato errors or the N ( 0,) cases are Coverage: 0.004 Mea wdth: 0.093 both o whch are wth SE s. Cauchy example: < s mportat. Sce the Cauchy has o te E s =, ad thus the mea terval The assumpto that E ( X) momets, [ ] wdth s. Thus the reported sample average ad stadard error s ot meagul. 6
However eve though the mea wdth s ot deed, the coverage rate s well deed. For every dataset, the dcator ucto s well deed ad the tegral o ay dcator ucto o probable terest s te. Sample sze: Whe desgg a Mote Carlo study, the sample sze m eeds to be determed. Usual approach s by boudg the SE. Wat whch gves SE σ σ SE where SE s the desred stadard error ad σ = Var ( X ). 7
There s the same problem here as wth determg the sample sze ecessary to boud the sze o a codece terval: What s σ? Sometmes you ca guess o what For example, the coverage rate case ( ) σ = C C σ mght be. Sce or the examples, C wll be aroud 0.95, use that to pck. It ca be tougher or other problems. For the wdth example, the questo comes dow to determg Var ( s ). Whle ths could be doe or the ormal (ad the Cauchy), t s tougher or the other dstrbutos. Oe approach s to do a small test sample to get a guess o σ ad use ths to gure out how may more samples eed to be added. 8
Sgle sample Multple questos I the example, a sgle sample was used to aswer both questos (.e. x ad s are the same averages). I could have used these same samples to aswer may more questos (e.g. Var ( w ), E x, E s + 4. mx, etc) Whe dealg wth multple quattes to be studed, you eed to pck a sample sze that meets the requremets or all quattes (at least the mportat oes). Implemetato S-Plus/R & Matlab Whe possble use vectorzed calculatos, ot loops, partcularly wth S-Plus. 9
Vectorzed rorm.vec <- ucto(, mu=0, sgma=) { Loop rorm.loop <- ucto(, mu=0, sgma=) { xbar<-rep(0,) s <- rep(0,) cover <- rep(0,) wdth <- rep(0,) dat<-matrx(rorm(0*, mu, sgma), col=0) xbar <- apply(dat,, mea) s <- sqrt(apply(dat,, var)) cover <- abs(xbar) * sqrt(0) / s <= qt(0.975, 9) wdth <- * qt(0.975, 9) * s / sqrt(0) C <- mea(cover) wbar <- mea(wdth) lst(cover=cover, C=C, wdth=wdth, wbar=wbar) } or( :) { x <- rorm(0, mu, sgma) xbar[] <- mea(x) s[] <- sqrt(var(x)) cover[] <- abs(xbar[]) * sqrt(0) / s[] <= qt(0.975, 9) wdth <- * qt(0.975, 9) * s[] / sqrt(0) } C <- mea(cover) wbar <- mea(wdth) lst(cover=cover, C=C, wdth=wdth, wbar=wbar) } Ru tmes whe = 0,000 R S-Plus Vectorzed.5 sec 35 sec Loop.5 sec 48 sec Loop/Vector.67.37 0
Tests doe o.6ghz Petum 4 rug Wdows XP R verso:.8. S-Plus verso: 6.0 Release Geeral cosesus about S-Plus would have suggested that the loop/vector rato should have bee hgher wth S-Plus tha wth R. Whle I m ot sure how to quaty t wth ths setup, the memory use or loops s usually worse tha or vectorzed setups, partcularly wth S-Plus. I Matlab, loopg s t as bad, though a procedure ca be doe wth vectorzed calculatos ts, usually preerable. Gettg more precse estmates ) crease ) deret samplg scheme
Strated Samplg Break the sample space S to dsjot regos S,, SK Sample pots xk,, xk rego k k Wth each rego get sample average The estmate k k = k = x µ by k ˆ µ K [ ] = P S k k k = Ths estmator s based o the dea = k E X E E X S The varace o ths estmator s Var K ( ˆ µ ) ( P[ Sk] ) = k = Var ( ( X) X Sk ) I the regos are pcked reasoably, ths wll have a smaller varace tha k
I P[ S ] Var ( ( X )) k = k (proportoal samplg), the varace o the strated estmator reduces to Var ˆ Var k = = E Var X Z K ( µ ) = P[ Sk] ( ( X) X Sk) ( ) where Z s a radom varable satsyg Z = k whe the sgle radom pot draw alls S. k Sce Var ( ) = Var E ( ( X) Z) X E X Z + Var The strated estmator has a smaller varace that the sample average estmator. Noproportoal samplg ca gve eve more ececy The optmal sample sze choces, subject to k = s 3
k = K j = [ ] Var k ( k) P S X X S ( ) j P S j Var X X S Ths mples that regos wth hgh varablty should get more samples tha regos o small varablty. Atthetc Varates Combes correlated estmators to acheve a more precse estmator. Based o the dea V + W Var = Var + Var 4 4 + Cov, ( V ) ( W ) ( VW) I V ad W are egatvely correlated, the you get a more precse estmate tha they were ucorrelated (or postvely correlated). So we eed to geerate coupled, egatvely correlated radom varables. 4
The ollowg proposto gves us a approach or dog ths Proposto.4.. Suppose X s a radom varable ad the uctos ( x ) ad g( x ) are both creasg or both decreasg. I the radom varables ( X ) ad g( X ) have te secod momets, the ( ) I ( x ) s creasg ad Cov X, g X 0 g x s decreasg (or vce-versa), the the covarace 0. Proo (See Lage, page 9) Suppose we wsh to calculate x g x dx where ( x ) s a creasg ucto ad the desty g( x ) has CDF ( G ( u) ) s creasg ad the ucto G ( u) s decreasg whe u [ 0,]. I U,, U G x. The the ucto s ad d sample rom U ( 0,), the 5
ad G ( U ) ( ) = ( ) E G U E G U = ( x) g( x) dx correlated. Thus the estmator = ad G ( U ) are egatvely ( ( ) ) + ( ) { G U G U } has a smaller varace tha = ( ( U )) G The dea behd ths estmator s that U,, U are uorm, so are U,, U. The ths mples that,, G U G ( U ) ad G ( U ),, G ( U ) are both sets o draws rom X ~ G. What ths estmator s dog s makg sure that the p th quatle s the sample o X, so s the p th quatle. I a sese, ts gvg a more balace sample. 6
Note that ths also works ( x ) s a decreasg ucto. Example: Let X ~ Exp ( ) ad we wat to d = = Γ.5 =.5334 x E X 0.5 xe dx Geerate = 000 values rom Exp ( ) ad use Atthetc varates Sampler Estmate SE U sample.39673 0.099 U sample.5836 0.005 Atthetc.4907 0.003 The error wth atthetc estmate s -0.0043. I a sgle sample o = 000 was take, the stadard error would be approxmately 0.043. The ga ececy due to atthetc varates s approxmately 0.5 (the square o the rato o the stadard errors). 7
To get the same ececy out o a sgle sample, almost 40,000 samples would be eeded. Atthetc varate geerato I x s cotaed the sample, the the correspodg sample that eeds to be added s * ( ( ) ) x = G G x Ths approach s reasoable whe G( x ) ad G ( x) are ce uctos. I act you oly eed G ( x) to be ce as you ca use the procedure descrbed at the start o the class based o the uorm dstrbuto to geerate the samples eeded. Symmetrc dstrbutos I X ~ G has a symmetrc dstrbuto aroud a mea µ (e.g. Normal, Logstc, etc), the atthetc varates approach s easy sce x * = µ x 8
Ths dea ca also be expaded h ( X ) s symmetrc or some mootoc ucto h. For example X s logormal, the log X s a symmetrc radom wth mea µ. The the atthetc varate s x I geeral the atthetc varate satses * = * e µ x ( µ ( ) ) x = h h x whe there s a symmetrzg ucto h ( x ). 9
Note or the logormal example, I would t mplemet t ths asho. Istead I would geerate z,, z ~ ( 0, N ) ad set x = e, x = e µ + σ z * µ σ z sce most logormal geerators start wth ormal radom varables the rst place. Cotrol Varates Smlar to atthetc varates where you wat to use correlato to reduce varablty The uderlyg dea s to look at = + E X E X g X E g X where E g( X) radom varables ( X ) ad correlated. Var ( ( X) g( X) ) s kow aalytcally ad the g X are postvely ( X ) ( ( X) g( X) ) ( g( X) ) = Var Cov, + Var 0
I ( X ) ad g X are hghly eough correlated, ths wll have a smaller varace Var X. tha Ths mples that the estmate o E ( X) ˆ µ, C ( x) g( x) g µ = = = wll have a smaller varace tha ˆ µ = = ( x ) Ths approach has tes to regresso. Let E g( X) µ g =. The the orgal ormulato ca be though o as lookg at X g X µ g Istead o ths, lets look at b ( g) X = X b g X µ For all b, E ( X) = E ( X) b.
Thus the orgal problem ca be moded by choosg the b to mmze Var ( b ( X )), whch ca be doe wth b ( ( X) g( X) ) g( X) Cov, σ = = ρ Var σ The dea behd ths method s that by usg the cotrol varate g( X ), we ca see how lkely the estmate o E ( X) just based o the sampled ( x ),, ( x ) s o. Wth ths adjustmet, the estmate o E ( X) s ˆ µ, Cb, ( x) b g( x) g µ = = = The varace o ths estmator s Var ˆ ( µ, Cb,) = ( σ bσ g + b σg) The varous varace ad covarace terms ca be estmated usg the stadard ubased estmators. g
Example: Let X ~ Exp ( ) ad we wat to d = = Γ.5 =.5334 x E X 0.5 xe dx Let ( x) = x; g( x) = x We kow that E[ X ] = ad t ca be show that.5 X X = Γ Γ Cov,.5.5 =.5334 Ths gves the optmum b o b = 0.33385 (Note most problems we ca t gure ths covarace out exactly) 3
Geerate = 000 values rom Exp ( ) ad use optmum b or g( x) data we get. = x. For the smulated x =.565 = x = =.0843 Sampler Estmate SE Crude MC.565 0.00993 Cotrol.50476 0.00606 I ths example, the cotrol varates approach s almost tmes more ecet that the stadard approach. We ca also look the ececy wth other choces o b. 4
0.0 0.0 0.03 0.04 0.0 0. 0.4 0.6 0.8.0 b Ececy 0 4 6 8 0 SE(mu-hat) 0.0 0. 0.4 0.6 0.8.0 b 5
Note that you usually do t kow the optmum Cov X, g X s usually b sce gettg tractable aalytcally. However you ca use Var g X your sample to estmate t (ad ecessary). Aother equvalet approach (assumg that Var g X ) s to ru the lear you estmate regresso o ( X ) o g X (.e. t model ( X) = a + bg( X) + ε { } { } wth the observed ( x ) ad g x ). Note that estmatg the optmal b wll troduce a slght bas the estmate o E ( X) ad a slghtly overoptmstc SE. However these problems usually are t eough to worry about ad asymptotcally t gves the correct aswer. 6
Rao-Blackwellzato The cotrol varate approach used the dea to try to do some aalytc computatos to mprove our estmator. Ths ext approach s based o the same dea, but ocuses more o the ucto o terest Suppose that X ca be decomposed to two ( ) ( ) parts X, X ad that we are terested estmatg ( ( ) ( ), ) E X = E X X. Oe approach s to sample pars Ths ca be doe by Sample ( ) X rom ( ( ) ) g X ( ( ) ( ), ) X X. Sample ( ) X rom ( ) g X X The estmate E ( X) by ( ) ( ) ˆ µ = x, x = 7
Suppose however that ( ) E X X = x ca be calculated aalytcally. The the expectato ca be estmated by ˆ µ, RB = E ( X) X = x = ( ) ( ) Both o these estmators are ubased. However Var ( ˆ µ, RB) = Var Var = Var ˆ Ths s based o Var ( ) = Var ( ) E X X ( ( X) ) ( µ ) ( ) X E X X + Var ( ) E X X Ths estmator suggests that wherever possble, do exact calculato over smulato. Rao-Blackwellzed estmators ca be used a wde rage o settgs, cludg mportace samplg, SIS, or MCMC. 8