Lecture 33: November 29

Transcription

1 36-705: Inermediae Saisics Fall 2017 Lecurer: Siva Balakrishnan Lecure 33: November 29 Today we will coninue discussing he boosrap, and hen ry o undersand why i works in a simple case. In he las lecure we discussed esimaing he variance of an esimaor. Today, we will discuss consrucing confidence inervals using he boosrap, and hen sar discussing model selecion Boosrap Confidence Inervals The boosrap can also be used o obain confidence inervals. If your esimaor has a normal limi hen you could jus use a Wald inerval wih he boosrap variance esimae, i.e. C n = [ θ n ŝz α/2, θ n + ŝz α/2 ]. I is ofen more accurae o use he disribuion of he boosrap esimaes iself o consruc he boosrap confidence inerval Hypoheical confidence inerval Suppose we knew he disribuion of our esimaor, in paricular suppose we knew he disribuion of n( θ n θ). Le us denoe he disribuion by G and denoe is α/2 and 1 α/2 quaniles by g α/2 and g 1 α/2. Then a 1 α confidence inerval would be: C n = [ θ n g 1 α/2, θ n g ] α/2. n n This migh seem a lile srange, bu his is probably because you are used o confidence inervals based on he normal disribuion which has symmeric quaniles. To verify his, P(θ C n ) = P (g α/2 ) n( θ n θ) g 1 α/2 = 1 α/2 α/2 = 1 α. Again he poin is ha we do no know he disribuion G above so we ry o approximae his using he boosrap. 33-1

2 33-2 Lecure 33: November Boosrap confidence inerval algorihm Boosrap Confidence Inerval 1. Draw a boosrap sample X 1,..., X n P n. Compue θ n = g(x 1,..., X n). 2. Repea he previous sep, B imes, yielding esimaors θ n,1,..., θ n,b. 3. Le Ĝ() = 1 B B j=1 ( I n( θ n,j θ ) n ). 4. Le C n = [ θ n g 1 α/2, θ n g ] α/2 n n where g α/2 = Ĝ 1 (α/2) and g 1 α/2 = Ĝ 1 (1 α/2). 5. Oupu C n Varians There are many many many papers ha have been wrien abou he boosrap. Paricularly, here are los of varians he sudenized boosrap where you hrow in some esimaes of he sandard deviaion in consrucing confidence inervals, he block boosrap for imeseries, he residual boosrap or he wild boosrap for regression, he parameric boosrap for parameric models, he smooh boosrap and ideas relaed o sub-sampling o avoid cerain regulariy condiions, he less compuaionally inensive bu less general Jackknife and so on Jusifying he Boosrap This par is going o be a lile bi echnical. Before we ge ino i, we should ry o figure ou wha i means o jusify he boosrap. Roughly, we wan ha he quaniles of he boosrap disribuion of our saisic should be close o he quaniles is acual disribuion, i.e. suppose we define: F n () = P n ( n( θ n θ n ) X 1,..., X n ),

3 Lecure 33: November o be he CDF of he boosrap disribuion, and F n () = P( n( θ n θ) ), o be he CDF of he rue sampling disribuion of our saisic, hen he boosrap works if for insance: sup F n () F n () 0. This urns ou o be rue in quie a bi of generaliy, only requiring mild condiions (Hadamard differeniabiliy, see Boosrap chaper in van der Vaar), bu we will prove i in he simples case: when θ n is a sample mean. In his case here are much simpler ways o consruc confidence inervals (using Normal approximaions) bu ha is no really he poin. Suppose ha X 1,..., X n P where X i has mean µ and variance σ 2. Suppose we wan o consruc a confidence inerval for µ. Le µ n = 1 n n i=1 X i and define We wan o show ha is close o F n. F n () = P( n( µ n µ) ). (33.1) ) F n () = P( n( µ n µ n ) X 1,..., X n Theorem 33.1 (Boosrap Theorem) Suppose ha µ 3 = E X i 3 <. Then, ( ) sup F 1 n () F n () = O P n. To prove his resul, le us recall ha Berry-Esseen Theorem. Theorem 33.2 (Berry-Esseen Theorem) Le X 1,..., X n be i.i.d. wih mean µ and variance σ 2. Le µ 3 = E[ X i µ 3 ] <. Le X n = n 1 n i=1 X i be he sample mean and le Φ be he cdf of a N(0, 1) random variable. Le Z n = n(x n µ). Then σ sup P(Z n z) Φ(z) 33 µ 3 4 σ 3 n. (33.2) z Proof of he Boosrap Theorem. Le Φ σ () denoe he cdf of a Normal wih mean 0 and variance σ 2. Le σ 2 = 1 n n i=1 (X i µ n ) 2. Thus, σ 2 = Var( n( µ n µ n ) X 1,..., X n ). Now, by he riangle inequaliy, sup F n () F n () sup F n () Φ σ () + sup = I + II + III. Φ σ () Φ σ () + sup F n () Φ σ ()

4 33-4 Lecure 33: November 29 F n O(1/ n) L O P (1/ n) F n O P (1/ n) L O(1/ B) F Figure 33.1: The disribuion F n () = P( n( θ n θ) ) is close o some limi disribuion L. Similarly, he boosrap disribuion F n () = P( n( θ n θ n ) X 1,..., X n ) is close o some limi disribuion L. Since L and L are close, i follows ha F n and F n are close. In pracice, we approximae F n wih is Mone Carlo version F which we can make as close o F n as we like by aking B large.

5 Lecure 33: November Le Z N(0, 1). Then, σz N(0, σ 2 ) and from he Berry-Esseen heorem, ( I = sup F n () Φ σ () = sup P n( µn µ) ) P (σz ) ( = sup P n( µn µ) ) ( P Z ) 33 µ 3 σ σ σ 4 σ 3 n. Using he same argumen on he hird erm, we have ha III = sup F n () Φ σ () 33 4 µ 3 σ 3 n where µ 3 = 1 n i=1 X i µ n 3 is he empirical hird momen. By he srong law of large numbers, µ 3 converges almos surely o µ 3 and σ converges almos surely o σ. So, almos surely, for all large n, µ 3 2µ 3 and σ (1/2)σ and III 33 4µ n 3. From he fac ha 4 σ σ = O P ( 1/n) i may be shown ha II = sup Φ σ () Φ σ () = O P ( 1/n). (This may be seen by Taylor expanding Φ σ () around σ.) This complees he proof. We have shown ha sup F ( ) 1 n () F n () = O P n. From his, i may be shown ha, for ( ) 1 each 0 < β < 1, β z β = O P n. So far we have focused on he mean. Similar heorems may be proved for more general parameers. The deails are complex so we will no discuss hem here Model Selecion In non-parameric regression we had an unknown bandwidh parameer. In pracice, his uning parameer is chosen using cross-validaion. Before we discuss cross-validaion les undersand a rain-es spli, i.e. suppose we spli he daa ino wo pars we can esimae our regression funcion for a grid of bandwidhs {h 1,..., h M } on one par of he daa. Now, we wan o pick one of hese bandwidhs. In his case, we could simply check how well we can predic on he es se, i.e., R( f h1, f) = 1 n es (Y i n f h1 (X i )) 2, es i=1 and repea his for each of he bandwidhs and hen pick he bandwidh ha minimizes his. Why does his work? Essenially, rain-es splis allow us o esimae he risk, and hen we are picking he value of he uning parameer ha minimizes our esimaed risk. We should be a lile bi careful abou wha risk we are esimaing: E( R( f h1, f)) = E(f(X) + ɛ f h1 (X)) 2 = E(f(X) f h1 (X)) 2 + σ 2.

6 33-6 Lecure 33: November 29 We can of course ignore he σ 2, bu one should noice ha he risk we are esimaing: E(f(X) f h1 (X)) 2 = (f(x) f h1 (x)) 2 p(x)dx, where p is he densiy of he covariaes. This is someimes called he L 2 (P)-risk, as opposed o he L 2 -risk which we defined earlier: R = (f(x) f h1 (x)) 2 dx. Mos people would consider he L 2 (P)-risk o be more naural, since i pus less weigh in places where you have less daa. Praciioners migh be concerned ha his he rain-es spli is waseful of he daa: you migh need a prey large es se o ge a good esimae and his is daa ha you migh have insead used o esimae he model. Also, rain-es splis have a loery effec: you migh ge unlucky in he way you spli he daa and his could affec resuls. K-fold cross-validaion ries o ge around his by spliing he daa ino K pieces (hink of K as a small number like 5). Now, we repea he rain-es spli K imes, each ime we use K 1 pieces for raining and he K-h piece for esing. In his way we ge, K esimaes of he error for each value of he bandwidh. We average hese K numbers o ge our risk esimae. Finally, we choose he value of he bandwidh ha minimizes he risk. The exreme case of K-fold cross-validaion is called leave-one-ou or n-fold cross-validaion. Here we leave ou one observaion, and ry o predic i and hen cycle hrough he observaions. The basic quesion is hen: is here a sense in which cross-validaion is doing he righ hing? 33.5 A simple analysis of he rain-es spli Les ry o undersand cross-validaion in a simple scenario. We will do his in he conex of poin esimaion, bu one could use exacly he same argumen for bandwidh selecion. Say we have models M 1,..., M M. These are differen models ha we hink migh be reasonable fis o he daa. Now, we observe our daa (X 1,..., X 2n ) and randomly spli i ino rain and es ses of size n each. We really should refer o he es se as a validaion se bu we will ignore his for oday. On he rain se, we fi our models (say using he MLE), and compue poin esimaes θ 1,..., θ M. Now, suppose ha we wan o selec he model/esimae ha fis he daa well. We will use he negaive log-likelihood as our measure, i.e., we wan an esimae ha has low negaive log-likelihood. This is he same as using he KL divergence as our loss funcion.

7 Lecure 33: November We can use he es se o esimae he negaive log-likelihood: R i = 1 n n log (X f θi n+i ). i=1 Noe ha: E(R i ) = E fθ log f θi (X) = KL(f θ f θi ) E fθ log f θ (X), so we are esimaing he KL divergence upo some erm ha does no depend on θ i. So minimizing E(R i ) is equivalen o minimizing he KL divergence. We can now use he LLN o argue ha if he es-se size goes o hen our risk esimaes converge o heir expecaions, and hen we will find he model/esimae wih he lowes KL o he rue model. Suppose however we waned o be more precise, and ry o undersand he role of he es se size and he number of models M? We could use Hoeffding s inequaliy. This will need an assumpion ha log f θ (X) B for every θ and X ha we care abou (his can be relaxed using more complex echniques). Now, noice ha he following is an imporan bu sraighforward consequence of Hoeffding s inequaliy: P(max R i E(R i ) ɛ) 2M exp( 2nɛ 2 /(4B 2 )). This is rue since for each i we know ha P( R i E(R i ) ɛ) 2 exp( 2nɛ 2 /(4B 2 )). so we can obain he desired inequaliy via a union bound (if he max exceeds ɛ a leas one of he erms mus exceed ɛ). Define, hen we know ha 4B2 log(2m/α) ɛ n =, n P(max R i E(R i ) ɛ n ) α. Suppose we selec he model î = arg min i R i, and le i = arg min i E(R i ), hen we have ha wih probabiliy a leas 1 α: E(Rî) Rî + ɛ n R i + ɛ n E(R i ) + 2ɛ n.

8 33-8 Lecure 33: November 29 So he model we selec will be sub-opimal by a mos 2ɛ n. In regression, we would use exacly he same reasoning, bu jus replace he risk wih he squared loss. Reasoning abou K-fold cross-validaion urns ou o be much more challenging, because he daa re-use breaks independence assumpions. The analysis above should remind you of he analysis we did before of Empirical Risk Minimizaion. The goals are slighly differen, as is he final guaranee. I is worh hinking abou wha exacly he daa spliing buys you. In paricular, we do no require uniform convergence of he empirical o he rue risk over all he model classes M 1,..., M M, raher we only require a good esimae of he risk for he fixed models indexed by θ 1,..., θ M.