3.1 More on model selection

Transcription

1 3. More on Model selecion 3. Comparing models AIC, BIC, Adjused R squared. 3. Over Fiing problem. 3.3 Sample spliing. 3. More on model selecion crieria Ofen afer model fiing you are lef wih a handful of models ha you deem good models. A final model mus be chosen. Should we jus ake he model wih he highes R squared? We know ha models wih more x variables will fi our sample of daa beer (have a higher R squared). R SSE SST

2 Model complexiy and model selecion The adjused R squared is one mehod o compare goodness of fi for models wih differen number of x s. Of a handful of possible models, choose he model wih he highes adjused R squared. The adjused R squared accouns for he number of x s included in he model and herefore accouns for he fac ha larger models canno perform worse han smaller models in sample. Aikaike Informaion Crieria (AIC) There are wo oher popular mehods for comparing models wih differen numbers of predicors. Aikaike Informaion Crieria is based on he sum of squared errors bu includes a penaly funcion for using more x s. Bayes Informaion Crieria (BIC) does he same hing, bu uses a differen penalizaion for using more x s.

3 AIC Where k is he number of x s, he AIC is given by: SSE AIC( k) ln k T T wan small penaly for larger k ln s k T The AIC presens a radeoff beween goodness of fi (small SSE) and model complexiy (k). Noice ha as he sample size T grows, he penaly par declines a rae /T. When comparing models, he one wih he smalles AIC is preferred. BIC Where k is he number of x s, he BIC is given by: SSE ln T BIC( k) ln k T T wan small penaly for larger k ln T ln s k T The BIC presens a differen radeoff beween goodness of fi (small SSE) and model complexiy (k). Noice ha as he sample size T grows, he penaly par declines a rae ln(t)/t his is slower han AIC. The penaly par has MORE weigh in BIC. When comparing models, he one wih he smalles BIC is preferred. 3

4 AIC, BIC, or Adjused R squared? There is no uniform ranking of preferences over AIC, BIC, or adjused R squared. Since he BIC pus more weigh on he penaly par compared o he AIC i will end o chose smaller models (fewer x s). If forecasing is he goal, ofen smaller models forecas beer ou of sample and BIC is preferred o AIC. Ineresingly, AIC is consisen in he sense ha if you are comparing esimaed models and one of he models is he rue model, AIC will selec ha model when he sample size ges large. 3. Over Fiing Problem For any given model selecion crieria we discussed in he las secion we run he risk of over fiing if we ry a lo of models. If predicion is our goal we would like a way of figuring ou how well he model will predic when we use he model on new daa. We will discuss he use of in sample and ouof sample echniques. 4

5 When we consider a large number of models i is likely ha we will sumble on a se of x s ha predics our y well jus by chance. This is known as he in sample over fiing problem. The greaer he number of models we consider, he greaer he likelihood of his happening. I would seem ha crieria ha conrol for he number of x s in he model (hink AIC, BIC, or adjused R squared) migh solve he over fiing problem. This inuiion is WRONG. The problem is ha if we look over a very large number of models we will sill find a model ha fis ANY crieria well jus by chance, no because i is selecing a good model. If we ry a lo of combinaions, somehing will fi jus by chance. 5

6 Mone Carlo We have 00 observaions and 50 x s. Y s are independen of X s. We do model selecion using 4 crieria. Forward selecion based on AIC and BIC Backward selecion based on AIC and BIC A each sage we use add or delee a variable ha improves he AIC or he BIC he mos. In doing so, we are comparing an awful lo of models. For example, in he forward selecion we are comparing 50 models in he firs variable selecion. We repea he experimen 000 imes and record he number of x s seleced in he final model. Number of x s seleced using forward selecion AIC 6

7 Number of x s seleced using forward selecion BIC Number of x s seleced using backward selecion AIC 7

8 Number of x s seleced using backward selecion BIC The fac ha we selec many x s in all hese approaches means hese models are over fiing. The procedures are selecing he x s ha happen o fi he wiggles in his sample bu will no forecas on a differen sample. 8

9 One soluion is o only esimae and compare a small number of models and only use variables ha you suspec should be in he model. Anoher soluion is o use one porion of your daa se for esimaion and model selecion and anoher porion of your daa for subsequen model evaluaion. 3.3 Sample spliing Use he firs porion o esimae and selec a small se of candidae models. Esimaion /model selecion Esimaion Use he second hold ou porion o evaluae your candidae models using a forecas accuracy crieris. Model Evaluaion 9

10 Sample spliing procedure Sep ) Use one or more of he discussed model selecion sraegies o selec several candidae models using he model selecion daa. Le s call he esimaed models model, model, and model 3. Sep ) Using each of he esimaed candidae models, predic he oucomes for he ou of sample, hold ou daa. Sep 3) compare he predicive accuracy of each of he models in he hold ou sample. Sep 4) Models wih higher accuracy are preferred. Commens The idea is simple. The over fiing problem is miigaed by using he hold ou sample o pick a final model. If some of he candidae models are over fi, hey will likely no perform well on a new sample of daa. How o spli daa? The opimal choice would require knowledge abou he rue model ha we don usually have. Ofen /3 esimae, /3 hold ou. 0

11 Which predicion crieria? Ofen mean squared error (sum of squared errors). In essence, chose he model ha minimizes he ou of sample forecas error variance. A beer alernaive, when possible, is o chose a crieria based on how he predicions will be used. Generally he predicions will be used as an inpu o some decision. For example, suppose we have a model ha predics reurns on he S&P500 on he subsequen day. If he reurns are expeced o be above a hreshold you buy, below anoher hreshold you sell. Evaluae profis and accoun for risk possibly wih a Sharp raio or some oher rade off beween risk and reward. Choice of hold ou sample The simples possibiliy is o randomly selec observaions o use for he hold ou sample. Of course here a los of oher choices.

12 4. Muliple period ahead forecass wih an AR model and heir properies 4. Muliple period ahead forecass 4. Muliple period ahead predicion inervals 4.3 Tess for random walk 4.4 Coninuously compounded reurns 4. Saionary vs. Non saionary ime series. An imporan aspec of a ime series model is wheher i is mean revering. Tha is, can we expec ha over he long run he process will end o an average value? Saionary processes will end o rever o heir mean value and a non saionary process will no. Obviously, he models have very differen implicaions abou he long run behavior of a process. These differences are imporan in looking a long range forecass. In his secion we will examine long range forecass and pay paricular aenion o he differences beween non saionary and saionary models.

13 Le s sar wih some simulaions jus o ge he inuiion abou wha we should expec he differences in saionary and non saionary long horizon forecass o look like. Here are 4 simulaions from he model AR() model wih =. Each series begins a he value 5 and has 00 observaions. Y 0 Y

14 Someimes he series wander up, someimes down, someimes up hen down. Where do you hink he process will be in period 00? There is no force driving he series back o a mean value. Here are 4 simulaions from he AR() model wih =.5. Each series begins a he value 5. Y.5.5Y

15 Conrary o he case when =, when =.5 he series is araced back o is mean value of Clearly, in period 00, we would expec he process o be somewhere around 5. A lile above, or a lile below. Le s look more closely a where we should expec an AR() model o be in he fuure and how sure we are. To keep he noaion simple, our discussion will use ime 0 as he poin ha we sar looking a fuure values of Y. Laer I ll show you he formula for he general case where we forecas ou k periods ino he fuure, saring a an arbirary ime. 5

16 Y Y Y Y 0 0 Y Y 0 = Y Y Y Y0 3 = Y Y You can see where his is going. For an arbirary forecas of >0 periods we can wrie (*) = + Y Y... Now, le s assume < Y = + Y Y 0 j = Y0 j j0 6

17 Y 0 j = Y0 j j0 = (The las line comes from 0 ) j Y0 j j0 This suff is known a ime 0 his is (weighed) sum of inervening (fuure) values of This is a useful express ha we can use o undersand forecass. Le s find he -sep ahead forecas ha we make a ime zero. E Y Y Y Y j 0 =E 0 j 0 j0 j = Y0 E j Y0 j0 j = Y E Y j 0 0 j0 = Y 0 7

18 This expression ells us how o forecas ou muliple periods given an iniial value of Y=Y 0. Clearly here is nohing special abou saring he forecas a ime period 0. Here s wha he formula looks like for a k period ahead forecas made a an k arbirary ime. Le Y denoe he EY k Y forecas of Y +k given Y. Then Y k k k = Y Y 0 Case : < If < hen he k period ahead forecas is given by: k k k Y = Y This makes he naure of he forecas perfecly clear. I says ha he forecas is a weighed average of he las value Y and he mean of Y. The furher ou we forecas he more weigh we pu on and he less weigh on Y. 8

19 Now le s see wha happens when = From (*) before, we have: Y = Y so wih = we ge: Y = Y = Y0 0 j j0 known a ime 0 Inervening (fuure) values of So, now he forecas is given by: EY Y0 EY0 0 j0 Y E Y Y 0 0 j 0 j0 0 0 j 9

20 More generally, when =, for an arbirary saring poin, and forecas horizon k Y Y k k 0 So as k ges large here, he forecas doesn converge o a mean. If 0 <0 he forecas becomes very negaive. If 0 >0 he forecas becomes very large. If 0 =0 he forecas is simply Y. 4. Forecas errors. How big will our forecas errors be? To answer his quesion we wan o know he difference beween he acual value of Y +k and he forecased value. k We again use Y denoe he forecas of EY k Y Y +k given Y. Using his noaion, we are ineresed in he error associaed wih he k sep ahead forecas error k k e Y Y k 0

21 k e In his secion we find he variance of. Jus like in he one sep ahead forecas, his ells us abou our uncerainy. Le s sar wih case i.e. <. We can wrie Y +k as k k k k j k j k= kj kj j0 j0 k Y again, his is (weighed) sum of inervening values of k k k j e Y ky kj j0 Y Y Y

22 Les find he forecas error variance: k k j Var e Var kj j0 bu he are iid. So k k k k j j Var e Var kj j0 j0 For large k his becomes k Var e Does his make sense? The size of he forecas error variance is deermined by and and he forecas horizon k. The size of he surprise in each period is deermined by. The larger is, he larger he swings are ha Y can ake away from he mean. The forecas error variance is increasing as we forecas furher ino he fuure. The forecas error variance converges o a fixed number as becomes large. The fixed number is jus he variance of Y.

23 Case i.e. = Here so k k k Y= Y k0 j Y j k j0 j0 Y k k k k j j0 k e e Y Y The variance of is hen: k k Var e Var jk j0 This is compleely differen from he case where <. The forecas error variance is proporional o he forecas horizon, k. The variance only depends on k, no. Of course, in pracice we don know he rue parameers so we plug in our bes guesses: The k sep ahead forecas when <is given by: where is he sample average of Y. b b Yˆ = b Y b Y b y k k k 0 k k b y The bes guess for he k sep ahead forecas error variance when < is given by: k b ˆk EY ky s b 3

24 Hence when < he 95% predicion inerval for is given by: Y k ˆ k Y b b k s For he case where = we have k k Var e Var jk j0 So he 95% predicion inerval for he k sep ahead forecas is given by: Y f ks 4

25 Forecas for =.8 and 0 =.5 = Forecas for = and 0 =0 =

26 Saionary vs. non saionary summary Saionary < Non-saionary = Forecass Y Mean rever k k k = Y Trend up or down depending on sign of 0 k Y Y k0 Forecas errors Iniially increase wih he forecas horizon. k Var e k Increases wih he forecas horizon. Var e k k More complicaed ARMA models Similar expressions can be worked ou for any ARMA model. The nex secion will give a es o see if any AR model is saionary or no. 6

27 4.3 Tess for a random walk We saw ha here is a big difference beween he properies of a saionary < and nonsaionary = model. Someimes i is difficul o ell he difference in a given sample beween he wo. One plo is = and he oher =

28 Clearly, we are ineresed in running he regression Y 0 Y and esing he null ha =. Why can we es he null of saionariy? The sa from his regression doesn follow a disribuion. Neiher he mean or variance of Y exiss! Forunaely some smar guys in he 70 s and 80 s figured ou how o do he es correcly. They figured ou he correc p values for he usual es. The criical value is no and depends on he sample size. The acual disribuion is called he Dickey Fuller disribuion. Forunaely, mos sofware does his es for us. 8

29 Here is he es for series P-value Since he p-value is small we can rejec he null of = a he 6% level Uni roo es for series P-value Since he p-value is large we don rejec he null of = 9

30 4.4 Coninuously Compounded Reurns There are wo ways o define reurns. Simple, which we have been using E B p p r r p r p B or p Coninuously compounded r ln E ln B or r ln p ln p The reurns will usually be very similar for he same price values p and p. However hey are no idenical. Recall ha ln(a*b)=ln(a)+ln(b) p r p ln pln rln p ln rln pln p Finally, for small r, ln r r so, ln r ln r ln p ln p r ln p p 30

31 There is more o he coninuously compounded reurns han jus his approximaion. As he name suggess, i has o do wih how ofen he ineres r is paid. For he simple reurns, i is all paid a he end of he period. For he coninuously compounded reurn ineres is coninuously paid ou. Since you are geing ineres on ineres in he coninuously updaing case, he coninuously compounded reurns are slighly smaller. Think abou aking a period of monh where E is he end of period value and B is he beginning period value. The simple reurn asks wha is r such ha E r B or E B r B iniial invesmen back reurn on iniial invesmen 3

32 We could have defined he reurn differenly. We could have asked wha if we go a reurn r/ over he firs half of he period and hen we reinvesed i in he asse and go a reurn of r/ in he second half. The value E afer he firs half would be: r E B A he end of he full period we would have r r r r E E B B If we broke he ime period up ino N periods, we would have: r E N In he limi as we break he period up ino a very large number of periods we ge: (*) N B r E lim N N N B A lile calculus can show ha he soluion o (*) is: ln r ln E B 3

33 Example: Simple reurn: Coninuously compounded reurn:.0953 ln 0 ln 00 The wo reurns are nearly idenical. However, he coninuously compounded reurns have a very nice properies. Log prices follow a random walk Suppose we have daily reurns r, r,,r T where r =ln(p ) ln(p ) Noice ha we can re arrange he model o wrie: ln(p ) =ln(p ) +r If r is iid (abou righ) hen he log of prices follows a random walk! 33

34 Anoher ineresing propery of coninuously compounded reurns The coninuously compounded reurn over many days can be compued by summing he reurns over each day. For example, if he reurn on day is r and he reurn on day is r, hen he reurn over he wo days: r days r r ln p ln p ln p ln p0 ln p ln p 0 r r More generally we can wrie he reurn over T days as he sum of he T daily reurns: R ln p ln p ln p ln p ln p ln p T T 0 T T T T rt rt p p p p p p ln ln... ln ln ln ln T T3 0 rt r r r r... r T T 34

35 This is way more convien han for simple reurns We saw: E r B or, p r p Which means: and p r p p r p 0 or p r r p r r rr p day reurn 0 0 More generally, for he simple reurns he muliday reurn is compued from:... p r r r p 0 This is much more cumbersome o work wih. For he coninuously compounded reurns he muliday reurn is compued by summing. For he simple reurns, he muliday reurn is compued by aking producs of (+r ). 35

36 Wha is he forecas of he cumulaive reurn over he nex k periods? If he reurns are coninuously compounded, he cumulaive reurn is jus r... r r So k... k E r pase r pas... E r pas E r r r pas k sep ahead forecas sep ahead forecas k sep ahead forecas or he cumulaive reurn is jus he sum of he k forecass. 36