Uncertainty in predictive modelling

Transcription

1 Uncerainy in predicive modelling Nikolaos Kourenzes Lancaser Universiy OR59 14/09/2017

2 Agenda 1. Forecasing & uncerainy 2. Models, assumpions & uncerainy 3. Managing modelling uncerainy 4. Approach I: Poin AIC 5. Approach II: Muliple Temporal Aggregaion 2/46

3 Forecasing & decision making Decision making in organisaions has a is core an elemen of forecasing Accurae forecass lead o reduced uncerainy beer decisions Forecass maybe implici or explici Forecass aims o provide informaion abou he fuure, condiional on hisorical and curren knowledge Company arges and plans aim o provide direcion owards a desirable fuure. Forecas Targe Presen Forecas Difference beween arges and forecass, a differen horizons, provide useful feedback 3/46

4 Forecasing & uncerainy The grea hing abou forecasing is ha you will ge i wrong and ha is fine mos probable oucome! All forecass come wih uncerainies We ry o idenify and manage hese uncerainies 4/46

5 Forecasing & uncerainy One of he mos imporan jumps in forecasing as a discipline has been o move from mehod based forecass o model based forecass. Opimal (maximum likelihood) parameers & model selecion. An explici represenaion of he uncerainy of he forecass. 80% and 95% predicion inervals Bu model based forecasing hides an insidious assumpion: he specified model is rue. 5/46

6 True models & unrue assumpions To beer undersand he problem le us consider he exponenial smoohing (ETS) family of models Since invenion counless advances and applicaions [Hol, 2004; Gardner, 2006] Recen survey: 32.1% of business forecass done exclusively by ETS [Weller & Crone, 2012] numerous more depend on some baseline ETS plus exper adjusmens. Wide applicaion: supply chain [Trapero e al., 2012], call cenres [Taylor, 2008], elecriciy load [Taylor, 2007], climae modelling [Fildes & Kourenzes, 2011], ec. Relaively good performance and aracive simpliciy [Makridakis & Hibon, 2000] Hyndman e al. [2002, 2008] embedded exponenial smoohing wihin he sae space framework giving i a saisical raionale parameer esimaion, model selecion, predicion inervals, ec. A he core of everyhing is he maximum likelihood esimaion (or similar cos). 6/46

7 The Exponenial Smoohing Family None Addiive Muliplicaive None Addiive Addiive Damped Muliplicaive Muliplicaive Damped Trend Seasonaliy h L F F a aa L ) (1 1 1 s L a S A L s S L A S ) (1 ) ( k s k S L F 1 1 ) (1 ) ( T L L T ( 1) 1 T L A S s S ) 1 ( h s h i i h S T L F 1 ) )( 1 ( 1 1 s T L a S A a L Use likelihood o find smoohing and iniial values. Use some informaion crierion (AIC, AICc, BIC, ec) o choose he appropriae model per series.

8 True models & unrue assumpions We opimise our models using MLE or (for ETS) equivalenly minimise he augmened sum of squared errors crierion: For addiive errors r(x -1 ) = 1, so his is equal o he well known MSE: Observe ha he cos funcion is based on 1-sep ahead errors. If he posulaed model is rue here is no problem, bu wha happens if no? 8/46

9 They used likelihood hey hough heir posulaed model was rue Now only a differen cos funcion can save hem!

10 True models & unrue assumpions Why is his such a problem? We are ypically ineresed in forecasing more han 1-sep ahead. If he model is rue hen i successfully describes he daa for one- or anyseps ahead, and he opimal parameers for one-sep ahead are appropriae for any forecas. If he model is no rue hen minimising he one-sep ahead error resuls in parameers ha make he model (approximaely) bes for predicing onesep ahead. Bu, if we were o predic 10-seps ahead hen he parameers are far from opimal and he forecas bound o large errors [Xia e al., 2011]. 10/46

11 True models & unrue assumpions Insead of going ino a saisical explanaion, le us consider an inuiive one: We wan o make a decision h-seps ahead, so we produce a +h forecas. Why should we opimise he model parameers for a shor erm objecive (+1)? I is well acceped ha shor and long erm predicions require differen mehods and in urn parameers [Clemens and Hendry, 1998; Chafield, 2000]. When was he las ime you happily acceped your model as being rue, raher han a useful approximaion? Well his invalidaes he basic assumpion behind maximum likelihood esimaion and in-urn predicive model building. 11/46

12 Unrue models & uncerainy Wha does his mean for our model based forecas uncerainy? Le us consider a very simple model, he Single Exponenial Smoohing (SES), or ETS(A,N,N) under he sae-space naming. The variance (uncerainy) of he forecass for muliple-seps ahead is given by: +1 esimaed variance model parameer forecas horizon 12/46

13 Unrue models & uncerainy If he model is unrue hen he model parameer for +1 and +h will be differen. So he uncerainy no longer varies simply due o he forecas horizon, bu also due o he smoohing parameer In shor, he predicion inervals for +h are wrong! 13/46

14 Uncerain models We esablished ha forecas uncerainy is no hones abou parameer uncerainy and his is commonplace in pracice (he model is no rue). I has a furher imporan implicaion: Model selecion (or specificaion for regression ype models) is ofen based on likelihood based informaion crieria. Number of model parameers Maximised double negaive log likelihood and he likelihood assumes a rue model, so evenually we choose models on very uncerain grounds (or a leas wih differen objecives han he ones hey are going o be used for!) 14/46

15 A way forward: incorporaing uncerainy in specificaion The double log negaive log likelihood evenually is a summary saisic ha ignores any uncerainy (ha is wha he sums do!) We could remove he sums and calculae a poin likelihood ha reains for each poin he uncerainy (no los in he sums): From ha we can wrie he poin AIC, which is a vecor ( = 1, n) for each model. 15/46

16 A way forward: incorporaing uncerainy in specificaion For a ime series, using AIC we could have and on op of i he poin AIC The boxplos of paic visualise he model selecion uncerainy! 16/46

17 A way forward: AIC wih uncerainy So he model selecion is ransformed from choosing beween single poins (AIC values, or CV saisics) choosing beween disribuions. If here is dominan disribuion choose ha model. If here is no any dominan one combine all op ones ha here is no evidence of difference. 17/46

18 A way forward: AIC wih uncerainy Does i work? Tes on hree daases (M3 compeiion, 3003 series; FMCG company sales, 229 series; FRED invenory saisics, 323 series) Rolling origin evaluaion using ETS model family Evaluae on AvRelMAE 18/46

19 A way forward: AIC wih uncerainy Bes AIC Bes paic Unweighed combinaion AIC weighed combinaion Top paic disribuions I always pays off o ake ino accoun he modelling uncerainy in model selecion! Improvemens up o 5% from he same forecass Working paper available soon TM 19/46

20 Agenda 1. Forecasing & uncerainy 2. Models, assumpions & uncerainy 3. Managing modelling uncerainy 4. Approach I: Poin AIC 5. Approach II: Muliple Temporal Aggregaion 20/46

21 Long erm forecasing We know ha differen forecasing models are beer for differen forecas horizons We also know ha i helps o forecas long horizons using aggregae daa Forecasing a quarer ahead using daily daa is `advenurous (90 seps ahead) Forecasing a quarer ahead using quarerly daa is easier (1 sep ahead) A differen daa frequencies differen componens of he series dominae ETS(M,A d,a) - AIC: x ETS(A,A,N) - AIC: d Monhs Years These forecass ofen do no agree, which one is `correc? 21/46

22 Any issues wih curren pracice? Issues wih auomaic modelling: Model selecion How good is he bes fi model? How reliable? Sampling uncerainy Idenified model/parameers sable as new daa appear? Model uncerainy Appropriae model srucure and parameers? Transparency/Trus Praciioners do no rus sysems ha change subsanially ETS(M,A d,a) - AIC: ETS(A,A d,m) - AIC: Monhs /46

23 Sales Any issues wih curren pracice? Wha can go wrong in parameer and model selecion: Business ime series are ofen shor Limied daa Esimaion of parameers can fail miserably (for monhly daa opimise up o 18 parameers, wih ofen no more han 36 observaions) Model selecion can fail as well (30 models over-fiing?) Boh opimisaion and model selecion are myopic Focus on daa fiing in he pas, raher han forecasabiliy Special cases: Demand Fi Forecas Monh True model: Addiive rend, addiive seasonaliy Idenified model: No rend, addiive seasonaliy Why? In-sample variance explained mosly by seasonaliy 23/46

24 A differen ake on modelling: emporal ricks! Tradiionally we model ime series a he frequency ha we sampled hem or ake decisions. However, a ime series can be view in many differen ways, adaping he noion of produc hierarchies o emporal hierarchies: 24/46

25 How emporal aggregaion changes he series Seasonal diagrams 25/46

26 The Idea Temporal aggregaion srenghens and aenuaes differen elemens of he series: a an aggregae level rend/cycle is easy o disinguish a a disaggregae level high frequency elemens like seasonaliy ypically dominae. Modelling a ime series a a very disaggregae level (e.g. weekly) shor-erm forecas. The opposie is rue for aggregae levels (e.g. annual) Propose Temporal Hierarchies ha provide a framework o opimally combine informaion from various levels (irrespecive of forecasing mehod) o: avoid over-reliance on a single planning level and merge informaive views avoid over-reliance on a single forecasing mehod/model manage uncerainy! 26/46

27 Temporal aggregaion and forecasing I is no new, bu he quesion has been a which single level o model he ime series. Economerics have invesigae he quesion for decades inconclusive Supply chain applicaions: ADIDA beneficial o slow and fas moving iems forecas accuracy (like everyhing no always!): Sep 1: Temporally aggregae ime series o he appropriae level Sep 2: Forecas Sep 3: Disaggregae forecas and use Selecion of aggregaion level No heoreical grounding for general case, bu good undersanding for AR(1)/MA(1) cases. 27/46

28 Muliple emporal aggregaion Wha if we do no selec an aggregaion level? use muliple 200 Aggregaion level 1 ETS(A,N,A) 200 ETS(A,M,A) Demand Demand Demand Demand Aggregaion level Period Aggregaion level 7 ETS(A,M,N) Period Aggregaion level 12 ETS(A,A,N) Issues: Differen model Differen lengh Combinaion Period Period 28/46

29 Demand Demand Forecas combinaion: Issue: Muliple emporal aggregaion Forecas combinaion is widely considered as beneficial for forecas accuracy Simple combinaion mehods (average, median) considered robus, relaively accurae o more complex mehods If here are differen model ypes o be combined hen he resuling forecas does no fi well a any componen! Period Period 29/46

30 y [1] y [2] y [3] Aggregae Fi sae space ETS Save saes Level y [10] y [11] y [12] Trend Season /46

31 Transform saes o addiive and o original sampling frequency Combine saes (componens) Produce forecass 31/46

32 Muliple Aggregaion Predicion Algorihm (MAPA) Sep 1: Aggregaion Sep 2: Forecasing Sep 3: Combinaion Y 1 ETS Model Selecion l 1 b s Ŷ 1 k 2 k 3... k K Y Y... Y 2 3 K ETS Model Selecion ETS Model Selecion... ETS Model Selecion l 2 b 2 s l 3 b s 3... l b s 2 3 K K K 1 K 1 K 1 K l b s Srenghens and aenuaes componens Esimaion of parameers a muliple levels Robusness on model selecion and parameerisaion 32/46

33 Muliple Aggregaion Predicion Algorihm (MAPA) 33/46

34 Some resuls I AvRelMAE on real & simulaed daa (Kourenzes, e al., 2017) 34/46

35 Some resuls II RMSE on FMCG forecasing (Barrow & Kourenzes, 2016) 35/46

36 Some resuls III Scaled Mean Absolue Error on SKUS wih promoions (Kourenzes & Peropoulos, 2016) 36/46

37 Muliple Aggregaion Predicion Algorihm (MAPA) MAPA was developed o ake advanage of emporal aggregaion and hierarchies: MAPA provides a framework o beer idenify and esimae he differen ime series componens Manage modelling uncerainy Robus agains model selecion and parameerisaion issues Beer forecass On average ouperforms ETS, one of he mos widely used, robus and accurae univariae forecasing mehods Shown o be useful for fas moving iems, promoional modelling and inermien ime series forecasing. 37/46

38 Temporal Hierarchies: A modelling framework MAPA demonsraed he srengh of he approach, bu i is no general: How o incorporae forecass from any model/mehod? How o incorporae judgemen? We can inroduce a general framework for emporal hierarchies ha borrows many elemens from cross-secional hierarchical forecasing. Objecive: differen model families (including mulivariae models) and human judgemen: consider differen sources informaion handle uncerainy differenly bu reain modelling uncerainy hemselves So merge model/mehod/exper forecass ge holisic view and reduce uncerainy 38/46

39 Cross-secional and Temporal Hierarchies We know how o do cross-secional hierarchies Top-down, boom-up, middle-ou Opimal combinaions Toal UK Spain Produc A Produc B Produc A Produc B We know how o do his! hen we know how o do his as well, wih some small-prin! bu we have o correc for he differen scales a each aggregaion level, which is no ha difficul due o he imposed emporal srucure. For ha we need o calculae he covariance marix beween he forecas errors a differen aggregaion levels. There are easy and no so easy ways o do his. 39/46

40 Some evidence ha i acually works! Comparison wih oher M3 resuls (symmeric Mean Absolue Percenage Error): Monhly daase Temporal (ETS based): 13.61% ETS: 14.45% [Hyndman e al., 2002] MAPA: 13.69% [Kourenzes e al., 2014] Thea: 13.85% (bes original performance) [Makridakis & Hibon, 2000] Quarerly daase Temporal (ETS based): 9.70% ETS: 9.94% [Hyndman e al., 2002] MAPA: 9.58% [Kourenzes e al., 2014] Thea: 8.96% (bes original performance) [Makridakis & Hibon, 2000] Deailed resuls available, if you are ineresed, a he end of he presenaion! 40/46

41 Applicaion: Predicing A&E admissions Collec weekly daa for UK A&E wards. 13 ime series: covering differen ypes of emergencies and differen severiies (measured as ime o reamen) Span from week (7 h Nov 2010) o week (7 h June 2015) Series are a England level (no local auhoriies). Accuraely predic o suppor saffing and raining decisions. Aligning he shor and long erm forecass is imporan for consisency of planning and budgeing. Tes se: 52 weeks. Rolling origin evaluaion. Forecas horizons of ineres: +1, +4, +52 (1 week, 1 monh, 1 year). Evaluaion MASE (relaive o base model) As a base model auo.arima (forecas package R) is used. 41/46

42 Applicaion: Predicing A&E admissions Toal Emergency Admissions via A&E Red is he predicion of he base model (ARIMA) Blue is he emporal hierarchy reconciled forecass (based on ARIMA) Observe how informaion is `borrowed beween emporal levels. Base models for insance provide very poor weekly and annual forecass 42/46

43 Applicaion: Predicing A&E admissions Accuracy gains a all planning horizons Crucially, forecass are reconciled leading o aligned plans 43/46

44 Producion ready? Muliple Aggregaion Predicion Algorihm (MAPA) Kourenzes, N.; Peropoulos, F. & Trapero, J. R. Improving forecasing by esimaing ime series srucural componens across muliple frequencies. Inernaional Journal of Forecasing, 2014, 30, (Deails) Kourenzes, N.; Rosami-Tabar B. & Barrow D. K. Demand forecasing by emporal aggregaion: using opimal or muliple aggregaion levels? Journal of Business Research, (Modelling uncerainy view) Oher papers exend MAPA for inermien demand and exogenous regressors. R code, MAPA package on CRAN: hps://cran.r-projec.org/package=mapa For inermien demand use R package sinermien. Temporal Hierarchies (THieF) Ahanasopoulos G.; Hyndman R.J.; Kourenzes, N.; Peropoulos. Forecasing wih Temporal Hierarchies. European Journal of Operaional Research, 2017, R code, hief package on CRAN: hps://cran.r-projec.org/package=hief All papers, code and examples available on my websie (hp://nikolaos.kourenzes.com) 44/46

45 Conclusions Forecas uncerainy is much bigger han he variance of he forecas errors! Uncerainy in he parameers & uncerainy in model form. Accouning for holisic uncerainy has implicaion for he consrucion and use of he forecass. Here we looked a wo poenial roues forward: paic and Muliple Temporal Aggregaion (MTA) paic is sill in is infancy, bu demonsraes he imporan of accouning model form uncerainy when choosing your forecasing model. MTA is implemened hrough MAPA (very good rack record bu resricive) and emporal hierarchies (more flexible, bu no universally beer han MAPA). Side benefi of MTA: reconciled forecass for operaional/acical/sraegic planning. 45/46

46 Thank you for your aenion! Quesions? Published, working papers and code available a my blog! Nikolaos Kourenzes nikolaos@kourenzes.com blog: hp://nikolaos.kourenzes.com

47 Appendix Deailed M3 resuls for emporal hierarchies

48 Some evidence ha i acually works! M3 quarerly daase % error change over base % error change over base ETS ARIMA BU: Boom-Up; WLS H : Hierarchy scaling; WLS v : Variance scaling; WLS s : Srucural scaling 756 series, forecas quarers ahead

49 Some evidence ha i acually works! M3 monhly daase % error change over base % error change over base ETS ARIMA BU: Boom-Up; WLS H : Hierarchy scaling; WLS v : Variance scaling; WLS s : Srucural scaling 1453 series, forecas monhs ahead

50 Appendix Calculaion deails for emporal hierarchies

51 Temporal Hierarchies - Noaion Non-overlapping emporal aggregaion o k h level: Annual Semi-annual Quarerly Observaions a each aggregaion level

52 Temporal Hierarchies - Noaion Collecing he observaions from he differen levels in a column: We can define a summing marix S so ha: Lowes level observaions Annual Semi-annual Quarer

53 Example: Monhly Aggregaion levels k are seleced so ha we do no ge fracional seasonaliies

54 Temporal Hierarchies - Forecasing We can arrange he forecass from each level in a similar fashion: The reconciliaion model is: Reconciled forecass Summing marix Reconciliaion errors how much he forecass across levels do no agree Unknown condiional means of he fuure values a lowes level The reconciliaion error has zero mean and covariance marix

55 Temporal Hierarchies - Forecasing If was known hen we can wrie (GLS esimaor): Bu in general i is no know, so we need o esimae i. I can be shown ha is no idenifiable (you need o know he reconciled forecass, before you reconcile hem), however: Reconciliaion errors So our problem becomes: Covariance of forecas errors

56 Temporal Hierarchies - Forecasing All we need now is an esimaion of W Sample covariance of in-sample errors In principle his is fine, bu is sample size is conrolled by he number of oplevel (annual) observaions. For example 104 observaions a weekly level, resuls in jus 2 sample poins (2 years). So he esimaion of is ypically weak in pracice.

57 Temporal Hierarchies - Forecasing We propose hree ways o esimae i, wih increasing simplifying assumpions. Using as example quarerly daa he approximaions are: Hierarchy variance scaling Diagonal of covariance marix less elemens o esimae Series variance scaling Srucural scaling Assume wihin level equal variances. This is wha convenional forecasing does. Increases sample size. Assume proporional error variances. No need for esimaes can be used when unknown (e.g. exper forecass).