Prediction and Change Detection

Transcription

1 Predcton and Change Detecton Mark Steyvers Scott Brown Unversty of Calforna, Irvne Unversty of Calforna, Irvne Irvne, CA Irvne, CA Abstract We measure the ablty of human observers to predct the next datum n a sequence that s generated by a smple statstcal process undergong change at random ponts n tme. Accurate performance n ths task requres the dentfcaton of changeponts. We assess ndvdual dfferences between observers both emprcally, and usng two knds of models: a Bayesan approach for change detecton and a famly of cogntvely plausble fast and frugal models. Some ndvduals detect too many changes and hence perform sub-optmally due to excess varablty. Other ndvduals do not detect enough changes, and perform sub-optmally because they fal to notce short-term temporal trends. Introducton Decson-makng often requres a rapd response to change. For example, stock analysts need to quckly detect changes n the market n order to adjust nvestment strateges. Coaches need to track changes n a player s performance n order to adjust strategy. When trackng changes, there are costs nvolved when ether more or less changes are observed than actually occurred. For example, when usng an overly conservatve change detecton crteron, a stock analyst mght mss mportant short-term trends and nterpret them as random fluctuatons nstead. On the other hand, a change may also be detected too readly. For example, n basketball, a player who makes a seres of consecutve baskets s often dentfed as a hot hand player whose underlyng ablty s perceved to have suddenly ncreased [,2]. Ths mght lead to sub-optmal passng strateges, based on random fluctuatons. We are nterested n explanng ndvdual dfferences n a sequental predcton task. Observers are shown stmul generated from a smple statstcal process wth the task of predctng the next datum n the sequence. The latent parameters of the statstcal process change dscretely at random ponts n tme. Performance n ths task depends on the accurate detecton of those changeponts, as well as nference about future outcomes based on the outcomes that followed the most recent nferred changepont. There s much pror research n statstcs on the problem of dentfyng changeponts [3,4,5]. In ths paper, we adopt a Bayesan approach to the changepont dentfcaton problem and develop a smple nference procedure to predct the next datum n a sequence. The Bayesan model serves as an deal observer model and s useful to characterze the ways n whch ndvduals devate from optmalty.

2 The plan of the paper s as follows. We frst ntroduce the sequental predcton task and dscuss a Bayesan analyss of ths predcton problem. We then dscuss the results from a few ndvduals n ths predcton task and show how the Bayesan approach can capture ndvdual dfferences wth a sngle twtchness parameter that descrbes how readly changes are perceved n random sequences. We wll show that some ndvduals are too twtchy: ther performance s too varable because they base ther predctons on too lttle of the recent data. Other ndvduals are not twtchy enough, and they fal to capture fast changes n the data. We also show how behavor can be explaned wth a set of fast and frugal models [6]. These are cogntvely realstc models that operate under plausble computatonal constrants. 2 A predcton task wth multple changeponts In the predcton task, stmul are presented sequentally and the task s to predct the next stmulus n the sequence. After t trals, the observer has been presented wth stmul y, y 2,, y t and the task s to make a predcton about y t+. After the predcton s made, the actual outcome y t+ s revealed and the next tral proceeds to the predcton of y t+2. Ths procedure starts wth y and s repeated for T trals. The observatons y t are D-dmensonal vectors wth elements sampled from bnomal dstrbutons. The parameters of those dstrbutons change dscretely at random ponts n tme such that the mean ncreases or decreases after a change pont. Ths generates a sequence of observaton vectors, y, y 2,, y T, where each y t = {y t, y t,d }. Each of the y t,d s sampled from a bnomal dstrbuton Bn(θ t,d,k), so y t,d K. The parameter vector θ t ={θ t, θ t,d } changes dependng on the locatons of the changeponts. At each tme step, x t s a bnary ndcator for the occurrence of a changepont occurrng at tme t+. The parameter α determnes the probablty of a change occurrng n the sequence. The generatve model s specfed by the followng algorthm:. For d=..d sample θ,d from a Unform(,) dstrbuton 2. For t=2..t, (a) Sample x t- from a Bernoull(α) dstrbuton (b) If x t- =, then θ t =θ t-, else for d=..d sample θ t,d from a Unform(,) dstrbuton (c) for d=..d, sample y t from a Bn(θ t,d,k) dstrbuton Table shows some data generated from the changepont model wth T=2, α=.,and D=. In the predcton task, y wll be observed, but x and θ are not. Table : Example data t x θ y

3 3 A Bayesan predcton model In both our Bayesan and fast-and-frugal analyses, the predcton task s decomposed nto two nference procedures. Frst, the changepont locatons are dentfed. Ths s followed by predctve nference for the next outcome based on the most recent changepont locatons. Several Bayesan approaches have been developed for changepont problems nvolvng sngle or multple changeponts [3,5]. We apply a Markov Chan Monte Carlo (MCMC) analyss to approxmate the jont posteror dstrbuton over changepont assgnments x whle ntegratng out θ. Gbbs samplng wll be used to sample from ths posteror margnal dstrbuton. The samples can then be used to predct the next outcome n the sequence. 3. Inference for changepont assgnments. To apply Gbbs samplng, we evaluate the condtonal probablty of assgnng a changepont at tme, gven all other changepont assgnments and the current α value. By ntegratng out θ, the condtonal probablty s (,, α) (, θ, α, ) P x x y = P x x y () θ where x represents all swtch pont assgnments except x. Ths can be smplfed by consderng the locaton of the most recent changepont precedng and followng tme L and the outcomes occurrng between these locatons. Let n be the number of tme steps from the last changepont up to and ncludng the current tme step such that x = and x = for <j< L R L n. Smlarly, let n n L n + j be the number of tme steps that follow tme step up to the next changepont such that x = and x = for + + R <j< R L n. Let y = L y and R n k y < k = y. The update equaton for the k< k + n R k changepont assgnment can then be smplfed to ( m x ) P x = R n D L R L R L R ( y, j y, j) ( Kn Kn y, j y, j) L R j= Γ ( 2 + Kn + Kn ) L L L R R R ( yj, ) ( Kn yj, ) ( yj, ) ( Kn yj, ) L R Γ ( 2+ Kn ) Γ ( 2+ Kn ) Γ + + Γ + + ( α ) m = D Γ + Γ + Γ + Γ + α j= m = We ntalze the Gbbs sampler by samplng each x t from a Bernoull(α) dstrbuton. All changepont assgnments are then updated sequentally by the Gbbs samplng equaton above. The sampler s run for M teratons after whch one set of changepont assgnments s saved. The Gbbs sampler s then restarted multple tmes untl S samples have been collected. Although we could have ncluded an update equaton for α, n ths analyss we treat α as a known constant. Ths wll be useful when characterzng the dfferences between human observers n terms of dfferences n α. n j (2)

4 3.2 Predctve nference The next latent parameter value θ t+ and outcome y t+ can be predcted on the bass of observed outcomes that occurred after the last nferred changepont: θ t ( θ ) = y / K, y = round K (3) t+, j, j t+, j t+, j * = t + where t * s the locaton of the most recent change pont. By consderng multple Gbbs samples, we get a dstrbuton over outcomes y t+. We base the model predctons on the mean of ths dstrbuton. 3.3 Illustraton of model performance Fgure llustrates the performance of the model on a one dmensonal sequence (D=) generated from the changepont model wth T=6, α=.5, and K=. The Gbbs sampler was run for M=3 teratons and S=2 samples were collected. The top panel shows the actual changeponts (trangles) and the dstrbuton of changepont assgnments averaged over samples. The bottom panel shows the observed data y (thn lnes) as well as the θ values n the generatve model (rescaled between and ). At locatons wth large changes between observatons, the margnal changepont probablty s qute hgh. At other locatons, the true change n the mean s very small, and the model s less lkely to put n a changepont. The lower rght panel shows the dstrbuton over predcted θ t+ values. x t.5 θ t+ y t Fgure. Results of model smulaton. 4 Predcton experment We tested performance of 9 human observers n the predcton task. The observers ncluded the authors, a vstor, and one student who were aware of the statstcal nature of the task as well as naïve students. The observers were seated n front of an LCD touch screen dsplayng a two-dmensonal grd of x buttons. The changepont model was used to generate a sequence of T=5 stmul for two bnomal varables y and y 2 (D=2, K=). The change probablty α was set to.. The two varables y and y 2 specfed the two-dmensonal button locaton. The same sequence was used for all observers. On each tral, the observer touched a button on the grd dsplayed on the touch screen. Followng each button press, the button correspondng to the next {y,y 2 } outcome n the sequence was hghlghted. Observers were nstructed to press the button that best predcted the next locaton of the hghlghted button. The 5 trals were dvded nto

5 three blocks of 5 trals. Breaks were allowed between blocks. The whole experment lasted between 5 and 3 mnutes. Fgure 2 shows the frst 5 trals from the thrd block of the experment. The top and bottom panels show the actual outcomes for the y and y 2 button grd coordnates as well as the predctons for two observers (SB and MY). The fgure shows that at tral 5, the y and y 2 coordnates show a large shft followed by an mmedate shft n observer s MY predctons (on tral 6). Observer SB wats untl tral 7 to make a shft. 5 5 outcomes SB predctons MY predctons Tral Fgure 2. Tral by tral predctons from two observers. 4. Task error We assessed predcton performance by comparng the predcton wth the actual outcome n the sequence. Task error was measured by normalzed cty-block dstance task error= T O O yt, yt, + yt,2 y (4) t,2 ( ) T t= 2 where y O represents the observer s predcton. Note that the very frst tral s excluded from ths calculaton. Even though more sutable probablstc measures for predcton error could have been adopted, we wanted to allow comparson of observer s performance wth both probablstc and non-probablstc models. Task error ranged from 2.8 (for partcpant MY) to 3.3 (for ML). We also assessed the performance of fve models ther task errors ranged from 2.78 to 3.2. The Bayesan models (Secton 3) had the lowest task errors, just below 2.8. Ths fts wth our defnton of the Bayesan models as deal observer models ther task error s lower than any other model s and any human observer s task error. The fast and frugal models (Secton 5) had task errors rangng from 2.85 to Modelng Results We wll refer to the models wth the followng letter codes: B=Bayesan Model, LB=lmted Bayesan model, FF..3=fast and frugal models..3. We assessed model ft by comparng the model s predcton aganst the human observers predctons, agan usng a normalzed cty-block dstance

6 model error= ( T ) M O M O yt, yt, + yt,2 y (5) t,2 T t= 2 where y M represents the model s predcton. The model error for each ndvdual observer s shown n Fgure 3. It s mportant to note that because each model s assocated wth a set of free parameters, the parameters optmzed for task error and model error are dfferent. For Fgure 3, the parameters were optmzed to mnmze Equaton (5) for each ndvdual observer, showng the extent to whch these models can capture the performance of ndvdual observers, not necessarly provdng the best task performance. 2 B LB FF FF2 FF3 Model Error.5.5 MY MS MM EJ PH NP DN SB ML Fgure 3. Model error for each ndvdual observer. 5. Bayesan predcton models At each tral t, the model was provded wth the sequence of all prevous outcomes. The Gbbs samplng and nference procedures from Eq. (2) and (3) were appled wth M=3 teratons and S=2 samples. The change probablty α was a free parameter. In the full Bayesan model, the whole sequence of observatons up to the current tral s avalable for predcton, leadng to a memory requrement of up to T=5 trals a psychologcally unreasonable assumpton. We therefore also smulated a lmted Bayesan model (LB) where the observed sequence was truncated to the last outcomes. The LB model showed almost no decrement n task performance compared to the full Bayesan model. Fgure 3 also shows that t ft human data qute well. 5.2 Indvdual Dfferences The rght-hand panel of Fgure 4 plots each observer s task error as a functon of the mean cty-block dstance between ther subsequent button presses. Ths shows a clear U-shaped functon. Observers wth very varable predctons (e.g., ML and DN) had large average changes between successve button pushes, and also had large task error: These observers were too twtchy. Observers wth very small average button changes (e.g., SB and NP) were not twtchy enough, and also had large task error. Observers n the mddle had the lowest task error (e.g., MS and MY). The left-hand panel of Fgure 4 shows the same data, but wth the x-axs based on the Bayesan model fts. Instead of usng mean button change dstance to ndex twtchness (as n Error bars ndcate bootstrapped 95% confdence ntervals.

7 the rght-hand panel), the left-hand panel uses the estmated α parameters from the Bayesan model. A smlar U-shaped pattern s observed: ndvduals wth too large or too small α estmates have large task errors. 3.3 SB ML 3.3 SB ML 3.2 NP DN 3.2 NP DN Task Error 3. 3 EJ MM PH Task Error 3. 3 EJ MM PH B MS MY MS MY α Mean Button Change Fgure 4. Task error vs. twtchness. Left-hand panel ndexes twtchness usng estmated α parameters from Bayesan model fts. Rght-hand panel uses mean dstance between successve predctons. 5.3 Fast-and-Frugal (FF) predcton models These models perform the predcton task usng smple heurstcs that are cogntvely plausble. The FF models keep a short memory of prevous stmulus values and make predctons usng the same two-step process as the Bayesan model. Frst, a decson s made as to whether the latent parameter θ has changed. Second, remembered stmulus values that occurred after the most recently detected changepont are used to generate the next predcton. A smple heurstc s used to detect changeponts: If the dstance between the most recent observaton and predcton s greater than some threshold amount, a change s nferred. We defned the dstance between a predcton (p) and an observaton (y) as the dfference between the log-lkelhoods of y assumng θ=p and θ=y. Thus, f f B (. θ, K) s the bnomal densty wth parameters θ and K, the dstance between observaton y and predcton p s defned as d(y,p)=log(f B (y y,k))-log(f B (y p,k)). A changepont on tme step t+ s nferred whenever d(y t,p t )>C. The parameter C governs the twtchness of the model predctons. If C s large, only very dramatc changeponts wll be detected, and the model wll be too conservatve. If C s small, the model wll be too twtchy, and wll detect changeponts on the bass of small random fluctuatons. Predctons are based on the most recent M observatons, whch are kept n memory, unless a changepont has been detected n whch case only those observatons occurrng after the changepont are used for predcton. The predcton for tme step t+ s smply the mean of these observatons, say p. Human observers were retcent to make predctons very close to the boundares. Ths was modeled by allowng the FF model to change ts predcton for the next tme step, y t+, towards the mean predcton (.5). Ths change reflects a two-way bet. If the probablty of a change occurrng s α, the best guess wll be.5 f that change occurs, or the mean p f the change does not occur. Thus, the predcton made s actually y t+ =/2 α+(-α)p. Note that we do not allow perfect knowledge of the probablty of a changepont, α. Instead, an estmated value of α s used based on the number of changeponts detected n the data seres up to tme t.

8 The FF model nests two smpler FF models that are psychologcally nterestng. If the twtchness threshold parameter C becomes arbtrarly large, the model never detects a change and nstead becomes a contnuous runnng average model. Predctons from ths model are smply a boxcar smooth of the data. Alternatvely, f we assume no memory the model must based each predcton on only the prevous stmulus (.e., M=). Above, n Fgure 3, we labeled the complete FF model as FF, the boxcar model as FF2 and the memoryless model FF3. Fgure 3 showed that the complete FF model (FF) ft the data from all observers sgnfcantly better than ether the boxcar model (FF2) or the memoryless model (FF3). Exceptons were observers PH, DN and ML, for whom all three FF model ft equally well. Ths result suggests that our observers were (mostly) dong more than just keepng a runnng average of the data, or usng only the most recent observaton. The FF model ft the data about as well as the Bayesan models for all observers except MY and MS. Note that, n general, the FF and Bayesan model fts are very good: the average cty block dstance between the human data and the model predcton s around.75 (out of ) buttons on both the x- and y-axes. 6 Concluson We used an onlne predcton task to study changepont detecton. Human observers had to predct the next observaton n stochastc sequences contanng random changeponts. We showed that some observers are too twtchy : They perform poorly on the predcton task because they see changes where only random fluctuaton exsts. Other observers are not twtchy enough, and they perform poorly because they fal to see small changes. We developed a Bayesan changepont detecton model that performed the task optmally, and also provded a good ft to human data when sub-optmal parameter settngs were used. Fnally, we developed a fast-and-frugal model that showed how partcpants may be able to perform well at the task usng mnmal nformaton and smple decson heurstcs. Acknowledgments We thank Erc-Jan Wagenmakers and Mke Y for useful dscussons related to ths work. Ths work was supported n part by a grant from the US Ar Force Offce of Scentfc Research (AFOSR grant number FA ). References [] Glovch, T., Vallone, R. and Tversky, A. (985). The hot hand n basketball: on the mspercepton of random sequences. Cogntve Psychology7, [2] Albrght, S.C. (993a). A statstcal analyss of httng streaks n baseball. Journal of the Amercan Statstcal Assocaton, 88, [3] Stephens, D.A. (994). Bayesan retrospectve multple changepont dentfcaton. Appled Statstcs 43(), [4] Carln, B.P., Gelfand, A.E., & Smth, A.F.M. (992). Herarchcal Bayesan analyss of changepont problems. Appled Statstcs 4(2), [5] Green, P.J. (995). Reversble jump Markov chan Monte Carlo computaton and Bayesan model determnaton. Bometrka 82(4), [6] Ggerenzer, G., & Goldsten, D.G. (996). Reasonng the fast and frugal way: Models of bounded ratonalty. Psychologcal Revew, 3,