Improved multi-level pedestrian behavior prediction based on matching with classified motion patterns

Transcription

1 itle Improved multi-level pedetrian behavior prediction baed on with claified motion pattern Author() Chen, Z; Yung, NHC Citation he 12th IEEE International Conference on Intelligent ranportation Sytem (ISC 2009), St. Loui, MO., 3-7 October In Proceeding of the 12th ISC, 2009, p Iued Date 2009 URL Right International Conference on Intelligent ranportation. Copyright IEEE.

2 Proceeding of the 12th International IEEE Conference on Intelligent ranportation Sytem, St. Loui, MO, USA, October 3-7, 2009 MoD4.4 Improved Multi-Level Pedetrian Behavior Prediction Baed on Matching with Claified Motion Pattern Zhuo Chen N.H.C. Yung Department of Electrical and Electronic Engineering he Univerity of Hong Kong Pofulam Road, Hong Kong, H.K.S.A.R. Abtract hi paper propoe an improved multi-level pedetrian behavior prediction method baed on our previou reearch wor on learning pedetrian motion pattern and predicting pedetrian long-term behavior a their motion intance are being oberved. he improvement mainly focue on the imilarity criteria between the trajectory and the clutered MP whoe main advantage are that (1) a reaonable imilarity range of MP i automatically calculated intead of manually et; (2) the ditance feature and the changing feature are conidered together for imilarity while only the ditance feature i conidered before. he improved method ha been implemented and a tudy of how the new prediction method perform in real world cenario i conducted. he reult how that it wor well in real DCE and the prediction i conitent with the actual behavior. Keyword-motion pattern; imilarity ; multi-level behavior prediction; dynamically changing environment I. INRODUCION raffic accident ha been decribed a one of the major caue of death and injurie around the world in a World Health Organization report [1]. Compared with vehicle occupant, the vulnerable road uer, uch a pedetrian, uffer higher ri of death in traffic accident [2]. Obviouly, if pedetrian behavior can be captured, analyzed and predicted, then many potential traffic accident may be avoided or the everity of the impact heavily reduced, which could eventually lower pedetrian fatalitie a a reult. In a Dynamically Changing Environment (DCE) uch a a buy treet in a build-up area involving both vehicle and pedetrian, Colliion Avoidance (CA) i no longer jut a matter between vehicle, but alo between vehicle and pedetrian. Auming that mot pedetrian have a certain ability of CA and would behave in a rational manner on the road, it would then be up to the vehicle or agent to navigate with a reaonably fat CA repone. In order to do that, reactive repone ha to be replaced by the ability to loo ahead into the future, i.e., prediction of pedetrian a well a vehicle movement or behavior in the proximity of the agent. Conventionally, object behavior prediction i performed in a hort-term manner which focue on pedetrian motion in the next time-tep [3-8]. hi ha the advantage that hort-term prediction i certainly more accurate than long-term prediction, although with only the next time-tep predicted, navigation planning can only be hort-term a well, without being able to conider a longer term path optimization. A uch, ome reearcher have recently attempted long-term behavior prediction for global and optimal CA [9, 10]. Our previou reearch wor alo concentrated on long-term behavior prediction which firt learned Motion Pattern (MP) from a erie of oberved pedetrian motion intance and then predicted long-term pedetrian behavior over a number of future time tep [11]. Compared with the other long-term behavior prediction method [9, 10], the main advantage of our previou method are that (1) no priori nowledge of pedetrian i needed for the contruction of MP; (2) it predict the entire path to be travelled by the pedetrian intead of jut the detination. In thi paper, we propoe an improved multi-level pedetrian behavior prediction method which ucceed the general idea of our previou reearch wor. he improvement mainly focue on the imilarity criteria between the trajectory and the clutered MP. he new criteria are uperior in two way: (1) a reaonable imilarity range of MP i automatically calculated intead of manually et; (2) the ditance feature and the changing feature are conidered together for imilarity while only the ditance feature i conidered before. In the improved method, the oberved pedetrian trajectorie (in term of patial, velocity or heading ) are clutered uing the clutering algorithm a decribed in [12]. For each clutered MP, it i either claified a a complete MP (MP_C), which repreent pattern that i more-or-le conitent over time, or a an incomplete MP (MP_I), which repreent an inconitent pattern that may be updated in the future [11]. Baed on thee MP, multi-level prediction i employed. It conit of three level of prediction, in which the high and middle level are both long-term prediction baed on the MP_C and MP_I that predict future trajectorie over a number of time tep. On the other hand, the low level prediction predict only the next time-tep, equivalent to a hort-term prediction. he improved method ha been implemented and a tudy of how the new prediction method perform in real world cenario i conducted. he reult how that it wor well in real DCE and the prediction i conitent with the actual behavior. he ret of thi paper i organized a follow. In Section II, give a brief introduction on the generalized multi-level prediction framewor. In Section III, preent the improved /09/$ IEEE 249

3 -baed pedetrian behavior prediction model. Section IV depict the experimental reult produced by the improved method, and Section V conclude the paper with a brief dicuion of future reearch direction. II. GENERALIZED MULI-LEVEL PREDICION FRAMEWORK he generalized multi-level framewor conit of four main function: (1) rajectory Formation; (2) MP Clutering; (3) MP Claification and Maintenance; and (4) Pedetrian Behavior Prediction; a depicted in Figure 1. When applied in a pecific cenario, at ome time tep t, the oberved new pedetrian intance are firt aociated with exiting trajectorie that have been aembled through the previou t-1 time tep. he aociation aim at optimizing a global hortet ditance for all intance. Baed on the newly formed trajectorie with patial feature, the trajectorie with velocity and heading feature can be accordingly derived. When trajectorie are obtained, MP clutering i performed for learning MP by uing an intance-baed clutering algorithm [12]. Each clutered MP repreent a ub-group of trajectorie that have imilar characteritic with patial, velocity or heading feature. In MP claification and maintenance module, by evaluating the number of obervable motion intance in each MP cluter, the MP i further claified into MP_C or MP_I [11]. MP MP Clutering MP Claification and Maintenance MP_I MP_C Hitorical trajectorie Obervable pedetrian intance (S i,t ) rajectory Formation Current trajectorie MP_C Matching MP_I Matching Long-term behavior High-level prediction Medium-term behavior Middle-level prediction Short-term Action Forecating action Low-level prediction Figure 1. Overview of the generalized multi-level prediction framewor. Finally, pedetrian behavior prediction i performed which will be decribed in detail in the following Section. Pedetrian behavior prediction conit of three level prediction. Highlevel prediction i firt performed in MP_C module if there i ome available MP_C. If a qualified match can be found between a trajectory and a MP_C, a long-term behavior of the trajectory i predicted to be imilar to the MP_C. If there i no available MP_C or no qualified match, middle-level prediction i then performed in MP_I module for the remaining unmatched trajectorie. If there i a qualified match when a trajectory with a MP_I, a medium-term behavior of the trajectory i predicted to be imilar to the MP_I. If no qualified exit, low-level prediction i accordingly performed in action forecating module, in which one ingle time-tep action i predicted. III. MACHING-BASED PEDESRIAN BEHAVIOR PREDICION MODEL A. General Algorithmic Flow MP_C (P_C n ) MP_I (P_I m ) Current trajectorie ( ) I MP_C empty? Current trajectorie ( ) Dimenion equalization Current trajectorie ( ) Ye Single-tep forecating Action Forecating No Dimenion equalization Pre-requiite Pre-requiite MP_C Matching Similarity MP_I Matching Short-term action (* ) Similarity I middle-level match found? No I high-level match found? No Ye Medium-term behavior (* ) Ye Long-term behavior (* ) Figure 2. Bloc diagram of -baed pedetrian behavior prediction model. he focu of thi multi-level prediction method i the -baed pedetrian behavior prediction model, which aim to predict pedetrian behavior in the mot appropriate manner baed on the MP_C, MP_I and current trajectorie, through a multiple prediction hierarchy a depicted in Figure 2. If there are available MP_C, the multi-level prediction tart from the high level, otherwie it tart from the middle level. In high-level prediction, when a current trajectory i matched with a MP_C, dimenion of the current trajectory and the MP_C are equalized firt. he proce in the improved method conit of two tage: pre-requiite and imilarity. In pre-requiite, a criterion baed on ditance between the current trajectory and the MP_C i propoed for deciding whether the current trajectory fall into a reaonable imilarity range of the MP_C. In our previou method, thi criterion depend on a manually etting parameter that define a imilarity range of the MP_C. It i improved that the imilarity range of the MP_C can be automatically generated baed on the left boundary and the right boundary of the MP_C. On the other hand, our previou method only conider the ditance between the current trajectory and the MP_C when meauring their imilarity. However, there are cae that the current trajectory may not be very imilar to the MP_C although they are cloe to each other. In the improved method, we propoed a new imilarity tage which i performed baed on a criterion that conider changing calculation and comparion. Current trajectorie are performed prerequiite firt, and thoe that have matched MP_C can be performed imilarity. If the current trajectory can further find a matched MP_C, then it ha a long-term predicted behavior baed on the matched MP_C, otherwie it i paed to middle-level prediction for predicting a medium-term 250

4 behavior. Middle-level prediction follow a imilar algorithmic flow a high-level prediction, while the difference i that MP_I are ued for in middle-level prediction intead of MP_C. If the prediction for a current trajectory fail in both high level and middle level, low-level prediction i performed in which ingle-tep forecating will be done by uing an Auto- Regreive (AR) model [13]. Let denote the obervable trajectory of the pedetrian PD, and P_I m and P_C n repreent the m th MP_I and the n th MP_C repectively. i given by {, v, φ } in which, v and φ repreent the pedetrian trajectory in patial, velocity and heading feature pace, repectively. P_I m and P_C n are given by {P_I m, P_I v m, P_I φ m } and {P_C n, P_C v n, P_C φ n } which imilarly repreent the MP_I and MP_C in three feature pace. Let * denote the predicted behavior of PD in any future motion. * i alo given by {*, * v, * φ } for repreenting the predicted behavior in three feature pace. If i defined up to t, then * i defined from t+1 onward. For illutration convenience, we chooe the patial feature a an example for preenting the multi-level prediction proce. hu, P_I m, P_C n and * in thi cae are all implified into {, Ø, Ø}, {P_I m, Ø, Ø}, {P_C n, Ø, Ø} and {*, Ø, Ø}, repectively, in which =t (n 1,n 2 )={r [n]} and * =t* (n 3,n 4 )={r* [n]}. LB i MV i RB i rajectorie in i th cluter Figure 3. Decription of MP. for all MP_C and MP_I [11]. It i more reaonable that the imilarity range of MP_C/MP_I can be automatically obtained baed on the group of trajectorie that generate the MP_C/MP_I. In the improved method, P_I m and P_C n are decribed by {LB_I m, MV_I m, RB_I m } and {LB_C n, MV_C n, RB_C n }, repectively, a depicted in Figure 3. Let P_C n be an example, beide uing the mean vector MV_C n =mv_c n (n 1,n 2 )={r (mv_c)n [n]} in our previou method, we further conider the deviation between the trajectorie in the cluter and the mean vector. In term of the moving direction of MV_C n, the left boundary LB_C n =lb_c n (n 1,n 2 )={r (lb_c)n [n]} and the right boundary RB_C n =rb_c n (n 1,n 2 )={r (rb_c)n [n]} repreent maximal deviated ditance in the left and the right ide of MV_C n, repectively. he following ub-ection will decribe the focu of our improved method in detail. B. Dimenion Equalization Since the current trajectorie and MP_C/MP_I conit of patial of different number of time tep, before i performed, their dimenion need to be equalized. o do that, we firt egment P_C n or P_I m into portion which have the ame data dimenion with. For example, if ha time tep, and P_C n ha K n P time tep (K n P > ). We elect the portion on P_C n which ha the mallet Euclidean ditance to a the repreentative of the whole P_C n a depicted in Figure 4. he repreentative portion of P_C n i denoted by P_C n(rp). P_C n(rp) i imilarly decribed by {LB_C n(rp), MV_C n(rp), RB_C n(rp) }, which i given a: MV_C n(rp) = mv_c n (Q+1, Q+ ), LB_C n(rp) = lb_c n (Q+1, Q+ ) 1 Q K n P -. (1) RB_C n(rp) = rb_c n (Q+1, Q+ ), C. Pre-requiite Matching In prerequiite, our concern i that whether a current trajectory fall into a reaonable imilarity range of a MP_C/MP_I. So a criterion i propoed baed on the ditance between a current trajectory and a MP_C/MP_I. he ditance function D(, P_C n(rp) ) between and P_C n(rp) i defined a: LB_C n MV_C n K n P RB_C n i D(, P_ Cn( rp) ) = d( r [ i], r( mv _ c) [ Q + i]), (2) n i= 1 H LB_C n(rp) { } Q+ Q } RB_C n(rp) MV_C n(rp) Figure 4. Dimenion equalization. In our previou method, P_I m and P_C n are only repreented by the mean vector of m th MP_I and n th MP_C cluter, repectively. When MP_C and MP_I are ued for with trajectorie, a global parameter i manually et through extenive experimentation to define a imilarity range where d(r [i], r (mv_c)n [Q+i]) refer to the Euclidean ditance between the correponding coordinate pair r [i] and r (mv_c)n [Q+i], and H i H = ( ) i a weight factor for each i= 1 time tep, which mean an older time tep ha le impact when. We regard P_Cn(rp) a a Gauian ditribution model where the Mean locate at MV_C n(rp). From MV_C n(rp) to LB_C n(rp) or RB_C n(rp), a larger ditance of away from MV_C n(rp) mean a le liely. If goe outide of LB_C n(rp) or RB_C n(rp), the fail a depicted in Figure 5. So the criterion for a ucceful between and P_C n(rp) i given a: i 251

5 i D(, P_ Cn( rp) ) DtMax( i) i= 1 H. (3) In (3), DtMax(i) define the larget acceptable ditance at each time tep by chooing the larger from the ditance between LB_C n(rp) and MV_C n(rp), and the ditance between RB_C n(rp) and MV_C n(rp), which i given a: DtMax(i) = Max{d(r (lb_c)n [Q+i], r (mv_c)n [Q+i]), d(r (rb_c)n [Q+i], r (mv_c)n [Q+i])}. (4) If atifie (3), it i paed to imilarity for further with P_C n, which i called a candidate MP_C after prerequiite. Otherwie, it i paed to middle-level prediction for predicting a medium-term behavior. LB_C n MV_C n RB_C n For the current trajectory, if more than one candidate MP_C/MP_I atifie (5), the candidate MP_C/MP_I which ha the mallet δ i choen for generating the predicted behavior for ince it ha the leat change in direction. For a current trajectory which could find a matched MP_C/MP_I in high-level/middle-level prediction, a longterm/medium-term predicted behavior i obtained baed on the correponding matched MP_C/MP_I. For example, if ha a matched MP_C P_C n, it long-term predicted behavior can be repreented a follow when only conidering patial feature: * ={*, Ø, Ø}={r* [n]}, +1 n +K n P -S, (7) where S repreent the time tep of P_C n which i cloet to the time tep of for performing prediction, and r* [n] repreent the predicted patial of * at each time tep after, which i defined a: r* [n] = r (mv_c)n [S+n- ] + (r [ ] - r (mv_c)n [S]). (8) Figure 5. Failed pre-requiite. D. Similarity Matching In imilarity, we conider the changing of the current trajectory at the time tep which the prediction i performed for further meauring the imilarity between the current trajectory and the candidate MP_C/MP_I. It i believed that a maller changing mean higher imilarity between the current trajectory and the candidate MP_C/MP_I ince there i le change in moving direction. he criterion for a qualified imilarity between and P_C n(rp) i given a: he medium-term predicted behavior of a current trajectory could be generated in the imilar way baed on the matched MP_I. E. Single-tep Forecating In low-level prediction, a ingle time tep action i predicted a the motion trategy. he next poition at time tep t+1 can be predicted by the following equation: w ( t + 1) = w( t) + v( t) + Ba( t) 2, (9) where w(t) mean the poition at time tep t, and v(t) and a(t) are correponding velocity value and acceleration value. i the durative time which a ingle time tep repreent. B i timedependent and i updated by the adaptive algorithm in [13]. IV. EXPERIMEN δ K δ δ, (5) d where δ i the changing of at the time tep which the prediction i performed, δ i the average of all changing of at hitorical time tep and δ d i the larget deviated when comparing all changing at hitorical time tep with δ, which i given a: ( δ i δ ), 3 i K 1 δ = arg. (6) d max Figure 6. he cenario of the real experiment. 252

6 In thi ection, we demontrate how the improved method wor in a dynamically changing real-world environment. he cenario of the experiment i baed on people waling in a hopping mall a hown in Figure 6. A fixed-bacground video for thi cenario wa taen over 10 minute. From the video recording, a total of 326 obervable pedetrian trajectorie were accordingly derived. We alo ue patial feature a an illutration in thi experiment. In thi cae, a continuou pedetrian trajectory i generally repreented by a erie of dicrete poition which are recorded at the ampling time of =1, which i a flexible parameter that can be changed depending on how trajectorie are extracted from the raw video data. Figure 9 depict the multi-level prediction reult among the remaining 26 trajectorie. In Figure 9(a) and (b), red olid-line and blue broen-line are ued for repreenting MP_C and MP_I, repectively, and correponding predicted long-term and medium-term behavior are hown by green-line. Blac-line repreent the actual behavior of the pedetrian for comparion and blac-circle label the time tep that the prediction wa performed. Figure 9(c) depict a ingle time tep action predicted at the low level by green-line, and blac-circle alo label the time tep that the prediction wa made. In order to evaluate the performance of the improved method, we compare the predicted behavior of each trajectory with the correponding actual behavior for analyzing the error of the improved prediction method. For the predicted behavior * of each pedetrian PD at the time tep t, prediction error e (t) i computed a: e () t e D() t =. (10) L where D e (t) i the deviated ditance between the predicted behavior and the actual behavior after time tep t, and L i the actual total travered ditance between the origin-detination pair of the pedetrian PD. In order to wor out a more accurate prediction error for each pedetrian, we calculate a erie of e (t), to generate a global prediction error ε of the pedetrian PD at all poible time tep t when a prediction can be performed. he calculation of ε i performed a: Figure 7. Obervable pedetrian trajectorie. Figure 8. Clutered MP. Out of all the obervable pedetrian trajectorie, we randomly elect 300 trajectorie for MP clutering and leave the remaining 26 trajectorie for behavior prediction. Figure 7 depict the elected 300 obervable pedetrian trajectorie in which red-line and green-line repreent double-directional trajectorie between each pair of entrance, repectively. here are altogether 20 MP which are clutered and the reult are hown in Figure 8, in which the arrow are ued for differentiating moving direction. By paing all 20 clutered MP to the MP claification, 5 of them are claified a MP_C a depicted by red olid-line, and the other 15 MP are claified a MP_I which are repreented by blue broen-line. ε = n 1 e() t t= 3. (11) N 3 where N i the total number of time tep of the pedetrian trajectory from the origin to the detination. For all the 26 trajectorie for prediction, we compare the improved multilevel prediction (IMP) method with our previou multi-level prediction (PMP) method and the recurive low-level prediction (RLP) method. For long-term or medium-term behavior predicted at high level or middle level from the IMP method, the RLP method alo generate the behavior with the ame number of future time tep by recurively predicting a ingle action in the next time tep. Figure 10 depict the calculated prediction error of all 26 trajectorie generated by the IMP method, the PMP method and the RLP method, repectively. It can be een that (1) the IMP method improved the prediction accuracy compared with PMP method; (2) the IMP method ha an obviouly better performance than the RLP method in mot teting cae. In a minority of teting cae which have very well-defined trajectorie, the IMP method i lightly wore than the RLP method. hi i to be expected a RLP method wor well with well-defined trajectorie, and could fail diatrouly when the trajectory change direction frequently. Furthermore, we alo compared the proceing time of IMP method and PMP method, and it i concluded baed on a very minor difference that the improved prediction accuracy of IMP method i not at the price of lower computational efficiency compared with PMP method. 253

7 (a) (b) (c) Figure 9. Multi-level prediction reult. he wor decribed in thi paper wa fully upported by a grant from the Reearch Grant Council of the Hong Kong SAR, China (Project No. HKU7196/06E). Figure 10. Prediction error comparion between the IMP method, PMP method and the RLP method. V. CONCLUSION In thi paper, we preented an improved multi-level behavior prediction method baed on a new algorithm with claified motion pattern. Baed on our previou multi-level behavior prediction framewor, the improved method propoed in thi paper concentrated on improving the imilarity criteria between the trajectory and the clutered MP for pedetrian behavior prediction. he new criteria are uperior in two area: (1) a reaonable imilarity range of MP i automatically calculated intead of manually et; (2) the ditance feature and the changing feature are conidered together for imilarity while only the ditance feature i conidered before. From the real-world experimental reult, it can be concluded that the improved method generate more accurate predicted trajectorie than our previou method, and it alo ha a better performance in mot teting cae compared with the recurive low-level prediction method. From the improved multi-level behavior prediction method, our future reearch will focu on three apect: (1) to integrate the prediction on patial, velocity and heading ; (2) to invetigate online learning of MP and to improve the accuracy of behavior prediction baed on updated MP; (3) to define behavior pattern baed on learned MP and to analyze pedetrian motion intention baed on their predicted behavior. ACKNOWLEDGMEN REFERENCES [1] M. Peden, R. Scurfield, D. Sleet, D. Mohan, A. A. Hyder, E. Jarawan, and C. Mather, Ed., World report on road traffic injury prevention. Geneva, Switzerland: World Health Organization, [2]. Gandhi and M. M. rivedi, Pedetrian protection ytem: iue urvey, and challenge, IEEE ranaction on Intelligent ranportation Sytem, Vol. 8, No. 3, pp , September [3] J. ani. Model-baed learning for mobile robot navigation from the dynamical ytem perpective, IEEE ranaction on Sytem, Man and Cybernetic - Part B, Vol. 26, No. 3, pp , June [4] Q. Zhu. Hidden Marov model for dynamic pedetrian avoidance of mobile robot navigation, IEEE ranaction on Robotic and Automation, Vol. 7, No. 3, June [5] Ahraf Elnagar. Prediction of moving object in dynamic environment uing Kalman Filter, Proceeding of IEEE International Sympoium on Computational Intelligence in Robotic and Automation. Banff, Alberta, Canada. July 29-Augut 1, [6] E. D. Dicmann, B. Myliwetz and. Chritian. An integrated patio-temporal approach to automatic viual guidance of autonomou vehicle, IEEE ranaction on Sytem, Man and Cybernetic, Vol. 20, No. 6, pp , Nov./Dec [7] A. Charavarthy and D. Ghoe. Pedetrian avoidance in a dynamic environment: A colliion cone approach, IEEE ranaction on Sytem, Man and Cybernetic - Part B, Vol. 28, No. 5, pp , [8] Z. Qu, J. Wang and C. E. Plaited. A new analytical olution to mobile robot trajectory generation in the preence of moving pedetrian, IEEE ranaction on Robotic, Vol. 20, No. 6, pp , December [9] Amalia F. Foa and Pano E. rahania. Predictive autonomou robot navigation, Proceeding of the 2002 IEEE/RSJ International Conference on Intelligent Robot and Sytem. EPFL, Lauanne, Switzerland. October [10] Maren Bennewitz. Mobile robot navigation in dynamic environment, PhD thei. June [11] Z. Chen, D.C.K. Ngai and N.H.C. Yung. Pedetrian Behavior Prediction baed on Motion Pattern for Vehicle-to-Pedetrian Colliion Avoidance, Proceeding of the 2008 IEEE International Conference on Intelligent ranportation Sytem. Beijing, China. October [12] N. H. C. Yung and A. H. S. Lai. Segmentation of color image baed on the gravitational clutering concept, Optical Engineering, Vol. 37, No. 3, March [13] Ye Cang. Behavior-Baed Fuzzy Navigation of Mobile Vehicle in Unnown and Dynamically Changing Environment, PhD thei. September