LARGE-SCALE TRAFFIC STATE ESTIMATION

Transcription

1 Hans van Lint, Yufei Yuan & Friso Scholten A localized deterministic Ensemble Kalman Filter LARGE-SCALE TRAFFIC STATE ESTIMATION

2 CONTENTS Intro: need for large-scale traffic state estimation Some Kalman Filter basics The Ensemble Kalman Filter getting rid of matrix inversions deterministic formulation localisation, Results and Discussions & outlook (big) plans

3 INTRODUCTION Dutch National Data Warehouse Traffic Information Network 2500 km freeways 3500 km prov/urban 24,000 measurement sites Real-time Data Dynamic: Flow, Speed, Occupancy, Travel time (lane / usr class specific), FCD Status: roadworks, incidents, events 150,000 measurements/min Historical database 200 TB Raw data 3

4 INTRODUCTION Dutch National Data Warehouse Traffic Information Ambition Build national traffic observatory (for RT and archived traffic info) From raw data to real information: Intelligent database (search terms related to traffic patterns and explanatory factors) Data-based routable network graphs Densities, space mean speeds Capacities, other FD params MFDs and other agg state vars Inflows, turns, OD flows Opensource tools to dive into all this 4

5 5

6 6

7 7

8 8

9 9

10 Treiber, M. and D. Helbing, Reconstructing the Spatio-Temporal Traffic Dynamics from Stationary Detector Data : p van Lint, J.W.C. and S.P. Hoogendoorn, A Robust and Efficient Method for Fusing Heterogeneous Data from Traffic Sensors on Freeways. CAICE, (8): p Van Lint, J.W.C., Empirical Evaluation of New Robust Travel Time Estimation Algorithms. Transportation Research Record, 2010(2160): p

11 INTRODUCTION Dutch National Data Warehouse Traffic Information Ambition Build national traffic observatory (for RT and archived traffic info) Next step: Scalable efficient model-based traffic state estimation: densities (inflows, turns) TRADE OFF Estimation accuracy vs computational efficiency 11

12 THE KALMAN FILTER For any system cast in discrete state space form the KF is an efficient recursive solution for estimating state vector x; x k+1 = F k x k + G k u k + w k y k = H k x k + v k that is optimal in three ways Minimum error (co)variance Maximal posterior probability Conditional mean estimation given a some (fairly strict) assumptions the validity of which strongly affect the quality of the result E(w k ) = 0, E(w k w l ) = E(v k ) = 0, E(v k v l ) = Q k R k k = l 0 k l k = l 0 k l 12

13 THE KALMAN FILTER Initial conditions (reasonable choices for Q and R must also be made) ˆx 0 0 = ˆx 0,P 0 0 = P 0 Time-propagation (prediction): ˆx k k 1 = F k 1ˆx k 1 k 1 + G k 1 u k 1 T P k k 1 = F k 1 P k 1 k 1 F k 1 + Q k 1 Measurement adaptation (correction): e k = y k H k ˆx k k 1 K k = P k k 1 H k ˆx k k = ˆx k k 1 + K k e k ( H k P k k 1 H k + R ) k P k k = [ I K k H k ]P k k 1 13

14 A CLOSER LOOK AT THE FILTER GAIN sensitivity observation model to changes in the state K k = P k k 1 H k H k P k k 1 H k + R k error covariance of the innovations (i.e. uncertainty in the new information we get from observations). 14

15 A CLOSER LOOK AT THE FILTER GAIN Kalman Gain = uncertainty in (process) model uncertainty in observations Sensitivity observation model So balance between uncertainty in process model and uncertainty in measurements / observation model! 15

16 TRAFFIC FLOW SIMULATION MODELS Are naturally cast in discrete state-space form Conservation of vehicles (+ possible additional pde for speed) Observation model typically fundamental diagram x k+1 = f ( x k,u k ) + w k typically f : ρ k+1 = ρ k + ΔL y k = h k ( x k ) + v k typically h :q k = Q k e ( ) ρ k Δt ( q in k q out k + r s) Q k e 100 ( ) V k e ρ k 50 ( ) = Q e k ( ρ k ) ρ k ρ k

17 TRAFFIC STATE ESTIMATION Extended Kalman filtering Nonlinear extension KF Assumptions Process model can be approximated by 1st order Taylor expansion Obs model can be approximated with 1st order Taylor expansion All other KF assumptions (iid Gaussian WN, etc) x k+1 = f k y k = h k F k = df k dx ( x ) k + w k ( x ) k + v k ( x) H k = dh k dx ( x) x= ˆx k x= ˆx k 17

18 EXTENDED KALMAN FILTER Many challenges linearisation may cause adaptation in the wrong direction Computing derivatives = pain in Scalability: matrix manipulations & inversions expensive (E)KF assumptions on violated all the time ˆx 0 0 = ˆx 0,P 0 0 = P 0 ˆx k k 1 = f ( ˆx k 1 k 1,u ) k T P k k 1 = F k 1 P k 1 k 1 F k 1 e k = y k h( ˆx ) k k 1 K k = P k k 1 H k ˆx k k = ˆx k k 1 + K k e k + Q k 1 ( H k P k k 1 H k + R ) k P k k = [ I K k H k ]P k k 1 18

19 ALTERNATIVE: ENSEMBLE KALMAN FILTER Use ensemble of N state vectors and noisy observation replications Predicting ensemble mean and variance is simple Update step similar to EKF Still requires linearisation? X 0 0 = [x 0 1,...,x 0 N ], X k k 1 = f ( X k 1 k 1,U k ) x k k 1 = E(X k k 1 ) = 1 N T P k k 1 = A k k 1 A k k 1 i=1:n D k = [d k 1,...,d k N ],with d k i E k = D k h X k k 1 K k = P k k 1 X k k = X k k 1 + K k E k i x k k 1 (N 1), with A x = X x E(X x ) = y k + e k i ( ) ( ) H k P k k 1 H k + R k 19

20 ALTERNATIVE: ENSEMBLE KALMAN FILTER Use ensemble of N state vectors and noisy observation replications Predicting ensemble mean and variance is simple ( ) T / P k k K k = 1 Update A N 1 k k 1 HA *, step similar to * with P k k EKF = 1 HA ( HA N 1 )T + R k and [HA Still i i ] = h(x requires linearisation k k 1 ) 1 j h(x N? k k 1 ) j=1:n X 0 0 = [x 0 1,...,x 0 N ] X k k 1 = f ( X k 1 k 1,U k ) x k k 1 = E(X k k 1 ) = 1 N T P k k 1 = A k k 1 A k k 1 i=1:n D k = [d k 1,...,d k N ],with d k i E k = D k h X k k 1 K k = P k k 1 X k k = X k k 1 + K k E k i x k k 1 (N 1), with A x = X x E(X x ) = y k + e k i ( ) ( ) H k P k k 1 H k + R k 20

21 A CLOSER LOOK AT THIS FILTER GAIN distance prior state to mean ensemble state distance predicted observations to mean ensemble prediction K = A ( k k 1 HA ) T k HA( HA) T + R k total error variance (uncertainty) measurement equation 21

22 A CLOSER LOOK AT THIS FILTER GAIN Kalman Gain = uncertainty in model predictions (sign?) uncertainty in observations So again balance between uncertainty in model and uncertainty in measurements! 22

23 EKF VS ENKF Improves upon the wrong sign adaptation problem density change in positive direction of derivative right direction Observation Prior mean state Prior/posterior Ensemble Posterior mean state 23

24 EKF VS ENKF Improves upon the wrong sign adaptation problem density change in positive direction of derivative right direction right direction Observation Prior mean state Prior/posterior Ensemble Posterior mean state 24

25 EKF VS ENKF Improves upon the wrong sign adaptation problem density change in negative direction of derivative right direction Observation Prior mean state Prior/posterior Ensemble Posterior mean state 25

26 EKF VS ENKF Improves upon the wrong sign adaptation problem density change in negative direction of derivative Given proper spread, the sign is (most of the time) correct right direction right direction Observation Prior mean state Prior/posterior Ensemble Posterior mean state 26

27 EKF VS ENKF Improves upon the wrong sign adaptation problem X k k = X k k 1 + K k E k In this case most ensemble members move in the correct direction: so does the ensemble mean K k :negative K k : positive E k :negative E k : positive * with P k k K k = 1 N 1 A k k 1 ( HA) T / P k k = 1 HA ( HA N 1 )T + R k and [HA i i ] = h(x k k 1 ) 1 N j=1:n *, j h(x k k 1 ) K k : positive K k :negative 27

28 ENKF FOR LARGE-SCALE ESTIMATION Three further adaptations Getting rid of expensive matrix inversion Deterministic instead of stochastic resampling of D Localisation of the filter 28

29 ENKF FOR LARGE-SCALE ESTIMATION Three further adaptations Getting rid of expensive matrix inversion Computational cost traditional : O(m 3 + m 2 N +mn 2 + nn 2 ) - cubic in nr measurements: (2 as much 8 times more cost) - quadratic in state variables and ensemble members Computational cost SMW alternative: O(N 3 +mn 2 +nn 2 ) - linear in measurements and state size - quadratic in ensemble members: 8 Sherman-Morrison-Woodbury (SMW) formula, e.g. W.Hager, SIAM Rev., 31(2), K k = 1 A ( HA N 1 k k 1 )T 1 ( HA ( HA N 1 )T + R k ) 1!## #" #### $ R 1 I 1 N 1 (HA) I + 1 (HA)T R 1 N 1 (HA) 1 (HA) T R 1 29

30 ENKF FOR LARGE-SCALE ESTIMATION Three further adaptations Getting rid of expensive matrix inversion Deterministic instead of stochastic resampling of D Correct: ensemble spans about the same (model) subspace as the forecasting errors (else: filter divergence looms ) Stochastic resampling may lead to clustering ensemble D k = [d k 1,...,d k N ],with d k i X k k = X k k 1 + K k = y k + e k i ( ) ( ) D k h X k k 1 x k k = x k k 1 + K k A k k = A k k 1 + K k A ( ) ( ) ( ) d k h x k k 1 D k AH k Deterministic approach manipulates so that innovation / residuals AH ALWAYS spread entire measurement space (with cov R) Sakov, P., & Oke, P. R. (2008). A deterministic formulation of the ensemble kalman filter: an alternative to ensemble square root filters. Tellus A, 60 (2), K k A 30

31 ENKF FOR LARGE-SCALE ESTIMATION Three further adaptations Getting rid of expensive matrix inversion Deterministic instead of stochastic resampling of D Localisation of the filter Further reduces computational costs (chops up Cov matrix => update equations in small bits) Avoids spurious correlations Increases effective ensemble size (and spread!) Note: localisation does minimise effect SMW formulation Different techniques: least efficient (state based) O(m 3 n + m 2 nn + mnn 2 ) most efficient (obs based) O(mnN 2 + nn) 31

32 LOCALISATION INCREASES EFFECTIVE ENSEMBLE SIZE nr.

33 LOCALISATION INCREASES EFFECTIVE ENSEMBLE SIZE nr.

34 LOCALISATION INCREASES EFFECTIVE ENSEMBLE SIZE nr.

35 LOCALISATION INCREASES EFFECTIVE ENSEMBLE SIZE nr.

36 LOCALISATION INCREASES EFFECTIVE ENSEMBLE SIZE nr.

37 LOCALISATION INCREASES EFFECTIVE ENSEMBLE SIZE Note: Localisation may induce jumps in the state, which may not be realistic nr.

38 SIMULATION NETWORK 264 km 4656 cells 2 seconds time step 592 (dual) loop detectors 38

39 EXPERIMENTAL SETUP 39

40 EXPERIMENTAL SETUP 40

41 EXPERIMENTAL SETUP 41

42 EXPERIMENTAL SETUP 42

43 RESULTS Ensemble size:10, 20, 30, 40, 50 respectively 43

44 RESULTS 44

45 CONCLUSIONS Localized D overall winner in this study Faster than Localised EKF on similar case More accurate and faster than all other alternatives No need (on the contrary) for SWM reformulation Localization of is absolutely crucial: Higher estimation accuracy (larger effective ensemble) Lower computational burden Deterministic sampling beneficial outperforms stochastic slightly more robust wrt smaller ensemble sizes 45

46 NEXT STEPS Dutch National Data Warehouse Traffic Information In the coming few years we are going to work on implementing and further developing these ideas on real-data Many additional puzzles: Network graph updating Parameter estimation (dual filtering?) Inflows / turns / ODs / Paths Urban / provincial (different data, different state variables) 46