Learning Air Traffic Controller Workload. from Past Sector Operations

Transcription

1 Learning Air Traffic Controller Workload from Past Sector Operations David Gianazza 1

2 Outline Introduction Background Model and methods Results Conclusion 2

3 ATC complexity Introduction Simple Complex 3

4 ATC complexity Introduction Simple Complex Workload & Airspace configuration 3

5 Questions related to ATC complexity and workload Can we predict the workload of ATC controllers, given a traffic situation as input? How to measure the ATC complexity of a traffic situation? How do we quantify workload? Can we build a predictive model of the workload, taking ATC complexity into account? What model should we choose? Many answers in the literature [Mogford et al., 1995, Hilburn, 2004, Delahaye and Puechmorel, 2000, Delahaye and Puechmorel, 2010, Flynn et al., 2005, Welch et al., 2007, Kopardekar and Magyarits, 2003, Chatterji and Sridhar, 1999], etc. 4

6 Focus of this publication Compare several machine learning methods Linear discriminant analysis (LDA), Quadratic discriminant analysis (QDA), Naive Bayes classifier (NBayes), Neural networks (NNet), Gradient boosted trees (GBM) On the workload prediction problem workload = f (complexity) complexity = g(traffic, sector, procedures) 5

7 Focus of this publication ATC complexity metrics taken from the literature Workload model is learned from past sector operations 6

8 Outline Introduction Background Model and methods Results Conclusion 7

9 ATC complexity Simple Complexe Literature reviews on ATC complexity factors R. Mogford, J. A. Guttman, S. L. Morrow, and P. Kopardekar, The complexity construct in air traffic control : A review and synthesis of the literature, tech. rep., FAA Technical Center : Atlantic City, B. Hilburn, Cognitive complexity in air traffic control, a litterature review, tech. rep., Eurocontrol experimental centre, E.g. : density, incoming flows, climbing/descending a/c, dispersion of speed direction/intensity, vertical velocity variance, number of potential conflicts, sector volume, number of crossing points, etc. 8

10 Research on ATC complexity Intrinsic complexity Delahaye, D. and Puechmorel, S. (2000). Air traffic complexity : towards intrinsic metrics. In Proceedings of the third USA/Europe Air Traffic Management R & D Seminar. Delahaye, D. and Puechmorel, S. (2010). Air traffic complexity based on dynamical systems. In Decision and Control (CDC), th IEEE Conference on, pages IEEE. Analogy with the entropy in Physics Air traffic = dynamic system (vector field + time) Complexity = system entropy Not related with workload 9

11 Cognitive approach of workload Flynn, G., Benkouar, A., and Christien, R. (2005). Adaptation of workload model by optimisation algorithms. Technical report, Eurocontrol. Welch, J. D., Andrews, J. W., Martin, B. D., and Sridhar, B. (2007). Macroscopic workload model for estimating en route sector capacity. In Proc. of 7th USA/Europe ATM Research and Development Seminar, Barcelona, Spain. Note : bibliographies are not exhaustive Mental/physical workload duration of ATC tasks G = τ j time required to execute task j λ j occurence ratio of task j J τ j λ j j=1 Complexity-dependent model parameters are weighed so that modeled workload threshold = declared sector capacity or modeled workload threshold = peak traffic count 10

12 Statistical models relating workload to complexity Collected data : Measures of complexity metrics extracted from radar data and sector geometry E.g. : density, incoming flows, climbing/descending a/c, dispersion of speed direction/intensity, vertical velocity variance, number of potential conflicts, sector volume, number of crossing points, etc. Workload measures : EEG, physiological parameters, pupil diameter, eye fixation points, duration and content of radio exchanges, subjectives ratings, etc. Assumption : workload = f (complexity) True relationship y = f (x) is unknown Approximation y = h(x) + ε h is tuned on a set of examples (x 1, y 1 ),..., (x N, y N ) 11

13 Statistical models relating workload to complexity Kopardekar, P. and Magyarits, S. (2003). Measurement and prediction of dynamic density. In Proceedings of the 5th USA/Europe Air Traffic Management R & D Seminar. Chatterji, G. B. and Sridhar, B. (1999). Neural network based air traffic controller workload prediction. In American Control Conference, Proceedings of the 1999, volume 4, pages IEEE. y = h(x) + ε y (response variable) : subjective workload ratings collected from ATC controllers x (explanatory variables) : measures of ATC complexity Different choices for model h : Linear model [Kopardekar and Magyarits, 2003] Neural network [Chatterji and Sridhar, 1999] 12

14 Outline Introduction Background Model and methods Results Conclusion 13

15 Our approach Basic assumption : Decisions to split or merge ATC sectors are caused by workload variations. Workload measure ATC sector status Low workload when sector s is collapsed with other sectors Normal workload when the sector s is operated as is High workload when s is split into several smaller sectors operated on different working positions 14

16 Supervised learning Our approach Unknown relationship y = f (x) between workload y and ATC complexity metrics x approximated by ŷ = h(x) using examples (x 1, y 1 ),..., (x N, y N ) Regression or classification problems Regression problem : y R or y R p as in [Kopardekar and Magyarits, 2003, Chatterji and Sridhar, 1999] Classification problem : y {C 1,... C P } is a category 15

17 Supervised learning Our approach Unknown relationship y = f (x) between workload y and ATC complexity metrics x approximated by ŷ = h(x) using examples (x 1, y 1 ),..., (x N, y N ) Regression or classification problems Regression problem : y R or y R p as in [Kopardekar and Magyarits, 2003, Chatterji and Sridhar, 1999] Classification problem : y {C 1,... C P } is a category Here, the output is a category : low, normal or high workload. 15

18 Output variable y Output can be modeled as posterior probabilities : ŷ 1 P(C 1 x) ŷ =. =. P(C P x) Non-overlapping classes : ŷ P ŷ p [0, 1], p {1,..., P} P P ŷ p = P (C p x) = 1 p=1 p=1 (1) 16

19 Output variable y Examples extracted from past sector operations : Collapsed sector y = (1, 0, 0) T Underload Opened sector y = (0, 1, 0) T Normal workload Split sector y = (0, 0, 1) T Overload Other Not used unknown workload Advantages, compared to other workload measures Historical data is easy to collect More than 510 ATC sectors for the 5 french ATCCs Not subjective Drawback Only three levels of workload 17

20 Output variable y Examples extracted from past sector operations : Collapsed sector y = (1, 0, 0) T Underload Opened sector y = (0, 1, 0) T Normal workload Split sector y = (0, 0, 1) T Overload Other Not used unknown workload Advantages, compared to other workload measures Historical data is easy to collect More than 510 ATC sectors for the 5 french ATCCs Not subjective Drawback Only three levels of workload 17

21 Input variables? What are the most relevant ATC complexity metrics for our problem? Previous research : D. Gianazza and K. Guittet. Evaluation of air traffic complexity metrics using neural networks and sector status. In Proceedings of the 2nd International Conference on Research in Air Transportation. ICRAT, D. Gianazza and K. Guittet. Selection and evaluation of air traffic complexity metrics. In Proceedings of the 25th Digital Avionics Systems Conference. DASC, D. Gianazza. Smoothed traffic complexity metrics for airspace configuration schedules. In Proceedings of the 3nd International Conference on Research in Air Transportation. ICRAT,

22 Input variables Among 27 ATC complexity metrics from the literature, we found 6 most relevant variables for our problem : vol, the airspace volume nb, the number of aircraft flow15, the incoming traffic flow within the next 15 minutes flow60, the incoming traffic flow within a 1 hour time horizon avg vs, the average absolute vertical speed of the aircraft within the sector, inter hori, the number of speed vector intersections with an angle greater than 20 degrees. Computed from radar tracks and sector geometries 19

23 Methods Objective : compare several classification methods on the workload prediction problem LDA : Linear discriminant analysis QDA : Quadratic discriminant analysis NBayes : Naive Bayes classifier NNet : Neural networks for classification GBM : Gradient-boosted classification trees 20

24 Dataset Historical data from 5 french ATC centers (2 weeks, Oct. 2016) : Radar tracks Past sector operations Data split into two datasets : Training set (Oct. 13 th to 19 th ) examples of workload and complexity measurements, from 511 different ATC sectors Test set (Oct. 20 th to 26 th ) examples, from 513 ATC sectors The three classes (low, normal, high) are equally represented 21

25 Hyperparameter tuning Some methods have hyperparameters that need to be selected E.g. : number of hidden units in a neural network 10-fold cross-validation, performed on the training set S Model selection: For each candidate model: For each fold : Adjust model using Assess error using Compute the cumulated error for the K folds Select model minimizing the error on the K validation subsets Prediction: Adjust the best model found Apply to fresh entries 22

26 Hyperparameter tuning 10-fold cross-validation to select hyperparameter values Method Hyperparameter grid LDA - QDA - NBayes - N NNet h = {15, 20, 25} (n,λ) λ = {10,1,1e-1,1e-2,1e-3,1e-5,0} m = {5000, 6000, 7000} GBM (m,j,ν) J = {2, 3, 4} ν = {1e-4, 5e-4, 1e-3,1e-2,1e-1} 23

27 Outline Introduction Background Model and methods Results Conclusion 24

28 Results Correct classification rates Method Overall Low Normal High LDA QDA NBayes NNet GBM

29 Why are correct classification rates not closer to 100%? nb flux15 flux avg.chgt_niv avg.inter_hori Sector N high low normal 26

30 Outline Introduction Background Model and methods Results Conclusion 27

31 Conclusion Nearly 82% of correct classification rates Neural nets (NNet) and gradient-boosted trees (GBM) performed best Poor performance of linear models 28

32 Have we answered our questions? Can we predict the workload of ATC controllers, given a traffic situation as input? How to measure the ATC complexity of a traffic situation? Combination of several factors. No single metric. How do we quantify workload? Categories (C 1, C 2, C 3 ) for low, normal and high workload. Examples are extracted from past sector operations, considering sector status (collapsed, opened, split). Can we build a predictive model of the workload, taking ATC complexity into account? Posterior probabilities P(C p x) in NNet, or classification trees in GBM. Models are learned from historical data What model should we choose? NNet and GBM perform better than the other tested models 29

33 Have we answered our questions? Can we predict the workload of ATC controllers, given a traffic situation as input? How to measure the ATC complexity of a traffic situation? Combination of several factors. No single metric. How do we quantify workload? Categories (C 1, C 2, C 3 ) for low, normal and high workload. Examples are extracted from past sector operations, considering sector status (collapsed, opened, split). Can we build a predictive model of the workload, taking ATC complexity into account? Posterior probabilities P(C p x) in NNet, or classification trees in GBM. Models are learned from historical data What model should we choose? NNet and GBM perform better than the other tested models 29

34 Have we answered our questions? Can we predict the workload of ATC controllers, given a traffic situation as input? How to measure the ATC complexity of a traffic situation? Combination of several factors. No single metric. How do we quantify workload? Categories (C 1, C 2, C 3 ) for low, normal and high workload. Examples are extracted from past sector operations, considering sector status (collapsed, opened, split). Can we build a predictive model of the workload, taking ATC complexity into account? Posterior probabilities P(C p x) in NNet, or classification trees in GBM. Models are learned from historical data What model should we choose? NNet and GBM perform better than the other tested models 29

35 Have we answered our questions? Can we predict the workload of ATC controllers, given a traffic situation as input? How to measure the ATC complexity of a traffic situation? Combination of several factors. No single metric. How do we quantify workload? Categories (C 1, C 2, C 3 ) for low, normal and high workload. Examples are extracted from past sector operations, considering sector status (collapsed, opened, split). Can we build a predictive model of the workload, taking ATC complexity into account? Posterior probabilities P(C p x) in NNet, or classification trees in GBM. Models are learned from historical data What model should we choose? NNet and GBM perform better than the other tested models 29

36 Have we answered our questions? Can we predict the workload of ATC controllers, given a traffic situation as input? How to measure the ATC complexity of a traffic situation? Combination of several factors. No single metric. How do we quantify workload? Categories (C 1, C 2, C 3 ) for low, normal and high workload. Examples are extracted from past sector operations, considering sector status (collapsed, opened, split). Can we build a predictive model of the workload, taking ATC complexity into account? Posterior probabilities P(C p x) in NNet, or classification trees in GBM. Models are learned from historical data What model should we choose? NNet and GBM perform better than the other tested models 29

37 Improved workload prediction Potential benefits Predict optimal sector configurations Anticipate overloads What-if evaluation of envisionned ATFCM measures 30

38 Further works Model performance on elementary sectors? Effects of seasonality? Use larger datasets (1 year of traffic and sector operations) 31

39 32

40 Linear and quadratic discriminant analysis Model : p(x, C k ) = p(x C k ) P(C k ) Assumptions P(C k ) = π k { p(x C k ) = (2π) P 2 Σ 1 2 exp 1 } 2 (x µ k) T Σ 1 k (x µ k) Parameters π k, µ k and Σ k approximated by maximum-likelihood estimates 33

41 LDA, QDA, and naive bayes (NBayes) ŷ 1 ŷ =. ŷ P = Bayes theorem to compute P(C j x) P(C j x) = p(x C j) P(C j ) p(x) Gaussian assumption for p(x C k ) P(C 1 x). P(C P x) = p(x C j) P(C j ) k p(x C k) P(C k ) LDA : Same covariance matrix Σ for all classes QDA : One covariance matrix Σ k for each class NBayes : Conditional independance of explanatory variables D p(x C j ) = d=1 p(x d C j ) when x = (x 1,..., x D ) T 34

42 Neural networks for classification Hidden layer Output layer Each hidden unit j computes z j = ϕ(a j ) where a j is a weighted sum of the inputs 1, x 0,..., x D ϕ(a j ) is the sigmoid function 1 1+e a j or the hyperbolic tangent function tanh 35

43 Neural networks for classification Hidden layer Output layer Output : ŷ 1 ŷ =. ŷ P = P(C 1 x). P(C P x) 35

44 Neural networks for classification Output interpreted as probabilities ŷ p [0, 1], p {1,..., P} P P ŷ p = P (C p x) = 1 p=1 p=1 Take ŷ = γ(a) where γ is the softmax function : exp{a 1 } a P 1 p=1 exp{ap} γ :. a P. exp{a P } P p=1 exp{ap} where each a p is a weighted sum of the outputs of the hidden layer 36

45 Training NNet on examples S T = {(x 1, y 1 ),..., (x N, y N )} Adjust weights so as to minimize error between the computed outputs ŷ 1,..., ŷ N and the target values y 1,..., y N Maximum-likelihood principle minimize cross-entropy : E(w) = N P n=1 p=1 y np ln ŷnp(w) y np Maximum posterior cross-entropy with weight decay E λ (w) = N P n=1 p=1 y np ln ŷnp(w) y np + λ D j=1 w 2 j 37

46 Classification and regression trees E l ement s of St at i st i cal L ear ning ( 2nd E d.) c H ast i e, T i bshirani & F r i edman 2009 C hap 9 X 1 t 1 X 2 t 2 X 1 t 3 X 2 t 4 R 1 R 2 R 3 R 4 R 5 X 2 X 1 T h T (x) = c j (x) 1 Rj (x) (2) j=1 Regression with quadratic loss : c j (x) = 1 N j x n R j y n Classification : c j (x) = C p where C p is the most occuring class in R j 38

47 Gradient-boosted trees Sum of tree models, iteratively adjusted. Model update : h m : x h m 1 (x) + ν R j T m γ mj 1 Rj (x) Parameters : J : tree size Usually, 4 J 8 M : number of boosting iterations Can be selected by an early stopping procedure ν : shrinkage parameter Usually, ν

48 40