Proposed Performance Metrics Block Time & Predictability

Transcription

1 Proposed Performance Metrics Block Time & Predictability Marc Rose, MCR/SETA-II Gabriela Rohlck, MCR/SETA-II 15 March, 2006 Presented at NEXTOR - Asilomar F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 1

2 Objective An evaluation of two specific metrics: Scheduled Block Time and Predictability A general discussion of types of metrics This work has been sponsored by two ATO Planning offices: 1) The Office of Strategy 2) The Office of Performance Analysis F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 2

3 Three Primary Types of Metrics System Activity or Workload Number of flights per month, overall average block time Provides an estimate of workload, generally NOT performance Impacted by non-performance based changes Avg Flight length - more longer flights, not performance related Performance Generally a system activity metric in constrained times Normalized block time, arrival delay (sometimes) Diagnostic Needed to understand changes in performance F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 3

4 Three Primary Types of Metrics ALL metrics have flaws/limitations: Identification of these limitations is vital to understanding and proper usage F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 4

5 Data and Definition Data Sources Airline Service Quality Performance (ASQP) Data available from 1/1995 to 12/2005 Aviation System Performance Metrics (ASPM) Data available from 1/2000 to 12/2005 Official Airline Guide (OAG) Data available from 1/1982 to 9/2005 Definition: Block Time: Airborne+Taxi-Out+Taxi-In (Gate-Out to Gate-in) Total Flight Time: Block Time + Departure Delay Or: Actual arrival (in) - Scheduled Departure Scheduled: Airline published departure/arrival times Based on OAG or ASPM data (pre departure) Actual: departure/arrival times as flown (post departure) Based on ASQP or ASPM data F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 5

6 Scheduled Block Time A Performance Metric (a lagging indicator) Metric based on OEP to OEP flights A current FAA Executive Council decision Considered the NAS Level (All* flights) and the OEP-OEP level (only flights to/from an OEP 35 airport) Used ASPM data, limited to those flights that have OAG data OAG= Y flag Does not include cancelled flights Two metrics proposed 1) Normalized average block time Normalization consists of identical set of O-D pairs weighted using the average number of flights/month across all datasets 2) Delta metric - A relative change from the overall average *Not truly ALL flights as will be seen in the development F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 6

7 Normalization of Block Time Mathematical Details Two intermediate values are calculated 1) the mean time µ 2) the average number of flights xijkl µ nˆ µ ijk ij k = = N l k = = 1 i, j n n N µ i, j ijk ijk ijk nˆ nˆ ij ij Where x is the block time for flight l for O-D pair ij within month k and n is the number of flights for O-D pair Within all (majority*) of data sets Combining into a normalized NAS level Performance measure: nˆij Where N is the number of months of data Where ij is an O-D pair, and k is month/year F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 7

8 Normalized Scheduled Block Time (OEP to OEP) Moving average (ttm) Jan-00 Apr Jul-00 Oct-00 Jan-01 Apr-01 Jul-01 Oct-01 Jan-02 Apr-02 Jul-02 Oct-02 Jan-03 Apr-03 Jul-03 Oct-03 Jan-04 Apr-04 Jul-04 Oct-04 Jan-05 Apr-05 Jul-05 Oct-05 Source: ASPM F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 8

9 Normalized Scheduled Block Time (NAS Level) Moving average (ttm) Easy to see missing data 80 Jan-82 Jan-83 Jan-84 Jan-85 Jan-86 Jan-87 Jan-88 Jan-89 Jan-90 Jan-91 Jan-92 Jan-93 Jan-94 Jan-95 Jan-96 Jan-97 Jan-98 Jan-99 Jan-00 Jan-01 Jan-02 Jan-03 Jan-04 Jan-05 Normalized Average Block Time (min) Source: OAG F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 9

10 Scheduled Block Times A Lagging Metric Scheduled Actual Scheduled Block Time (ttm) Total Flight Time (ttm) Actual Block Time (ttm) 0.99 Relative Changing Blocktime Flight Times (min) Dec-95 Apr-96 Aug-96 Dec-96 Apr-97 Aug-97 Dec-97 Apr-98 Aug-98 Dec-98 Apr-99 Aug-99 Dec-99 Apr-00 Aug-00 Dec-00 Apr-01 Aug-01 Dec-01 Apr-02 Aug-02 Dec-02 Apr-03 Aug-03 Dec-03 Apr-04 Aug-04 Dec-04 Apr-05 Source: ASPM, OAG and ASQP F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 10

11 Delta Metric Another Approach µ ijk n ijk l= = 1 N n µ x ijk ijkl n Where x is the block time for flight l and O-D pair ij within month k and n is the number of flights ijk ijk k = 1 ˆ µ ij = A multi-month average block time nijk k δ k = ij ( µ ˆ µ )n ijk ij n ijk ij ijk Which is a relative performance metric for the block time that includes ALL flights F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 11

12 Scheduled Block Time Delta Metric (OEP-OEP) Minutes Jan-00 Jul-00 Jan-01 Jul-01 Jan-02 Jul-02 Jan-03 Jul-03 Jan-04 Jul-04 Jan-05 Jul-05 Data Source: ASPM F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 12

13 Scheduled Block Time Delta Metric (NAS Level) Minutes Jan-00 Apr-00 Jul-00 Oct-00 Jan-01 Apr-01 Jul-01 Oct-01 Jan-02 Apr-02 Jul-02 Oct-02 Jan-03 Apr-03 Jul-03 Oct-03 Jan-04 Apr-04 Jul-04 Oct-04 Jan-05 Apr-05 Jul-05 Oct-05 Data Source: ASPM F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 13

14 Historical Scheduled Block Time Delta Metric (NAS Level) Moving average (ttm) -6-8 Jan-82 Jan-83 Jan-84 Jan-85 Jan-86 Jan-87 Jan-88 Jan-89 Jan-90 Jan-91 Jan-92 Jan-93 Jan-94 Jan-95 Jan-96 Jan-97 Jan-98 Jan-99 Jan-00 Jan-01 Jan-02 Jan-03 Jan-04 Jan-05 Block Time Changes (min) Data Source: OAG F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 14

15 Scheduled Block Time Changes Accounting for Aircraft Type Jan-82 Jan-83 Jan-84 Jan-85 Jan-86 Jan-87 Jan-88 Jan-89 Jan-90 Jan-91 Jan-92 Jan-93 Jan-94 Jan-95 Jan-96 Jan-97 Jan-98 Jan-99 Jan-00 Jan-01 Jan-02 Jan-03 Jan-04 Jan-05 Data Source: OAG F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 15

16 Total Flight Time Variability a Predictability Metric? F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 16

17 Percentiles as a Predictability Metric Start at origin-destination level: How does measured block time of airport-pair (O- D) differ from average time where: M = µ k x ij ij x µ ij ij is measured time for O-D pair (i, j) is mean time for O-D pair (i, j) Flight-level M k is generated for all flights and the Selected percentiles are output as a NAS level metric F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 17

18 NAS Predictability Performance Metric (Total Flight Time includes gate delay) Predictability Metric (min) Source: ASQP F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 18

19 Predictability Metric OEP Arrival Airports Sep/ ATL BOS BWI CLE CLT CVG DCA DEN DFW DTW EWR FLL HNL IAD IAH JFK LAS LAX LGA MCO MDW MEM MIA MSP ORD PDX PHL PHX PIT SAN SEA SFO SLC STL TPA F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 19

20 Observations Using 10 th, 90 th percentiles Eliminate significant outliers Analysis shows distribution right-skewed. Average closer to lower bound than upper bound. Metric defined as difference between lower, upper percentiles. Can be calculated with one month of data. Seasonality Does not require historical data. This Metric partially accounts for known seasonality (e.g., winter headwinds) by using current months averages rather than historical F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 20

21 Backup F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 21

22 Metric with Normalized Average Flight Time Average Total Flight Time (min) Metric (min) 0 0 Jul-05 Jan-05 Jul-04 Jan-04 Jul-03 Jan-03 Jul-02 Jan-02 Jul-01 Jan-01 Jul-00 Jan-00 Jul-99 Jan-99 Jul-98 Jan-98 Jul-97 Jan-97 Jul-96 Jan-96 Jul-95 Jan-95 F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 22

23 Actual Block Time Variability: Percentiles and Weighted Average (10 th and 90 th percentiles) Block time (min) F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 23

24 Predictability Metric Calculation - Example Origin Destination Flight Time Avg Flt tm Delta Sorted Delta A B A B A B A B C D C D C D C D C D C D E F E F E F E F E F E F E F Metric = to F E D E R A L A V I A T I O N A D M I N I S T R A T I O N A I R T R A F F I C O R G A N I Z A T I O N 24