Using the Empirical Probability Integral Transformation to Construct a Nonparametric CUSUM Algorithm

Transcription

1 Using the Empirical Probability Integral Transformation to Construct a Nonparametric CUSUM Algorithm Daniel R. Jeske University of California, Riverside Department of Statistics Joint work with V. Montes de Oca, W. Bischoff and M. Marvasti QPRC Quality Productivity and Research Conference June 4, 2009

2 Motivation An anomalous event is categorized as a departure from normal operating conditions due to either an existing or impending failure condition. In network surveillance it is important to detect anomalous activity as quickly as possible, and ideally before a major problem develops. There are 1,000 s of metrics to be monitored, therefore automated network management is important. Example: By-the-minute measurements of traffic (e.g., bytes or packets) carried over a network link, or passing through a network node. 2

3 ALIVE TM by Integrien Corporation Alive is a secure network monitoring system running Alive integrates monitoring, advanced analytics, root cause analysis, and recovery functions for the full scope of IT system components. Through advanced statistical technology, Alive detects deviation from normal operations and prevents problems from impacting the business. Alive Dashboard 3

4 Live Sessions Data Means and standard deviations for Live sessions obtained from 12 weeks of screened historical data. 4

5 Requirements 1) Versatility - the need for a nonparametric technique so that a wide variety of metrics can be included in the monitoring process 2) Adaptability - the need to handle time varying distributions for the metrics that reflect natural cycles in a work week 3) Efficiency - the need to be computationally efficient with the massive amounts of data that are available for processing 5

6 Structured Timeslot Non-Stationarity 161 Hours of the Week Historical Distributions (e.g., rolling window) 1 2. i 161 ˆ () F 1 Fˆ i () F ˆ () 161 Data for new monitoring week Y1 Y 2 Y30 Y30i 29 Y30i 28 Y 30i Y Y Y 4,801 4,802 4,830 6

7 Independence Assumptions 161 Hours of the Week Historical Distributions (e.g., rolling window) 1 2. i 161 ˆ () F 1 Fˆ i () F ˆ () 161 Data for new monitoring week Y1 2 Y Y30 Y30i 29 Y30i 28 Y 30i Y Y Y 4,801 4,802 4,830 Observations within timeslots (hour) are identically and independently distributed Observations between timeslots are independent but not identically distributed 7

8 Improving Plausibility of Independence Assumption X X X j1 j1 j2 jr = Y Y ρ Y = + µ 1 1 ρ j2 j j1 j 2 j 2 1 ρ j 1 ρ j Y ρ Y = + µ 1 1 ρ jr j j, r 1 j 2 j 2 1 ρ j 1 ρ j Under AR(1) model these observations are uncorrelated with common mean µ and common variance σ Other (application-dependent) transformations could be used. Our starting point is the transformed data. 2 j j 8

9 Effectiveness of Transformation For Raw Data Transformed Data Our Application Autocorrelation Functions of Raw Data (Column 1) and Corresponding Transformed Data (Column 2) for Three Timeslot-Week Combinations (Rows) 9

10 Recent Work in Network Surveillance Applications 1. Hajji, H. (2005), Statistical Analysis of Network Traffic for Adaptive Faults Detection, IEEE Transactions on Neural Networks, 16, Random regime switching assumption is not suitable for structured timeslot context 2. Kim, S., Alexopoulos, C., Tsui, K., and Wilson, J. R. (2007), A Distribution-free Tabular CUSUM Chart for Autocorrelated Data, IIE Transactions, 39, Applies to stationary contexts (which would be one timeslot for us) with known covariance structure 10

11 Empirical Probability Integral Transformation i 161 F ˆ () Fˆ () F ˆ () 161 i X1 2 X X 30 X X X 30i 29 30i 28 30i X X 4,801 4,802 X 4,830 F ˆ 1 ( X 1 ) F ˆ 1 ( X 2 ) F ˆ 1 ( X 30 ) ˆ ( ) Fi X30i 29 i 30 i 38 F ˆ ( X ) F ˆ ( ) i X 30 i Fˆ 161( X4,801) Fˆ 161( X4,802) F ˆ ( ) 161 X4,830 U U U 11

12 Discrete Probability Integral Transformation Conditional on the historical data, the U values for a given timeslot with m historical values take on m+ 1 values with probabilities: 0 1/ m ( m 1)/ m 1 F( x1: m) F( x2: m) F( x1: m) F( xmm : ) F( xm 1: m) 1 F( x mm : ) Probabilities are spacings of m observations from a U(0,1) distribution, so conditional distribution of U 's is approximately 0 1/ m ( m 1)/ m 1 1/( m+ 1) 1/( m+ 1) 1/( m+ 1) 1/( m+ 1) which depends on this historical data only through sample size m 12

13 Transformed Cusum (TC) Cusum Algorithm Sn = max 0, Sn 1+ ( F ( Xn) α), n= 1,, N, S0 = 0 n Sn = max 0, Sn 1+ (1 α F ( Xn)), n= 1,, N, S0 = 0 n Stopping Rule + S > H or S > H where n + P( S > H or S > H) = γ 0 n n H is obtained by simulation γ is the probability of a false alarm n Weeks of Historical Data Threshold Values (H) Probability of False Alarm (m=180) (m=360) (m=720) Infinity *Infinity corresponds to sampling from Continuous Uniform Distribution 13

14 Brownian Motion Approximation Let Z, Z,, Z 1 2 N be independent, identically distributed random variables with mean 0 and variance 1. Define n 1 Wn = Zi, n= 1,, N N i= 1 ( ) n Then for large N we have P W > H γ where H =Φ γ max 1 (1 / 2) 1 n N 14

15 Brownian Motion Cusum (BMC) Cusum Algorithm Ui = max 0,( F ( Xi) α) i S + n n 1 Ui E( Ui) = N i= 1 Var( Ui ) Vi = max 0,(1 α F ( X )) i, i= 1,, N i n 1 Vi E( Vi) Sn =, n= 1,, N N i= 1 Var( V) i means and standard deviations are computed as on next slide... Stopping Rule S H S H H γ + 1 n > or n > where =Φ (1 4) 15

16 Brownian Motion Cusum (BMC) E ( U ) = E ( V ) 0 l 0 l ( n n 1)( n n 2 n ) α + + α α 2 n ( n + 1) l l l l l l l V ( U ) = V ( V ) 0 l 0 l ( )( 2 ) ( 2 ) n n n 2 n n n n 2 n ( n n ) α + α + α + + α + α 2 3 l l l l l l l l l l + α n l 2 6 n ( n + 1) ( )( ) αn + 1 αn n αn l l l l l mm ( + 1) l ( n n 1)( n n 2 n ) 2 α + + α α 2 n ( n + 1) l l l l l l l 16

17 Illustration of TC and BMC Raw Data Transformed Data Y X TC BMC 17

18 Steps to Evaluate Cusum Algorithms Outer Loop 1. Generate historical data from a parametric model with m observations per timeslot 2. Generate 1000 monitoring weeks with no faults injected 3. Evaluate detect times and percentage of faults found 4. Repeat steps 1-3, 25 times Inner Loop 18

19 Conditional False Alarm Results m=180 m=360 m=720 m=inf For n=180 BMC has a preferred distribution of conditional false alarm rates since TC shows a skew toward high values. For the other values of m, both algorithms have conditional false alarm rate distributions that are satisfactorily centered on.10. However, the distributions for TC are more concentrated on

20 Unconditional False Alarm Results n TC BMC TC BMC TC BMC γ = 0.01 γ = 0.05 γ =

21 Steps to Evaluate Cusum Algorithms 1. Generate 12 weeks of historical data from parametric model Outer Loop 2. Generate 1000 monitoring weeks with 5 faults of the same type injected per week Fault-types are combinations of Magnitude 25%, 50%, 75%, 100% Duration 8, 16, 30, 60, 120 min Inner Loop 3. Evaluate detect times and percentage of faults found 4. Repeat steps 1-3, 25 times 21

22 Fault Injection Study Results Fault Pattern Detection Percentage Average Detect Time (min) Duration (min) TC BMC TC BMC Mean Increase 25% 50% 75% 100% Nominal 10% Two-Sided Cusum Algorithms, Random Fault Injection

23 Conclusions TC is the recommended algorithm based on fault injection study results Algorithm is being implemented in customer networks to gain field trial experience. Alternative nonparametric cusum algorithms being investigated. 23