Time Series Outlier Detection

Transcription

1 Time Series Outlier Detection Tingyi Zhu July 28, 2016 Tingyi Zhu Time Series Outlier Detection July 28, / 42

2 Outline Time Series Basics Outliers Detection in Single Time Series Outlier Series Detection from Multiple Time Series Demos Tingyi Zhu Time Series Outlier Detection July 28, / 42

3 Time Series Basics Tingyi Zhu Time Series Outlier Detection July 28, / 42

4 First-order Autoregression A model denoted as AR(1), in which the value of X at time t is a linear function of the value of X at time t 1: Assumptions: ε t i.i.d N(0, σ), stochastic term. ε t is independent of X t. X t = φx t 1 + ε t (1) Tingyi Zhu Time Series Outlier Detection July 28, / 42

5 General Autoregressive Model AR(p): X t = φ 1 X t 1 + φ 2 X t φ p X t p + ε t p = φ i X t i + ε t = i=1 p φ i B i X t + ε t i=1 where we use the backshift operator B (BX t = X t 1, B k X t = X t k ). Alternative notation: φ(b) is a polynomial of B, φ(b)x t = ε t φ(b) = 1 φ 1 B φ 2 B 2 φ p B p = 1 p φ i B i i=1 Tingyi Zhu Time Series Outlier Detection July 28, / 42

6 Moving Average Another approach for modeling univariate time series X t depends linearly on its own current and previous stochastic terms MA(1): MA(q): X t = ε t + θ 1 ε t 1 X t = ε t + θ 1 ε t θ q ε t q Tingyi Zhu Time Series Outlier Detection July 28, / 42

7 θ 1,..., θ q : parameters of MA model ε t,..., ε t q : stochastic terms Using backshift operator B, model simplified as X t = (1 + θ 1 B + + θ q B q )ε t q = (1 + θ i B i )ε t = θ(b)ε t i=1 Tingyi Zhu Time Series Outlier Detection July 28, / 42

8 ARMA Model A model consists of both autoregressive (AR) part and moving average (MA) part: X t = p q φ i X t i + ε t + θ i ε t i (2) i=1 i=1 referred to as the ARMA(p,q) model. p: the order of the autoregressive part q: the order of the moving average part More concisely, using backshift operator B, (2) becomes: φ(b)x t = θ(b)ε t Tingyi Zhu Time Series Outlier Detection July 28, / 42

9 Stationarity of Time Series In short, a time series is stationary if its statistical properties are all constant over time. To mention some properties: Mean: E[X t ] = E[X s ] for any t, s Z, Variance: Var[X t ] = Var[X s ] for any t, s Z, Joint distribution: Cov(X t, X t+1 ) = Cov(X s, X s+1 ) for any t, s Z. Tingyi Zhu Time Series Outlier Detection July 28, / 42

10 Tingyi Zhu Time Series Outlier Detection July 28, / 42

11 Requirements for a Stationary Time Series AR(1) X t = φx t 1 + ε t : φ < 1 AR(p) φ(b)x t = ε t : All the roots of φ(z) = 0 are outside unit circle. MA models are always stationary ARMA(p,q) φ(b)x t = θ(b)ε t : All the roots of φ(z) = 0 are outside unit circle. Tingyi Zhu Time Series Outlier Detection July 28, / 42

12 Non-stationary time series Trend effect Seasonal effect AirPassengers Time Figure: Monthly totals of international airline passengers, 1949 to Tingyi Zhu Time Series Outlier Detection July 28, / 42

13 Time Series Decomposition Think of a more general time series formulation including both trend and seasonal effect: X t = T t + S t + E t (3) X t is data point at time t Tt is the trend component at time t St is the seasonal component at time t E t is the remainder component at time t (containing AR and MA terms) Tingyi Zhu Time Series Outlier Detection July 28, / 42

14 Series with Trend, examples: Assuming no seasonal effect, i.e. S t = 0 Linear trend: X t = 2t + 0.5X t 1 + ε t Quadratic trend: X t = 2t + t X t 1 + ε t Goal: remove the trend, to transform the series to be stationary Solution: lag-1 differencing Tingyi Zhu Time Series Outlier Detection July 28, / 42

15 Differencing and Trend Define the lag-1 difference operator, where B is the backshift operator. If X t = β 0 + β 1 t + E t, then If X t = k i=0 β it i + E t, then X t = X t X t 1 = (1 B)X t, X t = β 1 + E t. k X t = (1 B) k X t = k!β k + k E t. we call k kth lag-1 difference operator. Tingyi Zhu Time Series Outlier Detection July 28, / 42

16 Lag-1 Differencing S&P 500 Quote Year To Date S&P 500 YTD Lag 1 Differencing Jan Mar May Jul Jan Mar May Jul Tingyi Zhu Time Series Outlier Detection July 28, / 42

17 Series with Seasonal Effect, example: For quarterly data, with possible seasonal (quarterly) effects, we can define indicator function S j. For j = 1, 2, 3, 4, { 1 if observation is in quarter j of a year, S j = 0 otherwise. A model with seasonal effects could be written as X t = α 1 S 1 + α 2 S 2 + α 3 S 3 + α 4 S 4 + ε t Goal: remove the seasonal effects Solution: lag-s differencing, where s is the number of seasons Tingyi Zhu Time Series Outlier Detection July 28, / 42

18 Differencing and Seasonal Effects Define the lag-s difference operator, s X t = X t X t s = (1 B s )X t, where B is the backshift operator. If X t = T t + S t + E t, and S t has period s (i.e. S t = S t s for all t), then s X t = (1 B s )X t = T t T t s + s E t. Tingyi Zhu Time Series Outlier Detection July 28, / 42

19 Non-seasonal ARIMA S t = 0 ARIMA stands for Auto-Regressive Integrated Moving Average, ARMA integrated with differencing. A nonseasonal ARIMA model is classified as ARIMA(p,d,q), where p is the order of AR terms, d is the number of nonseasonal differences needed for stationarity, q is the order of MA terms. Tingyi Zhu Time Series Outlier Detection July 28, / 42

20 Non-seasonal ARIMA, Cont. Recall ARMA(p,q): φ(b)x t = θ(b)ε t, φ(b) and θ(b) are polynomials of B of order p and q. Stationary requirement: all roots of φ(z) = 0 outside unit circle. ARIMA(p,d,q): φ(b)(1 B) d X t = θ(b)ε t, Xt is not stationary. Why? Z t = (1 B) d X t is ARMA(p,q), is stationary. Tingyi Zhu Time Series Outlier Detection July 28, / 42

21 Seasonal ARIMA A seasonal ARIMA model is classified as ARIMA(p, d, q) (P, D, Q) m p is the order of AR terms, d is the number of nonseasonal differences, q is the order of MA terms. P is the order of seasonal AR terms, D is the number of seasonal differences, Q is the order of seasonal MA terms. m is the number of seasons. Tingyi Zhu Time Series Outlier Detection July 28, / 42

22 Example: ARIMA(1, 1, 1) (1, 1, 1) 4 Tingyi Zhu Time Series Outlier Detection July 28, / 42

23 General ARIMA The ARIMA model can be generalized as follow: φ(b)α(b)x t = θ(b)ε t, φ(b): autoregressive polynomial, all roots outside unit circle α(b): differencing filter renders the data stationary, all roots on the unit circle θ(b): moving average polynomial, all roots outside unit circle (to assure θ(b) is invertible. Alternatively, X t = θ(b) φ(b)α(b) ε t. Tingyi Zhu Time Series Outlier Detection July 28, / 42

24 Outliers Detection in Single Time Series Tingyi Zhu Time Series Outlier Detection July 28, / 42

25 Automatic Detection Procedure Described in Chung Chen, Lon-Mu Liu. Joint Estimation of Model Parameters and Outlier Effects in Time Series,JASA, 1993 Based on the framework of ARIMA models R package tsoutlier written by YAHOO in 2014 Tingyi Zhu Time Series Outlier Detection July 28, / 42

26 Types of Outliers General representation: L(B)I t (t j ) L(B): a polynomial of lag operator B I t (t j ) = 1 there s outlier at time t = t j, and 0 otherwise. Types of outliers: Additive Outliers (AO): L(B) = 1; Level Shift (LS): L(B) = 1 1 B ; Temporary Change (TC): L(B) = 1 1 δb ; Seasonal Level Shift (SLS): L(B) = 1 1 B s ; Innovational Outliers (IO): L(B) = θ(b) φ(b)α(b). Tingyi Zhu Time Series Outlier Detection July 28, / 42

27 Types of Outliers Tingyi Zhu Time Series Outlier Detection July 28, / 42

28 Formulation ARIMA model: X t = θ(b) φ(b)α(b) ε t. Model with outliers at time t 1, t 2,..., t m : X t = m ω j L j (B)I t (t j ) + θ(b) φ(b)α(b) ε t. j=1 Lj (B) depends on pattern of the jth outlier I t (t j ) = 1 there s outlier at time t = t j, and 0 otherwise. ωj denotes the magnitude of the jth outlier effect Tingyi Zhu Time Series Outlier Detection July 28, / 42

29 Effect of One Outlier Assume the time series parameters are known, we examine the effect of one outlier: Define polynomial π(b) as: X t = ωl(b)i t (t 1 ) + θ(b) φ(b)α(b) ε t π(b) = φ(b)α(b) θ(b) = 1 π 1 B π 2 B, Contaminated by the outlier, the estimated residual ê t becomes (Without outlier, ê t = π(b)x t.) ê t = π(b)x t Tingyi Zhu Time Series Outlier Detection July 28, / 42

30 For the four types of outliers, IO: ê t = ωi t (t 1 ) + ε t, AO: ê t = ωπ(b)i t (t 1 ) + ε t, LS: ê t = ω π(b) 1 B I t(t 1 ) + ε t, TC: ê t = ω π(b) 1 δb I t(t 1 ) + ε t. Alternatively, ê t = ωx i,t + ε t, t = t 1, t 1 + 1,... and i = 1, 2, 3, 4 x i,t = 0 for all i and t < t 1, x i,t = 1 for all i, x 1,t1 +k = 0, x 2,t1 +k = π k, x 3,t1 +k = 1 k j=1 π j, x 4,t1 +k = δ k k 1 j=1 δk j π j π k. A simple linear regression! Tingyi Zhu Time Series Outlier Detection July 28, / 42

31 Estimate of ω The least square estimate doe the effect of a single outlier at t = t 1 can be expressed as Tingyi Zhu Time Series Outlier Detection July 28, / 42

32 Test Statistics τ From regression analysis, we have ˆω ω n ( x ˆσ i,t) 2 1/2 N(0, 1), a t=t 1 where ˆσ a is the estimation of residual standard deviation. We want to test whether ω = 0, then the following statistics are approximately N(0, 1): Tingyi Zhu Time Series Outlier Detection July 28, / 42

33 Procedure in the Presence of Multiple Ouliers In the presence of multiple outliers, recall the model X t = m ω j L j (B)I t (t j ) + θ(b) φ(b)α(b) ε t. j=1 where ˆσ a is the estimation of residual standard deviation. The estimated residual becomes ê t = m ω j π(b)l j (B)I t (t j ) + ε t j=1 Tingyi Zhu Time Series Outlier Detection July 28, / 42

34 Stage 1: Joint Estimation of Outlier Effect and Model Parameters Fitting the series by an ARIMA model (forecast package in R), obtain initial parameter (φ(b), θ(b), α(b)) estimation of the model. Detect outliers one by one sequentially Tingyi Zhu Time Series Outlier Detection July 28, / 42

35 Stage 2: Initial Parameter Estimation and Outlier Detection Tingyi Zhu Time Series Outlier Detection July 28, / 42

36 Tingyi Zhu Time Series Outlier Detection July 28, / 42

37 Outlier Series Detection from Multiple Time Series Tingyi Zhu Time Series Outlier Detection July 28, / 42

38 Detect Anomalous Series Goal: efficiently find the least similar time series in a large set Motivation: Internet companies monitoring the servers(cpu, Memory), find unusual behaviors Tingyi Zhu Time Series Outlier Detection July 28, / 42

39 Detect Anomalous Series Described in Rob J Hyndman et al. Large-Scale Unusual Time Series Detection, ICDM, 2015 Approach: Extract features from time series, PCA R package anomalous Test on real data from YAHOO server, 80% accuracy compared to 40% from previous methods Tingyi Zhu Time Series Outlier Detection July 28, / 42

40 Step 1: Extract Features from Time Series 15 features selected, each captures the global information of time series Tingyi Zhu Time Series Outlier Detection July 28, / 42

41 Step2: PCA to reduce dimension dim=15 initially, correlation existing between features The first 2 PCs are sufficient, capturing most of the variance Step 3: Implement multi-dimentional outlier detection algorithm to find outlier series Density based α-hull Tingyi Zhu Time Series Outlier Detection July 28, / 42

42 Demo Tingyi Zhu Time Series Outlier Detection July 28, / 42