Consistent change point estimation. Chi Tim Ng, Chonnam National University Woojoo Lee, Inha University Youngjo Lee, Seoul National University

Transcription

1 Consistent change point estimation Chi Tim Ng, Chonnam National University Woojoo Lee, Inha University Youngjo Lee, Seoul National University

2 Outline of presentation Change point problem = variable selection problem Penalized likelihood method Lasso (Tibshirani, 1996), scad (Fan and Li, 2001), bridge (Frank and Friedman, 1993), unbounded (Lee and Oh, 2009) New theory of selection consistency all local solutions are consistent! Simulation studies

3

4

5 Change point problem Home Depot stock returns ( ) Subseries: r t = log S t /S t 1. Sub I ( ) Sub II ( ) Sub III ( ) The length of full series and each subseries are respectively: 1510, 502, 506, 502.

6 Adjusted close price of Home Depot 3 rd,june, rd,june, Adj Close

7 0.1 Rate of return return

8 Change point problem Standard deviations of subseries Series Standard deviation Full Sub I Sub II Sub III

9 Change point problem Other applications Meteorology: Beaulieu, Chen, and Sarmiento (2012) Engineering: Gillet, Essid, and Richard (2007), Yao (1987) Econometrics: Perron (2005)

10 Change point problem Cumulative sum method (CUSUM) Tiao (1994), Lee, Ha, Na, and Na (2003), Kokoszka and Leipus (2003) Maximum likelihood + Dynamic programming Bai, Lumsdaine, and Stock (1998), Bai and Perron (1998) Bayesian method Lai and Xing (2013)

11 Change point problem = Variable selection Model: X t = µ t + ɛ t, t = 1, 2,..., n. ɛ t are independent N(0, σ 2 ). Re-parameterization: ξ 1 = µ 1 µ 2 ξ 2 = µ 2 µ 3. ξ n 1 = µ n 1 µ n ξ n = µ n µ 1 = ξ ξ n µ 2 = ξ ξ n. µ n 1 = ξ n 1 + ξ n µ n = ξ n

12 Change point problem = Variable selection Likelihood (ˆµ 1,..., ˆµ n ) = arg max 1 2 n t=1 (X t µ t ) 2 Equivalently, (ˆξ 1,..., ˆξ n ) = arg max { 12 } X Uξ 2 where X = X 1 X 2. X n, ɛ = ɛ 1 ɛ 2. ɛ n, U =

13 Change point problem = Variable selection Lasso penalized likelihood estimation: (ˆξ 1,..., ˆξ n ) = arg max 1 2 n t=1 (X t ξ t ξ t+1... ξ n ) 2 + λ n 1 t=1 ξ t. Equivalently, (ˆµ 1,..., ˆµ n ) = arg max 1 2 n t=1 (X t µ t ) 2 + λ n 1 t=1 µ t µ t+1. Computation algorithm?

14

15 Change point problem = Variable selection Local quadratic approximation Fan and Li (2001) and Hunter and Li (2005) ξ ξold 2 + ξ2 2 ξ old. ξ ξold 2 + ξ 2 2(δ + ξ old ). O(n) Iterative algorithm: arg max µ 1 2 n t=1 (X t µ t ) 2 + λ n 1 t=1 (µ t µ t+1 ) 2 2(δ + µ old t µ old t+1 ).

16 Change point problem = Variable selection Example: X = (0.12, 0.16, 0.76, 0.80) λ = 0.00: ˆξ = (0.04, 0.60, 0.04, 0.80) or ˆµ = (0.12, 0.16, 0.76, 0.80) λ = 0.01: ˆξ = (0.03, 0.60, 0.03, 0.79) or ˆµ = (0.13, 0.16, 0.76, 0.79) λ = 0.06: ˆξ = (0.00, 0.58, 0.00, 0.75) or ˆµ = (0.17, 0.17, 0.75, 0.75) λ = 1.00: ˆξ = (0.00, 0.00, 0.00, 0.46) or ˆµ = (0.46, 0.46, 0.46, 0.46)

17

18 Change point problem = Variable selection Alternative choice of penalized likelihood estimation: (ˆµ 1,..., ˆµ n ) = arg max 1 2 Scad: P λ (z) = n t=1 (X t µ t ) 2 + n 1 t=1 P λ ( µ t µ t+1 ). λ z, z n 1 λ, (nz 2 2aλ z + n 1 λ 2 )/[2(a 1)], n 1 λ < z an 1 λ, (a + 1)n 1 λ 2 /2, z > an 1 λ Fan and Li (2001) suggest a = 3.7. Bridge: P λ (z) = λ z γ (0 < γ < 1)

19 Change point problem = Variable selection Unbounded penalty P λ (z) = λ { where τ > 2, ν > 0, and g(z 2 ; τ, ν) = 1 4 log Γ(1/τ) + log τ τ + (τ 2) log g(z2 ; τ, ν) 2τ 2 τ + + z 2 2νg(z 2 ; τ, ν) + g(z2 ; τ, ν) τ (2 τ) 2 + 8τz2 ν }.,

20

21 Change point problem = Variable selection Local quadratic approximation (Fan and Li, 2001) Consider approximation P λ ( ξ ) b 0 + b 1 ξ 2 matching P λ ( ξ ) and P ( ξ ) at ξ old. That is P λ ( ξ ) {P λ (ξ old ) 12 P λ ( ξold )ξ old } + P λ ( ξold ) 2 ξ old ξ 2 Modification to avoid singularity (Hunter and Li, 2005): P λ ( ξ ) {P λ (ξ old ) 12 P λ ( ξold )ξ old } + P λ ( ξold ) 2(δ + ξ old ) ξ2 Computational complexity = O(n)

22 Evaluation of penalty function Gradual change Vs abrupt change Uniqueness of local solution Oracle: AT LEAST ONE OF the local maximums is good True identification: ALL local maximums are good

23 Evaluation of penalty function Technical definition of local maximum. Let Q be the penalized likelihood function and i be the gradient operator with respect to µ i. Then, ˆµ is said to be a local minimum if there exists a neighborhood N(ˆµ) such that for all µ N(ˆµ) {µ : ˆµ i = 0 i = 1, 2,..., n}, we have n i=1 (µ i ˆµ i ) T i Q(µ) < 0. This definition is applicable even in Bridge and unbounded penalty where P λ (0 +) =.

24

25 Evaluation of penalty function Gradual change One change or five changes??? ˆµ = (1, 1, 1, 1, 1, 1, 1.2, 1.4, 1.6, 1.8, 2, 2, 2, 2, 2) Abrupt change ˆµ = (1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2)

26 Evaluation of penalty function Gradual change Theorem: Suppose that the true model has only one change point. Let K L = 0, ±1, ±2... be a given constant. Let k > 1 and P k be the probability that there exists a local penalized likelihood solution with k > 1 changes at t (1) (= [nq (1) ] + K L ), t (1) + 1,..., t (1) + k 1. Then, (i) for unbounded penalty and Bridge, P k 0, (ii) for Lasso with λ = O(n α ) and α 1/2, P k C for some constant C < 1, (iii) for Scad with n 1/2 λ, P k 1.

27 Evaluation of penalty function Oracle Oracle: AT LEAST ONE OF the local maximums is good True model: changes at [nq (1) ], [nq (2) ],..., [nq (k) ] Estimation: changes at t (1), t (2),..., t (k )

28 Evaluation of penalty function Oracle Theorem: Suppose that the true model has k > 0 change points and k is finite. Then, (i) for Lasso with λ = O(n α ) and α 1/2, P k 0, (ii) for Scad with λ = O(n 1/2 ), P k C (0, 1), (iii) for Scad with λ = O(n α ) and α > 1/2, Bridge with α > γ/2, and unbounded penalty with α > 0, P k 1.

29 Evaluation of penalty function True identification True identification: ALL local maximums are good Penalized likelihood Convex: Lasso Non-convex: Scad, bridge, unbounded Inference on global maximum, e.g. Kim and Kwon (2012) Inference on the set of all local maximums

30 Evaluation of penalty function True identification Theorem: Suppose that the true model has k changes and k is finite. For bridge and unbounded, under some regularity conditions, the probability that all local solutions (fulfilling some criteria) have k k 2k goes to one. Ideal: All local solutions have k = k

31

32

33 Modified unbounded penalty Defined as P λ,λ (z) = { Pλ (z) if z > B, P λ (B) λ (B z ) otherwise, where P λ (z) is the unbounded penalty function. Here, λ = n β for some 1/2 < β < 1 and B is chosen such that n 1 P λ (B).

34 Modified unbounded penalty Theorem: Suppose that the true model has k changes and k is finite. Then, for modified unbounded, consistency in the number of changes: under some regularity conditions, the probability that all local solutions (fulfilling some criteria) have k = k goes to one. consistency in change dates: n 1 t (l) [nq (l) ] 0 for l = 1, 2,..., k consistency in parameter estimations: ˆµ i µ i 0 for i = 1, 2,..., n

35 Modified unbounded penalty True identification property = Trinity of consistency in the number of changes consistency in the change dates consistency in the parameter estimation

36 Modified unbounded penalty Uniqueness: Lasso No gradual change: Bridge, unbounded, modified Oracle: Scad, bridge, unbounded, modified True identification: modified

37 Simulation studies k = 0 Lasso, scad, bridge, unbounded penalty and modified unbounded penalty Detail settings: Scad with a = 3.7, bridge with γ = 1/2, unbounded penalty with τ = 30 and ν = 1, modified unbounded penalty with τ = 30, ν = 1, B = 1/n and λ = n 0.6. Bayesian information criterion is used to select λ. X 1,, X n (n = 500, 1000) are generated independently from N(µ i, 1) with µ i = 10 for 1 i n/2, and µ i = 20 for (n/2 + 1) i n.

38 Simulation studies k = 0 n = 500 Penalty Lasso Scad Bridge Unbounded Modified n = 1000 Penalty Lasso Scad Bridge Unbounded Modified

39 Simulation studies k = 0 Root means square error (RMSE): RMSE = 1 n n t=1 ˆµ t µ t 2.

40 Simulation studies k = 0 n = 500 Penalty median RMSE(ˆµ) mean RMSE(ˆµ) Lasso Scad Bridge Unbounded Modified n = 1000 Penalty median RMSE(ˆµ) mean RMSE(ˆµ) Lasso Scad Bridge Unbounded Modified

41 Simulation studies k = 1 X 1,, X n (n = 500, 1000) are generated independently from N(µ i, 1) with µ i = 10 for 1 i n/2, and µ i = 20 for (n/2 + 1) i n.

42

43 Simulation studies k = 1 n = 500 Penalty Lasso Scad Bridge Unbounded Modified n = 1000 Penalty Lasso Scad Bridge Unbounded Modified

44 Simulation studies k = 0 Root means square error (RMSE): RMSE = 1 n n t=1 ˆµ t µ t 2.

45 Simulation studies k = 1 n = 500 Penalty median RMSE(ˆµ) mean RMSE(ˆµ) Lasso Scad Bridge Unbounded Modified n = 1000 Penalty median RMSE(ˆµ) mean RMSE(ˆµ) Lasso Scad Bridge Unbounded Modified

46 Simulation studies k = 4 (X 1,, X 1500 ) is generated independently from N(µ i, 1) with µ i = 10 for 1 i 300, 601 i 900, and 1201 i 1500, and with µ i = 20 for 301 i 600 and 901 i 1200.

47

48 Simulation studies k = 4 n = 500 Penalty Lasso Scad Bridge Unbounded Modified n = 1000 Penalty Lasso Scad Bridge Unbounded Modified

49 Simulation studies k = 0 Root means square error (RMSE): RMSE = 1 n n t=1 ˆµ t µ t 2.

50 Simulation studies k = 4 n = 500 Penalty median RMSE(ˆµ) mean RMSE(ˆµ) Lasso Scad Bridge Unbounded Modified n = 1000 Penalty median RMSE(ˆµ) mean RMSE(ˆµ) Lasso Scad Bridge Unbounded Modified