Change Detection Using Directional Statistics

Transcription

1 Change Detection Using Directional Statistics Tsyoshi Idé Dzng T. Phan Jayant Kalagnanam IBM Research, T. J. Watson Research Center 1101 Kitchawan Rd., Yorktown Heights, NY 10598, USA Abstract This paper addresses the task of change detection from noisy mltivariate time-series data. One major featre of or approach is to leverage directional statistics as the noise-robst signatre of time-series data. To captre major patterns, we introdce a reglarized maximm likelihood eqation for the von Mises-Fisher distribtion, which simltaneosly learns directional statistics and sample weights to filter ot nwanted samples contaminated by the noise. We show that the optimization problem is redced to the trst region sbproblem in a certain limit, where global optimality is garanteed. To evalate the amont of changes, we introdce a novel distance measre on the Stiefel manifold. The method is validated with real-world data from an ore mining system. 1 1 Introdction The problem we wish to solve is change detection of mltivariate time-series data. Figre 1 shows a typical setting, where or task is to compte the degree of change, or the change score, of the data within the test window taken at time t in comparison to the training window. The task of change detection has a long history in statistics. The standard strategy is to se a parametric model for probability density and compte the likelihood ratio to qantify the degree of change between fitted distribtions [Chen and Gpta, 2012]. For a concise review from a statistical machine learning perspective, see [Yamada et al., 2013]. When applying a change detection method to real-world problems, the major reqirements are interpretability and robstness to nisance noise variables. To validate detected changes with domain knowledge, it is almost always reqired to explicitly present statistics (or featre) for the parametric model, sch as the mean for Gassian. Althogh it is possible to design an algorithm that skips the explicit step of featre extraction and jmps directly into score calclation [Li 1 The final version with corrections of T. Idé et al., Change Detection sing Directional Statistics, Proceedings of the Twenty- Fifrth International Joint Conference on Artificial Intelligence (IJ- CAI 16, Jly 9-15, 2016, New York, USA), pp N training window (fixed or sliding) D test window t (time) Figre 1: Change detection problem. et al., 2013], sch an approach is not highly appreciated in practice de to the lack of interpretability. In the mltivariate setting, the robstness to noise variables is the other critical reqirements since changes may not always occr in all of the variables simltaneosly. In fact, nder the existence of nisance noise variables, the performance of direct density-ratio estimation approaches is known to significantly degrade [Yamada et al., 2013]. Considering these two reqirements, we focs on change detection approaches having explicit two steps of featre extraction and score calclation. There are two important decision points here: (1) Parametric model for probability density fnction, and (2) scoring model for the change score. In this paper, we propose a novel framework for change detection for mltivariate sensor data. Or contribtion is threefold: We develop a novel featre extraction method based on reglarized maximm likelihood of the von Mises- Fisher (vmf) distribtion. We show that the featre extraction method is redced to an optimization problem called the trst-region sbproblem [Tao and An, 1998; Hager, 2001]. We propose a novel scoring method based on a parametrized Kllback-Leibler (KL) divergence. Implications of these contribtions are as follows. First, or featre extraction method is the first proposal to efficiently remove the mltiplicative noise that is biqitos in many physical systems. Thanks to an l 1 reglarization scheme, it is also capable of atomatically removing samples

2 contaminated by the nwanted noise. Second, the trst-region sbproblem garantees the global optimality in a certain limit, which is especially important for noisy data. Third, the parametrized Kllback-Leibler divergence for scoring provides s with a trstworthy way to qantify the discrepancy between different sbspaces with different dimensionalities. ln(c_m) / kappa kappa 2 Extracting featre matrix from noisy data The proposed method consists of two steps. The first step comptes an orthonormal matrix as the signatre of the flctation patterns of mltivariate data. The second step comptes the difference between two data sets in the past and present throgh the compted orthonormal matrices. This section explains the first step. 2.1 The von Mises-Fisher distribtion Assme we are monitoring a system with M sensors, and we are given N measrements x (1),..., x (N) R M, which may correspond to either the training or the test window in Fig. 1. Or principal probabilistic model is the von Mises- Fisher (vmf) distribtion [Mardia et al., 1980]: M(z, κ) c M (κ) exp ( κ z ) (1) c M (κ) κ M/2 1 (2π) M/2 I M/2 1 (κ), (2) where I M/2 1 ( ) denotes the modified Bessel fnction of the first kind with the order M 2 1. The random variable z is assmed to be normalized in the sense z z = 1, where represents transpose. The vmf distribtion has two parameters: the mean direction and the concentration parameter κ. As these names sggest, the vmf distribtion describes random variability of the direction arond the mean vector. The intition behind the se of vmf distribtion is as follows. When describing the measrements sing the vmf distribtion, we look only at the direction, disregarding flctations along the direction. This atomatically gives the robstness to mltiplicative noise, which is qite common in practice in systems with redndancy. As an example, we will look at an ore transfer system, where two belt conveyors are operated by the same electronic system. In this system, major flctations in one system de to e.g. dynamic load changes are shared by the other system. In systems having strongly correlated variables, the vmf distribtion can be a more natral tool for system monitoring than the Gassian. 2.2 Weighted joint maximm likelihood To simplify the notation, we introdce the data matrix X [x (1),..., x (N) ] = [b (1) z (1),..., b (N) z (N) ], (3) where z (n) 2 = 1 and b (n) x (n) 2 for n = 1,..., N and p being the p-norm. The parameters of the vmf distribtion may be inferred by maximizing a likelihood fnction: L(, w X) N w (n) b (n) ln M(z (n), κ), (4) n=1 Figre 2: γ/κ as a fnction of κ (M = 10). where we introdce sample weights w (w (1),..., w (N) ). We wish to captre major patterns represented as the directional data by optimally choosing the weights so that less trstworthy samples are down-weighted. The weighted likelihood L(, w X) has only a single pattern, and naively maximizing L(, w X) prodces only the single direction. To captre mltiple patterns of the change, we jointly fit m different distribtions while keeping the overlap minimal by imposing the orthogonality between different patterns: max U,W m L( i, κ, w i X) R(w i ) s.t. U U = I m, (5) where I m is the m-dimensional identity matrix, U [ 1,..., m ], and W [w 1,..., w m ]. The term R(w i ) is a reglarizer to remove the trivial soltion on the sample weights. Here we consider the elastic net reglarization [Zo and Hastie, 2005]: ( ) 1 R(w i ) = λ 2 w i ν w i 1, (6) where λ and ν are given constants, typically determined by cross-validation. The first term of the objective fnction is given by m L( i, κ, w i X) = κtr ( XWU ) + γ1 W b, (7) where we defined γ ln c M (κ), b [b (1),..., b (N) ], and 1 is a colmn vector of all ones. Throghot the paper we will treat κ as a given constant. Fortnately, γ/κ is qite insensitive to κ, as shown in Fig. 2. One sefl heristic is simply to set κ = M. Otherwise, we can se known approximation algorithms given in [Sra, 2012]. 2.3 Iterative seqential algorithm The optimization problem (5) can be seqentially solved for each of ( i, w i ). Imagine that the sample weight w is initialized to a vector. Given w, Eq. (5) is redced to max κ Xw s.t. = 1 (8) for the first. The soltion is readily obtained as = Xw Xw 2. (9)

3 Given this soltion, the problem (5) for w is now written as w q λ2 w w λν w 1 arg max w = arg min w where q is defined by 1 2 w q λ ν w 1, (10) q γb + κx. (11) This problem is a special case of LASSO regression, and has a closed-form soltion [Wen et al., 2010] as q w = sign(q) max λ ν1, 0, (12) where denotes the componentwise prodct, 0 is the zero vector, and q is an N-dimensional vector whose n-th entry is q n. With this new w, we can solve Eq. (8) again. We repeat solving Eqs. (8) and (10) alternatingly ntil convergence. Once we get the first soltion ( 1, w 1 ), we move on to the next soltion. We again start with initialized w and solve the following problem instead of Eq. (8): max κ Xw s.t. = 1, 1 = 0. (13) By introdcing Lagrange mltipliers α, β 1 for the two constraints, respectively, we have the condition of optimality as 0 = [κ Xw α ] 2 β 1 1 = κxw α β 1 1. (14) Using the constraints, the candidates for the soltion are κ(xw 1 1 Xw) (15) ± 2 (16) By plgging these two stationary points into the objective fnction in Eq. (13), we can get the maximizer. This soltion is inserted into Eq. (10) to get a new w. These steps are repeated ntil convergence. It is straightforward to generalize the above procedre for j = 2,..., m. We smmarize the procedre in Algorithm 1, which we call the REglarized Directional featre extraction (RED) algorithm: In Eq. (17), we define U j 1 [ 1,..., j 1 ]. I M is the M-dimensional identity matrix. The complexity of RED algorithm for each while loop iteration is O(NM). 3 Theoretical analysis 3.1 Convergence of RED algorithm We prove the following theorem: Theorem 1. Define X [I M U j 1 U j 1 ]X and g(w j, j ) κ Xw j j + γwj b R(w j) in the notation of Algorithm 1. For a fixed U j 1, the seqence g(wj t, t j ) t=0,1,... generated Eqs. (17)-(20) in the j-th while-loop has a finite limit. Algorithm 1 RED algorithm. Inpt: Initialized w. Reglarization parameters λ, ν. Concentration parameter κ. The nmber of major directional patterns m. Otpt: U = [ 1,..., m ] and W = [w 1,..., w m ]. for j = 1, 2,..., m do while no convergence do end while end for Retrn U and W. j κ[i M U j 1 U j 1]Xw j (17) j j sign( j Xw j ) (18) j 2 q j γb + κx j (19) qj w j sign(q j ) max λ ν1, 0 (20) Proof. First, we note that g(w j, j ) is the objective fnction to be maximized for each j nder the constraint j 2 = 1. The seqence g(wj t, t j ) is bonded above since g(w j, j ) κ Xw j j + γwj b λ 2 w j 2 2 (21) = λ 2 w j κ X 2 j + γb + κ X j + γb 2 2 λ 2λ 2 1 κ X j γb 2 2 2λ 1 κ 2 σ max ( X X ) + γb 2 2, (22) 2λ where σ max is the nonnegative maximm eigenvale of X X. The ineqality (21) is de to λ and ν being positive, the last ineqality (22) is derived from j 2 = 1. For a j, we have g(w t j, t j) g(w t j, t+1 j ) g(w t+1 j, t+1 j ), where the first ineqality comes from the fact that t+1 j is the maximizer given wj t, and the last one is becase wt+1 j is the maximizer given t+1 j. The bondedness and monotonically increasing of the seqence g(wj t, t j ) complete the proof. 3.2 Global optimality in ν 0 Here we look at the following theorem: Theorem 2. When ν tends to 0, the nonconvex problem (5) is redced to an optimization problem in the form of min Q + c s.t. = 1, (23) which has a global soltion obtained in polynomial time. Proof. The non-convex optimization problem (23) is known as the trst region sbproblem. For polynomial algorithms to

4 the global soltion, see [Sorensen, 1997; Tao and An, 1998; Hager, 2001; Toint et al., 2009]. Here we show how the algorithm is redced to the trst region sbproblem. When ν = 0, the objective fnction (5) leads to the optimality condition w.r.t. w as 0 = w w q λ 2 w w, which immediately gives an analytic soltion as w = q/λ. By inserting this soltion into the objective fnction in Eq. (5), we have an optimization problem only for : max κ XX + 2γ Xb s.t. = 1, which is obviosly eqivalent to Eq. (23) with Q κxx R M M and c 2γXb R M. Let 1 be the soltion of this problem. Once 1 is obtained, we again solve another trst-region sbproblem of the form (23) bt in the R M 1 space by adding an orthogonality condition 1 = 0. In particlar, withot loss of generality, we assme the last component of 1 is nonzero, i.e., 1,M 0. Define ũ ( 1,..., M 1 ) sing the first M 1 dimensions of, and a ( 1,1,..., 1,M 1 ) / 1,M in place of 1. The problem (23) with the additional linear eqality constraint is eqivalent to min ũ (ũ, a ũ ) Q ( ũ a ũ ) c i i c M a ũ M 1 + s.t. ũ (I M 1 + aa )ũ = 1. (24) Consider the Cholesky decomposition for rank-one pdate I M 1 + aa = LL and the change of variable y L ũ. The decomposition can be done efficiently. See, e.g., [Gill et al., 1974]. It is straightforward to see that the problem (24) is rewritten as min y Qy + c y s.t. y y = 1 y R M 1 for a certain ( Q, c), which has exactly the same format with (23). When more than two orthonormal vectors i are needed, a set of linear eqations is additionally considered i = 0, i = 1,..., j 1. A variable elimination method can be sed in order to work on a redced space. Note that an eigenvale decomposition for a matrix of dimension M j is needed to transform it into a (M j)-dimensional trst region sbproblem. In this way, we have a set of orthonormal vectors 1,..., m, where m is an inpt parameter representing the nmber of major directional patterns. Althogh the global optimality is no longer garanteed when ν > 0, we can take advantage of the global soltion at ν = 0 to initialize w in Algorithm 1. In practice, if there is a concern abot the qality of the soltion, we can gradally increase the vale of ν, and se the obtained w j s for the next trial for a larger ν. Althogh mathematically the RED algorithm is an iterative algorithm that may be trapped by sb-optimality, we can loosely say that it is an almost garanteed algorithm in practice. 4 Parameterized KL divergence for scoring Solving the optimization problem for the training and the test windows, we obtain two sets of orthonormal vectors 1,..., m and v 1,..., v r where m and r are the nmber of vectors given as inpt parameters. Compting the change score amonts to evalating the dissimilarity between the two vector spaces specified by orthonormal matrices U [ 1,..., m ], U (t) [v 1,..., v r ]. (26) Now or problem is to compte the dissimilarity on a Stiefel manifold, which is the space spanned by orthogonal matrices. As illstrated in Fig. 1, in practice, the window size D shold be chosen as small as possible to minimize the time lag in change detection, while the nmber of samples N in the training data shold be large enogh to make sre to captre major patterns. Depending on the natre of data, it makes sense to assme m r. To handle this general sitation, we consider linear combinations of i s and v i s and define the change score as a parameterized version of KL divergence: a (t) = min f,g dx M(x Uf, κ) ln M(x Uf, κ) M(x U (t) g, κ) (27) nder the constraint of f f = 1, g g = 1. This can be also viewed as an averaged version of the log likelihood ratio, which is the standard measre of change detection [Chen and Gpta, 2012]. By inserting the definition of the vmf distribtion in Eq. (1), we have a (t) = κ min f,g ( ) x Uf U (t) g, (28) where is the expectation w.r.t. M(x Uf, κ). Since x Uf follows from the basic property of the vmf distribtion [Mardia et al., 1980], we have a (t) = 1 max f,g f U U (t) g s.t. f f = 1, g g = 1, (29) where we dropped an nimportant prefactor. Solving the optimization problem (29) is easy. Introdcing Lagrange mltipliers π 1, π 2 for the two constraints, it is straightforward to obtain optimality conditions as U U (t) g = π 1 f, U (t) Uf = π 2 g. This means that f and g are fond via singlar-vale decomposition of U U (t), which is a small matrix of size m r. The maximm of the objective fnction is simply given by the maximm singlar vale, σ (t) 1 ; thereby we reach the final formla for the change score: a (t) = 1 σ (t) 1. (30) The maximm singlar vector is efficiently compted by the power method [Golb and Loan, 1996]. In most practical sitations, it reqires only iterations as many as minm, r. The complexity to compte the KL-based change score is (minm, r) 3. Since m and r can be jst several in most applications, it is negligible in practice.

5 5 Related work As described so far, the proposed method extensively ses the vmf distribtion. For change or anomaly detection, little work has focsed on the vmf distribtion. [Idé and Kashima, 2004] seems to be one of the earliest pieces of work that explicitly leverages the vmf distribtion for anomaly detection, bt it does not discss the particlar task of change detection and sample reglarization. In statistics, a lot of efforts have focsed on asymptotic analysis of the likelihood ratio [Chen and Gpta, 2012]. Recently, [Kncheva, 2013] compares varios change detection approaches and concldes that a model combining Gassian mixtre and Hotelling s T 2 statistic works best. However, it is widely known that stably learning Gassian mixtre is hard for physical sensor data we are interested in, which is qite noisy and incldes many otliers. Also, in general, accrately estimating densities itself is challenging when dimensionality is high (M 10). To address the challenge of density estimation, [Kawahara and Sgiyama, 2009; Li et al., 2013] proposed an interesting techniqe of direct density-ratio estimation, which integrates the two steps of parametric model estimation and scoring into a single step of density-ratio estimation. Thanks to the integration of the two steps, their approach is generally better than those estimates two densities individally. However, de to the very same reason, they lack practical interpretability, which is of critical importance in practice. Also, it has been pointed ot that the performance significantly degrades when some of the variables are jst non-informative nisance featres [Yamada et al., 2013]. Recently, to extract the most informative featres from mltivariate time-series data, [Blythe et al., 2012] proposed an interesting approach called stationary sbspace analysis (SSA). Althogh SSA proposed for the task of time-series segmentation, which is different from the on-line changedetection we are dealing with, we experimentally compare it with the proposed method in the next section. 6 Experimental reslts 6.1 Methods compared We compare the proposed method denoted by RED+KL, which ses Algorithm 1 for compting U and U (t) in Eqs. (26) and (30) for the change score, with the following alternative methods: RED+tr: Use Algorithm 1 for U bt replace Eq. (30) with the trace norm where r = m: a (t) = 1 1 m Tr(U U (t) ). (31) SSA: Use SSA for U. First identify the most stationary sbspace by solving min V V=I M m E ln V Σ i V + V µ i 2, (32) where E is the nmber of epochs defined by a nonoverlapping time window of size D, and µ i, Σ i are the sample mean and covariance matrix compted in the i-th time point index Figre 3: Synthetic three-dimensional time-series data. RED SSA PCA Figre 4: Comparison of 1 (Synthetic data). epoch. represents the determinant in this case. Once V is fond, U is obtained as the complement space of V. The change score is compted by a (t) = ln U Σ (t) U + U µ (t) 2 + Tr(U Σ (t) U), where µ (t) and Σ (t) are the mean and the covariance matrix over D (t), which is defined as the set of the samples in the sliding window at time t. Before compting these, the data is whitened with the same pooled mean and covariance of the training data. PCA: Use the principal component analysis for U. Employ the mean reconstrction error for scoring [Papadimitrio and Y, 2006] : a (t) = 1 (I m UU )x (n) 2. (33) D n D (t) T2: Use the mean Hotelling s T 2 statistic a (t) = 1 (x (n) µ) Σ 1 (x (n) µ). (34) D n D (t) for scoring. Here µ and Σ are the mean and the covariance matrix of the training data. T2 is compared only in change detection since it has no explicit featre extraction step. 6.2 Synthetic data: comparison of featre extraction methods Figre 3 shows three-dimensional synthetic time-series data we generated. In this data, x 1 and x 2 are qite noisy bt significantly correlated, while x 3 is ncorrelated to the others. Ths we expect the principal direction wold point to the 45 line between x 1 and x 2.

6 x st component x5 x7 x6 x nd component time point index x9 0 Figre 5: Ore transfer data nder the normal operation. We compared the proposed algorithm (denoted simply by RED) with two alternatives SSA and PCA. All methods take the data matrix X as the inpt, and retrn an orthonormal matrix U as the otpt. Figre 4 shows the coefficients of the first component 1. For RED, we sed (λ, ν) = (1, 4), while for SSA we sed D = 25. We see that RED sccessflly captres the expected 45 direction as the principal direction. PCA has a similar trend, bt de to the major otlier in x 2 at 94 (see Fig. 3), it has more weight in x 2. Close inspection shows that RED atomatically removes this otlier by ptting zero weight, demonstrating the atomated noise-filtering capability. The basic assmption of SSA is that the stationary components are most likely noise, and the non-stationary components are more informative for change detection. In this particlar example, the major pattern is the correlation strctre between x 1 and x 2, and the other variable x 3 is the noise. However, de to the heavy noise especially in the first half of x 3, SSA fails to disregard the noise variable. 6.3 Real-world data: comparison of on-line change detection performance We applied the proposed method to a real-world on-line change detection task. Figre 5 shows time-series data from an ore transfer system being monitored by ten sensors measring physical qantities sch as speed, crrent, load, temperatre, and displacement. The data itself was generated by a testbed system to simlate the normal operating condition and ths sed as the training data. As introdced in Sec. 2.1, the system consists of two almost eqivalent sbsystems, and some of the variables are significantly correlated, and incr the mltiplicative noise. As seen, the data is extremely noisy and sometimes exhibits implse-like noise de to the physical operating condition of the otdoor ore transfer system. For this training data, we applied the RED algorithm to find U and W. Figre 6 shows W = [w 1, w 2 ], where we sed (λ, ν) = (2, 9) that maximized the F score for the test data (see the next paragraph). We see that many samples are driven to zero. In fact, 43 and 39 percents of the samples have exact zero in w 1 and w 2, respectively, ot of which abot 16 percent are zero in both. Close inspections show that almost all of ot-of-context otliers are ct off, while informative otliers that are consistent to the major correlation strctre srvive. This is exactly what we expected. Figre 6: The magnitde of the sample weights w (n) i for the first and second components (shold be aligned with Fig. 5). negative sample accracy (positive sample accracy) RED+KL RED+tr SSA PCA T2 Figre 7: Comparison of the ROC crve. We evalated the performance of on-line change detection sing another data set, where changes de to system malfnctions are recorded. Most of the failre symptoms are associated with nreasonable changes in the correlation strctre between the variables. We simply se the negative-sample and positive-sample accracies as the performance metric. Here, the negative samples are defined by those not belonging to change-points, while the positive samples are those belonging to change-points. We sed the harmonic average (F score) of them to determine the (λ, ν) vales. We se the window of D = 60 over abot 1200 samples. Figre 7 compares the ROC crve. Clearly, RED-KL with (m, r) = (2, 3) otperforms the alternatives. It is interesting to see RED-tr gives the worst, which demonstrates the importance the proposed sbspace comparison techniqe of Eq. (30). For this data set, SSA fails to captre change points. The main reason is that SSA is not necessarily robst to spiky otliers as seen in Fig. 5. Since the correlational strctre is critical to detect change points in this data, it makes sense that the PCA and T2 do a good job. 7 Conclsion We have proposed a new on-line change detection algorithm for mltivariate time-series data. Or algorithm extracts ma-

7 jor directional patterns while atomatically disregarding less informative samples. We proved the convergence of the algorithm, and showed that the qality of the soltion is spported by the global optimality of the trst-region sbproblem. We also proposed a new method of scoring the change based on a parameterized KL divergence. We validated the algorithm sing real-world data. References [Blythe et al., 2012] D.A.J. Blythe, P. von Bna, F.C. Meinecke, and K.-R.Mller. Featre extraction for changepoint detection sing stationary sbspace analysis. IEEE Transactions on Neral Networks and Learning Systems, 23(4): , [Chen and Gpta, 2012] Jie Chen and Arjn K. Gpta. Parametric Statistical Change Point Analysis. Birkh aser Applied Probability and Statistics. Springer Verlag, [Gill et al., 1974] P. E. Gill, G. H. Golb, W. Mrray, and M. A. Sanders. Methods for modifying matrix factorizations. Mathematics of Comptation, 28(126): , [Golb and Loan, 1996] Gene H. Golb and Charles F. Van Loan. Matrix comptations (3rd ed.). Johns Hopkins University Press, Baltimore, MD, [Hager, 2001] W. W. Hager. Minimizing a qadratic over a sphere. SIAM Jornal on Compting, 12: , [Idé and Kashima, 2004] T. Idé and H. Kashima. Eigenspace-based anomaly detection in compter systems. In Proc. ACM SIGKDD Intl. Conf. Knowledge Discovery and Data Mining, pages , [Kawahara and Sgiyama, 2009] Yoshinob Kawahara and Masashi Sgiyama. Change-point detection in time-series data by direct density-ratio estimation. In Proc. of 2009 SIAM Intl. Conf. on Data Mining (SDM 09), [Kncheva, 2013] Ldmila I. Kncheva. Change detection in streaming mltivariate data sing likelihood detectors. IEEE Transactions on Knowledge and Data Engineering, 25(5): , [Li et al., 2013] S. Li, M. Yamada, N. Collier, and M. Sgiyama. Change-point detection in time-series data by relative density-ratio estimation. Neral Networks, 43:72 83, [Mardia et al., 1980] K. V. Mardia, J. T. Kent, and J. M. Bibby. Mltivariate Analysis. Academic Press, [Papadimitrio and Y, 2006] Spiros Papadimitrio and Philip Y. Optimal mlti-scale patterns in time series streams. In Proc ACM SIGMOD Intl. Conf. Management of Data, pages , [Sorensen, 1997] D. C. Sorensen. Minimization of a largescale qadratic fnction sbject to a spherical constraint. SIAM Jornal on Optimization, 7(1): , [Sra, 2012] Svrit Sra. A short note on parameter approximation for von Mises-Fisher distribtions: and a fast implementation of I s (x). Comptational Statistics, 27(1): , March [Tao and An, 1998] Pham Dinh Tao and Le Thi Hoai An. A D.C. optimization algorithm for solving the trst-region sbproblem. SIAM J. Optimization, 8: , [Toint et al., 2009] P. L. Toint, D. Tomanos, and M. Weber- Mendonca. A mltilevel algorithm for solving the trstregion sbproblem. Optimization Methods and Software, 24(2): , [Wen et al., 2010] Zaiwen Wen, Wotao Yin, Donald Goldfarb, and Yin Zhang. A fast algorithm for sparse reconstrction based on shrinkage, sbspace optimization, and contination. SIAM Jornal on Scientific Compting, 32(4): , [Yamada et al., 2013] M. Yamada, A. Kimra, F. Naya, and H. Sawada. Change-point detection with featre selection in high-dimensional time-series data. In Proc. Twenty- Third International Joint Conference on Artificial Intelligence, IJCAI 13, pages , [Zo and Hastie, 2005] Hi Zo and Trevor Hastie. Reglarization and variable selection via the elastic net. Jornal of the Royal Statistical Society, 67: , 2005.