The Complexity of Primal-Dual Fixed Point Methods for Ridge Regression

Transcription

1 The Complexity of Primal-Dual Fixed Point Methods for Ridge Regression Ademir Alves Ribeiro Peter Richtárik August 28, 27 Abstract We study the ridge regression L 2 regularized least squares problem and its dual, which is also a ridge regression problem. We observe that the optimality conditions describing the primal and dual optimal solutions can be formulated in several different but equivalent ways. The optimality conditions we identify form a linear system involving a structured matrix depending on a single relaxation parameter which we introduce for regularization purposes. This leads to the idea of studying and comparing, in theory and practice, the performance of the fixed point method applied to these reformulations. We compute the optimal relaxation parameters and uncover interesting connections between the complexity bounds of the variants of the fixed point scheme we consider. These connections follow from a close link between the spectral properties of the associated matrices. For instance, some reformulations involve purely imaginary eigenvalues; some involve real eigenvalues and others have all eigenvalues on the complex circle. We show that the deterministic Quartz method which is a special case of the randomized dual coordinate ascent method with arbitrary sampling recently developed by Qu, Richtarik and Zhang can be cast in our framework, and achieves the best rate in theory and in numerical experiments among the fixed point methods we study. Keywords. descent. Unconstrained minimization, primal-dual methods, ridge regression, gradient Introduction Given matrices A,..., A n R d m encoding n observations examples, and vectors y,..., y n R m encoding associated responses labels, one is often interested in finding a vector w R d such that, in some precise sense, the product A T i w is a good approximation of y i for all i. A fundamental approach to this problem, used in all areas of computational practice, is to formulate the problem as an L 2 -regularized least-squares problem, also known as ridge regression. In particular, we consider The results of this paper were obtained between October 24 and March 25, during AR s affiliation with the University of Edinburgh. School of Mathematics, University of Edinburgh, United Kingdom peter.richtarik@ed.ac.uk, supported by the EPSRC Grant EP/K2325X/, Accelerated Coordinate Descent Methods for Big Data Optimization. Department of Mathematics, Federal University of Paraná, Brazil ademir.ribeiro@ufpr.br, supported by CNPq, Brazil, Grants 285/24-3 and 39437/26-4.

2 the primal ridge regression problem def min P w w Rd 2n n i A T i w y i 2 + λ 2 w 2 2n AT w y 2 + λ 2 w 2, where λ > is a regularization parameter, denotes the standard Euclidean norm. In the second and more concise expression we have concatenated the observation matrices and response vectors to form a single observation matrix A [A, A 2,, A n ] R d N and a single response vector y y, y 2,, y n R N, where N nm. With each observation A i, y i we now associate a dual variable, α i R m. The Fenchel dual of is also a ridge regression problem: where α α, α 2,..., α n R N. def max Dα α RN 2 2 Aα 2 + n αt y 2n α 2, 2 Optimality conditions. The starting point of this work is the observation that the optimality conditions for the primal and dual ridge regression problems can be written in several different ways, in the form of a linear system involving the primal and dual variables. In particular, we find several different matrix-vector pairs M, b, where M R d+n d+n and b R d+n, such that the optimality conditions can be expressed in the form of a linear system as where x w, α R d+n. x Mx + b, 3 Fixed point methods. With each system 3 one can naturally associate a fixed point method performing the iteration x k+ Mx k + b. However, unless the spectrum of M is contained in the unit circle, such a method will not converge [?]. To overcome this drawback, we utilize the idea of relaxation. In particular, we pick a relaxation parameter θ and replace 3 with the equivalent system x G θ x + b θ, where G θ θi + θm and b θ θb. The choice θ recovers 3. We then study the convergence of the primal-dual fixed point methods x k+ G θ x k + b θ through a careful study of the spectra of the iteration matrices G θ. The central question of this work is: While all these formulations are necessarily algebraically equivalent, they give rise to different fixed-point algorithms. 2

3 . Contributions and literature review It is well known that the role of duality in optimization and machine learning is very important, not only from the theoretical point of view but also computationally [,, 5]. However, a more recent idea that has generated many contributions is the usage of the primal and dual problems together. Primal-dual methods have been employed in convex optimization problems where strong duality holds, obtaining success when applied to several types of nonlinear and nonsmooth functions that arise in various application fields, such as image processing, machine learning, inverse problems, among others [2, 7, 8]. On the other hand, fixed-point-type algorithms are classical tools for solving some structured linear systems. In particular, we have the iterative schemes developed by the mathematical economists Arrow, Hurwicz and Uzawa for solving saddle point problems [, 3]. In this paper we develop several primal-dual fixed point methods for the Ridge Regression problem. Ridge regression was introduced by Hoerl and Kennard [5, 6] as a regularization method for solving least squares problems with highly correlated predictors. The goal is to reduce the standard errors of regression coefficients by imposing a penalty, in the L 2 norm, on their size. Since then, numerous papers were devoted to the study of ridge regression or even for solving problems with a general formulation in which ridge regression is a particular case. Some of these works have considered its dual formulation, proposing deterministic and stochastic algorithms that can be applied to the dual problem [3, 4, 9,, 4, 5]. To the best of our knowledge, the only work that considers a primal-dual fixed point approach to deal with ridge regression is [2], where the authors deal with ill-conditioned problems. They present an algorithm based on the gradient method and an accelerated version of this algorithm. Here we propose methods based on the optimality conditions for the problem of minimizing the duality gap between the ridge regression problems and 2 in different and equivalent ways by means of linear systems involving structured matrices. We also study the complexity of the proposed methods and prove that our main method achieves the optimal accelerated Nesterov rate..2 Outline In Section 2 we formulate the optimality conditions for the problem of minimizing the duality gap between and 2 in two different, but equivalent, ways by means of linear systems involving structured matrices. We also establish the duality relationship between the problems and 2. In Section 3 we describe a family of parameterized fixed point methods applied to the reformulations for the optimality conditions. We present the convergence analysis and complexity results for these methods. Section 4 brings the main contribution of this work, with an accelerated version of the methods described in Section 3. In Section 5 we discuss some variants of our accelerated algorithm. Finally, in Section 6 we perform some numerical experiments and present concluding remarks. We left the proofs of the results of this paper to the appendix. 2 Separable and Coupled Optimality Conditions Defining x w, α R d+n, our primal-dual problem consists of minimizing the duality gap between the problems and 2, that is def min fx P w Dα. 4 x Rd+N 3

4 This is a quadratic strongly convex problem and therefore admits a unique global solution x R d+n. 2. A separable system P w Note that fx Dα, where P w n AAT w y + λw and Dα 2 AT Aα n α + y. 5 n So, the first and natural way of writing the optimality conditions for problem 4 is just to set the expressions given in 5 equal to zero, which can be written as w α 2.2 A coupled system AA T A T A w α + Ay y. 6 In order to derive the duality between and 2, as well as to reformulate the optimality conditions for problem 4, note that P w n φ i A T i w + λgw, 7 n where φ i z 2 z y i 2 and gw 2 w 2. i Now, recall that the Fenchel conjugate of a convex function ξ : R l R is ξ : R l R { } defined by ξ u def sup{s T u ξs}. s R l Note that if ξ is strongly convex, then ξ u < for all u R l. Indeed, in this case ξ is bounded below by a strongly convex quadratic function, implying that the sup above is in fact a max. It is easily seen that φ i s 2 s 2 + s T y i and g u 2 u 2. Furthermore, we have If we write the duality gap can be written as Dα λg n A i α i n i ᾱ def Aα P w Dα λ gw + g ᾱ w T ᾱ + n and the weak duality follows immediately from the fact that n φ i α i. 8 i n A i α i, 9 i n i φ i A T i w + φ i α i + αi T A T i w gw + g ᾱ w T ᾱ and φ i A T i w + φ i α i + α T i A T i w. 4

5 Strong duality occurs when these quantities vanish, which is precisely the same as or, equivalently, w g ᾱ and α i φ i A T i w ᾱ gw and A T i w φ i α i. Therefore, another way to see the optimality conditions for problem 4 is by the relations This is equivalent to 2.3 Compact form w α w ᾱ Aα and α y AT w. A A T w α + y. Both reformulations of the optimality conditions, 6 and, can be viewed in the compact form x Mx + b, 2 for some M R d+n d+n and b R d+n. Let us denote M AA T A T and M A 2 A A T 3 the matrices associated with the optimality conditions formulated as 6 and, respectively. Also, let b Ay and b y 2. 4 y Thus, we can rewritte 6 and as respectively. 3 Primal-Dual Fixed Point Methods x M x + b and x M 2 x + b 2, 5 A method that arises immediately from the relation 2 is given by the scheme x k+ Mx k + b. However, unless the spectrum of M is contained in the unit circle, this scheme will not converge. To overcome this drawback, we utilize the idea of relaxation. More precisely, we consider a relaxation parameter θ and replace 2 with the equivalent system x θx + θmx + b. Note that the choice θ recovers 2. The proposed algorithm is then given by the following framework. 5

6 Algorithm 3. Primal-Dual Fixed Point Method input: matrix M R d+n d+n, vector b R d+n, parameter θ > starting point: x R d+n repeat for k,, 2,... Set x k+ θx k + θmx k + b As we shall see later, the use of the relaxation parameter θ enables us to prove convergence of Algorithm 3. with M M and b b or M M 2 and b b 2, chosen according to 3 and 4, independent of the spectral radius of these matrices. Let us denote Gθ θi + θm 6 and let x be the solution of the problem 4. Then x Mx + b with M M and b b or M M 2 and b b 2. Therefore, x Gθx + θb. Further, the iteration of Algorithm 3. can be written as x k+ Gθx k + θb. Thus, x k x Gθ k x x 7 and consequently the convergence of the algorithm depends on the spectrum of Gθ. More precisely, it converges if the spectral radius of Gθ is less than, because in this case we have Gθ k. In fact, we will address the following questions: What is the range for θ so that this scheme converges? What is the best choice of θ? What is the rate of convergence? How the complexity of this algorithm compares with the known ones? 3. Convergence analysis In this section we study the convergence of Algorithm 3. and answer the questions raised above. To this end we point out some properties of the iteration matrices and uncover interesting connections between the complexity bounds of the variants of the fixed point scheme we consider. These connections follow from a close link between the spectral properties of the associated matrices. For this purpose, let A UΣV T 8 be the singular value decomposition of A. That is, U R d d and V R N N are orthogonal matrices and Σ p Σ d p 9 p N p 6

7 where Σ diagσ,..., σ p brings the nonzero singular values σ σ p > of A. The next result is crucial for the convergence analysis and complexity study of Algorithm 3.. Lemma 3. The characteristic polynomials of the matrices M and M 2, defined in 3, are respectively. p t t N+d 2p p j t + σ2 j 2 and p 2 t t N+d 2p p j t 2 + σ2 j, The following result follows directly from Lemma 3. and the fact that M is symmetric. Corollary 3.2 The spectral radii of M and M 2 are, respectively, ρ M σ2 A 2 and ρ 2 σ A. From Corollary 3.2 we conclude that if σ <, then ρ ρ 2 <. So, M k and M k 2, which in turn implies that the pure fixed point method, that is, Algorithm 3. with θ, converges. However, if σ, we cannot guarantee convergence of the pure method. Now we shall see that Algorithm 3. converges for a broad range of the parameter θ, without any assumption on σ, λ or n. We begin with the analysis of the framework that uses M and b, defined in 3 and Fixed Point Method based on M Algorithm 3.2 Primal-Dual Fixed Point Method; M M input: M M, b b, parameter θ > starting point: x R d+n repeat for k,, 2,... Set x k+ θx k + θmx k + b Theorem 3.3 Let x R d+n be an arbitrary starting point and consider the sequence x k k N 2 generated by Algorithm 3.2 with θ, + σ 2. Then the sequence x k converges to the unique solution of the problem 4 at a linear rate of { ρ θ def max θ + σ2, θ} Furthermore, if we choose θ it is equal to def σ 2, then the theoretical convergence rate is optimal and ρ def σ σ 2 7 θ.

8 The top picture of Figure illustrates the eigenvalues of G θ def θi + θm magenta together with the eigenvalues of M blue, for a fixed value of the parameter θ. The one farthest from the origin is θ θσ2 or θ. On the bottom we show the two largest in absolute value eigenvalues of G θ corresponding to the optimal choice of θ. Figure : Eigenvalues of G θ and M. Now we analyze the fixed point framework that employs M 2 and b 2, defined in 3 and Fixed Point Method based on M 2 Algorithm 3.3 Primal-Dual Fixed Point Method; M M 2 input: M M 2, b b 2, parameter θ > starting point: x R d+n repeat for k,, 2,... Set x k+ θx k + θmx k + b Theorem 3.4 Let x R d+n be an arbitrary starting point and consider the sequence x k k N 2 generated by Algorithm 3.3 with θ, + σ 2. Then the sequence x k converges to the unique solution of the problem 4 at an asymptotic convergence rate of Furthermore, if we choose θ 2 is equal to def ρ 2 θ def θ 2 + θ2 σ 2. + σ 2, then the theoretical convergence rate is optimal and it ρ 2 def σ + σ 2 θ 2. The left picture of Figure 2 illustrates, in the complex plane, the eigenvalues of G 2 θ def θi + θm 2 magenta together with the eigenvalues of M 2 blue, for a fixed value of the parameter θ. On the right we show, for each θ,, one of the two eigenvalues of G 2 θ farthest from the origin. The dashed segment corresponds to the admissible values for θ, that is, the eigenvalues with modulus less than one. The square corresponds to the optimal choice of θ. 8

9 Figure 2: Eigenvalues of G 2 θ and M Comparison of the rates We summarize the discussion above in Table which brings the comparison between the pure θ and optimal θ θj, j, 2 versions of Algorithm 3.. We can see that the convergence rate of the optimal version is times the one of the pure version if M 2+σ 2 is employed and times the pure version when using M 2. Moreover, in any case, employing M provides +σ 2 faster convergence. This can be seen in Figure 3, where Algorithm 3. was applied to solve the problem 4. The dimensions considered were d 2, m and n 5 so that the total dimension is d + N d + nm 52. We also mention that the pure version does not require the knowledge of σ, but it may not converge. On the other hand, the optimal version always converges, but θ depends on σ. Range for θ MFPM, θ MFPM 2, θ 2 2, + σ 2, + σ 2 Pure θ σ 2 Optimal θ θ j σ σ 2 θ σ 2 + σ 2 σ θ 2 Table : Comparison between the convergence rates of pure and optimal versions of Algorithm Direct relationship between the iterates of the two methods Another relation regarding the employment of M or M 2 in the pure version of Algorithm 3., which is also illustrated in Figure 3, is that one step of the method with M corresponds exactly to 9

10 2 MFPM,θ MFPM2,θ MFPM,θopt MFPM2,θopt Figure 3: Performance of pure and optimal versions of Algorithm 3.. two steps of the one with M 2. Indeed, note first that M2 2 M. Thus, denoting the current point by x and the next iterate by x + M, in view of 5 we have x ++ M 2 M 2 x + M 2 + b 2 M 2 M 2 x + b 2 + b 2 M x + M 2 b 2 + b 2 M x + b x + M. In Section 4 we shall see how this behavior can invert with a small change in the computation of the dual variable. 3.6 Complexity results In order to establish the complexity of Algorithm 3. we need to calculate the condition number of the objective function, defined in 4. Note that the Hessian of f is given by 2 f AA T + I n AT A + I Let us consider two cases: If, then σ 2 + σ2 + σ2 +, which in turn implies that the largest eigenvalue

11 of 2 f is L σ2 +. The smallest eigenvalue is n n, if d < N σd n, if d N { min λ, σ2 N 2 + n }, if d > N. Therefore, if d < N, the condition number of 2 f is the condition number of 2 f is σ If <, then σ 2 + < σ2 + < σ2 +, which in turn implies that the largest eigenvalue of 2 f is L σ The smallest eigenvalue is { σ 2 min d n + λ, }, if d < N n σ 2 d + λ, if d N n λ, if d > N. So, assuming that d < N, the condition number is deficient, then the condition number is σ 2 { + σ 2 2 min d n + λ, }. If A is rank n σ We stress that despite the analysis was made in terms of the sequence x k w k, α k, the linear convergence also applies to objective values. Indeed, since f is L-smooth, we have fx k fx + fx T x k x + L 2 xk x 2 L 2 xk x 2, where the equality follows from the fact that the optimal objective value is zero. Therefore, if we want to get fx k fx < ε and we have linear convergence rate ρ on the sequence x k, then it is enough to enforce L 2 ρ2k x x 2 < ε, or equivalently, k > 2 log ρ log x x 2 L 2ε. 22

12 Using the estimate log θ θ, we can approximate the second hand side of 22 by x x 2 L log, 23 2ε in the case M is used and by 2θ θ 2 x x 2 L log, 24 2ε if we use M 2. In order to estimate the above expressions in terms of the condition number, let us consider the more common case. Then the condition number of 2 f is given by 2, that is, So, if we use M, the complexity is proportional to κ def σ θ σ κ If we use M 2, the complexity is proportional to θ 2 + σ2 κ Accelerated Primal-Dual Fixed Point Method Now we present our main contribution. When we employ Algorithm 3.3, the primal and dual variables are mixed in two equations. More precisely, in view of 9 the iteration in this case can be rewritten as { w k+ θw k + θᾱ k α k+ θα k + θy A T w k. The idea here is to apply block Gauss-Seidel to this system. That is, we use the freshest w to update α. Let us state formally the method by means of the following framework. Algorithm 4. Accelerated Fixed Point Method input: matrix A R d N, vector y R N, parameter θ, ] starting points: w R d and α R N repeat for k,, 2,... Set w k+ θw k + θᾱ k Set α k+ θα k + θy A T w k+ Due to this modification, we can achieve faster convergence. This algorithm is a deterministic version of a randomized primal-dual algorithm Quartz proposed and analyzed by Qu, Richtárik and Zhang [8]. 2

13 4. Convergence analysis In this section we study the convergence of Algorithm 4.. We shall determine all values for the parameter θ for which this algorithm converges as well as the one giving the best convergence rate. To this end, we start by showing that Algorithm 4. can be viewed as a fixed point scheme. Then we determine the dynamic spectral properties of the associated matrices, which are parameterized by θ. First, note that the iteration of our algorithm can be written as w k+ I θa T I or in a compact way as with I G 3 θ θa T I and α k+ θi θ A θi θ A θi w k α k + θy x k+ G 3 θx k + f 28 θi I f θa T I θy θi + θ θy. A θ A T θ AT A We know that if the spectral radius of G 3 θ is less that, then the sequence defined by 28 converges. Indeed, in this case the limit point is just x, the solution of the problem 4. This follows from the fact that x G 3 θx + f. Next lemma provides the spectrum of G 3 θ. Lemma 4. The eigenvalues of the matrix G 3 θ, defined in 29, are given by } {2 θ θ 2 σj 2 ± θσ j θ 2 2 σj 2 4 θ, j,..., p { θ}. Figure 4 illustrates, in the complex plane, the spectrum of the matrix G 3 θ for many different values of θ. We used n 25, d 3, m therefore N 25, λ.3 and a random matrix A R d N. The pictures point out the fact that for some range of θ the spectrum is contained in a circle and for other values of θ some of the eigenvalues remain in a circle while others are distributed along the real line, moving monotonically as this parameter changes. These statements will be proved in the sequel. In what follows, let us consider the functions δ j : [, ] R defined by 29 δ j θ θ 2 σ 2 j 4 θ. 3 The following result brings some basic properties of them, illustrated in Figure 5. Lemma 4.2 Each function δ j, j,..., p, is strictly increasing, from 4 to σj 2 as θ goes from zero to. Furthermore, these functions are sorted in decreasing order, δ δ 2 δ p, and their zeros, σ def j 2 θ j, 3 are sorted in increasing order: < θ θ 2 θ p <. 3 σ 2 j

14 θ θ.25 θ.4 θ.45 θ θ.4755 θ θ θ.5659 θ θ θ θ.6275 θ.6775 θ.6587 Figure 4: The spectrum of G 3 θ for many different values of θ. The first 4 pictures satisfying the condition θ 2 σ 2 4 θ < ; in the fifth picture we have θ2 σ 2 4 θ and the remaining ones represent the case where θ 2 σ 2 4 θ >. 4

15 Proof. It is straightforward δ δ 2 δ 3 δp Figure 5: The functions δ j, j,..., p and the properties stated in Lemma 4.2. Now we shall study the spectrum of G 3 θ, given in Lemma 4.. For this, let us denote and, for j,..., p, λ j λ + j where δ j is defined in 3. λ θ def θ 32 def θ 2 θ θ 2 σ 2 j θσ j δj θ, 2 33 def θ 2 θ θ 2 σ 2 j + θσ j δj θ 2 34 Lemma 4.3 Consider θ as defined in 3. If θ [, θ ], then λ + j θ λ j θ θ for all j,..., p, which in turn implies that the spectral radius of G 3 θ is θ. It can be shown that the parameter θ θ, defined in Theorem 3.3, satisfies the conditions of Lemma 4.3. So, the spectral radius of G 3 θ is θ, exactly the same spectral radius of G θ θ I + θ M. This is shown in Figure 6, together with the spectrum of G 2 θ2 θ2 I + θ 2 M 2. We also show in this figure the right picture the spectrum of G 3 θ3, where θ 3 is the optimal parameter. This parameter will be determined later, in Theorem 4.6. Lemma 4.4 Consider θ j, j,..., p, as defined in 3. If θ [ θ l, θ l+ ], then the eigenvalues λ + j θ and λ j θ, j,..., l, are real numbers satisfying λ θ λ l θ θ λ+ l θ λ+ θ. On the other hand, for j l +,..., p we have Thus, the spectral radius of G 3 θ is λ θ. λ + j θ λ j θ θ 5

16 Figure 6: On the left, the spectrum of G θ magenta, G 2θ2 blue and G 3θ red. On the right, the spectrum of G 3 θ3, where θ 3 is the optimal parameter. From Lemmas 4.3 and 4.4 we can conclude that θ is the threshold value for θ after which the eigenvalues of G 3 θ start departing the circle of radius θ. The next result presents the threshold after which the eigenvalues are all real. Lemma 4.5 Consider θ p as defined in 3. If θ θ p, then the eigenvalues λ + j θ and λ j θ, j,..., p, are real numbers satisfying λ θ λ p θ θ λ + p θ λ + θ. Thus, the spectral radius of G 3 θ is λ θ. Proof. The same presented for Lemma 4.4. Using the previous results, we can finally establish the convergence of Algorithm 4. Deterministic Quartz. Theorem 4.6 Let w R d and α R N be arbitrary and consider the sequence w k, α k k N 2 generated by Algorithm 4. with θ,. Then the sequence w k, α k converges to the + σ unique solution of the problem 4 at an asymptotic linear rate of θ, if θ, θ ] where ρ 3 θ def Furthermore, if we choose θ3 to 2 θσ δ θ + θ 2 σ 2 2 θ, if θ θ, θ σ 2. σ 2 def θ, then the theoretical convergence rate is optimal and it is equal ρ 3 def θ 3. 6

17 It is worth noting that if the spectral radius of M is less than, that is, if σ 2 <, then Algorithms 3.2, 3.3 and 4. converge for any choice of θ, ]. Indeed, in this case we have 2 + σ 2 > 2 + σ >, which implies that the set of admissible values for θ established in Theorems 3.3, 3.4 and 4.6 contains the whole interval, ]. On the other hand, if σ 2, the convergence of these algorithms is more restrictive. Moreover, in this case we have 2 + σ 2 2, + σ which means that Algorithm 4. has a broader range for θ than Algorithms 3.2 and Complexity results Taking into account 22, 25, the relation log θ θ and Theorem 4.6, we conclude that the complexity of our Accelerated Fixed Point Method, Algorithm 4., is proportional to 2θ 3 σ σ 2 κ 4 κ. 35 Note that in the case when λ /n, as is typical in machine learning applications, we can write 35 κ κ + 4 κ. 4 2θ 3 This is very surprising as it means that we are achieving the optimal accelerated Nesterov rate Õ κ. 5 Extensions In this section we discuss some variants of Algorithm 4.. The first one consists of switching the order of the computations, updating the dual variable first and then the primal one. The second approach updates the primal variable enforcing the first relation of the optimality conditions given by and using the relaxation parameter θ only to update the dual variable. 5. Switching the update order This approach updates the dual variable α first and then updates the primal variable w using the new information about α. This is summarized in the following scheme. α k+ θα k + θy A T w k w k+ θw k + θ Aαk

18 As we shall see now, this scheme provides the same complexity results as Algorithm 4.. To see this, note that the iteration 36 is equivalent to I θ A w k+ θi w k I α k+ θa T θi α k + θy or in a compact way, with Gθ I θ A I θi θa T x k+ Gθx k + f 37 θi θi + θ θ AAT θ A A T. 38 It can be shown that the matrix Gθ has exactly the same spectrum of G 3 θ, defined in 29. So, the convergence result is also the same, which we state again for convenience. Theorem 5. Let w R d and α R N be arbitrary and consider the sequence w k, α k k N 2 defined by 36 with θ,. Then the sequence w k, α k converges to the unique + σ solution of the problem 4 at an asymptotic linear rate of θ, if θ, θ ] ρ 3 θ where 2 θσ δ θ + θ 2 σ 2 2 θ, if θ θ, θ σ 2. Furthermore, if we choose θ 3 θ, then the theoretical convergence rate is optimal and it is equal to ρ 3 θ Maintaining primal-dual relationship The second approach updates the primal variable enforcing the first relation of the optimality conditions given by and uses the relaxation parameter θ only to update the dual variable, as described in the following scheme. σ 2 w k+ Aαk α k+ θα k + θy A T w k+. Differently from the previous case, this scheme cannot achieve accelerated convergence. As we did before, note that the scheme 39 can be written as I w k+ θa T I α k+ A w k θi α k + θy 8 39

19 or in a compact way, with I Gθ θa T I x k+ Gθx k + f 4 A θi We can conclude that the eigenvalues of this matrix are { θ θσ2 j, j,..., p A θi θ AT A } { θ},. 4 exactly the same of the matrix G θ, the iteration matrix of Algorithm 3. with employment of M. So, the complexity analysis here is the same as that one established in Theorem Maintaining primal-dual relationship 2 For the sake of completeness, we present next the method where we keep the second relationship intact and include θ in the first relationship. This leads to α k+ y A T w k w k+ θw k + θ 42 Aαk+. Here we obtain the same convergence results as the ones described in Section 5.2. In fact, the relations above can be written as I w k+ A T w k y I θ A α k+ θi α k + or in a compact way, with I A T Gθ I θ A θi We can conclude that the eigenvalues of this matrix are { x k+ Gθx k + f 43 θ θσ2 j, j,..., p } θi θ AAT { θ}, A T. 44 exactly the same of the matrix G θ, the iteration matrix of Algorithm 3. with employment of M. So, the complexity analysis here is the same as that one established in Theorem 3.3. Observe that in 39 we have On the other hand, in 42 we have w k+ φ α k and α k+ φ 2 θ, α k, w k+. α k+ φ 3 w k and w k+ φ 4 θ, w k, α k+. 9

20 It is worth noting that if we update the variables as or we obtain θ AAT θa T α k+ φ 2 θ, α k, w k and w k+ φ α k+ w k+ φ 4 θ, w k, α k and α k+ φ 3 w k+ θ A θi and θi θ A θa T θ AT A as the associated iteration matrices, respectively. Moreover, we can conclude that they also have the same spectrum of G θ. So, the complexity analysis is the same as that one established in Theorem Numerical Experiments In this section we present a comparison among the methods discussed in this work. Besides a table with the convergence rates and complexity bounds, we show here some numerical tests performed to illustrate the properties of Algorithms 3.2, 3.2 and 4. as well as of the extensions 36 and 39 applied to solve the primal-dual ridge regression problem stated in 4. We refer to Algorithm 4. as Quartz and the extensions 36 and 39 as New Quartz and Modified Quartz, respectively. We summarize the main features of these methods in Table 2 which brings the range for the parameter to ensure convergence, the optimal convergence rates, the complexity and the cost per iteration of each method. For instance, the two versions of Algorithm 3. have the same range for theta. The usage of M provides best convergence rate compared with using M 2. However, it requires more calculations per iteration: the major computational tasks to be performed are computation of the matrix-vector products AA T w and A T Aα, while the use of M 2 needs the computation of Aα and A T w. Surprisingly, Algorithm 4. has shown to be the best from both the theoretical point of view and the numerical experiments and with the same cost as the computation of Aα and A T w. We also point out that the modified Quartz, 39, did not have here the same performance as the randomized version studied in [8]. Figures 7 and 8 illustrate these features, showing the primal-dual objective values against the number of iterations. The dimensions considered were d, m and n 5. We adopted the optimal parameters associated with each method. Figure 7 compares the two variants of Algorithm 3. against Algorithm 4., while Figure 8 shows the three variants of algorithm Quartz and MFPM. We can see the equivalence between Quartz and New Quartz and also the equivalence between Modified Quartz and MFPM. 2

21 Range for θ Optimal rate MFPM, θ MFPM 2, θ Qtzθ NewQtzθ ModQtzθ , + σ 2, + σ 2,, 2, + σ + σ + σ 2 σ σ 2 σ 2 + σ 2 θ 3 θ 3 σ σ 2 Complexity Cost/iteration AA T w, A T Aα Aα, A T w Aα, A T w + A T w, Aα + Aα, A T w + Table 2: Comparison between the convergence rates and cost per iteration of the different versions of Algorithms 3. and 4.. MFPM MFPM2 Qtz Figure 7: Comparison among the two versions of Algorithm 3. and Deterministic Quartz. MFPM Qtz Qtzn Qtzm Figure 8: Comparison among the different versions of Deterministic Quartz and Algorithm

22 7 Conclusion In this paper we have proposed and analyzed several algorithms for solving the ridge regression problem. The proposed methods are based on the optimality conditions for the problem of minimizing the duality gap between the primal and dual ridge regression problems. In fact, we have described a family of parameterized fixed point methods applied to the reformulations for the optimality conditions. We have presented the convergence analysis and complexity results for these methods and proved that our main method achieves the optimal accelerated Nesterov rate. We have performed some numerical experiments to illustrate the properties of our algorithms. References [] K J Arrow and L Hurwicz. Gradient method for concave programming I: Local results. In K J Arrow, L Hurwicz, and H Uzawa, editors, Studies in Linear and Nonlinear Programming, pages Stanford University Press, Stanford, 958. [2] A Chambolle and T Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis., 4:2 45, 2. [3] M El-Dereny and N I Rashwan. Solving multicollinearity problem using ridge regression models. Int. J. Contemp. Math. Sci., 6:585 6, 2. [4] D M Hawkins and X Yin. A faster algorithm for ridge regression of reduced rank data. Comput. Statist. Data Anal., 42: , 22. [5] A E Hoerl. Application of ridge analysis to regression problems. Chem. Eng. Prog., 58:54 59, 962. [6] A E Hoerl and R W Kennard. Ridge regression: biased estimation for nonorthogonal problems. Technometrics, 2:55 67, 97. [7] N Komodakis and J C Pesquet. Playing with duality: An overview of recent primal-dual approaches for solving large-scale optimization problems. IEEE Signal Process. Mag., 326:3 54, 25. [8] Z Qu, P Richtárik, and T Zhang. Quartz: Randomized dual coordinate ascent with arbitrary sampling. In Adv. Neural Inf. Process. Syst. 28, pages , 25. [9] C Saunders, A Gammerman, and V Vovk. Ridge regression learning algorithm in dual variables. In Proceedings of the 5th International Conference on Machine Learning, pages Morgan Kaufmann Publishers Inc., 998. [] S Shalev-Shwartz and T Zhang. Stochastic dual coordinate ascent methods for regularized loss. J. Mach. Learn. Res., 4: , 23. [] S Shalev-Shwartz and T Zhang. Accelerated proximal stochastic dual coordinate ascent for regularized loss minimization. Math. Program., Ser. A, 55:5 45, 26. [2] T C Silva, A A Ribeiro, and G A Periçaro. A new accelerated algorithm for ill-conditioned ridge regression problems. Comp. Appl. Math., 27. To appear. 22

23 [3] H Uzawa. Iterative methods for concave programming. In K J Arrow, L Hurwicz, and H Uzawa, editors, Studies in Linear and Nonlinear Programming, pages Stanford University Press, Stanford, 958. [4] H D Vinod. A survey of ridge regression and related techniques for improvements over ordinary least squares. The Review of Economics and Statistics, 6:2 3, 978. [5] T Zhang. On the dual formulation of regularized linear systems with convex risks. Machine Learning, 46:9 29,

24 A Proofs of Results from Section 3 First, let us see a basic linear algebra result whose proof is straightforward by induction. Proposition A. Let Q j R l l, j,..., 4, be diagonal matrices whose diagonal entries are components of α, β, γ, δ R l, respectively. Then Q Q det 2 Q 3 Q 4 l α j δ j β j γ j. A. Proof of Lemma 3. Let c. From 8 and 9, we can write M W Σ W T and M 2 W Σ 2 W T, where W U V cσσ T, Σ cσ T Σ j and Σ 2 c Σ Σ p d p p N p The evaluation of p t detti M detti Σ is straightforward and p 2 t detti M 2 det The result then follows from Proposition A.. A.2 Proof of Theorem 3.3 ti c Σ ti Σ ti ti det ti c Σ Σ ti ti ti. p d p p N p We claim that the spectral radius of G θ def θi + θm is ρ θ and also coincides with G θ. Using Lemma 3., we conclude that the eigenvalues of this matrix are { } So, its spectral radius is θ θσ2 j, j,..., p { θ}. { max θ + σ2, θ} ρ θ. Since G θ is symmetric, this quantity coincides with G θ. Furthermore, the admissible values for θ, that is, the ones such that the eigenvalues have modulus less than one, can be found by solving θ + σ2 <, 24

25 which immediately gives < θ < 2 + σ 2. So, the linear convergence of Algorithm 3. is guaranteed 2 for any θ, + σ 2. Finally, note that the solution of min ρ θ θ> is achieved when yielding θ σ 2 θ + σ2 θ, and the optimal convergence rate ρ σ σ 2. A.3 Proof of Theorem 3.4 First, using Lemma 3., we conclude that the eigenvalues of G 2 θ def θi + θm 2 are { θ ± θσ } j i, j,..., p { θ}, where i. The two ones with largest modulus are θ ± θσ i see Figure 2. So, the spectral radius of G 2 θ is θ 2 + θ2 σ 2 ρ 2θ. Further, the values of θ for which the eigenvalues of G 2 θ have modulus less than one can be found by solving θ 2 + θ2 σ 2 < giving < θ < 2 + σ 2. The asymptotic convergence follows from the fact that G 2 θ k /k ρ 2 θ. Indeed, using 7 we conclude that x k x /k x x G 2 θ k /k ρ 2 θ. This means that given γ >, there exists k N such that x k x ρ 2 θ + γ k x x for all k k. Finally, the optimal parameter θ2 and the corresponding optimal rate ρ 2 obtained directly by solving min θ> θ2 + θ2 σ 2. can be 25

26 B Proofs of Results from Section 4 B. Proof of Lemma 4. Consider the matrix M 3 θ def A θ A T θ AT A Using the singular value decomposition of A, given in 8, we can write U M 3 θ V Σ θ Σ T θ ΣT Σ Therefore, the eigenvalues of M 3 θ are the same as the ones of Σ θ Σ T θ ΣT Σ. U T V T c Σ θ Σ θc Σ 2 p d p p N p. p d p p N p where c and Σ is defined in 9. The characteristic polynomial of this matrix is ti c Σ ti c Σ p θ t det ti θ Σ ti θc Σ 2 det θ Σ ti θc Σ 2 ti. ti ti Using Proposition A., we obtain p θ t t N+d 2p p j Thus, the eigenvalues of M 3 θ are p tt θcσj 2 c θσj 2 t N+d 2p j t 2 + θσ2 j t + θσ2 j } { θσj 2 ± σ j θ 2 2 σj 2 4 θ, j,..., p {}, so that the eigenvalues of G 3 θ θi + θm 3 θ are } {2 θ θ 2 σj 2 ± θσ j θ 2 2 σj 2 4 θ, j,..., p { θ}, giving the desired result.. 26

27 B.2 Proof of Lemma 4.3 Note that in this case we have δ j θ for all j,..., p. So, λ + j θ 2 λ j θ 2 4λ 2 n 2 2 θ θ 2 σ 2 j 2 θ 2 σ 2 j δ jθ 4λ 2 n 2 4 θ 2 λ 2 n 2 4 θθ 2 σj 2 + θ4 σj 4 θ2 σj 2 θ 2 σj 2 4 θ θ 2, yielding the desired result since θ. B.3 Proof of Lemma 4.4 We have δ j θ for all j,..., l. So, Furthermore, λ + j θ θ 2 2 δj θ + θσ j δj θ 2 δj θ θσj δ j θ. 2 2 θ θ 2 σj 2 + θσ j δj θ + θ 4 θ θ 2 σj 2 + θσ j δj θ 2 2. θσ j δj θ θ 2 σj θ 2 2 σj 2 4 θ θ 2 σj 2 2 θ Since θ 2 σ 2 j 2 θ δ jθ + 2 θ, Now, note that λ + j θ 2 θ θ 2 σ 2 j + θσ j δj θ. 2 λ j θ θ θ θ 2 σj 2 θσ j δj θ + θ δ j θ θσ j δj θ. Moreover, from Lemma 4.2 and the definition of σ j, we have δ θ δ l θ and σ σ l, which imply that λ θ λ l θ. The inequality λ+ l θ λ+ θ follows from the fact that the function [ a, s s 2 + s s 2 a is increasing. Finally, for j l +,..., p we have δ j θ and, by the same argument used in Lemma 4.3, we conclude that λ + j θ λ j θ θ. 27

28 B.4 Proof of Theorem 4.6 Since Algorithm 4. can be represented by 28, we need to show that ρg 3 θ, the spectral radius of G 3 θ, is less than. First, note that by Lemmas 4.3, 4.4 and 4.5, we have ρg 3 θ ρ 3 θ. Using Lemma 4.2 we conclude that the function θ ρ 3 θ is increasing on the interval [ θ,, which means that its minimum is attained at θ. To finish the proof, it is enough to prove that ρ 3 θ if and only if θ 2. + σ Note that and θ ρ 3 θ θσ δ θ + θ 2 σ 2 2 θ 2 2 θ 2 σ 2δ θ 22 θ θ 2 σ 2 2 θ θ σ θ 2 + σ 2 θ 2 σ 2 2 θ2 + σ completing the proof. θσ δ θ + θ 2 σ 2 2 θ 2, 28