arxiv: v1 [math.oc] 17 Nov 2017

Transcription

1 Optimal rates of linear convergence of the averaged alternating modified reflections method for two subspaces Francisco J Aragón Artacho Rubén Campoy November 3, 208 arxiv:70652v [mathoc] 7 Nov 207 Abstract The averaged alternating modified reflections AAMR method is a projection algorithm for finding the closest point in the intersection of convex sets to any arbitrary point in a Hilbert space This method can be seen as an adequate modification of the Douglas Rachford method that yields a solution to the best approximation problem In this paper we consider the particular case of two subspaces in a Euclidean space We obtain the rate of linear convergence of the AAMR method in terms of the Friedrichs angle between the subspaces and the parameters defining the scheme, by studying the linear convergence rates of the powers of matrices We further optimize the value of these parameters in order to get the minimal convergence rate, which turns out to be better than the one of other projection methods Finally, we provide some numerical experiments that demonstrate the theoretical results Keywords Best approximation problem Linear convergence Averaged alternating modified reflections method Linear subspaces Friedrichs angle MSC200: Introduction 65F0 65K05 65F5 5A08 47H09 90C25 4A25 The averaged alternating modified reflections AAMR algorithm, introduced in [2], is a projection method for solving best approximation problems in the convex setting A best approximation problem consists in finding the closest point to any given point in the intersection of a collection of sets In this work we study problems involving two subspaces U and V in R n Given a point z R n, the corresponding best approximation problem is defined as Find w U V such that w z = For any initial point x 0 R n, the AAMR algorithm is iteratively defined by inf x z x U V x k+ := αx k + α2βp V z I2βP U z Ix k, k = 0,, 2 2 When α, β ]0, [, the generated sequence {x k } k=0 converges to a point x such that P U x + z = P U V z, which solves problem Furthermore, the shadow sequence {P U x k + z} k=0 is convergent to the solution P U V z even if α =, see [2, Theorem 4] In fact, when the sets involved are subspaces, we prove that the sequence {x k } k=0 is also convergent for α =, see Corollary 3 Department of Mathematics, University of Alicante, Spain franciscoaragon@uaes Department of Mathematics, University of Alicante, Spain rubencampoy@uaes

2 Several projection methods have been developed for solving convex feasibility problems in Hilbert spaces, see eg [3, 6, 8,, 4] In the case where the sets are subspaces, some of these methods converge to the closest point in the the intersection to the starting point, providing thus a solution of the best approximation problem Among these schemes, probably the two most well-known are the method of alternating projections AP, which was originally introduced by John von Neumann [24], and the Douglas Rachford method DR [2, 20], which is also referred as averaged alternating reflections The rate of linear convergence of these methods is known to be the cosine of the Friedrichs angle between the subspaces for DR [4], and the squared cosine of this angle for AP [0] Several relaxations and generalizations of these methods have been proposed, such as the relaxed and the partial relaxed alternating projections RAP, PRAP [, 7, 23], the generalized alternating projections GAP [5, 7], the relaxed averaged alternating reflections RAAR [2], and the generalized Douglas Rachford GDR [3], among others We note that AAMR can also be seen as a modified version of DR, since both methods coincide when α = 2 and β = in 2 Thanks to the linearity of the projector operator onto subspaces, projection methods reduce to matrix iterations Taking advantage of this fact, optimal convergence rates have been obtained in [5] for RAP, PRAP and GDR By following an analogous matrix analysis, the rate of convergence with optimal parameters for GAP has been recently given in [6] In the current setting, the rate of convergence of the AAMR algorithm was numerically analyzed in various computational experiments in [2, Section 7] The goal of this work is to provide the theoretical results that substantiate the behavior of the algorithm that was numerically observed By following the same approach as in [5], we analyze the linear rate of convergence of the AAMR method by studying the convergence rates of powers of matrices The rate obtained depends on both the Friedrichs angle and the parameters defining the algorithm In addition, we also obtain the optimal selection of the parameters according to the Friedrichs angle, so that the rate of convergence is minimized This rate coincides with the one for GAP, which is the best among the rates of all the projection methods mentioned above This is not just by chance: the shadow sequences of GAP and AAMR coincide for linear subspaces under some conditions see Theorem 4 and Figure 4 The remaining of the paper is structured as follows We present some definitions and preliminary results in Section 2 In Section 3 we collect our main results regarding the rate of convergence of the AAMR method We compare the rate with optimal parameters of AAMR with the rate of various projection methods in Section 4 In Section 5 we perform two computational experiments that validate the theoretical results obtained We finish with some conclusions and future work in Section 6 2 Preliminaries In this work, our setting is the Euclidean space R n with inner product, and induced norm For a given set C R n we denote by C = {x H : c, x = 0, c C} the orthogonal complement of C Given x R n, a point p C is said to be a best approximation to x from C if p x = dx, C := inf c x c C The operator P C x := argmin { x c, c C} is called the projector onto C When C is closed and convex, P C is single-valued In the case when C is a subspace, P C x is sometimes called the orthogonal projection of x to C, due to the fact that x P C x C Throughout this paper, we assume without lost of generality that U and V are two subspaces of R n such that p := dim U dim V =: q n, with U U V and U V {0} otherwise, problem would be trivial We now recall the concept of principal angles and Friedrichs angle between a pair of subspaces, and a result relating both concepts 2

3 Definition 2 The principal angles between U and V are the angles 0 θ θ 2 θ p π 2 whose cosines are recursively defined by cos θ k := u k, v k = max { u, v : u U, v V, u = v =, u, u j = v, v j = 0 for j =,, k }, with u 0 = v 0 := 0 Definition 22 The Friedrichs angle between U and V is the angle in θ F ]0, π 2 ] whose cosine is { } c F U, V := sup u, v : u U U V, v V U V, u, v Fact 2 Let θ, θ 2,, θ p be the principal angles between U and V, and let s := dimu V Then we have θ k = 0 for k =,, s and θ s+ = θ F > 0 Proof See [5, Proposition 33] Remark 2 By our standing assumption that U U V, we have s = dimu V < p The projector operator onto subspaces is known to be a linear mapping The following result provides a matrix representation of the projectors onto U and V, according to their principal angles We denote by I n, 0 n and 0 m n, the n n identity matrix, the n n zero matrix, and the m n zero matrix, respectively For simplicity, we shall omit the subindices when the size can be deduced Fact 22 If p + q < n, we may find an orthogonal matrix D R n n such that I p C 2 CS 0 0 P U = D 0 0 p q p 0 D and P V = D CS S I q p 0 D, n p q n p q where C and S are two p p diagonal matrices defined by C := diagcos θ,, cos θ p and S := diagsin θ,, sin θ p, with θ,, θ p being the principal angles between U and V Proof See [5, Proposition 34] 2 The averaged alternating modified reflections operator for two subspaces The AAMR operator was originally introduced for two arbitrary closed and convex sets [2, Definition 32] In this section, we present the scheme in the case of two subspaces, as well as some properties of the operator and its set of fixed points within this context Definition 23 Given α ]0, ] and β ]0, [, the averaged alternating modified reflections AAMR operator is the mapping T U,V,α,β : R n R n given by T U,V,α,β := αi + α2βp V I2βP U I Where there is no ambiguity, we shall abbreviate the notation of the operator T U,V,α,β by T α,β Fact 23 Let α ]0, ] and β ]0, [ Then, the AAMR operator T α,β is nonexpansive Proof See [2, Proposition 33] 3

4 Proposition 2 Let α ]0, ] and β ]0, [ Then Proof Observe that Moreover, by [2, Proposition 34], Fix T U,V,α,β = U V x Fix T U,V,α,β P V 2βP U x x = P U x 4 P U x = P U V 0, for all x Fix T U,V,α,β Therefore, if x Fix T U,V,α,β, then 0 = P U V 0 = P U x Using this equality, together with 4, we deduce that P U x = P V x = 0, which implies x U V To prove the converse implication, pick any x U V Then we trivially have that P V 2βP U x x = P U x, and thus x Fix T U,V,α,β The AAMR scheme is iteratively defined by 2 Using the linearity of the projector operator onto subspaces, we deduce that the iteration takes the form x k+ = αx k + α2βp V z I2βP U z Ix k = αx k + α2βp V z I 2β P U x k + z z x k = αx k + α 2βP V z 2β P U x k + z z x k 2β P U x k + z z + x k = x k + 2αβ P V 2β P U x k + z z x k + z P U x k + z = x k + 2αβ 2βP V P U x k + z + 2βP V z P V x k P U x k + z 5 Fact 24 Let α ]0, ], β ]0, [ and z R n Then, one has Fix T U z,v z,α,β and Furthermore, for any x R n, Fix T U z,v z,α,β = x + U V, x Fix T U z,v z,α,β T U z,v z,α,β x = T U,V,α,β x x + x, x Fix T U z,v z,α,β Proof Since U and V are subspaces in a finite dimensional space, by [2, Fact 2 and Corollary 3] we get that Fix T U z,v z,α,β The remaining assertions are obtained by applying Proposition 2 and [2, Proposition 36] to U q,v q and q U q V q, noting that [2, Proposition 36] also holds for α = 22 Optimal convergence rate of powers of matrices We denote by C n n R n n, the space of n n complex real matrices, equipped with the induced matrix norm A := max { Ax : x C n, x } The kernel of a matrix A C n n is denoted by ker A := {x C n n : Ax = 0} and the set of fixed points of A is denoted by Fix A := kera I We say A is nonexpansive if Ax Ay x y for all x, y C n n Definition 24 A matrix A C n n is said to be convergent to A C n n if and only if lim k Ak A = 0 We say A is linearly convergent to A with rate µ [0, [ if there exist a positive integer k 0 and some M > 0 such that A k A Mµ k, for all k k 0 In this case, µ is called a linear convergence rate of A When the infimum of all the convergence rates is also a convergence rate, we say this minimum is the optimal linear convergence rate 4

5 For any matrix A C n n, we denote by σa the spectrum of A the set of all eigenvalues An eigenvalue λ σa is said to be semisimple if its algebraic multiplicity coincides with its geometric multiplicity cf [22, p 50], or, equivalently, if kera λi = ker A λi 2 see [5, Fact 23] The spectral radius of A is defined by ρa := max{ λ : λ σa}, and the second-largest modulus of the eigenvalues of A after is denoted by γa := max { λ : λ {0} σa \ {}} An eigenvalue λ σa with λ = γa is called a subdominant eigenvalue Fact 25 Let A C n n Then A is convergent if and only if one of the following holds: i ρa < ; ii ρa = and λ = is semisimple and is the only eigenvalue on the unit circle When this happens, A is linearly convergent with any rate µ ]γa, [, and γa is the optimal linear convergence rate of A if and only if all the subdominant eigenvalues are semisimple Furthermore, if A convergent and nonexpansive, then lim k A k = P Fix A Proof See [22, pp 67 68, 630] and [5, Theorem 22, Theorem 25 and Corollary 27ii] 3 Convergence rate analysis We begin this section with the following theorem that establishes the rate of convergence of the AAMR algorithm in terms of α, β and the Friedrichs angle between the subspaces We denote the positive part of x R by x + := max{0, x} Theorem 3 Let α ]0, ] and β ]0, [ Then, the AAMR operator T α,β := αi n + α2βp V I n 2βP U I n is linearly convergent to P U V γt α,β = with any rate µ ]γt α,β, [, where 4αβ β, if 0 c F < cα, β; 4 ααβ 2 c 2 F + 2αβ2, if cα, β c F < ĉ β ; + 2αβ βc 2 F + c F β 2 c 2F 2β +, if ĉ β c F < ; with c F := cos θ F and θ F being the Friedrichs angle between U and V, 2β + 4αβ β 2 2αβ 2 + ĉ β := and cα, β :=, if α < ; 4 ααβ β 2 0, if α = 6 7 Furthermore, γt α,β is the optimal linear convergence rate if and only if β +sin θ F or θ F = π 2 Proof To prove the result, we consider two main cases Case : p + q < n By Fact 22, we can find an orthogonal matrix D R n n such that 3 holds After some calculations, we obtain M α,β 0 0 T α,β = D 0 2αβI q p I n p q D, 8 5

6 where M α,β := 2αβ2β C 2 + 2αβI p 2αβ2β CS 2αβCS 2αβC 2 + 2αβI p Let s := dimu V and let = c = = c s > c s+ = c F c s+2 c p 0 be the cosine of the principal angles 0 = θ = = θ s < θ s+ = θ F θ s+2 θ p π 2 between U and V see Fact 2 By the block determinant formula see, eg, [8, 0853], we deduce after some algebraic manipulation that the spectrum of T α,β is given by p { } + 2αβ βc 2 k ± c k β 2 c 2k 2β + {}, if q = p; σt α,β = k= p { } + 2αβ βc 2 k ± c k β 2 c 2k 2β + {} { 2αβ}, if q > p k= Then λ k,r := + 2αβ βc 2 k + r c k β 2 c 2 k 2β + are eigenvalues of T α,β, with r =, 2 and k =,, p Observe that λ k,r R if c k β 2β + =: ĉ β, while λ r k C otherwise To study the modulus of the eigenvalues λ k,r, consider the function f α,β,r : [0, ] R given by + 2αββc 2 2 4α 2 β 2 c 2 β 2 c 2 2β +, if c < ĉ β ; f α,β,r c := + 2αβ βc 2 + r c 2 β 2 c 2 2β +, if c ĉβ Hence, one has λ k,r 2 = f α,β,r c k Let us analyze some properties of the function f α,β,r When ĉ β > 0, observe that f α,β,r is continuous at ĉ β, since lim c ĉ β f α,β,r c = lim c ĉ + β f α,β,r c = 2α β 2 Define the auxiliary function g β,r c := βc 2 + r c β 2 c 2 2β + for c ĉ β Then, { 4 ααβ 2 c 2 + 2αβ 2, if c < ĉ β ; f α,β,r c = + 2αβg β,r c 2, if c ĉ β The derivative of f α,β,r is given for c ĉ β by 8 ααβ 2 c, if c < ĉ β ; f α,β,r c = 2 β 4αβ + 2αβg β,r c r 2 c 2 2β++ βc r, if c > ĉ β β 2 c 2 2β+ Further, we claim that + 2αβg β,2 c > + 2αβg β, c 0 for all c > ĉ β Indeed, since 2β 2 c 2 2β + 2 = 4β 2 c 2 β 2 c 2 2β + + 2β 2 2βc 2 β 2 c 2 2β +, we deduce, after taking square roots and reordering, that 2β βc 2 c β 2 c 2 2β + = 2βg β, c < 2βg β,2 c, from where the assertion easily follows All the above properties of the function f α,β,r can be summarized as follows: For all 0 c < d ĉ β, F 0 α,β := 2αβ2 f α,β,r c f α,β,r d 2α β 2 6

7 For all ĉ β c < d, F α,β := 4αβ β2 f α,β, d f α,β, c 2α β 2 f α,β,2 c < f α,β,2 d In view of Fact 25, we have to show that the eigenvalue λ = is semisimple and the only eigenvalue in the unit circle According to the monotonicity properties of f α,β,r, we have that λ k,r for all k =,, p and r =, 2 Further, λ k,r = k {, 2,, s} and r = 2, in which case λ k,2 = Thus, we have shown that ρ T α,β = and λ = is the only eigenvalue in the unit circle Let us see now that λ = is semisimple First observe that, for any λ C, given the block diagonal structure of T α,β, one has ker T α,β λi = ker T α,β λi 2 ker M α,β λi = ker M α,β λi 2 Then, we can compute M α,β I = 2αβ 2β C 2 I p CS 2β CS S 2 Observe that the matrices C and S can be decomposed as I s 0 0 s 0 C = and S = 0 C 0 S, 9 where both C and S are diagonal matrices and S has strictly positive entries Hence, 2 βi s M α,β I = 2αβ 0 2β C 2 I p s 0 C S , 0 2β C S 0 S 2 and one has that ker M α,β I = ker M α,β I 2 if and only if ker M 0 = ker M0 2, where M 0 := 2β C 2 I p s C S 2β C S S 2 Since det M 0 = det S2 0 again, by the block determinant formula, we conclude that λ = is a semisimple eigenvalue Then, since T α,β is nonexpansive by Fact 23, we have by Fact 25 that T α,β is linearly convergent to P Fix Tα,β with any rate µ ]γt α,β, [, and Fix T α,β = U V by Proposition 2 Furthermore, we can also deduce from the monotonicity properties of f α,β,r that the subdominant eigenvalues of T α,β are determined by γt α,β = max { λ s+,2, λ, } { } = max + 2αβ βc 2F + c F β 2 c 2 F 2β +, 4αβ β 7

8 To prove 6, let us compute the value of γt α,β If c F > ĉ β, then λ, < λ s+,2 Otherwise, λ s+,2 λ, f α,β,2 c F f α,β, 4 ααβ 2 c 2 F F α,β F 0 α,β Consequently, if we define cα, β := F α,β F 0 α,β 4 ααβ 2, if F α,β > F 0 α,β ; 0, otherwise; which is equivalent to the expression in 7, we obtain 6 Three possible scenarios for f α,β,r and the constants cα, β and ĉ β depending on the values of α and β are shown in Figure 08 f α,β,2 c f α,β, c α = 05, β = F 0 α,β F α,β 02 y = f α,β, cα, β = ĉ β a 0 = cα, β = ĉ β < α = 095, β = 075 α = 035, β = 058 f α,β,r c, c < ĉ β f α,β,r c, c < ĉ β 08 f α,β,2 c, c ĉ β 08 f α,β,2 c, c ĉ β f α,β, c, c ĉ β f α,β, c, c ĉ β F α,β F 0 α,β y = f α,β, F 0 α,β F α,β cα, β y = f α,β, ĉ β b 0 = cα, β < ĉ β < cα, β ĉ β 08 c 0 < cα, β < ĉ β < Figure : The three possible scenarios for the function f α,β,r c To conclude the proof, let us see that the subdominant eigenvalues are semisimple if and only if β 2 c 2 F 2β + 0 or c F = 0 The candidate eigenvalues to be subdominant are λ, and λ s+,2, possibly simultaneously Consider first the case where λ, = 4αβ β is subdominant, and compute 2β S 2 CS M α,β λ, I = 2αβ 2β CS C 2 2β I p 8

9 Using the decomposition of C and S given in 9, we get M α,β λ, I = 2αβ 0 2β S 2 0 C S βi s 0, 0 2β C S 0 C2 2β I p s and one has that ker M α,β λ, I = ker M α,β λ, I 2 if and only if ker M = ker M 2, where M := 2β S 2 C S 2β C S C2 2β I p s Since we are assuming that λ, is subdominant, it necessarily holds that 2 det M = det 2β 2 S2 0, < β < Hence, and one trivially has that ker M = ker M 2, which proves that λ, is semisimple Consider now the case where λ s+,2 = + 2αβ βc 2 F + c F β 2 c 2F 2β + is a subdominant eigenvalue Denote by F := β 2 c 2 F 2β + and compute M α,β λ s+,2 I = 2αβ 2β C 2 c F βc F + F I p CS 2β CS C 2 c F βc F + F I p Let k {,, p s} be such that c F = c s+ = c s+2 = = c s+k > c s+k+ Then C = I s c F I k C and S = 0 s s F I k S where both C and S are diagonal matrices and C has entries strictly smaller than c F Hence, one has m m m m m 36 M F := M α,β λ s+,2 I = 2αβ, m m m m m 6 where m := 2β c F βc F + F I s, m 2 := c F F + βc F I k, m 3 := 2β C 2 c F βc F + F I p k s, m 4 := c F βc F + F I s, m 5 := c F F βc F I k, m 6 := C 2 c F βc F + F I p k s, m 25 := c F s F I k, m 36 := C S, m 52 := 2β c F s F I k and m 63 := 2β C S m 2 m 25 m 3 m 36 Thus, if we denote by M {2,5} := and by M {3,6} :=, we m 52 m 5 m 63 m 6 get that ker M F = ker MF 2 ker M{2,5} = ker M{2,5} 2, and ker M {3,6} = ker M 2 {3,6} 9

10 On the one hand, by the block determinant formula we have that det M {3,6} = detm3 m 6 m 63 m 36 = det 2β C 4 + c 2 F βc F + F 2 I p k s 2βc F βc F + F C 2 + 2β C 2 S2 = det 2β 2βc F βc F + F C 2 + c 2 F βc F + F 2 I p k s = det β 2 c 2 F 2β + β 2 c 2 F 2βc F F C 2 + c 2 F βc F + F 2 I p k s = det βc F + F 2 C2 c 2 F I p s k Observe that βc F + F = 0 if and only if β = 2 and c F = 0, in which case M {3,6} = 0 2p k s If βc F + F 0, then det M {3,6} 0 Thus, in either case, we get ker M{3,6} = ker M{3,6} 2 On the other hand, we can rewrite F + βc F I k s F I k M {2,5} = c F, 2β s F I k F βc F I k and one has M 2 {2,5} = c2 F Observing that F + βc F 2 2β s 2 F Ik 2 F s F I k 2 F 2β s F I k F βc F 2 2β s 2 F Ik F βc F 2 2β s 2 F = 2 F F βc F, F + βc F 2 2β s 2 F = 2 F F + βc F, we deduce that M{2,5} 2 = 2 F c F M {2,5} If c F = 0, then M {2,5} = 0 2k Therefore, ker M {2,5} = ker M{2,5} 2 if and only if F 0 or c F = 0 Summarizing the discussion above, we have shown that ker M F = ker M 2 F F 0 or c F = 0 Finally, observe that F = 0 if and only if c F = ĉ β, in which case λ s+, is a subdominant eigenvalue, and this proves the last assertion in the statement Case 2: p + q n We can take some k such that n := n + k > p + q, and consider U := U {0 k } R n, V := V {0 k } R n, and T α,β := T U,V,α,β = αi + α2βp V I2βP U I Since P U = P U 0 P V 0 and P V = 0 0 k 0 0 k, it holds that T α,β = T α,β 0 0 I k Therefore, σt α,β {} = σt α,β and γt α,β = γt α,β Note that the principal angles between U and V are the same that the ones between U and V Hence, the result follows from applying Case to T α,β Remark 3 For simplicity, we have assumed that dim U = p q = dim V If this is not the case and q < p, observe that one has to exchange the matrix decomposition of P U and P V given in 3 In this case, one can check that the matrix T α,β obtained corresponds to the transpose of the one given in 8 Hence, the spectrum of T α,β remains the same and thus all the results in Theorem 3 also hold 0

11 Remark 32 The expression in 6 corroborates what it was numerically observed in [2]: there are values of α and β for which the rate of convergence of AAMR does not depend on the value of the Friedrichs angle for all angles larger than arccos cα, β We now look for the values of the parameters α and β and V, in order to that minimize the rate of convergence of the AAMR method obtained in Theorem 3 Theorem 32 The infimum of the linear convergence rates of the AAMR operator T α,β attains its smallest value at α = and β = +sin θ F, where θ F is the Friedrichs angle between U and V ; ie, it holds sin θ F + sin θ F = γ T,β γ T α,β for all α, β ]0, ] ]0, [ Furthermore, γ T,β is an optimal linear convergence rate if and only if θ F = π 2 Proof Let us look for the values of parameters α and β that minimize the rate γt α,β given by 6 Define the sets D := ]0, ] ]0, [, { D := α, β D : β < }, + s F { } D 2 := α, β D : β + s F + s 2 or α β + s2 F F β, 4 β 2 s 2 β > F + s 2, F { } D 3 := α, β D : α < β + s2 F β, 4 β 2 s 2 β > F + s 2, F and the functions Γ α, β := + 2αβ βc 2F + c F β 2 c 2F 2β +, for α, β D, Γ 2 α, β := 4 ααβ 2 c 2 F + 2αβ2, for α, β D, Γ 3 α, β := 4αβ β, for α, β D, having D = D D 2 D 3 Hence, we can define the convergence rate in terms of the parameters α and β through the function Γ α, β, if α, β D, Γα, β := γt α,β = Γ 2 α, β, if α, β D 2, Γ 3 α, β, if α, β D 3, see Figure 2 The function Γ is piecewise defined, continuous and differentiable on the interior of each of the three regions D, D 2 and D 3, but is not differentiable on the boundaries Let us analyze the three problems of minimizing the function Γ over the closure of each of the three pieces The gradient of the functions Γ, Γ 2 and Γ 3 are given by 2β βc 2 F + c F β 2 c 2F 2β + Γ α, β = 2α βc 2 F + c F β 2 c 2 F 2β + βc F + β 2 c 2 F 2β+, β 2 c 2 F 2β+ 2β βc 2 F Γ 2 α, β = + 2αβ c2 F 4 ααβ 2 c 2 F + 2αβ2 2α 2βc 2 F + 2αβ c2 F, 4β β Γ 3 α, β = 4α 2β

12 D D 2 D 3 β = β = + s F + s 2 F αβ 4 β 2 s 2 F = β + s 2 F α α α β a Friedrichs angle: π 6 radians β b Friedrichs angle: π 3 radians β c Friedrichs angle: π 2 radians Figure 2: Piecewise domain of the function Γα, β for three different values of the Friedrichs angle To minimize Γ over D, we assert that Γ α α, β < 0, Γ β α, β < 0, for all α, β D 0 Indeed, on the one hand, since c 2 F <, then c 2 F β 2 c 2 F 2β + = β 2 c 4 F 2βc 2 F + c 2 F < β 2 c 4 F 2βc 2 F + = βc 2 2 F Thus, taking square roots and reordering, we get that βc 2 F + c F the other hand, one has βc F + β 2 c 2 F 2β + < 0 On β 2 c 2 F 2β + 0, with equality if and only if c F = 0 and β = 2 In this case, the point has the form α, 2 / D for α ]0, ] We have therefore shown that 0 holds, and thus the unique minimum of Γ over D is attained at, +s F Let us consider now the problem of minimizing Γ over D 2 To address this problem, we consider two cases Suppose first that c F = 0 and observe that Γ 2 α, β = 2αβ 2 0, for all α, β D 2, having Γ 2 α, β = 0 if and only if 2αβ = Since, 2 is the only point in D2 satisfying this equation, we deduce that it is the unique minimum Suppose now that c F [ > 0 In] this case, we claim that Γ 2 attains its minimimum over the region D 2 D 3 = [0, ] +s F, at the point, +s F D 2, and so does Γ over D 2 Indeed, observe that Γ 2 is smooth on the interior of the set D 2 D 3 Moreover, Γ 2 only vanishes at 0, 0 Therefore, [ the ] minimum has to be attained at some point in the boundary Note that, for all β +s F,, the following holds: i Γ 2 0, β = ii Γ 2, β = 2β, which attains its minimum at β = +s F iii The function α Γ 2 α, β is the square root of a positive non-degenerated convex parabola, α Γ 2 α, β = 4β 2 s 2 F α2 4β βc 2 F α + ; which attains its minimum at α β := βc2 F 2βs 2 F Γ 2, < Γ 2 α, + s F + s F 2 Since α +s F = +s F 2s F, for all α [0, ], we have

13 On the other hand, α = 2, which implies Γ 2 2, < Γ 2 α,, for all α [0, ] Then, noting that Γ 2, + s F = s F < c F = Γ 2 + s F 2,, we have shown by i iii that Γ 2 attains its minimum over D 2 D 3 at claimed Finally, observe that if α, β D 3, it holds that 2 < β < Then, +s F D 2, as Γ 3 α α, β < 0, Γ 3 β α, β > 0, for all α, β D 3 Thus, there exists some point α3, β 3 D 3 with α3 β 3 4 β 3 2 s 2 F = β 3 + s 2 F such that Γ 3 α3, β3 < Γ 3 α, β, for all α, β D 3 Note that α3, β 3 lies on the boundary curve between D 2 and D 3 Since Γ is continuous on D, it holds that Γ 2 α3, β 3 = Γ 3α3, β 3 and hence, Γ, Γα + s 3, β3 < Γα, β, for all α, β D 3 F Hence, all the reasoning above proves that with Γ, +s F argmin Γα, β = α,β D, + s F, = s F +s F Finally, by the last assertion in Theorem 3, γ optimal linear convergence rate if and only if c F = 0, as claimed T, +s F is an Corollary 3 Let α ]0, ] and β ]0, [ Given z R n, choose any x 0 R n and consider the sequence generated, for k = 0,, 2,, by x k+ = T U z,v z,α,β x k = αx k + α2βp V z I2βP U z Ix k Let γt α,β be given by 6 Then, for every µ ]γt α,β, [, the sequence x k k 0 is R-linearly convergent to P Fix TU z,v z,α,β x 0 and the shadow sequence P U z + x k k 0 is R-linearly convergent to P U V z, both with rate µ, in the sense that there exists a positive integer k 0 such that P U z + x k P U V z x k P Fix TU z,v z,α,β x 0 µ k, for all k k 0 Proof According to Fact 24 we have that Fix T U z,v z,α,β and x k+ = T U z,v z,α,β x k = T U,V,α,β x k x + x, for x := P Fix TU z,v z,α,β x 0 Hence, one has Again by Fact 24, one has x k x = T U,V,α,β x k x = = T k U,V,α,β x 0 x Fix T U z,v z,α,β = x + U V, 3

14 and by the translation formula for projections cf [, 27ii], x = P Fix TU z,v z,α,β x 0 = P x +U V x 0 = P U V x 0 x + x, which implies P U V x 0 x = 0, and therefore, x k x = TU,V,α,β k x 0 x T = U,V,α,β k PU V x 0 x Let ν ]γt α,β, µ[ Since ν > γt α,β, by Theorem 3, there exists a positive integer k and some M > 0 such that TU,V,α,β k P U V Mν k, for all k k Let k 0 k be a positive integer such that µ ν k M x0 x, for all k k 0 Then, we deduce that x k x T k U,V,α,β P U V x0 x Mν k x 0 x µ k, for all k k 0, which proves the second inequality in By [2, Proposition 34] and the translation formula for projections cf [, 27ii], we can deduce that P U z + PFix TU z,v z,α,β x 0 = P U V z Thus, the first inequality in is a consequence of this, and the linearity and nonexpansiveness of P U Indeed P U z + x k P U V z = P U z + x k P U z + P Fix TU z,v z,α,β x 0 which completes the proof = P U x k P Fix TU z,v z,α,β x 0 x k P Fix TU z,v z,α,β x 0, 4 Comparison with other projection methods In this section, we compare the rate of AAMR with optimal parameters obtained in Section 3 with the rates of various projection methods analyzed in [5, 6] We summarize the key features of these schemes in Table, where we recall the operator defining the iteration of each method, as well as the optimal parameters and rates of convergence when these schemes are applied to linear subspaces Note that all these rates only depend on the Friedrichs angle θ F between the subspaces On the one hand, we observe that the rates for AAMR and GAP coincide Moreover, their optimal parameters are closely related, in the sense that α AAMR = α GAP and α,gap = α 2,GAP = 2β AAMR We explain this behavior in Section 4, where under some conditions, we show that the shadow sequences of GAP and AAMR coincide for linear subspaces Theorem 4 On the other hand, we note that the rate for AAMR/GAP is considerably smaller than the one of other methods, see Figure 3 We numerically demonstrate this with a computational experiment in Section 5 4

15 Method Optimal parameters Rate Alternating Projections AP = P V P U Relaxed Alternating Projections RAP = αi + αpv P U cos 2 θ F α = 2 +sin 2 θ F Generalized Alternating Projections α = GAP = αi + α α P V + α I α 2 P U + α 2 I α = α 2 = 2 +sin θ F Douglas Rachford DR = αi + α2p V I2P U I Averaged Alternating Modified Reflections AAMR = αi + α2βpv I2βP U I α = 2 α =, β = +sin θ F sin 2 θ F +sin 2 θ F sin θ F +sin θ F cos θ F sin θ F +sin θ F Table : Rates of convergence with optimal parameters of AP, RAP, GAP, DR and AAMR when they are applied to two subspaces Rate of convergence AP RAP DR AAMR GAP π 8 π 4 Friedrichs angle in radians 3π 8 π 2 Figure 3: Comparison of the rates of linear convergence with optimal parameters of AP, RAP, DR, AAMR and GAP 4 Relationship between AAMR and GAP for subspaces Fält and Giselsson have recently obtained in [6] the rate of convergence with optimal parameters for the generalized alternating projections GAP method for two subspaces This iterative scheme is defined by z k+ = αz k + α α P V + α I α 2 P U + α 2 I z k, 2 where α ]0, ] and α, α 2 ]0, 2] The next result shows that, for subspaces, the shadow sequences of GAP and AAMR coincide when α = α 2 = 2β and the starting point of AAMR is chosen as x 0 = 0; see Figure 4 for a simple example in R 2 This is not the case for general convex sets, as shown in Figure 5: GAP gives a point in the intersection of the sets, while AAMR solves the best approximation problem Figures 4 and 5 were created with Cinderella [9] 5

16 V x 4 + z x 3 + z x 2 + z x + z z 4 z 3 z 2 z z = z 0 x 4 x 3 U x 2 x w w 2 w 3 w 4 Figure 4: Graphical representation of Theorem 4 for two lines in R 2 The sequence {x k + z} k=0 is generated by AAMR with x 0 = 0, while the sequence {z k } k=0 is generated by GAP We also represent w k := 2β z k z z = z 0 x + z x 2 + z x 3 + z x 4 + z x 5 + z z z 2 z 3 z 4 A B P A B z Figure 5: The sequence {x k + z} k=0, with x 0 = 0, generated by AAMR converges to P A B z and thus solves the best approximation problem, while the sequence {z k } k=0, with z 0 = z, generated by GAP only converges to some point in A B Theorem 4 Given z R n, set x 0 = 0 and consider the AAMR sequence {x k } k=0 generated by 2 with parameters α ]0, ] and β ]0, [ Let {z k } k=0 be the sequence generated by GAP 2 with parameters α and α = α 2 = 2β and starting point z 0 = z Then, one has for all k = 0,, 2, P U x k + z = P U z k and P V x k = 2β P V z k z, 3 Proof To simplify the notation, let η := 2β We shall prove 3 by induction Since both equalities clearly hold for k = 0, we can assume that they are valid for some k 0 By 5, the 6

17 sequence generated by the AAMR scheme satisfies and, P U x k+ = P U x k + αη ηp U P V P U x k + z + ηp U P V z P U P V x k P U x k + z = αη 2 P U P V P U αηp U P V + αηp U xk + α η 2 P U P V P U + η ηp U P V ηp U z, 4 P V x k+ = P V x k + αη η P V P U x k + z + P V ηz x k = αηη P V P U x k + z + P V αη ηz + αηx k 5 Thanks to the linearity of the projectors onto subspaces and using α = α 2 = η, the GAP iteration 2 takes the form z k+ = αη2 η z k + α η 2 P V P U z k + η ηp V z k + η ηp U z k ; and thus this scheme verifies and P U z k+ = αη 2 P U P V P U + αη ηp U P V + αηp U zk, 6 P V z k+ = αηp V z k + αηp V P U z k 7 Then, by 5, the induction hypothesis 3 and 7, we obtain P V x k+ = αηη P V P U z k + αηp V x k + P V αη ηz = αηη P V P U z k + αηη P V z k z + P V αη ηz = η αηp V P U z k + αηp V z k + ηp V z = η P V z k+ z, which proves the second equation in 3 for k + Finally, by 6, 3 and 4, we have that P U z k+ = αη 2 P U P V P U x k + z + αηp U P V x k + ηz + αηp U x k + z = αη 2 P U P V P U αηp U P V + αηp U xk + α η 2 P U P V P U + η ηp U P V ηp U z + PU z = P U x k+ + z, which proves the first equation in 3 for k + and completes the proof 5 Computational experiments In this section we demonstrate the theoretical results obtained in the previous sections with two different numerical experiments In both experiments we consider randomly generated subspaces U and V in R 50 with U V {0} We have implemented all the algorithms in Python 27 and the figures were drawn with Matplotlib [9] The purpose of our first computational experiment is to exhibit the piecewise expression of the convergence rate γt α,β given in Theorem 3 To this aim, we generated 500 pairs of random subspaces For each pair of subspaces, we chose 0 random starting points with x 0 = Then, for each of these instances, we ran the AAMR method with α = 08 and β {05, 06, 07, 08, 09} The algorithm was stopped when the shadow sequence satisfies P U x n + x 0 P U V x 0 < ɛ := 0 8, 8 7

18 where x n + n=0 is the sequence iteratively defined by 2 with z = x 0 According to Corollary 3, for any µ ]γt α,β, [, the left-hand side of 8 is bounded by µ n for n big enough Therefore, an estimate of the maximum number of iterations is given by log ɛ log γt α,β 9 The results are shown in Figure 6, where the points represent the number of iterations required by AAMR to satisfy 8, and the lines correspond to the estimated upper bounds given by 9 We clearly observe that the algorithm behaves in accordance with the theoretical rates We emphasize the fact that 9 is expected to be a good upper bound on the number of iterations only when this number is sufficiently large We can indeed find a few instances in the plot, especially those which require a small number of iterations, exceeding its estimated upper bound Number of iterations β=05 β=06 β=07 β=08 β= Friedrichs angle in radians Figure 6: Number of iterations required to converge for the AAMR algorithm with α = 08 and five different values of the parameter β, with respect to the Friedrichs angle The lines correspond to the approximate upper bounds given by 9 and the theoretical rates 6 In our second experiment we compare the performance of AP, RAP, DR and AAMR, when their parameters are selected to be optimal see Table For 00 pairs of subspaces, we generated 50 random starting points with x 0 = For a fair comparison, we monitored the shadow sequence for all the algorithms We also used the stopping criterion 8, with ɛ = 0 8 The results of this experiment are summarized in Figure 7, where we show in three different graphics the median, the difference between the maximum and the median, and the coefficient of variation of the number of iterations needed to converge for each pair of subspaces As expected, since the rate of convergence of AAMR is the smallest amongst all the compared methods see Table and Fig 3, this algorithm is clearly the fastest, particularly for small angles Moreover, we can observe that AAMR is one of the most robust methods together with RAP, which makes the median to be a good representative of the rate of convergence 8

19 Median of iterations AP RAP DR AAMR Friedrichs angle in radians Max-median of iterations AP RAP DR AAMR Friedrichs angle in radians Coeff of variation of iterations AP RAP DR AAMR Friedrichs angle in radians Figure 7: Median, difference between the maximum and the median, and coefficient of variation of the required number of iterations with respect to the Friedrichs angle of AP, RAP, DR and AAMR for their respective optimal parameters 9

20 6 Conclusions We have computed the rate of linear convergence of the averaged alternating modified reflections AAMR method, which was originally introduced in [2], for the case of two subspaces in a Euclidean space We have additionally found the optimal selection of the parameters defining the scheme that minimizes this rate, in terms of the Friedrichs angle The rate with optimal parameters coincides with the one of the generalized alternating projections GAP method, which is the best among all known rates of projection methods This coincidence motivated us to study the relationship between AAMR and GAP when they are applied to subspaces We have discovered that, under some conditions, their associated shadow sequences overlap, which explains the coincidence of the rates This is not the case for general convex sets The developed theoretical results validate the conclusions drawn in the numerical analysis of the convergence rate developed in [2, Section 7] The sharpness of these theoretical results were additionally demonstrated in this work with two computational experiments The analysis in this work was done for the case of linear subspaces in a finite-dimensional space It would be interesting to investigate in future research whether the results can be extended to infinite-dimensional spaces; or even more, to study the rate of convergence of the method when it is applied to two arbitrary convex sets Acknowledgements The authors would like to thank one of the anonymous referees of [2] for suggesting us this interesting project This work was partially supported by Ministerio de Economía, Industria y Competitividad MINECO of Spain and European Regional Development Fund ERDF, grant MTM C2--P FJ Aragón Artacho was supported by the Ramón y Cajal program by MINECO and ERDF RYC and R Campoy was supported by MINECO and European Social Fund BES under the program Ayudas para contratos predoctorales para la formación de doctores 205 References [] Agmon, S: The relaxation method for linear inequalities Canad J Math 63, [2] Aragón Artacho, FJ, Campoy, R: A new projection method for finding the closest point in the intersection of convex sets Comput Optim Appl, DOI: 0007/s [3] Bauschke, HH, Combettes, PL: Convex analysis and monotone operator theory in Hilbert spaces Springer, New York 20 [4] Bauschke, HH, Cruz, JB, Nghia, TT, Phan, HM, Wang, X: The rate of linear convergence of the Douglas Rachford algorithm for subspaces is the cosine of the Friedrichs angle J Approx Theory 85, [5] Bauschke, HH, Cruz, JB, Nghia, TT, Phan, HM, Wang, X: Optimal rates of linear convergence of relaxed alternating projections and generalized Douglas Rachford methods for two subspaces Numer Algor 73, [6] Cegielski, A: Iterative methods for fixed point problems in Hilbert spaces Lecture Notes in Mathematics, 2057 Springer, Heidelberg 202 [7] Cegielski, A, Suchocka, A: Relaxed alternating projection methods SIAM J Optim 93,

21 [8] Censor, Y, Cegielski, A: Projection methods: an annotated bibliography of books and reviews Optimization 64, [9] Cinderella software [0] Deutsch, F: The angle between subspaces of a Hilbert space In: Singh SP eds Approximation Theory, Wavelets and Applications, pp Springer, Dordrecht 995 [] Deutsch, F: Best approximation in inner product spaces CMS Books in Mathematics/Ouvrages de Mathématiques de la SMC, 7 Springer-Verlag, New York 200 [2] Douglas, J, Rachford, H H: On the numerical solution of heat conduction problems in two and three space variables Trans Amer Math Soc 82, [3] Eckstein, J, Bertsekas, DP: On the Douglas Rachford splitting method and the proximal point algorithm for maximal monotone operators Math Program 55, [4] Escalante, R, Raydan, M: Alternating projection methods Fundamentals of Algorithms, 8 Society for Industrial and Applied Mathematics SIAM, Philadelphia, PA 20 [5] Fält, M, Giselsson, P: Line search for generalized alternating projections Preprint 206 arxiv: [6] Fält, M, Giselsson, P: Optimal convergence rates for generalized alternating projections Preprint 207 arxiv: [7] Gubin, LG, Polyak, BT, Raik, EV: The method of projections for finding the common point of convex sets USSR Comp Math Math Phys 76, [8] Horn, RA, Johnson, CR: Matrix analysis Second edition Cambridge University Press, Cambridge 203 [9] Hunter, JD: Matplotlib: A 2D graphics environment Comput Sci Eng 93, [20] Lions, PL, Mercier, B: Splitting algorithms for the sum of two nonlinear operators SIAM J Numer Anal 66, [2] Luke, DR: Finding best approximation pairs relative to a convex and a prox-regular set in a Hilbert space SIAM J Optim 92, [22] Meyer, CD: Matrix analysis and applied linear algebra SIAM, Philadelphia 2000 [23] Motzkin, TS, Schoenberg, IJ: The relaxation method for linear inequalities Canad J Math 63, [24] Von Neumann, J: Functional operators II: The geometry of orthogonal spaces Princeton University Press 950 Reprint of mimeographed lecture notes first distributed in 933 2