A NEW GEOMETRIC ACCELERATION OF THE VON NEUMANN-HALPERIN PROJECTION METHOD

Transcription

1 Electronic Transactions on Numerical Analysis. Volume 45, , Coyright c 2016,. ISSN ETNA A NEW GEOMETRIC ACCELERATION OF THE VON NEUMANN-HALPERIN PROJECTION METHOD WILLIAMS LÓPEZ Abstract. We develo a geometrical acceleration scheme for the von Neumann-Halerin alternating rojection method, when alied to the roblem of finding the rojection of a oint onto the intersection of a finite number of closed subsaces of a Hilbert sace. We study the convergence roerties of the new scheme. We also resent some encouraging reliminary numerical results to illustrate the erformance of the new scheme when comared with a well-nown geometrical acceleration scheme, and also with the original von Neumann-Halerin alternating rojection method. Key words. von Neumann-Haerin algorithm, alternating rojection methods, orthogonal rojections, acceleration schemes AMS subject classifications. 52A20, 46C07, 65H10, 47J25 1. Introduction. Let H be a Hilbert sace with inner roduct.,.. For a given 0 H and many closed subsaces M 1,..., M in H, we consider the best aroimation roblem: find the closest oint to 0 in M = i=1 M i, which can be stated as an otimization roblem as follows: 1.1) minimize 0 subject to M, where, for any z H, z 2 = z, z. The unique solution of roblem 1.1) is called the rojection of 0 onto M and is denoted as P M 0 ). In 1933, von Neumann solved roblem 1.1) for the articular case of two closed subsaces. THEOREM 1.1 von Neumann [28]). If M 1 and M 2 are closed subsaces in H, then for each 0 H, lim P M 2 P M1 ) 0 ) = P M1 M 2 0 ). Figure 1.1 shows the geometric interretation of Theorem 1.1. The etension to more than two subsaces was established in 1962 by Halerin. THEOREM 1.2 I. Halerin [23]). If M 1,..., M are closed subsaces in H, then for each 0 H, lim P M P M 1 P M1 ) 0 ) = P i=1 Mi 0). Theorem 1.2 suggests an algorithm, called the method of alternating rojections or MAP for short); see [11, 16], which can be described as follows: for any 0 H, set 1.2) 0 = 1 i = P Mi i 1) i = 1, 2,...,, for Z +, with initial value 0 = 0. The MAP is closely related to Kaczmarz alternating rojection method [25] for solving linear systems of equations. Received October 27, Acceted June 9, Published online on Setember 2, Recommended by Fiorella Sgallari. Deartamento de Física y Matemáticas, Núcleo Universiraio Rafael Rangel, Universidad de Los Andes, Trujillo, Venezuela wloez@ula.ve). 330

2 ACCELERATION OF THE VON NEUMANN-HALPERIN PROJECTION METHOD 331 P M2 P M1 ) 0 ) M 2 0 P M1 M 2 0 ) M 1 P M1 0 ) FIG lim P M 2 P M1 ) 0 ) = P M1 M 2 0 ). THEOREM 1.3. For any i = 1, 2,...,, the sequence { i } generated by 1.2) converges to P M 0 ). Proof. Let i {1, 2,..., }. Then i = P Mi P Mi 1 P M1 ) ), for all 0, where = P M P M 1 P M1 ) 0 ). From Theorem 1.2 we have that P M 0 ) when and since P Mi, P Mi 1,..., P M1 are bounded linear oerators, then i = P Mi P Mi 1 P M1 ) ) P Mi P Mi 1 P M1 )P M 0 )) = P M 0 ) when. Alications of MAP in various areas of mathematics can be found in [16, 9, 10, 3, 7, 8, 22]. The MAP has an r-linear rate of convergence that can be very slow when the angles between the subsaces in the sense of Friedrichs [20]) are small; see, e.g., [16]. Franchetti and Light [19] showed that the convergence in Theorem 1.1 may be arbitrarily slow if M 1 + M 2 is not closed i.e., if the angle between M 1 and M 2 is zero). Consequently, several acceleration schemes have been roosed; see, e.g., [4, 21, 24, 12]. Motivated by a recent wor of Lóez and Raydan [26], in this wor we resent and analyze a geometrical acceleration scheme for MAP in 1.2), which solves roblem 1.1). The new scheme involves finding oints closest to a solution, ) in the roduct sace H H, where is the solution of 1.1). The rest of this aer is organized as follows. In Section 2 we rovide information about already eisting acceleration schemes for MAP. In Section 2.1 we develo the new acceleration scheme and discuss its theoretical roerties. In Section 3 we resent encouraging reliminary numerical results. 2. Accelerations for MAP. The sequences generated by the method of alternating rojections often converge very slowly. Such a slow convergence is observed, e.g., if M 1 and M 2 are two closed subsaces with angles close to 0 see, e.g., [19]). Since the method of alternating rojections has many ractical alications, any acceleration technique seems to be imortant. Several acceleration schemes with a geometrical flavor have been roosed to imrove the erformance of MAP; see, e.g., De Pierro and Iusem [13], Dos Santos [14], Gearhart and Koshy [21], Bausche et al. [4], Gubin, Polya and Rai [22], Martínez [27], Aleby and Smolarsi [1] and Censor [6]. In Section 2.1, we describe a geometrically aealing acceleration scheme following the resentation in [21]. Some other different acceleration ideas have also been develoed see, e.g., Echebest et al. [17, 18] and Scolni et al. [29, 30]) based on the so-called rojected aggregation method PAM). Other secialized acceleration scheme

3 332 W. LÓPEZ ideas have been develoed by Hernández-Ramos et. al. [24] based on the use of the conjugate gradient method for minimizing a related conve quadratic ma. Usually vector etraolation algorithms are used for accelerating the convergence of MAP; see, e.g., [3, 8, 7]. Dystra s algorithm [15, 5] solves roblem 1.1) when the involved sets are closed and conve not necessarily closed subsaces). Dystra s algorithm can be viewed as a natural etension to conve sets of von Neumann-Halerin s MAP for subsaces in a Hilbert sace. The sequences generated by Dystra s algorithm often converge very slowly. Such a slow convergence is observed, e.g., if M 1 is a hyerlane and M 2 is a ball which is tangent to M 1 ; see, e.g., [2, Eamle 5.3]. An acceleration scheme has been develoed for Dystra s algorithm [26]. In Section 2.1 we develo a new acceleration scheme for von Neumann-Halerin s MAP on closed subsaces, which is related to the scheme given in [26] A new acceleration scheme. We now discuss an acceleration scheme for MAP to solve roblem 1.1). For that we need to consider an auiliary sequence in the roduct sace H H that will be denoted as H 2. If H is a Hilbert sace with inner roduct.,., we will use the inner roduct in H 2 defined by, y), w, z) =, w + y, z, for all, y), w, z) H 2 and norm, y) 2 =, y),, y) = 2 + y 2, for all, y) H 2. Thus H 2 is a Hilbert sace. For each Z +, let us define 2.1) = 1, ) H 2, where 1 and are defined in 1.2). Let us also denote = P M 0 ), P M 0 )) H 2. It follows that { } is a sequence in H 2 that converges to. Our idea to accelerate the convergence of MAP consists in building another sequence in H 2 which converges faster to than { }. For that we need to define a suitable subsace Π of H 2 that contains, as follows: Π = {, ) : H}. Clearly, Π is a closed subsace in H 2 and Π. Let us denote by P Π ) the orthogonal rojection of H 2 onto Π. For each Z +, let us consider P Π ) Π, where { } is given by 2.1). THEOREM 2.1. For all 1, P Π ). Proof. For all 1, since P Π ) is orthogonal to Π, and since P Π ) and belong to Π, then P Π ) is orthogonal to P Π ). Hence 2 = P Π ) + P Π ) 2 = P Π ) 2 + P Π ) 2. Consequently, 2 P Π ) 2, thus P Π ). The sequence {P Π )} will lay a ey role in the develoment of the acceleration scheme for MAP. REMARK 2.2. If, y) H 2, then see, e.g., [26]) 2.2) P Π, y) = 1/2 + y, + y). Notice that comuting the rojection onto Π only requires us to comute the average of the two involved vectors, i.e., it requires a very ineensive calculation. Let { } be the sequence defined by 2.1). For any 1, with P Π ) P Π ), let ô Π be the rojection of onto the line that goes through P Π ) and P Π ) see

4 ACCELERATION OF THE VON NEUMANN-HALPERIN PROJECTION METHOD 333 Figure 2.1). Then ô, P Π ) P Π ) = 0 and there eists a unique α R such that Therefore ô = P Π ) + α P Π ) P Π )). 2.3) P Π ) + α P Π ) P Π )), P Π ) P Π ) = 0. Solving for α, from 2.3), gives 2.4) α = P Π ), P Π ) P Π ) P Π ) P Π ) 2. In the following result we will give a formula for α in 2.4) that does not require nowledge of. Π ô P Π ) P Π ) FIG ô, P Π ) P Π ) = 0. THEOREM 2.3. Let { } be defined by 2.1). Then, P Π ) P Π ) = 0 for all 1. Proof. Let 1. Since Π and the rojection P Π is self-adjoint, we obtain, P Π ) P Π ) =, =, ), 1 1, ) =, 1 1 +,. On the other hand, since M = i=1 M i and the rojections P Mi are self-adjoint, we have, =,, =,, P M P... P M 1 M 1 ) ) =,, = 0. Similarly, we also obtain, 1 1 = 0. COROLLARY 2.4. Let α, for some, be given by 2.4). Then 2.5) α = P Π ), P Π ) P Π ) P Π ) P Π ) 2. Proof. This is a consequence of Theorem 2.3 and 2.4).

5 334 W. LÓPEZ Let us define the sequence {ô } in Π, for 1, as follow { P Π ) if P Π ) P Π ) = 0, 2.6) ô = P Π ) + α P Π ) P Π )) if P Π ) P Π ) 0, where α is given by 2.5). THEOREM 2.5. Let {ô } be given by 2.6). Then ô for all 1. Proof. Let 1. If P Π ) P Π ) = 0, then from Theorem 2.1 and 2.6) it follows that ô = P Π ). On the other hand, if P Π ) P Π ) 0, then ô is chosen to minimize the distance to along the line connecting P Π ) and P Π ). Therefore ô P Π ) and, from Theorem 2.1, it follows that ô. Theorem 2.5 guarantees than the sequence {ô } defined by 2.6) accelerates the convergence of the sequence { } defined by 2.1). REMARK 2.6. For the articular case of closed subsaces, the sequence {ô } defined by 2.6) accelerates the convergence of the sequence {ẑ } defined in [26]. For any 1, ẑ defined in [26] is the intersection of the subsace Π with the line that goes through and see Figure 2.2). Consequently ẑ = + α ) Π, for some α R. Since ẑ Π and P Π is a linear oerator, ẑ = P Π ẑ ) = P Π ) + α P Π ) P Π )). Therefore we have that ẑ belongs to the line connecting P Π ) and P Π ). Since ô is chosen to minimize the distance to along the line connecting P Π ) and P Π ), ô ẑ see Figure 2.2). Π P ẑ Π ) ô P Π ) FIG ô vs. ẑ. REMARK 2.7. Since the sequence {ô } defined by 2.6) belongs to Π, for any 1, there eists a unique o H such that 2.7) ô = o, o ) Π.

6 ACCELERATION OF THE VON NEUMANN-HALPERIN PROJECTION METHOD 335 THEOREM 2.8. Let {o } the sequence defined by 2.7). Then, for each 1, o P M 0 ) ma{ 1 P M 0 ), P M 0 ) }. Proof. Let 1. From Theorem 2.5, we have that 2 o P M 0 ) = o P M 0 ) 2 + o P M 0 ) 2 = ô = 1 P M 0 ) 2 + P M 0 ) 2 2 ma{ 1 P M 0 ) 2, P M 0 ) 2 } = 2 ma{ 1 P M 0 ), P M 0 ) }, and the result is established. COROLLARY 2.9. The sequence {o } defined by 2.7) converges to P M 0 ). Proof. This is a consequence of Theorem 2.8 and Theorem 1.3. REMARK Observe that the sequence {ô } in H 2 accelerates the convergence of the sequence { } in H 2 see Theorem 2.5). However, Theorem 2.8 does not guarantee acceleration when the sequence {o } in H is comared to the original MAP sequence { } in H, since a maimal is involved. Nevertheless, based on numerical eerimentation, the sequence {o } almost always accelerates the convergence of the original MAP sequence { }. We will now describe elicitly the sequence {o } defined by 2.7). If P Π ) P Π ) = 0, then from 2.1) and 2.2) we have that ô = P Π ) = P Π 1, In this case o = 1/2 1 + then where ) = 1/2 ) = o, o ) Π. 1 +, 1 + ) H. If, on the other hand, PΠ ) P Π ) 0, ô = P Π ) + α P Π ) P Π )) = o, o ) Π, o = 1/2 + α From Corollary 2.9 it follows that o = 1/2 + α ) α 1 1)) H. ) α 1 1)) PM 0 ) when. Since P M is a bounded linear oerator where P M is the rojection oerator onto M ), [ P M 1/2 + α ) α 1 1))] PM P M 0 )) when. Consequently 1/2 [ + α ) + + α )] PM 0 )

7 336 W. LÓPEZ when. Therefore 2.8) + α ) PM 0 ) when. Equation 2.8) suggests how to define an accelerated sequence in H and a secialized algorithm for which convergence to the solution P M 0 ) will be later established. Nevertheless, to achieve this it is convenient to write α given by 2.5) as a function of the original MAP iterations in H. LEMMA Let α, for some, be given by 2.5). Then 2.9) α = 1 +, , Proof. Let 1. The numerator in 2.5) can be written as P Π ), P Π ) P Π ) = P Π ), = 1/2 ) 1 +, 1 +, 1 1, ) = 1/2 1 +, 1 ) ,. On the other hand P Π ) P Π ) = 1/2 1 +, 1 + ) 1/2 1 +, 1 + ) = 1/ , ). Therefore the denominator in 2.5) can be written as P Π ) P Π ) 2 = 1/ ) = 1/ , and the result holds. Let { i } be the MAP s iterates given by 1.2). Let us now define a new sequence {o } in H, for 1, as follows { 2.10) o = if = 0, + α ) if , where α is given by 2.9). THEOREM The sequence {o } defined by 2.10) converges to P M 0 ). Proof. This is a consequence of Corollary 2.9, equation 2.8), and Theorem 1.3. We are now ready to resent our acceleration scheme for MAP. Algorithm 1. Let M i, i = 1,...,, be closed subsaces in H. Given 0 H; set = 1. set 0 = 0 1 = P M 1... P M1 ) 1 ) = P M 1) for = 1, 2,... do 1 = P M 1... P M1 ) ) = P M 1 )

8 ACCELERATION OF THE VON NEUMANN-HALPERIN PROJECTION METHOD 337 if then comute α using 2.9), and set o = + α ) else set o = end if end for In the following, we will comare our scheme, given in Algorithm 1, with the acceleration scheme given by Gearhart and Koshy [21]. Therefore, we will give a brief elanation of the Gearhart-Koshy scheme. Let us denote by 0 the given starting oint and by Q the comosition of the rojection oerators, i.e, Q = P M P M 1 P M1, where P Mi is the rojection oerator onto M i for all i. Let be the th iterate, and let Q be the net iterate after alying a swee of MAP. The idea is to search along the line through the oints and Q to obtain the oint closest to the solution = P i=1 Mi 0). Let us reresent any oint on this line as t) = tq + 1 t) = + tq ), for some real number t. Let t be the value of t for which this oint is the closest to. Then, 2.11) t ), Q = 0. Now, since i=1 M i and the rojections P Mi are self-adjoint, it follows, Q = P M1 P M2 P M, =,. Consequently,, Q = 0, and so can be eliminated from 2.11) to obtain t ), Q = 0. Solving for t gives To summarize, we have the following stes. t =, Q Q 2. Algorithm 2. Gearhart-Koshy) Let M i, i = 1,...,, be closed subsaces in H. Given 0 H. for = 1, 2,... do Q = Q t =, Q / Q 2 = t Q + 1 t ) end for 3. Numerical eeriments. We comare our acceleration scheme, given in Algorithm 1, with the acceleration scheme given by Gearhart and Koshy Algorithm 2), and with the original MAP with no acceleration given by 1.2). All comutations were erformed in MATLAB. For all eeriments we now the eact solution, and we sto each algorithm when the absolute error the distance from the -th iterate to ) is less than or equal to 10 6.

9 338 W. LÓPEZ For our first eeriment, we consider the following three subsaces of the sace of square real matrices R 3 3, with the Frobenius norm A 2 F = A, A = traceat A): M 1 = { A R 3 3 : A T = A }, M 2 = { A R 3 3 : a i,j+1 = 0, i = 1, 2, j = i + 1,..., 3 }, M 3 = { A R 3 3 : a 1,1 = a 1,3 = a 3,1 = a 3,3 }. If A = A ij ) R 3 3, then P M1 A) = A T + A)/2, A P M2 A) = A 21 A 22 0, and P M3 A) = A 31 A 32 A 33 where P = A 11 + A 13 + A 31 + A 33 )/4. We choose as the initial given oint. Then since A 0 = P M1 M 2 M 3 A 0 ) = P A 12 P A 21 A 22 A 23 P A 32 P, A : R = M 1 M 2 M 3, A, A 0 P M1 M 2 M 3 A 0 ) = trace The erformance of each method is shown in Figure 3.1. T , = 0. Consider the roblem of solving the linear system of equations A = 0, where A R m n and R n. This roblem can be generalized to any Hilbert sace H by considering the following formulation: find in the intersection of m closed subsaces given by H i = { H : a i, = 0}, for every i = 1,..., m. Here a i denotes the ith row of A or, in general, a fied given vector in H. If z H, then P Hi z) = z a i, z / a i, a i )a i. In the following two test roblems we will solve the linear equations A = 0, where A is a square nonsingular matri which is randomly generated in MATLAB we verified that deta) 0 after randomly generating the matri A in MATLAB). Therefore the redetermined solutions is always = 0. In both eeriments we chose 0 = 100, 100,..., 100) as the initial given oint. Figure 3.2 shows the erformance for each methods for A R 5 5 while Figure 3.3 shows the erformance for each method for A R Our reliminary numerical eeriments seem to indicate that our acceleration scheme, when comared to the other two cometitors, benefits from a reduction in the number of cycles, as well as in the comutational wor.

10 ACCELERATION OF THE VON NEUMANN-HALPERIN PROJECTION METHOD 339 FIG Acceleration for three subsaces in R 3 3. FIG Acceleration for solving A = 0, where A R 5 5. Acnowledgements. I than the three anonymous referees for constructive remars and additional references. REFERENCES [1] G. APPLEBY AND D. C. SMOLARSKI, A linear acceleration row action method for rojecting onto subsaces, Electron. Trans. Numer. Anal, ), htt://etna.ricam.oeaw.ac.at/vol / dir/ df [2] H. H. BAUSCHKE AND J. M. BORWEIN, Dystra s alternating rojection algorithm for two sets, J. Aro.

11 340 W. LÓPEZ FIG Acceleration for solving A = 0, where A R Theory, ), [3] H. H. BAUSCHKE, P. L. COMBETTES, AND S. G. KRUK, Etraolation algorithm for affine-conve feasibility roblems, Numer. Algorithms, ), [4] H. H. BAUSCHKE, F. DEUTSCH, H. HUNDAL, AND S. H. PARK, Accelerating the convergence of the method of alternating rojections, Trans. Amer. Math. Soc., ), [5] J. P. BOYLE AND L. DYKSTRA, A method for finding rojections onto the intersection of conve sets in Hilbert saces, in Advances in Order Restricted Statistical Inference, R. Dystra, T. Robertson, and F. T. Wright, eds., Lecture Notes in Statistics, 37, Sringer, Berlin, 1986, [6] Y. CENSOR, Comutational acceleration of rojection algorithms for the linear best aroimation roblem, Linear Algebra Al., ), [7] P. L. COMBETTES, Conve set theoretic image recovery by etraolated iterations of arallel subgradient rojections, IEEE Trans. Image Process., ), [8], The conve feasibility roblem in image recovery, in Advances in Imaging and Electron Physics, 95 P. W. Hawes, ed., Academic Press, New Yor, 1996, [9] F. DEUTSCH, Alications of von Neumann s alternating rojections algorithm, in Mathematical Methods in Oerations Research, P. Kenderov, ed., Bulgarian Academy of Science, Sofia, 1983, [10], The method of alternating orthogonal rojections, in Aroimation Theory, Sline Functions and Alications, S. P. Singh, ed., Kluwer, Netherlands, 1992, [11], Best Aroimation in Inner Product Sace, Sringer, New Yor, [12], Accelerating the convergence of the method of alternating rojections via a line search: a brief survey, in Inherently Parallel Algorithms in Feasibility and Otimization and their Alications, D. Butnariu, Y. Censor, S. Reich, eds., Studies in Comutational Mathematics, 8, North-Holland Publishing, Amsterdam, 2001, [13] A. DE PIERRO AND A. N. IUSEM, A arallel rojection method for finding a common oint of a family of conve sets, Pesquisa Oeracional, ), [14] L. T. DOS SANTOS, A arallel subgradient rojections method for the conve feasibility roblem, J. Comut. Al. Math., ), [15] R. L. DYKSTRA, An algorithm for restricted least-squares regression, J. Amer. Statist. Assoc., ), [16] R. ESCALANTE AND M. RAYDAN, Alternating Projection Methods, SIAM, Philadelhia, [17] N. ECHEBEST, M. T. GUARDARUCCI, H. SCOLNIK, AND M. C. VACCHINO, An acceleration scheme for solving conve feasibility roblems using incomlete rojection algorithms, Numer. Algorithms, ), [18], An accelerated iterative method with diagonally scaled oblique rojections for solving linear feasibility roblems, Ann. Oer. Res., ), [19] C. FRANCHETTI AND W. LIGHT, On the von Neumann alternating algorithm in Hilbert sace, J. Math. Anal.

12 ACCELERATION OF THE VON NEUMANN-HALPERIN PROJECTION METHOD 341 Al., ), [20] K. FRIEDRICHS, On certain inequalities and characteristic value roblems for analytic functions and for functions of two variables, Trans. Amer. Math. Soc., ), [21] W. GEARHART AND M. KOSHY, Acceleration schemes for the method of alternating rojection, J. Comut. Al. Math., ), [22] L. G. GUBIN, B. T. POLYAK, AND E. V. RAIK, The method of rojections for finding the common oint of conve sets, USSR Comut. Math. Math. Phys., ), [23] I. HALPERIN, The roduct of rojection oerators, Acta Sci. Math. Szeged), ), [24] L. M. HERNÁNDEZ-RAMOS, R. ESCALANTE, AND M. RAYDAN, Unconstrained otimization techniques for the acceleration of alternating rojection methods, Numer. Funct. Anal. Otim., ), [25] S. KACZMARZ, Angenäherte Auflösung von Systemen linearer Gleichungen, Bull. Internat. Acad. Pol. Sci. Lett. Ser. A, ), [26] W. LÓPEZ AND M. RAYDAN, An acceleration scheme for Dystra s algorithm, Comut. Otim. Al., ), [27] J. M. MARTÍNEZ, An accelerated successive orthogonal rojection method for solving large-scale linear feasibility roblems, Comut. Math. Al., ), [28] J. VON NEUMANN, Functional oerators. Vol II. The Geometry of Orthogonal Saces, Annals of Math. Studies, vol. 22, Princeton University Press, Princeton, [29] H. SCOLNIK, N. ECHEBEST, M. T. GUARDARUCCI, AND M. C. VACCHINO, Acceleration scheme for arallel rojected aggregation methods for solving large linear systems, Ann. Oer. Res., ), [30], A class of otimized row rojection methods for solving large nonsymmetric linear systems, Al. Numer. Math., ),