A Parallel Best-Response Algorithm with Exact Line Search for Nonconvex Sparsity-Regularized Rank Minimization

Transcription

1 A Parallel Bes-Response Algorihm wih Exac Line Search for Nonconvex Sparsiy-Regularized Rank Minimizaion Yang Yang and Marius Pesaveno Universiy of Luxembourg, Technische Universiä Darmsad, arxiv:704489v [csdc] 3 Nov 07 Absrac In his paper, we propose a convergen parallel bes-response algorihm wih he exac line search for he nondiffereniable nonconvex sparsiy-regularized rank minimizaion problem On he one hand, i exhibis a faser convergence han subgradien algorihms and block coordinae descen algorihms On he oher hand, is convergence o a saionary poin is guaraneed, while ADMM algorihms only converge for convex problems urhermore, he exac line search procedure in he proposed algorihm is performed efficienly in closed-form o avoid he meiculous choice of sepsizes, which is however a common boleneck in subgradien algorihms and successive convex approximaion algorihms inally, he proposed algorihm is numerically esed Index Terms Backbone Nework, Big Daa Analyics, Line Search, Rank Minimizaion, Successive Convex Approximaion I INTRODUCTION In his paper, we consider he esimaion of a low rank marix X R N K and a sparse marix S R I K from noisy measuremens Y R N K such ha Y = X + DS + V, where D R N I is a known marix The rank of X is much smaller han N and K, ie, rank(x min(n, K, and he suppor size of S is much smaller han IK, ie, S 0 IK A naural measure for he daa mismach is he leas square error augmened by regularizaion funcions o promoe he rank sparsiy of X and suppor sparsiy of S: (SRRM : X,S X + DS Y + λ X + µ S, where X is he nuclear norm of X This sparsiy-regularized rank minimizaion (SRRM problem plays a fundamenal role in he analysis of raffic anomalies in large-scale backbone neworks [] In his applicaion, X = RZ where Z is he unknown raffic flows over he ime horizon of ineres, R is a given fa rouing marix, S is he raffic volume anomalies The marix X inheris he rank sparsiy from Z because common emporal paerns among he raffic flows in addiion o heir periodic behavior render mos rows/columns of Z linearly dependen and hus low rank, and S is assumed o be sparse because raffic anomalies are expeced o happen sporadically and las shorly relaive o he measuremen inerval, which is represened by he number of columns K Alhough problem (SRRM is convex, i canno be easily solved by sandard solvers when he problem dimension is large, for he reason ha he nuclear norm X is neiher differeniable nor decomposable among he blocks of X I follows from he fac [, 3] ( X = min (P,Q + Q, s PQ = X ha i may be useful o consider he following opimizaion problem where he nuclear norm X is replaced by + Q : P,Q,S PQ + DS Y + λ ( + Q + µ S, ( where P R N ρ and Q R ρ K for a ρ ha is usually much smaller han N and K: ρ min(n, K Despie he fac ha problem ( is nonconvex, i is shown in [4, Prop ] ha every saionary poin of ( is a global opimal soluion of (SRRM under some mild condiions A block coordinae descen (BCD algorihm is adaped in [5] o find a saionary poin of he nonconvex problem ( In he BCD algorihm, he variables are updaed in a cyclic order or example, when P (or Q is updaed, he variables (Q, S (or (P, S are fixed However, when fixing (P, Q and updaing S, he elemens of S are updaed elemen-wise in a sequenial order o reduce he complexiy This is because he opimizaion problem wr s i,k, he (i, k-h elemen of S, has a closed-form soluion: s i,k PQ + DS Y + λ ( + Q + µ S, while he join opimizaion problem wih respec o (wr all elemens of he marix variable S does no have a closed-form soluion and is hus no easy o solve Neverheless, a drawback of he sequenial elemen-wise updae is ha i may incur a large delay because s i+,k canno be updaed unil s i,k is updaed and he delay may be very large when I is large, which is a norm raher han an excepion in big daa analyics [6] The alernaing direcion mehod of mulipliers (ADMM algorihm enables he simulaneous updae of all elemens of S, bu i does no have a guaranee convergence o a saionary poin because he opimizaion problem ( is nonconvex [4] Noe ha here is some recen developmen in ADMM for nonconvex problems, see [7, 8] for example and he references herein The ADMM algorihm proposed in [7] is designed for nonconvex sharing/consensus problems, and canno be applied o solve problem ( The ADMM algorihm proposed in [8] converges if he marix D in ( has full row rank, which is however no necessarily he case The nondiffereniable nonconvex problem ( can also be solved by sandard subgradien and/or successive convex approximaion (SCA algorihms [9] However, convergence of subgradien and SCA algorihms is mosly esablished under diminishing sepsizes, which is someimes difficul o deploy in pracice because he convergence behavior is sensiive o he decay rae As a maer of fac, he meiculous choice of sepsizes severely limis he applicabiliy of subgradien and SCA algorihms in nonsmooh opimizaion and big daa analyics [6] In his paper, we propose a convergen parallel bes-response algorihm, where all elemens of P, Q and S are updaed simulaneously This is a well known concep in opimizaion and someimes lised under differen names, for example, he parallel block coordinae descen algorihm (cf [0] and he Jacobi algorihm (cf [] To accelerae he convergence, we compue he sepsize by he exac line search procedure proposed in []: he exac line search is performed over a properly designed differeniable funcion and he resuling sepsize can be expressed in a closed-form expression, so

2 ha he compuaional complexiy is much lower han he radiional line search which is over he original nondiffereniable objecive funcion The proposed algorihm has several aracive feaures: i he variables are updaed simulaneously based on he bes-response; ii he sepsize is compued in closed-form based on he exac line search; iv i converges o a saionary poin, and is advanages over exising algorihms are summarized as follows: eaure i is an advanage over he BCD algorihm; eaures i and ii are advanages over subgradien algorihms; eaure ii is an advanage over SCA algorihms; eaure iii is an advanage over ADMM algorihms The above advanages will furher be illusraed by numerical resuls II THE PROPOSED PARALLEL BEST-RESPONSE ALGORITHM WITH EXACT LINE SEARCH In his secion, we propose an ieraive algorihm o find a saionary poin of problem ( I consiss of solving a sequence of successively refined approximae problems, which are presumably much easier o solve han he original problem To his end, we define f(p, Q, S PQ + DS Y + λ ( + Q, g(s µ S Alhough f(p, Q, S in ( is no joinly convex wr (P, Q, S, i is individual convex in P, Q and S In oher words, f(p, Q, S is convex wr one variable while he oher wo variables are fixed Preserving and exploiing his parial convexiy considerably acceleraes he convergence and i has become he cenral idea in he successive convex approximaion and he successive pseudoconvex approximaion [, ] To simplify he noaion, we use Z as a compac noaion for (P, Q, S: Z (P, Q, S; in he res of he paper, Z and (P, Q, S are used inerchangeably Given Z = (P, Q, S in ieraion, we approximae he original nonconvex funcion f(z by a convex funcion f(z; Z ha is of he following form: where f(z; Z = f P (P; Z + f Q(Q; Z + f S(S; Z, ( f P (P; Z f(p, Q, S = PQ + DS Y + λ, f Q(Q; Z f(p, Q, S = P Q + DS Y + λ Q, f(p, Q, s i,k, (s j,k j i, (s j j i f S(S; Z i,k = i,k P q k + d is i,k + d js j,k y k j i = r(s T d(d T DS (3a (3b r(s T (d(d T DS D T (DS Y + P Q, (3c wih q k (or y k and d i denoing he k-h and i-h column of Q (or Y and D, respecively, while d(d T D denoes a diagonal marix wih elemens on he main diagonal idenical o hose of he marix D T D Noe ha in he approximae funcion wr P and Q, he remaining variables (Q, S and (P, S are fixed, respecively Alhough i is emping o define he approximae funcion of f(p, Q, S wr S by fixing P and Q, minimizing f(p, Q, S wr he marix variable S does no have a closed-form soluion and mus be solved ieraively Therefore he proposed approximae funcion f S(S; Z in (3c consiss of IK componen funcions, and in he (i, k-h componen funcion, s i,k is he variable while all oher variables are fixed, namely, P, Q, (s j,k j i, and (s j j i As we will show shorly, minimizing f(s; Z wr S exhibis a closed-form soluion We remark ha he approximae funcion f(z; Z is a (srongly convex funcion and i is differeniable in boh Z and Z urhermore, he gradien of he approximae funcion f(p, Q, S; Z is equal o ha of f(p, Q, S a Z = Z To see his: P f(z; Z = P fp (P; Z ( = P PQ + DS Y + λ = P f(p, Q, S Z=Z, Q f(z; Z = Q fq(q; Z ( = Q P Q + DS Y + λ Q = Q f(p, Q, S Z=Z and S f(z; Z = ( si,k f(z; Z i,k while si,k f(z; Z = si,k fs(s; Z = si,k f(p, Q, s i,k, s i, k, s i = si,k f(p, Q, S Z=Z P=P Q=Q In ieraion, he approximae problem consiss of minimizing he approximae funcion over he same feasible se as he original problem (: Z=(P,Q,S f(z; Z + g(s (4 Since f(z; Z is srongly convex in Z and g(s is a convex funcion wr S, he approximae problem (4 is convex and i has a unique (globally opimal soluion, which is denoed as BZ = (B P Z, B QZ, B SZ The approximae problem (4 naurally decomposes ino several smaller problems which can be solved in parallel: B P Z arg min P k fp (P; Z = (Y DS (Q T (Q (Q T + λi, (5a B QZ arg min Q f Q(Q; Z = ((P T P + λi (P T (Y DS, (5b B SZ arg min f S(S; Z + g(s S = d(d T D S µ (d(d T DS D T (DS Y + P Q, (5c where S µ(x is an elemen-wise sof-hreshold operaor: he (i, j-h elemen of S µ(x is [X ij λ] + [ X ij λ] + As we can readily see from (5, he approximae problems can be solved efficienly because he opimal soluions are provided in an analyical expression Since f(z; Z is convex in Z and differeniable in boh Z and Z, and has he same gradien as f(z a Z = Z, i follows from [, Prop ] ha BZ Z is a descen direcion of he original

3 objecive funcion f(z + g(s a Z = Z The variable updae in he -h ieraion is hus defined as follows: P + = P + γ(b P Z P, Q + = Q + γ(b QZ Q, S + = S + γ(b SZ S, (6a (6b (6c where γ (0, ] is he sepsize ha should be properly seleced A naural (and radiional choice of he sepsize γ is given by he exac line search: min 0 γ { f(z + γ(bz Z + g(s + γ(b SZ S }, (7 in which he sepsize ha yields he larges decrease in objecive funcion value along he direcion BZ Z is seleced Neverheless, his choice leads o high compuaional complexiy, because g(s is nondiffereniable and he exac line search involves minimizing a nondiffereniable funcion Alernaives include consan sepsizes and diminishing sepsizes However, hey suffer from slow convergence (cf [] and parameer uning (cf [] As a maer of fac, he meiculous choice of sepsizes have become a major boleneck for subgradien and successive convex approximaion algorihm [6] I is shown in [, Sec III-A] ha o achieve convergence, i suffices o perform he exac line search over he following differeniable funcion: f(z + γ(bz Z + g(s + γ(g(b SZ g(s, (8 which is an upper bound of he objecive funcion in (7 afer applying Jensen s inequaliy o he convex nondiffereniable funcion g(s: g(s + γ(b SZ S g(s + γ(g(b SZ g(s This exac line search procedure over he differeniable funcion (8 achieves a good radeoff beween performance and complexiy urhermore, afer subsiuing he expressions of f(z and g(s ino (8, he exac line search boils down o minimizing a four order polynomial over he inerval [0, ]: γ = arg min 0 γ where = arg min 0 γ a P Q, { f(z + γ(bz Z + γ(g(b SX g(s } { 4 aγ4 + 3 bγ3 + } cγ + dγ, (9 b 3r( P Q (P Q + P Q + D S T, c r( P Q (P Q + DS Y T + P Q + P Q + D S + λ( P + Q, d r((p Q + P Q + D S (P Q + DS Y + λ(r(p P + r(q Q + µ( B SX S, for P B P Z P, Q B QZ Q and S B SZ S inding he opimal poins of (9 is equivalen o finding he nonnegaive real roo of a hird-order polynomial Making use of Cardano s mehod, we could express γ defined in (9 in a closedform expression: γ = [ γ ] 0, γ = 3 Σ + Σ + Σ3 Σ + 3 Σ + Σ3 b 3a, (0a (0b where [ γ ] 0 = max(min( γ,, 0 is he projecion of γ ono he inerval [0, ], Σ (b/3a 3 + bc/6a d/a and Σ c/3a Algorihm The parallel bes-response algorihm wih exac line search for problem ( Daa: = 0, Z 0 (arbirary bu fixed, eg, Z 0 = 0, sop crierion δ S: Compue (B P Z, B QZ, B SZ according o (5 S: Deermine he sepsize γ by he exac line search (0 S3: Updae (P, Q, Z according o (6 S4: If r((bz Z T f(z δ, STOP; oherwise go o S (b/3a Noe ha in (0b, he righ hand side has hree values (wo of hem could be complex numbers, and he equal sign reads o be equal o he smalles one among he real nonnegaive values The proposed algorihm is summarized in Algorihm, and we draw a few commens on is aracive feaures and advanages On he parallel bes-response updae: In each ieraion, he variables P, Q, and S are updaed simulaneously based on he besresponse The improvemen in convergence speed wr he BCD algorihm in [5] is noable because in he BCD algorihm, he opimizaion wr each elemen of S, say s i,k, is implemened in a sequenial order, and he number of elemens, IK, is usually very large in big daa applicaions To avoid he meiculous choice of sepsizes and furher accelerae he convergence, he exac line search is performed over he differeniable funcion f(z +γ(bz Z + γ(g(b SZ g S(Z and i can be compued by a closed-form expression The yields easier implemenaion and faser convergence han subgradien and SCA algorihms wih diminishing sepsizes On he complexiy: The complexiy of he proposed algorihm is mainained a a very low level, because boh he bes-responses (B P Z, B QZ, B SZ and he exac line search can be compued by closed-form expressions, cf (3 and (0 Only basic linear algebraic operaions are required, reducing he requiremens on he hardware s compuaional capabiliies On he convergence: The proposed Algorihm has a guaraneed convergence in he sense ha every limi poin of he sequence {Z } is a saionary poin of problem ( This claim direcly follows from [, Theorem ], and i serves as a cerificae for he soluion qualiy compared wih ADMM algorihms A Decomposiion of he Proposed Algorihm The proposed Algorihm can be furher decomposed o enable he parallel processing over a number of L nodes in a disribued nework To see his, we firs decompose he marix variables P, D and Y ino muliple blocks (P l L l=, (D l L l= and (Y l L l=, while P l R Nl ρ, D l R Nl I and Y l R Nl K consiss of N l rows of P, D and Y, respecively: P = P P P L, D = D D D L, Y = Y Y Y L, where each node l has access o he variables (P l, Q, S The compuaion of B P Z in (6a can be decomposed as B P Z = (B P,l Z L l=: B P,l Z = (Y l D l S (Q T (Q (Q T + λi, l =,, L Accordingly, he compuaion of B QZ and B SZ in (6b and (6c can be rewrien as ( B QZ L ( = l= (P l T P L l + λi l= (P l T (Y l D l S, ( L B SZ = d l= DT l D l ( L S µ (d l= DT l D l S L l= DT l (D l S Yl + P lq

4 0 5 regularizaio scaling facor elaive error in objecive funcion value regularizaio scaling facor BCD algorihm in [5] Algorihm (proposed ADMM algorihm in [4] ieraion relaive error in objecive funcion value regularizaion scaling facor r=0 regularizaion scaling facor r=05 BCD algorihm in [5] Algorihm (proposed running ime (seconds igure Relaive error in objecive funcion value versus ieraions igure Relaive error in objecive funcion value versus he CPU ime Before deermining he sepsize, he compuaion of a in (0 can also be decomposed among he nodes as a = L l= a l, where a l P l Q The decomposiion of b, c, and d is similar o ha of a, where b l 3r( P l Q (P l Q + P lq + D l S T, c l r( P l Q (P lq + D l S Y l T + P l Q + P lq + D l S + λ P l + λ Q l I, d l r((p l Q + P lq + D l S (P lq + D l S Yl + λr(p l P l + λ I r(q Q + µ I ( BSX S To compue he sepsize as in (0, he nodes muually exchange (a l, b l, c l, d l The four dimensional vecor (a l, b l, c l, d l provides each node wih all he necessary informaion o individually calculae (a, b, c, d and (Σ, Σ, Σ 3, and hen he sepsize γ according o (0 The signaling incurred by he exac line search is hus small and affordable III NUMERICAL SIMULATIONS In his secion, we perform numerical ess o compare he proposed Algorihm wih he BCD algorihm proposed in [5] and he ADMM algorihm proposed in [4] We sar wih a brief descripion of he ADMM algorihm: he problem ( can be rewrien as P,Q,A,B PQ + DA Y + λ ( + Q + µ B, subjec o A = B ( The augmened Lagrangian of ( is L c(p, Q, A, B, Π = PQ + DA Y + λ ( + Q + µ B + r(π T (A B + c A B, where c is a posiive consan In ADMM, he variables are updaed in he -h ieraion as follows: (Q +, B + = arg min Q,A L c(p, Q, A, B, Π, P + = arg min L c(p, Q +, A +, B, Π, P A + = arg min L c(p +, Q +, A, B +, Π, B Π + = Π + c(a + B + Noe ha he soluions o he above opimizaion problems have an analyical expression; see [4] for more deails We se c = 00 in he following simulaions The simulaion parameers are se as follows N = 06, K = 380, I = 380, ρ = 3 The elemens of D are generaed randomly and hey are eiher 0 or The elemens of V follow he Gaussian disribuion wih mean 0 and variance 00 Each elemen of S can ake hree possible values, namely, -, 0,, wih he probabiliy P (S i,k = = P (S ik = = 005 and P (S ik = 0 = 09 We se Y = PQ + DS + V, where he elemens of P (Q are generaed randomly following he Gaussian disribuion wih mean 0 and variance 00/I (00/K The sparsiy regularizer λ = r Y ( Y is he specral norm of Y and µ = r D T Y, where r is he regularizaion scaling facor ha is eiher 0 or 05 In igure, we show he relaive error in objecive funcion value versus he number of ieraions achieved by differen algorihms, where he opimal objecive funcion value is compued by running Algorihm for a sufficien number of ieraions As we can see from igure, he ADMM does no always converge for boh regularizaion parameers r = 0 and 05, as he opimizaion problem ( (and ( is nonconvex Noe ha for he BCD algorihm in igure, all elemens of S are updaed once, in a sequenial order, in one ieraion We can see from igure ha he BCD algorihm converges in less number of ieraions han he proposed Algorihm Bu he incurred delay of each ieraion in he BCD algorihm is ypically very large, because all elemens are updaed sequenially On he oher hand, in he proposed algorihm, all variables are updaed simulaneously and he CPU ime (in seconds needed for each ieraion is relaively small This is illusraed numerically in igure, where wo regularizaion scaling facors are esed, namely, r = 0 and r = 05 We see ha he improvemen is noable when he regularizaion parameer is small IV CONCLUDING REMARKS In his paper, we have proposed a parallel bes-response algorihm for he nonconvex sparsiy-regularized rank minimizaion problem The proposed algorihm exhibis fas convergence and low complexiy, because he variables are updaed simulaneously based on heir bes response; he sepsize is based on he exac line search and i is performed over a differeniable funcion; and 3 boh he bes response and he sepsize are compued by closed-form expressions urhermore, he proposed algorihm has a guaraneed convergence o he saionary poin The advanages of he proposed algorihm are also consolidaed numerically

5 REERENCES [] M Mardani, G Maeos, and G B Giannakis, Recovery of low-rank plus compressed sparse marices wih applicaion o unveiling raffic anomalies, IEEE Transacions on Informaion Theory, vol 59, no 8, pp , 03 [] B Rech, M azel, and P A Parrilo, Guaraneed Minimum-Rank Soluions of Linear Marix Equaions via Nuclear Norm Minimizaion, SIAM Review, vol 5, no 3, pp 47 50, jan 00 [3] C Seffens and M Pesaveno, Block- and Rank-Sparse Recovery for Direcion inding in Parly Calibraed Arrays, pp 9, 07 [4] M Mardani, G Maeos, and G B Giannakis, Decenralized sparsiyregularized rank minimizaion: Algorihms and applicaions, IEEE Transacions on Signal Processing, vol 6, no, pp , 03 [5], Dynamic anomalography: Tracking nework anomalies via sparsiy and low rank, IEEE Journal on Seleced Topics in Signal Processing, vol 7, no, pp 50 66, 03 [6] K Slavakis, G B Giannakis, and G Maeos, Modeling and Opimizaion for Big Daa Analyics: (Saisical learning ools for our era of daa deluge, IEEE Signal Processing Magazine, vol 3, no 5, pp 8 3, sep 04 [7] M Hong, Z-q Luo, and M Razaviyayn, Convergence Analysis of Alernaing Direcion Mehod of Mulipliers for a amily of Nonconvex Problems, SIAM Journal on Opimizaion, vol 6, no, pp , jan 06 [8] B Jiang, T Lin, S Ma, and S Zhang, Srucured Nonconvex and Nonsmooh Opimizaion: Algorihms and Ieraion Complexiy Analysis, pp 30, 06 [9] acchinei, G Scuari, and S Sagraella, Parallel Selecive Algorihms for Nonconvex Big Daa Opimizaion, IEEE Transacions on Signal Processing, vol 63, no 7, pp , nov 05 [0] M Elad, Why simple shrinkage is sill relevan for redundan represenaions? IEEE Transacions on Informaion Theory, vol 5, no, pp , 006 [] G Scuari, acchinei, P Song, D P Palomar, and J-S Pang, Decomposiion by Parial Linearizaion: Parallel Opimizaion of Muli-Agen Sysems, IEEE Transacions on Signal Processing, vol 6, no 3, pp , feb 04 [] Y Yang and M Pesaveno, A Unified Successive Pseudoconvex Approximaion ramework, IEEE Transacions on Signal Processing, vol 65, no 3, pp , 07