Online Compleion of Ill-condiioned Low-Rank Maices Ryan Kennedy and Camillo J. Taylo Compue and Infomaion Science Univesiy of Pennsylvania Philadelphia, PA, USA keny, cjaylo}@cis.upenn.edu Laua Balzano Elecical Engineeing and Compue Science Univesiy of Michigan Ann Abo, MI, USA giasole@umich.edu Absac We conside he poblem of online compleion of illcondiioned low-ank maices. While many maix compleion algoihms have been poposed ecenly, hey ofen suggle wih ill-condiioned maices and ake a long ime o convege. In his pape, we pesen a new algoihm called Pola Incemenal Maix Compleion (PIMC) o addess his poblem. Ou mehod is based on he algoihm, and we show how a pola decomposiion can be used o mainain an esimae of he singula value maix o bee deal wih ill-condiioned poblems. The mehod is also online, allowing i o be applied o seaming daa. We evaluae ou algoihm on boh synheic daa and a eal sucue fom moion daase fom he compue vision communiy, and show ha PIMC oupefoms simila mehods. Index Tems maix compleion, online opimizaion, condiion numbe I. INTRODUCTION Low-ank maix sucue has found applicaions in a gea numbe of domains, and he applicabiliy of low-ank maix compleion esuls o eal daa poblems is quie pomising since daases ofen have missing o unobseved values. Since he seminal esuls of 6, 7, many algoihms have been developed fo low-ank maix compleion 1, 5, 9, 10, 11, 13. Howeve, he low-dimensional sucue found in eal daa is aely well-behaved: singula values of lage daa maices ofen dop off in such a way ha i is no obvious a wha poin we ae disinguishing signal fom noise. In hese scenaios, he suie of exising maix compleion algoihms all suggle o find he ue low-ank componen, boh wih egads o achieving low eo and wih egads o he numbe of algoihm ieaions i akes o ge a good esul. Recenly, seveal algoihms have been poposed which impove pefomance fo maices wih lage condiion numbes 10, 9, bu hese algoihms sill have difficuly fo exemely illcondiioned poblems. Fuhemoe, hese algoihms ae bach and canno easily be used fo seaming daa. This pape makes he following conibuions. Fis, we show how he algoihm fo online maix compleion 1 can be e-inepeed via he Incemenal Singula Value Decomposiion (ISVD) 4 as finding he soluion o a specific leas-squaes poblem. Based on his inepeaion, we hen pesen a modificaion o his algoihm which dasically impoves is pefomance fo maices wih lage condiion numbe. We also demonsae expeimenally ha ou algoihm oupefoms ohe bach maix compleion algoihms on exemely ill-condiioned poblems. II. THE ISVD FORMULATION OF We begin by biefly descibing he algoihm 1 and is elaion o he incemenal singula value decomposiion (ISVD) 2. The ISVD algoihm 4, 3 is a simple mehod fo compuing he SVD of a maix by updaing an iniial decomposiion one column a a ime. Given a maix A R n m a ime whose SVD is A U Σ V T, we wish o compue he SVD of a new maix wih a single column added: A +1 A v. Defining weighs w U T v and esidual v U w, we have A +1 U Σ w 0 V T 0. (1) We compue an SVD of he cene maix, Σ w 0 ˆΣ ˆV T, (2) which yields he new SVD, A +1 U +1 Σ +1 V+1 T whee U +1 U ; Σ+1 ˆΣ; V 0 V +1 ˆV. (3) If only he op k singula vecos ae needed, hen we can apply he heuisic of dopping he smalles singula value and he associaed singula veco afe each such updae. I has ecenly been shown ha he algoihm 1 has a close elaionship o his ISVD algoihm 2. Le  U R T be an esimaed ank-k facoizaion of A such ha U has ohonomal columns. Given a new column v wih missing daa, le Ω 1,..., N} be he se of obseved enies. If w and ae now he leas-squaes weigh and esidual veco, especively, defined wih espec o only he se of obseved indices Ω, hen we can wie U R T ṽ U I w 0 whee ṽ has impued values, defined as vω on Ω ṽ U w ohewise T R 0, (4).
Noe he similaiy of Equaions (1) and (4). Taking he SVD of he cene maix o be I w 0 ˆΣ ˆV T, (5) i was shown in 2 ha updaing U o U +1 U (6) and subsequenly dopping he las column is equivalen o fo a specific sep size, which pefoms gadien descen diecly on he Gassmann manifold. Combining Equaions (4) and (5), updaing R hen becomes R 0 R +1 ˆV ˆΣ, (7) and dopping he las column povides a coesponding updae fo he maix R. The esul is a new ank-k facoizaion  +1 U +1 R+1. T We may ge insigh ino his vesion of by examining his inepeaion using wha we know abou he SVD. By he Ecka-Young heoem 8, he pocess of Equaions (5) and (6) ae finding he closes ank-k maix o U ṽ wih espec o he Fobenius nom. In ohe wods, we can inepe his new algoihm as solving he minimizaion poblem min ank(m)k U ṽ M 2 F. (8) The updaed U +1 is hen given by he op k lef singula vecos of M (o any ohonomal vecos which span his subspace). Le M ẐT, whee Rn k, and ẑ 1 Ẑ. ẑ k w Ẑk R (k+1) k ; (9) w noe his enfoces he ank-k consain on M. By plugging ino (8), we see ha each ieaion of his algoihm amouns o minimizing he following cos funcion: U +1 ag min min U Ẑk 2 F + min ṽ Ẑ k w w 2 2 (10) This has an inuiive inepeaion: he fis em equies ha U +1 have a column span close o ha of he cuen subspace U and he second em equies ha he new veco ṽ can be well-appoximaed by a linea combinaion of he columns of U +1. The updaed maix is he one ha minimizes he combinaion of hese wo compeing coss. The fis em of his minimizaion poblem can be scaled by a paamee λ in ode o allow fo a ade-off beween he wo ems, and by binging λ inside he nom and incopoaing i ino Ẑk, his is equivalen o scaling U : ag min min U λ Ẑk 2 F + min ṽ Ẑ k w w 2 2 } } (11) A lage λ will lead o a smalle change; i can be used as a egulaizaion paamee. In conas o he ISVD algoihm, does no make any use of he singula values of he maix. By no using an esimae of he singula values, can have difficuly conveging fo ill-condiioned maices. This is demonsaed in Figue 1, whee was un on a ank-5 maix wih no missing daa and no noise. Even in his ideal seup, he condiion numbe of he maix has a lage effec on he convegence ae of. # passes ove he daa o each an RMSE of 1x10 5 10 4 10 3 10 2 1 PIMC (poposed) 1 10 2 Condiion numbe ( σ 1 / σ 5 ) Fig. 1: Effec of he condiion numbe of a maix on he convegence. We conside a ank-5 maix of size 500 500 wih no noise o missing daa. We plo he numbe of passes ove he daa ha wee equied o each an RMSE eo of 1 10 5. As he condiion numbe inceases, convegence slows while ha of ou poposed algoihm PIMC emains consan. This convegence issue has been peviously noiced in bach maix compleion algoihms, and seveal algoihms have been pesened which ale he opimizaion on he Gassmann manifold in ode o ake ino accoun he non-isoopic scaling of he space by incopoaing he singula values ino he opimizaion 9, 10. These algoihms have demonsaed impoved pefomance on ill-condiioned maices, bu ae limied o he bach seing. Fuhemoe, as we show in Secion IV, even hese algoihms have ouble wih exemely ill-condiioned maices. We ake a simila appoach and incopoae he use of singula values ino, which allows fo accuae maix compleion even fo vey ill-condiioned maices in an online manne. III. PIMC FOR MATRIX COMPLETION In ode o impove he convegence of fo ill-condiioned maices, we would like o use U S as a epesenaive of he cuen subspace, whee S is an esimae of he singula values, ahe han jus U. Howeve, we canno diecly use ISVD and jus dop he las column a each ieaion o mainain a consan ank fo wo easons. Fis, he esuling singula values may no be a good esimae fo he eal singula values because of he missing daa.
Second, he ISVD equies V o be ohogonal, so while wih i is saighfowad o e-pocess a daa veco ha has peviously been pocessed by emoving he column fom R, wih ISVD i is no possible. We heefoe popose a new algoihm, which we call Pola Incemenal Maix Compleion (PIMC). Le  U R T be he cuen esimae of a maix compleion poblem a ime. We epesen R by is pola decomposiion R Ṽ S, (12) whee Ṽ R m k has ohonomal columns and S R k k is posiive semidefinie. This pola decomposiion exiss fo any maix R, and if Ū S V T R is an SVD of R, hen he facos can be wien explicily as Ṽ Ū V T and S V S V T. (13) The maix Ṽ now has ohonomal columns, simila o V fom ISVD. Likewise S is an esimae of he singula values of he space, alhough i may no longe be diagonal. We addiionally choose o scale S o accoun fo he fac ha U S is sill only an appoximaion o pas daa due o he missing enies. When daa ae missing, he weighs w ae defined wih espec o only he daa ha ae obseved, bu we use he inepolaed veco ṽ U w + in ou updae. Recalling ha he sum of squaes of he singula values is equal o he sum of column 2-noms, he singula values will heefoe be inceasing wih espec o his inepolaed veco ahe han wih espec o only he obseved daa as we would like. Insead, we will e-scale he singula value maix S o accoun fo only he obseved enies. To do so, we keep a unning sum of he nom of he acual obseved daa, s 2 s 2 1 + v Ω 2 2, (14) and a each ieaion scale S by γ s S F, whee γ is a fixed consan. The esuling facoizaion is given by γs S A +1 U S F S w F γs R T 0. 0 (15) Ou mehod, PIMC, hen finds he SVD of he cene maix and subsequenly dops he las singula value and he coesponding singula vecos a each ieaion. Noe ha he use of S effecively scales U a each ieaion, in a simila way o adding a egulaizaion paamee λ in Equaion 11, and so we do no explicily se λ in ou expeimens. The full algoihm is shown in Algoihm 1. IV. EXPERIMENTS We compae ou poposed algoihm PIMC o he ISVD fomulaion of, 13, 11, ScGad 10, and 9; he lae wo ae bach algoihms designed o pefom well on ill-condiioned maices by modifying he meic on he Gassmann manifold. We used MATLAB code fom he especive auhos wih defaul paamees. Fo PIMC, γ was se o 0.01 fo all expeimens. Algoihm 1 PIMC fo maix compleion 1: pocedue PIMC (A, γ, max ) 2: Iniialize U 1, S 1, R 1, s 0 3: fo 1,..., k max do 4: Selec a column i of A: v A(:, i) 5: Esimae weighs: w ag min a U Ω a v Ω 2 2 6: Updae he scaling weigh: s 2 s 2 1 + v Ω 2 2 7: Compue esidual: Ω v Ω U Ω w ; Ω C 0 8: Zeo-ou ow of R : R (i, :) 0 9: if e-ohogonalizing R hen 10: Compue pola decomposiion: R Ṽ S 11: Updae maices: R Ṽ; S S ST 12: end if 13: Compue SVD of cene ( maix: 14: Ŝ ˆV γs ) T SV D S F S w 0 15: Updae U : U +1 U 16: Updae S : S +1 Ŝ 17: Se up las column fo R updae: 18: z 0... 0 T ; z(i) 1 19: Updae R : R +1 S F γs R z ˆV 20: Dop las singula value and coesponding singula vecos 21: end fo 22: eun U max, S max, R max 23: end pocedue A. Synheic daa wihou noise We geneaed a 5000 5000 maix W of ank 5 as he poduc W XSY T whee X and Y ae andom 5000 5 maices wih ohonomal columns and S is a 5 5 diagonal maix conaining he singula values. The smalles singula value was se o be σ 5 1 10 3 and hey vaied logaihmically up o σ 1. 95% of he enies wee emoved unifomly a andom. Resuls ae shown in Figue 2 fo wo values of σ 1. In all cases, he algoihms ha ook accoun of an esimae of he singula values of he space PIMC, ScGad and oupefomed he ohe maix compleion algoihms. Howeve, wih an incease of one ode of magniude, he pefomance of ScGad and suffes (Figue 2b). We noe ha he auhos of boh of hese algoihms only pefomed expeimens fo condiion numbes up o aound 10, while hee we have gone up o 1000. Ou poposed algoihm PIMC conveges in oughly he same amoun of ime egadless of he condiion numbe. We hypohesize ha his may be due o he fac ha ScGad and boh pefom an alenaing opimizaion, having o eac back ono he manifold a each ieaion, while PIMC has no alenaion and emains ohogonal he enie ime.
10 8 10 10 1 10 2 10 3 (a) σ 1 σ 5 1 10 2 PIMC 10 8 ScGad 10 10 1 10 2 10 3 (b) σ 1 σ 5 1 10 3 Fig. 2: Compaison wihou noise. Random 5000 5000, ank-5 maices wih no noise and 95% of hei enies missing wee geneaed wih singula values ha vaied logaihmically fom σ 1 1 10 3 up o σ 5. In all cases, PIMC conveges in oughly he same amoun of ime. of a video. These acks can be aanged in a measuemen maix whee evey pai of columns gives he x and y locaions of poins ove all fames and each ow conains he 2D locaions of all poins in a given fame. If he camea is assumed o be affine, hen i can be shown ha his maix has ank a mos 4 12. Missing daa occu when poins acks ae los o become occluded. (a) Banded sucue of he daa PIMC ScGad 1 10 2 10 3 (b) Compaison of algoihms 1 10 2 10 3 (a) τ 0.3 PIMC ScGad 1 10 2 10 3 (b) τ 0.1 Fig. 3: Compaison wih noise. Singula values wee se o decay exponenially fom σ 1 1 10 7 as σ i σ i 1 τ and 95% of hei enies missing wee geneaed. The ank o esimae was se o 5 and we measue he eo wih espec o he bes ank-5 maix aken fom he full daa. τ was se o 0.3 and 0.1, esuling in maices wih σ 1 being 123 and 10000 imes lage han σ 5. B. Synheic daa wih noise We nex es how he algoihms pefom wih espec o noise using a andom 5000 5000 maix wih singula values ha decay exponenially as σ i τσ i 1 wih σ 1 1 10 7, fo some consan τ. 95% of he daa wee andomly emoved and he esimaed ank was se o 5. Resuls ae shown in Figue 3 fo τ 0.3, and 0.1. The eo was measued wih espec o he bes ank-5 maix as calculaed using he SVD of he daa maix befoe any daa wee emoved. This siuaion is much moe difficul and no algoihm is able o find he opimal soluion in any siuaion due o he lack of sepaaion beween he signal and noise subspaces. Howeve, i is again he case ha PIMC oupefoms ohe algoihms when he spead of singula values is lage. C. Sucue fom moion daa Sucue-fom-moion involves ecoveing he full 3D locaions of poins given hei 2D locaions acked ove he fames Fig. 4: Compaison of algoihms on sucue-fommoion daase. All algoihms have ouble eaching he opimum due o he banded sucue of he daa maix. PIMC conveges he fases and has he leas eo afe 1 10 3 seconds. See ex fo deails on he daase. We geneaed a synheic cylinde of adius 10 and heigh 5000 wih 500 poins acked ove 1000 fames. Afe emoving poins acked fo fewe han five fames, he esuling measuemen maix has size 2484 2000. The cylinde oaed once evey 500 fames, esuling in 80.13% missing daa. This maix has an exac ank-4 soluion wih a condiion numbe σ 1 /σ 4 290. An ineesing aspec of his daase is ha he daa ae no andomly obseved, bu appea wihin a band down he diagonal of he maix (Figue 4a). This sands in conas o he heoeical guaanees of convegence fo maix compleion which assume ha daa ae obseved unifomly a andom 6, 7. Figue 4 shows esuls on he sucue-fom-moion daase. All algoihms pefom elaively similaly wih PIMC conveging fases and achieving he lowes eo, bu all ae unable o find he opimal soluion. We have found ha he banded sucue of he daa maix hee makes opimizaion moe difficul han if he daa wee sampled unifomly a andom, and when combined wih a lage condiion numbe his opimizaion poblem is quie challenging fo all algoihms. V. CONCLUSION In his pape we have pesened a novel algoihm fo maix compleion based on he incemenal singula value decomposiion. Ou mehod is online and akes ino accoun an esimae of he singula values duing opimizaion o impove convegence fo maices which ae ill-condiioned. We have demonsaed ha i oupefoms ohe bach algoihms fo exemely ill-condiioned maices.
REFERENCES 1 L. Balzano, R. Nowak, and B. Rech. Online idenificaion and acking of subspaces fom highly incomplee infomaion. In Communicaion, Conol, and Compuing (Alleon), pages 704 711. IEEE, 2010. 2 L. Balzano and S. J. Wigh. On and incemenal SVD. In IEEE Wokshop on Compuaional Advances in Muli-Senso Adapive Pocessing (CAMSAP), 2013. 3 M. Band. Incemenal singula value decomposiion of unceain daa wih missing values. Euopean Confeence on Compue Vision (ECCV), pages 707 720, 2002. 4 J. R. Bunch and C. P. Nielsen. Updaing he singula value decomposiion. Numeische Mahemaik, 31:111 129, 1978. 10.1007/BF01397471. 5 J.-F. Cai, E. J. Candès, and Z. Shen. A singula value hesholding algoihm fo maix compleion. SIAM Jounal on Opimizaion, 20(4):1956 1982, 2008. 6 E. Candès and B. Rech. Exac maix compleion via convex opimizaion. Foundaions of Compuaional Mahemaics, 9(6):717 772, Decembe 2009. 7 E. Candès and T. Tao. The powe of convex elaxaion: Nea-opimal maix compleion. IEEE Tansacions on Infomaion Theoy, 56(5):2053 2080, May 2010. 8 C. Ecka and G. Young. The appoximaion of one maix by anohe of lowe ank. Psychomeika, 1(3):211 218, 1936. 9 B. Misha, A. A. Kaavadi, and R. Sepulche. A iemannian geomey fo low-ank maix compleion. Tech Repo, 2012. 10 T. Ngo and Y. Saad. Scaled gadiens on gassmann manifolds fo maix compleion. In Advances in Neual Infomaion Pocessing Sysems, 2012. 11 K.-C. Toh and S. Yun. An acceleaed poximal gadien algoihm fo nuclea nom egulaized linea leas squaes poblems. Pacific Jounal of Opimizaion, 6(615-640):15, 2010. 12 C. Tomasi and T. Kanade. Shape and moion fom image seams unde ohogaphy: a facoizaion mehod. Inenaional Jounal of Compue Vision, 9(2):137 154, 1992. 13 Z. Wen, W. Yin, and Y. Zhang. Solving a low-ank facoizaion model fo maix compleion by a nonlinea successive ove-elaxaion algoihm. Mahemaical Pogamming Compuaion, 2012.