Bounds for block sparse tensors

Transcription

1 A Bouds for bock sparse tesors Oe of the ma bouds to cotro s the spectra orm of the sparse perturbato tesor S The success of the power teratos ad the mprovemet accuracy of recovery over teratve steps of RTD requres ths boud Lemma 4 Spectra orm bouds for bock sparse tesors Let M R satsfy the bock sparsty assumpto S The M = Od 15 M 10 Proof Let Ψ R be a tesor that ecodes the sparsty of M e Ψ,j,k = 1 ff S,j,k 0 for a, j, k [] We have that M = max M,j,k uujuk u: u =1,j,k = max u: u =1,j,k max u: u =1,j,k M M,j,k Ψ,j,k uujuk M,j,k Ψ,j,k uujuk max u: u =1,j,k Ψ,j,k uujuk = M Ψ, where the ast equaty s from Perro Frobeus theorem for o-egatve tesors [9] Note that Ψ s oegatve by defto Now we boud Ψ o es of [3, Lemma 4] Reca that [B], j [], Ψ = B ψ ψ ψ, ψ 0 d, ψ j = 0 or 1 =1 By defto ψ = d Defe ormazed vectors ψ := ψ / ψ We have Ψ = d 15 B =1 ψ ψ ψ Defe matrx ψ := [ ψ 1 ψ, ψb ] Note that ψ ψ R B B s a matrx wth ut dagoa etres ad absoute vaues of off-dagoa etres bouded by η, by assumpto From Gershgor Dsk Theorem, every subset of L coums ψ has sguar vaues wth 1 ± o1, where L < 1 η Moreover, from Gershgor Dsk Theorem, ψ < 1 + Bη For ay ut vector u, et S be the set of L dces that are argest ψ u By the argumet above we kow ψ S u ψ S u 1 + o1 I partcuar, the smaest etry ψ S u s at most / L By costructo of S ths mpes for a ot S, ψ u s at most / L Now we ca wrte the 3 orm of ψ u as ψ u 3 3 = S S ψ u 3 + S ψ u 3 ψ u + / L 3 S 1 + η ψ 1 + Bη 15 ψ u Here the frst equaty uses that every etry outsde S s sma, ad ast equaty uses the boud argued o ψ S u, the spectra orm boud s assumed o A S c Sce B = Oη 15, we have the resut Aother mportat boud requred s -orm of certa cotractos of the ormazed sparse tesor ad ts powers, whch we deote by M beow We use a oose boud based o spectra orm ad we requre M < 1/ However, ths costrat w aso be eeded for the power teratos to succeed ad s ot a addtoa requremet Thus, the oose boud beow w suffce for our resuts to hod Lemma 5 Ifty orm bouds Let M R satsfy the bock sparsty assumpto S Let u, v satsfy the assumpto u, v µ 1/ The, we have

2 Robust Tesor Decomposto uder Bock Sparse Perturbato 1 Mu, v, I κµ 1/ M, where κ := Bd µ [Mu, v, I] p κµ M M p 1 for p > 1 3 p 1 [Mu, I, I]p v κµ M M 1 M whe M < 1/ Proof We have from orm coverso Mu, v, I u v max MI, I, e j 1 11 j µ Bd M, 1 where 1 orm e sum of absoute vaues of etres of a sce MI, I, e j s Bd, sce the umber of o-zero etres oe bock a sce s d Let Z = Mu, I, I R Now, Mu, I, I p v = Z p v = Z p 1 a where a = Zv Now, Z p 1 a = max e T j Z p 1 a Z p 1 a Z p 1 a M p 1 a κµ M M p 1 j Hece, p 1 [Mu, I, I]p v κµ M M 1 M B Proof of Theorem 1 Lemma 6 Let L, S be symmetrc ad satsfy the assumptos of Theorem 1 ad et S t be the t th terate of the th stage of Agorthm 1 Let 1,, r be the egevaues of L, such that 1 r 0 ad λ 1,, λ r be the egevaues of T S t such that λ 1 λ r 0 Reca that E t := S S t Suppose further that 1 E t 8µ3 k 3/ supp E t supp S t 1, ad The, for some costat c [0, 1, we have t 1 1 c +1 + λ +1 + t 1 λ 1 + c t 13 Proof Note that T S t = L + E t Now, where γ t := So we have: λ +1 + λ+1 +1 E 8 t 8d 3/ E t 8µ3 rγ t 3/ d3/, t 1 That s, λ µ 3 r d t 1 λ +1 + t 1 8µ3 r 3/ γt Smary, λ 8µ3 r d 3/ γt 3/ d γ t 1 + 3/ d 16µ 3 r γ t t 1 c +1 +, t 1 /3 where the ast equaty foows from the boud d c µ 3 k for some costat c

3 Lemma 7 Assume the otato of Lemma 6 Aso, et L t, S t be the t th terates of r th stage of Agorthm 1 ad L t+1, S t+1 be the t + h terates of the same stage Aso, reca that E t := S S t ad E t+1 := S S t+1 Suppose further that 1 E t 8µ3 r 3/ supp E t supp S t 1, ad 3 E t < C, where C < 1/ s a suffcety sma costat The, we have: L t+1 L µ3 r 3/ t Proof Let L t+1 = =1 λ u t+1 be the ege decomposto obtaed usg the tesor power method o T S t at the t + h step of the th stage Aso, reca that T S t = L + E t where L = r j=1 j u 3 j Defe E t := S S t Defe E := E t u t+1, I, I Let E t := ɛ Cosder the egevaue equato T S t u t+1, u t+1, I = λ u t+1 L u t+1, u t+1, I + E t u t+1, u t+1, I = λ u t+1 r, u j uj + E t u t+1, u t+1, I = λ u t+1 j=1 u t+1 [λ I E t u t+1, I, I]u t+1 = u t+1 = r j=1 I + p 1 : u t+1, u j uj E p λ r j=1 λ u t+1, u j uj Now, L t+1 L [] λ u t+1 3 [] 3 u + λ u t u + [] r =+1 r =+1 3 u u 3 For a fxed, usg λ + ɛ [] ad usg Lemma 11, we obta λ u t u + ɛu t u u t u + ɛ u t u + ɛ [3 u t+1 + ɛ u t u t+1 u t+1 u t+1 u u + 3 u t u u + ɛ u t+1 3 u u + u t+1 u 3 ]

4 Robust Tesor Decomposto uder Bock Sparse Perturbato Now, u t+1 u = r r u t+1, u j uj u + λ j=1 j=1,p 1 1 u u t+1, u + λ λ j + u t+1 E p, u u + λ λ p 1 λ u t+1 p,j u t+1 λ, u j E p u j, u j uj u t+1 E p, u j u j λ For the frst term, we have 1 λ u t+1, u u where we substtute for ɛ the ast step For the secod term, we have λ j whch s a ower order term Next, λ p 1 u t+1 u t ɛ µ ɛ Cµ 1/, u j uj E p, u u λ ɛ 1 u ɛ ɛ µ ɛ u, 1/ p 1 E p u λ λ λ p 1 µ E t /λ λ 1 E t /λ κ t µ E t 1 C λ from Lemma 5, ad the assumpto o spectra orm of E t, where κ t := Bd µ 1 1 ɛ µ 1/ E p u λ For the remag terms, we have u t+1 E p, u j u j λ λ p,j j whch s a ower order term u t+1 λ, u j p 1 E λ p u 1 Combg the above ad recag ɛ, [] u t+1 8 κ t µ u E t 1 C λ Aso, from Lemma 1 λ 8 E t 8ɛ λ p 1 E λ p u 1 ɛ,

5 Thus, from the above two equatos, we obta the boud for the parameters egevectors ad egevaues of the ow-rak tesor u t+1 u ad λ We combe the dvdua parameter recovery bouds as: λ u t u r[7 u t+1 u u + ɛ u t+1 3 ] [] [] ad the other term r =+1 Combg boud 14 wth the above, we have L t+1 L rµ κ t µ 3 r 1 C 15 Et 14 u 3 +1 rµ C κ t E t + +1 < 1 4 Et where the ast equaty comes from the fact that rµ Et 8 ad the assumpto that C < 1/, ad we ca choose B ad d st 448rµ 3 15 κ t < 1 8 Ths s possbe from assumpto S The foowg emma bouds the support of E t+1 ad E t+1, usg a assumpto o L t+1 L Lemma 8 Assume the otato of Lemma 7 Suppose The, we have: L t+1 L µ3 r 3/ 1 supp E t+1 supp S E t+1 7 µ3 r 3/ +1 +, ad Proof We frst prove the frst cocuso Reca that, t S t+1 = H ζ T L t+1 = H ζ L L t+1 + S, where ζ = 4 µ3 r λ 3/ +1 + λ s as defed Agorthm 1 ad λ 1,, λ are the egevaues of T S t such that λ 1 λ If S = 0 the Et+1 jk = 1 L L t+1 >ζ L Lt+1 The frst part of the emma ow foows by usg the assumpto that L t+1 L µ3 r 3/ +1 + ζ1 4 µ3 r λ 3/ +1 + λ = ζ, where ζ 1 foows from Lemma 6 We ow prove the secod cocuso We cosder the foowg two cases: 1 T L t+1 > ζ: Here, S t+1 µ3 r / T L t+1 ζ + L Lt+1 3/ = S + L Lt+1 Hece, S t+1 S L Lt+1 ζ: I ths case, S t+1 = 0 ad S + L Lt+1 ζ So we have, E t+1 7 µ3 r +1 + The ast equaty above foows from Lemma 6 = S

6 Robust Tesor Decomposto uder Bock Sparse Perturbato Ths proves the emma Theorem 9 Let L, S be symmetrc ad satsfy L ad S, ad β = 4 µ3 r The outputs L ad ts parameters 3/ û ad ˆλ ad Ŝ of Agorthm 1 satsfy whp: δ û u µ r 1/ m, ˆλ δ, [], L L F δ, Ŝ S δ, ad supp Ŝ supp 3/ S Proof Reca that the th stage, the update L t+1 s gve by: L t+1 = P T S t ad S t+1 s gve by: S t+1 = H ζ T L t+1 Aso, reca that E t := S S t ad E t+1 := S S t+1 We prove the emma by ducto o both ad t For the base case = 1 ad t = 1, we frst ote that the frst equaty o L 0 L s trvay satsfed Due to the threshodg step step 3 Agorthm 1 ad the coherece assumpto o L, we have: So the base case of ducto s satsfed 8µ 3 r 3/ E 0 8µ3 r 3/ + 1, ad supp E 0 supp S We frst do the ductve step over t for a fxed r By ductve hypothess we assume that: a E t , b supp E t supp S The by Lemma 7, we have: Lemma 8 ow tes us that 1 E t+1 8µ3 r 3/ +1 +, ad supp E t+1 supp S L t+1 L µ3 r 3/ t Ths fshes the ducto over t Note that we show a stroger boud tha ecessary o E t+1 We ow do the ducto over Suppose the hypothess hods for stage Let T deote the umber of teratos each stage We frst obta a ower boud o T Sce T S 0 L E 0 1 d 3/ E , we see that T 10 og 3µ 3 r 1/δ So, at the ed of stage r, we have: 1 E T 7µ3 r 3/ T supp E T supp S 7µ3 r +1 3/ + δ 10, ad Reca, r+1 T S T r+1 E T d µ 3 r r+1 + δ We w ow cosder two cases: 1 Agorthm ermates: Ths meas that β r+1 T S T < δ whch the mpes that 3/ r+1 < δ So we have: T L L = L T L µ3 r 1 3/ r+1 + r δ 5 3/ Ths proves the statemet about L ad ts parameters egevaues ad egevectors A smar argumet proves the cam o Ŝ S The cam o supp Ŝ foows sce supp ET supp S 6µ 3 r

7 Agorthm 1 cotues to stage r + 1: Ths meas that β r+1 L T δ r+1 > So we have: δ 8µ 3 r Smary for L T L Ths fshes the proof E T 8µ3 r 3/ r+1 + 8µ3 r 3/ 8µ3 r 3/ T r δ 10µ 3 r 3/ µ3 r 3/ / whch the mpes that B1 Short proof of Coroary 1 The state of art guaratees for robust matrx PCA requres that the overa sparsty aog ay row or coum of the put matrx be D = O rµ whe the put matrx s R Uder S, the tota sparsty aog ay row or coum of M s gve by D := db Now, Theorem 1 hods whe the sparsty codto 5 s satsfed That s, RTD succeeds whe 4/3 D = Od B = O m r 1/3 µ, /3 r /3 µ r 1/3 = O rµ Hece, RTD ca hade arger amout of corrupto tha the matrx methods ad the ga becomes more sgfcat for smaer η B Some auxary emmas We reca Theorem 51 from [] Let ɛ = 8 E t where E t := S S t Lemma 10 Let L t+1 = k =1 λ u t+1 be the ege decomposto obtaed usg Agorthm 1 o T S t The, 1 If u t+1 u ɛ, the dstu t+1 m, u ɛ m j u t+1, u j ɛ m 3 u t+1 µ + ɛ 1/ m 4 ɛ λ + ɛ Proof 1 Let z u ad z = 1 u t+1 = u t+1 u = u t+1, u u t+1, u dstu t+1, u u + dstu t+1, u z 1 u + dstu t+1, u z + 0 The usg Theorem 51 from [], we obta the resut Next, sce u t+1, u + dstu t+1, u = 1, we have u t+1, u ɛ 1 m

8 Robust Tesor Decomposto uder Bock Sparse Perturbato Note that u t+1 = k j=1 u t+1, u j u j + dstu t+1, Uz where z U such that z = 1 Usg u t+1 = 1 ad the Pythagoras theorem, we get 1 u t+1, u = u t+1 t+1, u j + dstu, U 1 u t+1, u j j j Usg part 1 of Lemma 10, we get j u t+1, u j ɛ m 3 We have u t+1 = u t+1 u t+1, u u + dstu t+1, u z u t+1, u u + dstu t+1 µ, u z 1 + 1/ ɛ m 4 Ths foows from Theorem 51 from [], e,, λ ɛ Lemma 11 Let a = b+ɛ 1 where a, b are ay vectors ad ɛ > 0 The, a 3 b 3 a b b +Oɛ Proof We have a 3 b 3 = b + ɛ 1 3 b 3 Let, j, k be the maxmum eemet Therefore, b + ɛ 1 3 b 3 = b + ɛb j + ɛb k + ɛ b b j b k = ɛb b j + b j b k + b k b + ɛ b + b j + b k + ɛ 3 Wth b c for some c > 0 ad ɛ = a b, we have a 3 b 3 3ɛc + Oɛ C Symmetrc embeddg of a asymmetrc tesor We use the symmetrc embeddg syml of a tesor L as defed Secto 3 of [4] We focus o thrd order tesors whch have ow CP-rak We have three propertes to derve that s reevat to us: 1 Symmetry: From Lemma of [4] we see that syml for ay tesor s symmetrc CP-Rak: From Equato 65 of [4] we see that CP-raksymL 6CP-rakL Sce ths s a costat, we see that the symmetrc embeddg s aso a ow-rak tesor 3 Icoherece: Theorem 47 of [4] says that f u 1, u ad u 3 are ut moda sguar vectors of T, the the vector ũ = 3 1/ [u 1 ; u ; u 3 ] s a ut egevector of symt Wthout oss of geeraty, assume that T s of sze 1 3 wth 1 3 I ths case, we have ũ µ 3 1 1/ 15 ad ũ µ / 16 for µ = cµ for some costat c to be cacuated Equatg the rght had sdes of Equatos 15 ad 16, we obta c = [ /3 1 ] 1/ Whe Θ 1 = Θ = Θ 3, we see that the egevectors ũ of symt as specfed above have the coherece-preservg property

9 D Proof of Theorem Let L be a symmetrc tesor whch s a perturbed verso of a orthogoa tesor L, L = L + E R, L = [r] u 3, where 1 r > 0 ad {u 1, u,, u r } form a orthoorma bass The aayss proceeds teratvey Frst, we prove covergece to egepar of L, whch s cose to top egepar 1, u 1 of L We the argue that the same hods o the defated tesor, whe the perturbato E satsfes 8 from Ths fshes the proof of Theorem To prove covergece for the frst stage, e covergece to egepar of L, whch s cose to top egepar 1, u 1 of L, we aayze two phases of the shfted power terato I the frst phase, we prove that wth N 1 tazatos ad N power teratos, we get cose to true top egepar of L, e 1, u 1 After ths, the secod phase, we prove covergece to a egepar of L The proof of the secod phase s outed the ma text Here, we ow provde proof for the frst phase D1 Aayss of frst phase of shfted power terato I ths secto, we prove that the output of shfted power method s cose to orga egepars of the uperturbed orthogoa tesor, e Theorem hods, except for the property that the output correspods to the egepars of the perturbed tesor We adapt the proof of tesor power terato from [] but here, sce we cosder the shfted power method, we eed to modfy t We adopt the otato of [] ths secto Reca the update rue used the shfted power method Let θ t = k =1 θ,tv R k be the ut vector at tme t The θ t+1 = k θ,t+1 v := T I, θ t, θ t + αθ t / T I, θ t, θ t + αθ t =1 I ths subsecto, we assume that T has the form where {v 1, v,, v k } s a orthoorma bass, ad, wthout oss of geeraty, T = k =1 λ v 3 + Ẽ 17 λ 1 θ 1,t = max λ θ,t > 0 [k] Aso, defe λ m := m{ λ : [k], λ > 0}, λmax := max{ λ : [k]} We assume the error Ẽ s a symmetrc tesor such that, for some costat p > 1, ẼI, u, u ɛ, u Sk 1 ; 18 ẼI, u, u ɛ/p, u Sk 1 st u v ɛ/ λ 1 19 I the ext two propostos Propostos D1 ad D ad Lemmas D1, we aayze the power method teratos usg T at some arbtrary terate θ t usg oy the property 18 of Ẽ But throughout, the quatty ɛ ca be repaced by ɛ/p f θ t satsfes θt v ɛ/ λ 1 as per property 19 Defe for τ {t, t + 1} θ 1,τ R τ := 1 θ1,τ γ τ := 1 1/, r,τ := λ 1 θ 1,τ λ θ,τ, 1 m 1 r,τ, δ τ := ɛ, κ := λ max λ 1 θ1,τ λ 1 0

10 Robust Tesor Decomposto uder Bock Sparse Perturbato Proposto D1 Proposto D m r,t R t 1 κ, γ t 1 κ, θ1,t = R t R t 1 + Rt r,t+1 r,t α λ 1 + κδ t r,t + α λ θ,t, [k], 1 R t+1 = R t 1 θ 1,t 1 γ t + δ t + α1 θ1,t1/ R λ 1θ1,t t κ R t α λ 1 θ 1,t + δ t + α1 θ1,t1/ λ 1θ 1,t Proof Let ˇθ t+1 := T I, θ t, θ t + αθ t, so θ t+1 = ˇθ t+1 / ˇθ t+1 Sce ˇθ,t+1 = T v, θ t, θ t = T v, θ t, θ t + αθ t + Ev, θ t, θ t, we have ˇθ,t+1 = λ θ,t + Ev, θ t, θ t + αθt v, [k] By defto, we have θ,t = θt v Usg the trage equaty ad the fact Ev, θ t, θ t ɛ, we have ˇθ,t+1 λ θ,t ɛ + αθ,t θ,t λ θ,t ɛ/ θ,t + α ad for a [k] Combg 3 ad 4 gves r,t+1 = λ 1 θ 1,t+1 λ θ,t+1 = λ 1 ˇθ1,t+1 ˇθ,t+1 λ θ,t + ɛ + αθ,t θ,t λ θ,t + ɛ/ θ,t + α λ ˇθ,t+1 r,t 1 + λ ɛ + α θ,t λ θ,t = r,t 1 + λ / λ 1 δ t r,t + θ,t Moreover, by the trage equaty ad Höder s equaty, 1/ 1/ [ˇθ,t+1 ] = λ θ,t + Ev, θ t, θ t + αθ,t = Combg 3 ad 5 gves = = 1/ λ θ,t 4 + max λ θ,t 1 = = 1 θ1,t 1/ θ 1,t+1 1 θ1,t+1 = ˇθ 1,t+1 1/ 1/ = [ˇθ,t+1 ] I terms of R t+1, R t, γ t, ad δ t, ths reads R t+1 1 γ t = 1 γ t R t + = θ,t 1/ + ɛ + max 1 1/ k Ev, θ t, θ t + α k = θ,t = 1/ r,t α θ,t κδ t r,t + θ,t 1/ λ θ,t + ɛ/1 θ1,t 1/ + α 5 θ 1,t 1 θ 1,t 1/ 1 θ 1,t 1/ + δt + α1 θ1,t1/ λ 1θ1,t 1 θ 1,t θ 1,t 1 θ 1,t δ t + α1 θ 1,t 1/ λ 1θ1,t κ R t λ 1 θ 1,t ɛ/ θ 1,t + α max 1 λ θ,t + ɛ/1 θ 1,t 1/ + α α = R t 1 γ t + α λ 1 θ 1,t + δ t + α1 θ 1,t 1/ λ 1θ 1,t λ 1 θ 1,t δ t + α1 θ 1,t 1/ λ 1θ 1,t R t where the ast equaty foows from Proposto D1

11 Lemma D1 Fx ay ρ > 1 Assume ad γ t > 1 + κρ δ t 1 If r,t ρ, the r,t+1 r,t 1 + γt Proof By 1 from Proposto D, r,t+1 r,t { 1 0 δ t < m 1 + κρ, 1 1/ρ } 1 + κρ α λ 1 + κδ t r,t + α λ θ,t r,t where the ast equaty s see as foows: Let ξ = 1 1 γ t 1 + κρ δ t + θ,t 1 + κρ δ t + r,t θ,t 1 + γ t The, we have γt + γ t + ξ 0 The postve root s 1+9 4ξ1/ Sce γ t 0, we have 9 4ξ 1/ 1, so α we assume ξ for the equaty to hod, e, 1 + κρ λ δ t θ,t The rest of the proof s aog the smar es of [], except that we use SVD tazato stead of radom tazato The proof of SVD tazato s gve [3]