Supplementary Material: Accelerating Adaptive Online Learning by Matrix Approximation

Transcription

1 Supplemenary Maerial: Acceleraing Adapive Online Learning by Marix Approximaion Yuanyu Wan and Lijun Zhang Naional Key Laboraory for Novel Sofware Technology Nanjing Universiy, Nanjing 1003, China A Theoreical Analysis In his supplemenary maerial, we provide proof of Theorems 1 and. A.1 Supporing Resuls The following resuls are used hroughou our analysis. Lemma 1. (Proposiion 3 of [1]). Le sequence {β } be generaed by ADA-RP. We have R(T ) 1 η [ BΨ+1(β, β + B Ψ (β, β + ] + 1 η B Ψ 1 (β, β + η f (β ) Ψ. Lemma. Le X = i=1 x ix i and A denoe he pseudo-inverse of A, hen x, (X 1/ ) x x, (X 1/ T ) x = r(x 1/ T ). Lemma can be proved in he same way as Lemma 10 of [1]. Theorem 3. (Theorem.3 of [11]). Le 0 <, δ < 1 and S = 1 k R R k n where he enries R i,j of R are independen sandard normal random variables. Then if k = Θ( d+log(1/δ) ), hen for any fixed n d marix A, wih probabiliy 1 δ, simulaneously for all x R d, (1 ) Ax SAx (1 + ) Ax. Based on he above heorem, we derive he following corollary. Corollary 1. Le 0 <, δ < 1 and each enry of r R τ is a Gaussian random variable drawn from N (0, 1/ τ) independenly. Then, if τ = Ω( r+log(t/δ) ), wih probabiliy 1 δ, simulaneously for all = 1,..., T, (1 )C C S S (1 + )C C.

2 Theorem. (Theorem 10 of [0]). Le R be a Gaussian random marix of size p n. Le C = diag(c 1,..., c p ) and S = diag(s 1,..., s p ) be p p diagonal marices, where c i 0 and c i + s i = 1 for all i. Le M = C + 1 n SRR S and r = i s i. P r(λ 1 (M) 1 + ) q exp ( cn ), max i (s i )r ) P r(λ p (M) 1 ) q exp ( cn max i (s i )r, where he consan c is a leas 1/3, and q is he rank of S. Based on he above heorem, we derive he following corollary. σ i, r α+σi = max 1 T σ 1. Le K = αi d + C C, K = αi d + S S, and Corollary. Le c 1/3, α > 0, σ i = λ i(c C ), r = i max r and σ 1 = k T Ĩ = K / K K / r. If τ σ 1 c (α+σ 1 1 δ, simulaneously for all = 1,, T, dt log ) δ (1 )I d Ĩ (1 + )I d., hen wih probabiliy a leas A. Proof of Theorem 1 Le X denoe S S. Firs, we consider bounding he firs erm in he upper bound of Lemma 1. Wih probabiliy 1 δ, we have B Ψ+1 (β, β + B Ψ (β, β + = 1 β β +1, ( X 1/ 1/ +1 X )(β β + 1 β β +1, 1 + X 1/ +1 (β β + 1 β β +1, 1 X 1/ (β β + 1 β β +1, (X 1/ +1 X1/ )(β β + + β β +1, (X 1/ +1 + X1/ )(β β + 1 β β +1 (X 1/ +1 X1/ ) + β β +1, (X 1/ +1 + X1/ )(β β + 1 β β +1 r(x 1/ +1 X1/ ) + β β +1, (X 1/ +1 + X1/ )(β β +

3 where he firs inequaliy is due o Corollary 1. Thus, we can ge 1 [ BΨ+1(β, β + B Ψ (β, β + ] β β +1 r(x 1/ +1 X1/ ) + β β +1, (X 1/ +1 + X1/ )(β β + 1 max T β β r(x 1/ T ) 1 β β 1 r(x 1/ + max T β β X 1/ β β 1 r(x 1/. Noe ha β 1 = 0, hen B Ψ1 (β, β = 1 β 1/, (σi d + X β 1 σ β + + () β r(x 1/ where he inequaliy is due o Corollary 1. Then, we consider he upper bound of T f (β ) Ψ. Wih probabiliy 1 δ, we have 1 f (β ) Ψ = 1/ g, (σi d + X ) g 1 g, (X ) 1/ g = l (β x ) x, (X ) 1/ x 1 1 (3) where he inequaliy is due o Corollary 1. According o Lemma, we have f (β ) Ψ T l (β x ) x, (X ) 1/ x 1 max T l (β x ) T x, (X ) 1/ x 1 1 max T l (β x ) r(x 1/ T ). We complee he proof by subsiuing (3), (), and (5) ino Lemma 1. (5) A.3 Proof of Theorem Inspired by he proof of Theorem 1, we can derive Theorem by respecively bounding each erm in he upper bound of Lemma 1. Before ha, we need o derive he lower and upper bounds of (S S ) 1/ based on Corollary.

4 Le he SVD of C be C = UΣV where U R d d, Σ R d d, V R d. According o Corollary, wih probabiliy a leas 1 δ, simulaneously for all = 1,..., T, S S = K αi d = K 1/ Ĩ K 1/ αi d and (1 + )K αi d = (1 + )C C + αi d = U ( (1 + )ΣΣ + αi d ) U S S + αi d = K αi d + αi d = K 1/ Ĩ K 1/ αi d + αi d (1 )K αi d + αi d = (1 )C C. Then simulaneously for all = 1,..., T, we have and (S S ) 1/ 1 + U(ΣΣ) 1/ U + αui d U = 1 + X 1/ + αi d (6) (S S ) 1/ = (S S ) 1/ + αi d αi d ( (S S ) + αi d ) 1/ αid 1 X 1/ αi d. Then we consider bounding he firs erm in he upper bound of Lemma 1. Le X denoe S S. Simulaneously for all = 1,..., T, we have B Ψ+1 (β, β + B Ψ (β, β + = 1 β β +1, ( X 1/ 1/ +1 X )(β β + 1 β β +1, 1 + X 1/ +1 (β β + 1 β β +1, 1 X 1/ )(β β β β +1, αi d (β β + = 1 β β +1, 1 + X 1/ +1 (β β + 1 β β +1, 1 X 1/ )(β β + + α (β β + 1 β β +1 r(x 1/ +1 X1/ ) + β β +1, (X 1/ +1 + X1/ )(β β + + α (β β + (7)

5 where he firs inequaliy is due o (6), (7) and he las inequaliy has been proved in he proof of Theorem 1. Thus, we can ge [ BΨ+1 (β, β + B Ψ (β, β + ] Noe ha β 1 = 0, hen 1 max T β β r(x 1/ T ) 1 β β 1 r(x 1/ + max T β β β β 1 r(x 1/ X 1/ + α(t 1) max T β β. B Ψ1 (β, β = 1 β 1/, (σi d + X β 1 σ β + + β r(x 1/ + 1 α β. (8) (9) Before considering he upper bound of T f (β ) Ψ, we need o derive he upper bound of H. Le he SVD of S be S = UΣV where U R d d, Σ R d d, V R d. We also have, for all = 1,..., T, H = σi d + (S S ) 1/ = U ( σi d + (ΣΣ) 1/) U U ( αi d + (ΣΣ) ) 1/ U = (αi d + S S ) 1/ due o σ α λ i (S S ) + α λ i (S S ) for all i = 1,, d. Then according o Corollary, wih probabiliy a leas 1 δ, simulaneously for all = 1,..., T, H ( (αi d + S S ) 1/) ( 1/ = (K Ĩ K 1/ 1 (K ) 1/ 1 ( = (αid + X ) ) 1/. 1 1 ) ) 1/ Thus, we can ge f (β ) Ψ = g, H g g, ( (αi d + X ) ) 1/ g 1 = l (β x ) 1 x, (X ) 1/ x.

6 According o Lemma, we have f (β ) Ψ 1 max T l (β x ) x, (X ) 1/ x (10) max 1 T l (β x ) r(x 1/ T ). We complee he proof by subsiuing (8), (9), and (10) ino Lemma 1. A. Proof of Corollary 1 Le C = UΣV be he singular value decomposiion of C. Noice ha U R r, ΣV R r d. According o Theorem 3, we have if τ = Θ( r+log(1/δ) ), hen simulaneously x R r, wih probabiliy 1 δ, (1 ) Ux R Ux (1 + ) Ux Le y R d be arbirary vecor, hen C y = UΣV y = Ux where x = ΣV y R r. Then we have y S S y = y C R R C y = R Ux (1 + ) Ux = (1 + )y C C y and y S S y = y C R R C y = R Ux (1 ) Ux = (1 )y C C y. Then, we have (1 )C C S S (1+)C C wih probabiliy 1 δ, provided τ = Ω( r+log(1/δ) ). Using he union bound, we have if τ = Ω( r+log(t/δ) ), wih probabiliy 1 δ, simulaneously for all = 1,..., T, (1 )C C S S (1 + )C C. A.5 Proof of Corollary Le he SVD of C be C = UΣV where U R d d, Σ R d d, V R d. Then we have K = U(αI d + ΣΣ )U and Ĩ = K / K K / = K / (αi d + C R R C )K / ( = U αi d (αi d + ΣΣ) + (αi p + ΣΣ) / ΣV R R VΣ(αI d + ΣΣ ) /) U ( = U αi d (αi d + ΣΣ) + (αi p + ΣΣ) / ΣRR Σ(αI d + ΣΣ ) /) U where R = V R R d τ is a Gaussian random marix due o ha V is an orhogonal marix and R is a Gaussian random marix.

7 α α+σ i Le c i = a leas 1 δ, and s i = r σ 1 σ i α+σi. Then according o Theorem, wih probabiliy (1 )I d Ĩ (1 + )I d provided τ c (α+σ1 δ where he consan c is a leas 1/3. Using he union bound, we complee he proof. ) log d