Marx reconrucon wh he local max norm Rna oygel Deparmen of Sac Sanford Unvery rnafb@anfordedu Nahan Srebro Toyoa Technologcal Inue a Chcago na@cedu Rulan Salakhudnov Dep of Sac and Dep of Compuer Scence Unvery of Torono ralakhu@uaoronoedu Supplemenary Maeral A Proof of Theorem 1 Specal cae: elemen-we upper bound r, we aume ha he general reul rue, e R,C = nf r A + c B AB =, 1 and prove he reul n he pecal cae, where R = r [n] : r R and C = c [m] : c C Ung rong dualy for lnear program, we have r A = n + = nf a R,a 1 R n + r A : r R, r = 1 a + R a 1 : a + a 1 A In h la lne, f we fx a and wan o mnmze over a 1 R n +, clear ha he nfmum obaned by eng a 1 = A a + for each Th prove ha r A = nf a + A R a a R + Applyng he ame reaonng o he column and pluggng everyhng n o 1, we ge R,C = nf a + AB =, a,b R R A a + b + + C B b + General facorzaon reul In he proof kech gven n he man paper, we howed ha R,C nf r 1 / A + c 1 / B AB = We now wan o prove he revere nequaly Snce R,C = R,C by defnon where S denoe he cloure of a e S, we can aume whou lo of generaly ha R and C are boh cloed and compac e 1
r, we rerc our aenon o a pecal cae he pove cae, where we aume ha for all r R and all c C, r > 0 and c > 0 for all and We wll rea he general cae below Therefore, nce rr,c connuou a a funcon of r, c for any fxed and nce R and C are cloed, we mu have ome r R and c C uch ha R,C = rr,c, wh r > 0 for all and c > 0 for all Nex, le UDV = r 1 / c 1 / be a ngular value decompoon, and le A = r 1 / UD 1 / and B = c 1 / V D 1 / Then A B =, and r 1 / A = UD 1 / = raceudu = raced = rr,c = R,C Below, we wll how ha r = arg max r 1 / A Th wll mply ha R,C = r 1/ A, and followng he ame reaonng for B, we wll have proved R,C = r 1 / A + c 1 / B nf r 1 / A + c 1 / B, AB = whch uffcen I reman only o prove Take any r R wh r r and le w = r r We have r 1/ A r 1 / A = w A = w r UDU, and wll be uffcen o prove ha h quany 0 To do h, we fr defne, for any [0, 1], f := r 1 + w r UDU 1/ + w = race r UDU Ung he fac ha race r for all marce, we have r 1/ + w f r UDU = r + w 1 / c 1 / V U r r = r + w 1 / c 1 / = rr +w,c R,C = UDU = f0, r where he la nequaly come from he fac ha r + w R by convexy of R Therefore, 0 d d =0 f = d 1 + w d r UDU = =0 1 w r UDU, a dered Here we ake he rgh-ded dervave, e akng a lm a approache zero from he rgh, nce f only defned for [0, 1] Th conclude he proof for he pove cae Nex, we prove ha he general facorzaon 1 hold n he general cae, where we mgh have R R n ++ and/or C R m ++ If for any [n] we have r = 0 for all r R, we can dcard h row of, and ame for any [m] Therefore, whou lo of generaly, for all [n] here ome r R wh r > 0 Takng a convex combnaon, r + = 1 n r + R R n ++ Smlarly, we can conruc c + C R m ++ r R, we have x any ɛ > 0, and le δ = mnmn r +, mn c + 1+ɛ > 0, and defne cloed ube R 0 = r R : mn r δ R and C 0 = c C : mn c δ C Snce we know ha he facorzaon reul hold for he pove cae, we have r 1/ nf A c 1/ + B = R0,C AB = 0 0 0 r 1/ = c 1 / r 1 / c 1 / = R,C 0, 0 r, r ɛ
Now chooe any facorzaon à B = uch ha r 1/à c 1/ + B 0 0 r 1/ Nex, we need o how ha à for B Chooe any r R, and le r = mn r 1 δ mn r +, r 1 / c 1 / 1 + ɛ / 3 r r 1/ no much larger han 0 à r δ + r + R Then δ mn r + mn r + mn r + = δ, 1 and o r R 0 We alo have r 1 δ mn r r + for all Therefore, /à r 1 1 δ mn r + 1/ r 1 /à 1 δ mn r + Snce h rue for any r R, applyng he defnon of δ, we have r 1 /à 1 δ mn r + 1/ r 1/à 0 1/ 1/ 1 + ɛ/ 1 + ɛ r 1/à 0 and ame r 1/à 0 Applyng he ame reaonng for B and hen pluggng n he bound 3, we have nf r 1 / A + c 1 / B r 1 /à + c 1 / B AB = 1 1 + ɛ/ r 1/à c 1 + ɛ + 1/ B 0 0 1 1 + ɛ/ 1 + 1 + ɛ ɛ / = 1 + ɛ R,C R,C Snce h analy hold for arbrary ɛ > 0, h prove he dered reul, ha nf r 1 / A + c 1 / B R,C AB = B Proof of Theorem We follow mlar echnque a ued by Srebro and Shrabman [1] n her proof of he analogou reul for he max norm We need o how ha Conv uv : u R n, v R m, u R = 1 : R,C 1 K G Conv uv : u R n, v R m, u R = 1 or he lef-hand ncluon, nce R,C a norm and herefore he conran R,C 1 convex, uffcen o how ha uv R,C 1 for any u R n, v R m wh u R = 1 Th a rval conequence of he facorzaon reul n Theorem 1 Now we prove he rgh-hand ncluon Grohendeck Inequaly ae ha, for any Y R n m and for any dmenon k, Y, UV : U R n k, V R m k, U 1, V 1 K G Y, uv : u R n, v R m, u 1, v 1, 3
where K G 167, 179 Grohendeck conan We now exend h o a lghly more general form Take any a R n + and b R m + Then, eng Ũ = daga+ U and Ṽ = dagb+ V where M + he peudonvere of M, and ame for ũ and ṽ, we ee ha Y, UV : U R n k, V R m k, U a, V b = daga Y dagb, ŨṼ : Ũ Rn k, Ṽ Rm k, Ũ 1, Ṽ 1 K G daga Y dagb, ũṽ : ũ R n, ṽ R m, ũ 1, ṽ 1 = K G Y, uv : u R n, v R m, u a, v b 4 Now ake any Y R n m Le R,C be he dual norm o he R, C-norm To bound h dual norm of Y, we apply he facorzaon reul of Theorem 1: Y R,C = R,C 1 = U,V = U,V = Y, Y, UV : 1 Y, UV : a R n + : a R 1 U,V b R m + : b C 1 r U + c V 1 r U = c V 1 Y, UV : U a, V b K G Y, uv : u a, v b a R n + : a R 1 U,V b R m + : b C 1 = K G Y, uv : u R 1, v C 1 u,v = K G Y, : Conv uv : u R n, v R m, u R = 1 = Y, : KG Conv uv : u R n, v R m, u R = 1 A n [1], h uffcen o prove he reul Above, he ep marked * rue becaue, gven any U and V wh 1 r U + c V 1, we can replace U and V wh U := U ω and V := V ω 1, where ω := 4 Th wll gve U V = UV, and U r = V c = 1 r U c V r U c V r U + c V 1 4
C Proof of Theorem 3 ollowng he raegy of Srebro & Shrabman 005, we wll ue he Rademacher complexy o bound h exce rk By Theorem 8 of Barle & Mendelon 00 1, we know ha Y p nf p Y E S R,C k = O where he expeced Rademacher complexy defned a [ E S RS R n m : R,C ] k := 1 E S,ν [ E S RS R n m : R,C ] k ν R,C, k, 5 where ν ±1 a random vecor of ndependen unbaed gn, generaed ndependenly from S Now we bound he Rademacher complexy By calng, uffcen o conder he cae k = 1 The man dea for h proof o fr how ha, for any wh R,C 1, we can decompoe no a um + where max K G and r p K G γ 1 /, where p repreen he moohed row and column margnal wh moohng parameer ζ = 1 /, and where K G 179 Grohendeck conan We wll hen ue known Rademacher complexy bound for he clae of marce ha have bounded max norm and bounded moohed weghed race norm To conruc he decompoon of, we ar wh a vecor decompoon lemma, proved below Lemma 1 Suppoe R R 1/,γ Then for any u Rn wh u R = 1, we can decompoe u no a um u = u + u uch ha u 1 and u prow := p u γ 1 / Nex, by Theorem, we can wre = K G l u l vl, l=1 where l 0, l=1 l = 1, and u l R = v l C = 1 for all l Applyng Lemma 1 o u l and o v l for each l, we can wre u l = u l + u l and v l = v l + v l, where u l 1, u l prow γ 1 /, v l 1, v l pcol γ 1 / Then = K G l u lv l + l u lv l + l=1 l=1 l=1 l u l v l =: K G 1 + + 3 urhermore, u l p row u l 1, and v v l prow l C 1 Applyng Srebro and Shrabman [1] convex hull bound for he race norm and max norm aed n Secon 4 of he man paper, we ee ha 1 max 1, and ha ha r p γ 1 / for =, 3 Defnng = 1 and = + 3, we have he dered decompoon Applyng h reul o every n he cla R n m : R,C 1, we ee ha [ ] E S RS R n m : R,C 1 [ ] [ E S RS : max K G + E S RS K G O n + K G γ 1 / O ] : r p K G γ 1 / n logn + n logn, 1 The aemen of her heorem gve a reul ha hold wh hgh probably, bu n he proof of h reul hey derve a bound n expecaon, whch we ue here 5
where he la ep ue bound on he Rademacher complexy of he max norm and weghed race norm un ball, hown n Theorem 5 of [1] and Theorem 3 of [], repecvely nally, we wan o deal wh he la erm, n logn, ha oude he quare roo Snce n by aumpon, we have n logn n log n, and f n logn, hen we can mprove h o n logn n logn Reurnng o 5 and pluggng n our bound on he Rademacher complexy, h prove he dered bound on he exce rk C1 Proof of Lemma 1 or u R n wh u R = 1, we need o fnd a decompoon u = u + u uch ha u 1 and u prow = p u γ 1 / Whou lo of generaly, aume u 1 u n nd N 1,, n and 0, 1] o ha N 1 =1 p + p N = γ 1, and le r = γ p 1,, p N 1, p N, 0,, 0 [n] Clearly, r γ p for all, and o r R 1/,γ R Now le u = u 1,, u N 1, u N, 0,, 0, and e u = u u We hen calculae u p row = N 1 =1 p u + p N u N = γ 1 n =1 r u γ 1 u R γ 1 nally, we wan o how ha u 1 Snce u = 0 for < N, we only need o bound u for each N We have n 1 = u R r u N r u =1 =1 u N =1 r # = u u, where he ep marked * ue he fac ha u u for all N, and he ep marked # come from he fac ha r pored on 1,, N Th uffcen D Proof of Propoon 1 Le L 0 = Lo Then, by defnon, = arg mn Penaly β,τ : Lo L 0 Then o prove he lemma, uffcen o how ha for ome [0, 1], = arg mn R,C : Lo L 0, where we e R = r [n] : r 1 + n 1, C = c [m] : c 1 + m 1 Trvally, we can rephrae hee defnon a R = 1 + n 1 1,, 1 + 1 1 + n 1 r : r [n] and C = 1 + m 1 1,, 1 + 1 1 + m 1 c : c [m] 6 Noe ha for any vecor u R n + and v R m +, r u = max u and r [n] c [m] c v = max v 7 6
Applyng he SDP formulaon of he local max norm proved n Lemma below, we have R,C = 1 nf U r U + c V : 0 V By 6 and 7 1 = nf 1 + n 1 1 U + 1 + n 1 max U + 1 + m 1 1 U V + 1 + m 1 max V : 0 V = ω nf A + 1 max A + A B + 1 max B : 0 B = ω 1 nf MA, B + TA, B :, 8 where for he nex-o-la ep, we defne A = U 1 + m 1 1 + n 1, B = V 1 + n 1 1 + m 1, ω 1 =, 1 + n 1 1 + m 1 and for he la ep, we defne TA, B = racea + raceb, MA, B = max A + max B, and = A : B 0 Nex, we compare h o he β, τ penaly formulaed n our man paper Recall Penaly β,τ = nf max A =AB + max B A + B Applyng Lemma 3 below, we can oban an equvalen SDP formulaon of he penaly Penaly β,τ = nf MA, B TA, B : 9 Snce MA, B TA, B maxn, mma, B, and nce for any x, y > 0 we know xy α x + α 1 y for any α > 0 wh equaly aaned when α = y/x, we ee ha 1 Penaly β,τ = 1 nf = nf α [1, maxn,m] nf α [1, maxn,m] [ 1 nf α MA, B + α 1 TA, B : α MA, B + α 1 TA, B : ] Snce he quany nde he quare bracke nonnegave and connuou n α, and we are mnmzng over α n a compac e, he nfmum aaned a ome α, o we can wre Penaly β,τ = 1 α nf MA, B + α 1 TA, B : 7
Recall ha mnmze Penalyβ,τ ubec o he conran Lo L 0 Seng := α 1 α+ α 1, we ge arg mn = arg mn nf a dered = arg mn nf α MA, B + α 1 TA, B : : Lo L0 α MA, B + α 1 α + α 1 nf α + α 1 TA, B : = arg mn : Lo L 0 1 MA, B + TA, B : : Lo L 0 R,C : Lo L 0, E Compung he local max norm wh an SDP Lemma Suppoe R and C are convex, and are defned by SDP-repreenable conran Then he R, C-norm can be calculaed wh he emdefne program R,C = 1 nf A r A + c B : 0 B In he pecal cae where R and C are defned a n 8 n he man paper, hen he norm gven by R,C = 1 a nf + R a 1 + b + C b 1 : a 1 0 and a + a 1 A, A b 1 0 and b + b 1 B, B 0 Proof or he general cae, baed on Theorem 1 n he man paper, we only need o how ha nf A r A + c B : 0 B = nf r A + c B : AB = Th proved n Lemma 3 below or he pecal cae where R and C are defned by elemen-we bound, we reurn o he proof of Theorem 1 gven n Secon A, where we ee ha R,C = nf a+r a 1 +b+c b 1 : a+a 1 A, b+b AB 1 B =,a,b R a 1 R n +,b1 Rm + Nong ha A AA and B BB, we agan ue Lemma 3 o ee ha h equvalen o he SDP nf a + R a 1 + b + C b 1 : a 1 0 and a + a 1 A, A b 1 0 and b + b 1 B, 0 B 8
Lemma 3 Le f : R n R m R be any funcon ha nondecreang n each coordnae and le R n m be any marx Then A1 nf f,, An, B1,, Bm : AB = Φ = nf f Φ 11,, Φ nn, 11,, mm : 0, where he facorzaon AB = aumed o be of arbrary dmenon, ha, A R n k and B R m k for arbrary k N Proof We follow mlar argumen a n Lemma 14 n [3], where h equaly hown for he pecal cae of calculang a race norm or convenence, we wre and ga, B = f Then we would lke o how ha A1,, An, B1,, Bm hφ, = f Φ 11,, Φ nn, 11,, mm nf ga, B : AB = = nf Φ hφ, : 0 Φ r, ake any facorzaon AB = Le Φ = AA and = BB Then 0, and we have ga, B = hφ, by defnon Therefore, nf ga, B : AB = Φ nf hφ, : 0 Φ Nex, ake any Φ and uch ha Φ = A 0 B C A 0 B C 0 Take a Choleky decompoon AA AB = BA BB + CC rom h, we ee ha AB =, ha Φ = A for all, and ha B for all Snce f nondecreang n each coordnae, we have hφ, ga, B Therefore, we ee ha nf ga, B : AB = Φ nf hφ, : 0 Reference [1] N Srebro and A Shrabman Rank, race-norm and max-norm 18h Annual Conference on Learnng Theory COLT, page 545 560, 005 [] R oygel, R Salakhudnov, O Shamr, and N Srebro Learnng wh he weghed race-norm under arbrary amplng drbuon Advance n Neural Informaon Proceng Syem, 4, 011 [3] N Srebro Learnng wh marx facorzaon PhD he, Ceeer, 004 9