Gradient Descent and Non-Linear Iterative Methods

Size: px

Start display at page:

Download "Gradient Descent and Non-Linear Iterative Methods"

Florence Sullivan
5 years ago
Views:

1 CS7540 Spctral Algorithms, Spring 207 Lctur #25 #28 Gradint Dscnt and Non-Linar Itrativ Mthods Prsntr: Richard Png Apr &3&8&20, 207 DISCLAIMER: Ths nots ar not ncssarily an accurat rprsntation of what I said during th class Thy ar mostly what I intnd to say, and hav not bn carfully ditd Th plan for th nxt fw days is to talk about th mor gnral problm of minimizing a convx function, min x f (x ) Gradint Dscnt Th main ida is to tak gradint stps, x x η f (x ) whr η is th stp siz, and η f(x ) can b viwd as th stp, or x To bound th convrgnc of this mthod, a standard assumption is that th gradint of f is L-Lipschitz continuous Formally this is for any x and x w hav This implis: f (x ) f (x ) 2 L x x 2 f (x + x ) f (x ) + f (x ) T x + L 2 x Th Hssian matrix H(x ), which consists of all scond-ordr driviativs at x, satisfying H(x ) LI vrywhr Proof Intgrating th Lipschitz continuity quation, whr th incrmnt is rprsntd via a dirctional drivativ (t is a vctor in th dirction of x ), t=0 f(x + t x )δt f(x ) δt L t=0 t f (x + x ) f (x ) f (x ) T x + L 2 x t x δt whr th last stp is from taking dot product with x on both sids of th quation

2 2 Th dfinition of Hssian givs: x T H (x ) x = f (x + t x ) t=0 f (x + t x ) t=t f (x + t x ) t=0 = lim t 0 t = lim x T ( f (x + t x ) f (x )) t 0 t Th diffrnc btwn th gradints can in turn b boundd by th norm btwn thm, giving x T ( f (x + t x ) f (x )) x 2 t L x 2, which whn substitutd back in givs x T H (x ) x L x 2 2 x, Not that also w can gt () from (2) via using th 3-trm Taylor xpansion of f at f(x + x ) = f(x ) + f(x ) T x + 2 x T H( x ) x, for som x = x + θ x with 0 θ Now, using th fact that H( x ) LI, w gt This suggsts taking th stp f (x + x ) f (x ) + f (x ) T x + L 2 x 2 2 x = f (x ), L which givs a progrss of 2L f (x ) 2 2 So w nd to lowr bound f (x ) 2 As f is a convx function, th gradint condition givs: f (x ) f (x ) + f (x ) (x x ), which rarrangs into: f (x ) f (x ) f (x ) (x x ), upon which Cauchy Schawrz inquality givs: f (x ) f (x ) f(x ) 2 x x 2, 2

3 or that th progrss is at last f(x ) 2 f (x ) f (x ) x x 2 That is, if w know that at all points, x is clos to x, thn th progrss at ach stp is at last (f (x ) f (x )) 2 L x x 2 2 Strong Convxity If w also hav a lowr bound on th condition numbr of th Hssian, g λ min I H (x ), thn th condition givn by th intgral also xtnds to f (x ) + f (x ) T x + λ min 2 x 2 2 f (x + x ) Putting in x thn givs: f (x ) T (x x ) + λ min 2 x x 2 2 f (x ) f (x ), or that λ min 2 x x 2 2 f (x ) f (x ) This now mans th progrss mad at ach stp is at last λ min L (f (x ) f (x )), which is xactly th guarants of Richardson itration 2 Th Linar Cas W now look backwards at itrativ mthods for solving linar systms undr this sam viw Rcall that th norm of convrgnc is: f (x ) = x x 2 M Not that w do not hav accss to x, but can still comput gradints as it simplifis to: f (x ) = M (x x ) = M x b, Furthrmor, th Hssian of this, which is th gradint of th gradint, is just H (x ) = 2M Th standard condition for itrativ mthods is I M κi, which translats to smoothnss of κ, and strong convxity of 3

4 3 Th l Vrsion At fac valu, a function of th form f is not diffrntiabl and w cannot apply gradint dscnt to it Howvr, som form of prconditioning also xists for minimizing non-linar functions Th norm can b turnd into a smooth convx function with a wll dfind gradint: Givn a vctor v R d, th soft-max function is dfind as ) s max,t (f ) df = t ln ( xp (tf ) + xp ( tf ) Hr t > 0 is a paramtr which controls th accuracy at th xpns of smoothnss Th smoothnss of this function can in turn b masurd wrt th l norm: Lmma For any x and y, s max,t (x ) s max,t (y) t x y Proof This follows simply from th fact that soft-max is -Lipschitz wrt th l -norm So if w just want to minimiz f, gradint dscnt works Lmma 2 Givn π R n and any, th updat of π π satisfis: s max,t (π ) s max,t (π) T s max,t (π) + 2 Proof By th convxity of soft-max, s max,t (π ) s max,t (π) T s max,t (π ) = T s max,t (π) + T [ s max,t (π) s max,t (π )] Using Holdr s inquality, s max,t (π ) s max,t (π) T s max,t (π) + s max,t (π) s max,t (π ) T s max,t (π) + 2 Th last stp follows from Lmma 2 Prconditioning for Max-Flow Hr w nd to b mor spcific wrt th problms, spcifically maximum flow: min f () st B T f = d (2) 4

5 From th prvious sction, w can us soft-max to turn our objctiv into a smooth convx function Th main part that maks maximum flow difficult is th additional flow consrvation constraint B T f = d W can rlax this constraint into an objctiv via B T f d, but this nds to b solvd to high accuracy to nsur actual consrvation To gt somthing that w nd to solv to lowr accuracy, w will instad us an oblivious routing schm / cut approximat Rcall that th dual of th maximum flow problm is th minimum cut problm With our congstion minimization formulation, it can rprsnt th most congstd cut: opt (d) = max S V d S w (S, V \ S) Hr d S is simply th total amount of dmand in S, aka th minimum amount that nds to b routd from S to V \ S Th ky obsrvation is that this is linar, so th spac of all cuts is also a linar oprator Spcifically thr xists R such that: opt (d) = Rd What is particularly surprising is that for graphs, thr also xist approximations with far fwr rows Lmma 2 Givn any graph, thr xists a matrix R with about O(n) rows such that for any dmand d Rd opt (d) O(log 4 n) Rd Improving this O(log 4 n), or mor gnrally th trad-offs btwn th siz of R and this rror is opn What is vn mor surprising is that th following objctiv is O(polylog(n))-Lipschitz in th l l sns: s max,t (f ) + s max,t ( R ( B T f d )) To show this, it suffics to show that if f f 2 is small, thn RB T (f f 2 ) is small, and that if (x x 2 ) is small, thn so is B R T (x x 2 ) For th first on, not that d = B T (f f 2 ) is a valid dmand that can b satisfid by a flow with congstion at most f f 2 Th bound on Rd thn follows from Rd opt For th othr dirction, w can show th following Lmma instad: 5

6 Lmma 22 Proof Rcall A = A T A = max x Ax, it is givn by th maximum l norm of a row of A: w can gt quality by picking x to b th signs of that row On th othr hand, A T = max A T x x is looking for th colum of A T with th maximum norm: w can pick it out by stting x to b in that column and 0 vrywhr ls 2 Dtails on th Updat Stps Dfinition 23 (Oblivious Routing Oprator) An α-flow routing oprator Z is matrix st for any flow f, B T Z d = d, Z B T f α f Dfinition 24 (Flow Corrction Oprator) A flow corrction oprator P is matrix st for any flow f, Not that this can simply b dfind as B T (Pf + Z d) = d P df = I Z B T Lmma 25 For any st of vrtx labls x, thr xist f st x T ( B T f d ) 0 Proof This follows from dfinition of duality W actually only nd wak duality Th fact that B T f = d givs d T x = (f ) T Bx (3) Lt f b th flow with opposit signs as Bx Thn w hav f Bx (4) Bx (5) f T Bx = Bx d T x, 6

7 or x T (B T f d) = f T Bx d T x 0 W want a squnc of flows f (), f (2),, f (t) such that B T f (i) = d i t (6) Φ (t) df = ( ) t ( ) 2t xp f (i) xp (7) β β E i= To mak th ( notation asir ) lt Exp(β, t) dnot th E sizd vctor, whr th th lmnt is xp t i= β f (i) So, Φ (t) = Exp(β, t) Now, if w mak sur that ach incrmntal flow is rlativly small, thn w can stimat th convrgnc of th abov function Formally, lt f (j+) ɛβ Thn Φ (t+) = Exp(β, t + ) = ( ) t+ xp f (i) β E i= = ( ) t xp f (i) + β β f (i+) E i= Using th following proprty: w gt x ɛ = xp( + x) + ɛ 2 + x, Φ (t+) Φ (t) ( + ɛ 2 ) + E xp ( β t i= f (i) ) β f (i+) Now, w know from Lmma 25 that thr xists a ± vctor f such that [ Exp(β, t)t ( Z) ] T [B T f d] 0 So, stting f (i+) to cβ(pf + Z d) = cβ(f Z (B T f d)), w gt So, th rcurrnc bcoms ) (Exp(β, t) T f (i+) c ( Exp(β, t) T f ) β c Exp(β, t) 7

8 Φ (t+) ( + ɛ 2 + c)φ (t) = Φ (t) ( + ɛ 2 + c) t Φ (0) If w start with Φ (0) as polynomial in m, Φ (t) xp(tc + tɛ 2 + O(log m)) For convrgnc, w nd tc + tɛ 2 + O(log m) 2tc = ɛ 2 + O(log m)/t c Taking c ɛ 2 and t = O(log nɛ 2 /c 2 ) suffics 3 Oblivious Routing Schms W now discuss th construction of congstion approximators, which is a subst of cuts such that for any dmand d, approximats th valu of opt(d) via a small numbr of (fficintly accssibl) cuts Spcifically, w want to produc an fficintly computabl oprator R such that opt (d) Rd Whr ach trm in Rd has th form of: d S w (S, V \ S), for som subst of vrtics S, and d S is th total dmand of vrtics in S This algorithm that w will show is a variant of a schm by [Räc08], and w ll also discuss an xtnsion from [Mad0] Th main ida is to rprsnt a graph as a collction of trs, or mor gnrally, tr-lik graphs Spcifically, crating a collction of trs T T k such that: G T i, 2 k i= T i Õ(k) G Both of ths ar in th congstion-dilation routing sns: G H if vry dg in G can b attributd to a path in H such that th dgs in H ar not ovrcongstd Spcifically, on way to crat a tr T such that G T is to simply rrout vry dg in G through its tr path This rrouting incrass th wights of th tr dgs, 8

9 which w will addrss latr Howvr, this dos guarant that vry cut in th tr is at last th siz of th corrsponding cut in th graph Spcifically for vry S, w G (S, V \ S) w T (S, V \ S), which in turn guarnats that th routing schm for (ach of) ths trs, R T i, satisfis R (Ti) d opt G (d) What is trickir to s is that th condition of T i Õ(k) G implis that polylog(n) max i R (T i ) d opt G (d) For this, lt C dnot this maximum congstion Thn w can rout d on ach T i with congstion at most C Thn routing ach d on ach tr mans w can rout k d on i T i, which is in turn routabl onto Õ(k)(G) This mans that d is routabl with congstion C on polylog(n)(g), and in turn opt G (d) polylog(n) C So th qustion thn bcoms how to dcompos a graph into a small numbr of trs, upon ach w r abl to rout th ntir original graph G W will actually satsify th condition of i T i polylog(n)g poitn-wis: w mak sur that ach dg isn t subjctd to too much congstion ovr all th trs Spcifically for an dg T i, w can dfin its congstion as: cong G,T () = E(G), P ath T () w w W will kp ths valus small ovr a squnc of k trs via a rgrt-minimization algorithm vry dg in G has a wight rprsnt how congstd it is, and w will show that low strtch spanning trs nabl us to produc a tr with low avrag congstion for any st of wights First considr th cas of unit wightd graphs, th total strtch of a tr T can b rwrittn as: str T (G) = P ath T ( ) = P ath T ( ) = { G, P ath T ( )}, G T, G T whr th last trm is xactly th congstion of th dg Furthrmor, if w hav importanc valus that ar: At last, 2 Sum up to O(m), 9

10 computing low strtch spanning trs with ach dg rplacd by l dgs ach of lngth l givs th gurant of ( ) l cong G,T () Õ l T To utiliz such an oracl, w thn turn to th xponntial function to gnrat such wights Hr th soft max function is usful: Φ (T T i ) df = ( ) xp cong G,Tj (), t whr t is a paramtr that w ll choos latr (somwhr around /m) Whn w add a tr to this collction, say i +, Taylor xpansion of th xponntial function (hr is whr w nd t larg) givs: Φ (T T i, T i+ ) df = xp ( t j i ) ( cong G,Tj () + ) t cong G,T i (), and th argumnt abov allows us to bound this by ( Φ (T T i, T i+ ) + polylog(n) ) Φ (T T i ) t j i Th ovrall guarant of polylog(n) thn coms from th initial potntial of m, and running this about Õ(t) stps 3 Acclration via J-trs This routin can b mad fastr in ways not too unlik how th tr-basd linar systm solvrs gt acclratd Th convrgnc, which dpnds on maximum congstion, can b dcrasd by a factor of k via adding back k of th most congstd dgs 2 Th tr-plus-k-dgs graph can b shrunk th sam way as Gaussian limination: rpatdly rmov dgr and 2 vrtics 3 At som point a graph sparsifir on k vrtics nds to b invokd, but that dons t brak things ithr sinc w only car about th sizs of cuts Such a routin lads to (anothr!!!) rcurrnc of th form T (m) = Õ (k T (m/k) + m), which solvs to about m xp (Õ ( log n ) ) 0

11 4 Non-Linar Prconditioning with Application to Minimum Cost Transshipmnt W now finish by dscribing ths in th biggr pictur of non-linar norms This part follows th prsntation from a rcnt papr by Jonah Shrman [Sh7] In gnral, th norm minimization problms can b formulatd as: min x p subjc to: Ax = b, Howvr, as w saw bfor, on of th biggst issus with maximum flow is satisfying th constraint B x = d, whr B is th dg-vrtx incidnc matrix: this condition cannot b mt locally bcaus th qualitis may nd to b propagatd down vry long paths Ths conditions can b rlaxd via notions of approximat solutions: Dfinition 4 W say that x is an (α, β)-solution if: x p α x p, whr x is th optimum solution 2 Ax b q β A p q x q Hr Ax b livs in th vrtx spac, and is th spac of rsidus Th crucial obsrvation is that this spac also contains a norm, th q norm, and w can dfin th p q norm of A as: df Ax q A p q = max x x p Th most important dfinition wrt th composition of ths oprators is th notion of linar and non-linr condition numbrs Th linar condition numbr of A, κ p q (A), can b dfind as: κ p q (A) df = A p q min G G:AGA=A q p Th condition of AGA = A mans that it is ssntially a psudo-invrs of A: if w hav y = Ax, thn th vctor satisfis x = Gy Ax = AGAx = Ax = y So with this kind of formulation, what is important is among th many possibl psudoinvrss of A, which on do w choos What s a bit strang from this viw is that it s actually quit asy to construct (α, β) solvrs using tools such as gradint dscnt

12 Lmma 42 Thr ar (+ɛ, ɛ) solvrs for both and that taks O(log nɛ 2 ) itrations, ach prforming on matrix-vctor multiplication involving A and A So th hardr part is to turn an ( + ɛ, ɛ) solvr into a ( + O(ɛ), 0) solvr Lmma 43 If F is an (α, β/κ p q (A))-algorithm for A in th p q norm sns, thn th algorithm givn by F F (b) df = F (b) + F (b AF (b)) is an ( α + αβ, β 2 /κ p q (A) ) algorithm for A in th p q norm Proof Dnot th rsult of th first itration with x (0), and rsidu aftr th first invocation with b = b AF (b) Th guarants on F givs x df (0) p α x p, (8) β b A p q q κ x p (9) Th condition AGA = A mans th vctor x = G b has Ax = AG b = AGA x = A x = b, whr x b th optimum solution for b This in turn mans: x p x κ p q (A) p G q p b b q q A p q aftr which th guarants follows from applying what w know about F again, and thn triangl inquality It s worth noting that this kind of composition works with diffrnt algorithms, say F and F 2 that ar (α, β ) and (α 2, β 2 ) solvrs rspctivly A mor complt vrsion of this statmnt can b found in Lmma 3 in Sction 3 of [Sh7] A dirct consqunc of such a solvr is: Lmma 44 Thr xists ( + ɛ, δ) p p solvrs for A and p = {, } with itrations count: O ( κ p p (A) log mɛ 2 + log(δ ) ) 2

13 To put this into th contxt of maximum flows, rcall that w had a linar oprator P such that: ( ) PB T O log O() n, and w solv th problm min subjc to: x PB T x = Pd whr A = PB T Th trm A in κ (A) follows immdiatly, so all w nd to do is to construct th G oprator Th simplst form of this is to hav G rturn a flow, instad of valus on cuts Not that for trs, this can happn bcaus th mbdding of G into T, and hnc th routing of th dmands, is uniqu That ssntially guarants B T Pd = d, or B T P = I, which if w plug in into A = PB T and G = I givs and hnc w gt G q p = 0 AGA = B T PB T P = B T P = A, 4 l -Oblivious Routing Schms In th last bit, I ll talk about a solution for th -norm cas Spcifically, solving th problm min x subjc to: Ax = b, which in turn boils down to trying to construct an oprator P such that B T P = I and PB T is small Th plan is sam as bfor, find a collction of trs T () T (k), along with routing schms on ach of thm such that B T P (i) = I This is accomplishd by routing ach dmand uniquly along th tr So it rmains to bound th oprator norm of th routing, k P (i) B T k i= Rcall (from last wk) that Z can b intrprtd as th maximum norm of a column of Z Hr, bcaus w r daling with B T, this in turn bcoms: k P (i) B T k = max P k k (i) χ, i= 3 i=

14 whr χ is th indicator vctor for th dg That is, w just want to find a routing oprator such that th distortion of ach dg is small Low strtch spanning tr algorithms actually produc a distribution ovr spanning trs such that for a random spanning tr pickd from this distribution: For vry dgg, th xpctd strtch is O(log n log log n) 2 Maximum strtch is O(n) So onc again, avraging togthr n low strtch spanning trs givs a good solution: th O(n) plays a rol similar to width Similar to th flow cas, thr is also a rcursiv vrsion of this that givs bttr paramtrs Th ida is also simplr than th cas : Pick a random LSST, 2 Tak th dgs with high strtch, aka mor than O(m log n log log n/h), 3 Shrink th graph onto thos O(h) dgs, rcurs on thm Working out this rcursion lads to anothr O(m xp( log n)) typ routin Howvr, somhow unlik flows, it s opn whthr a log O() n quality oprator xists for this#$#$#$??? Thr ar strong indications that somthing spannr lik can work hr [BKKL6] Th othr intrsting obsrvation about this algorithm is that in som paramtr rgims, it givs th bst paralll algorithm for computing shortst paths in paralll that dosn t caus som trribl xplosion in work Rfrncs [BKKL6] Rubn Bckr, Andras Karrnbaur, Sbastian Krinningr, and Christoph Lnzn Approximat undirctd transshipmnt and shortst paths via gradint dscnt CoRR, abs/ , 206 Availabl at: [Mad0] Alksandr Madry Fast approximation algorithms for cut-basd problms in undirctd graphs In Foundations of Computr Scinc (FOCS), 200 5st Annual IEEE Symposium on, pags IEEE, 200 Availabl at [Räc08] Harald Räck Optimal hirarchical dcompositions for congstion minimization in ntworks In Procdings of th 40th annual ACM symposium on Thory of computing, STOC 08, pags , Nw York, NY, USA, 2008 ACM Michal Cohn attributs this ida to Aaron Sidford first 4

15 [Sh7] Jonah Shrman Gnralizd prconditioning and undirctd minimum-cost flow In Procdings of th Twnty-Eighth Annual ACM-SIAM Symposium on Discrt Algorithms, SODA 7, pags , 207 Availabl at: 5

The Matrix Exponential

The Matrix Exponential Th Matrix Exponntial (with xrciss) by D. Klain Vrsion 207.0.05 Corrctions and commnts ar wlcom. Th Matrix Exponntial For ach n n complx matrix A, dfin th xponntial of A to b th matrix A A k I + A + k!