On Convergence Rate of Concave-Convex Procedure

Transcription

1 On Converence Rae o Concave-Conve Proceure Ian E.H. Yen Nanun Pen Po-We Wan an Shou-De Ln Naonal awan Unvers OP 202

2 Oulne Derence o Conve Funcons.c. Prora Applcaons n SVM leraure Concave-Conve Proceure CCCP Majorzaon-Mnzaon MM alorh Block Coornae Descen BCD Converence Analss Alernave BCD Forulaon Converence heore

3 D.C. Prora Le u v be conve uncon ene on R n j be ane uncon on R n. A Derence o Conve Funcon D.C. Prora s ene as: n s.. u v 0... p j 0 j... q

4 D.C. Prora Le u v be conve uncon ene on R n j be ane uncon on R n. A Derence o Conve Funcon D.C. Prora s ene as: n s.. u v 0... p j 0 j... q E. Srucural SVM wh hen varables: [C.N.J. Yu an. Joachs 2009]

5 D.C. Prora Le u v be conve uncon ene on R n j be ane uncon on R n. A Derence o Conve Funcon D.C. Prora s ene as: n s.. u v 0... p j 0 j... q E. Srucural SVM wh hen varables: [C.N.J. Yu an. Joachs 2009] E. Srucural SVM wh non-conve her boun: [C. B. Do e al. 2009] n w where 2 w 2 C N l w l w sup[ ' w ' ] sup ' ' ' w ' l w ' 0 sup ' w '

6 D.C. Prora Le u v be conve uncon ene on R n j be ane uncon on R n. A Derence o Conve Funcon D.C. Prora s ene as: n s.. u v 0... p j 0 j... q Converence rae s har o analze n non-sooh proble. In hs work we hanle he specal case when he sooh par o u s srcl conve quarac an v s pecewse-lnear. n w where 2 w 2 C N l w l w sup[ ' w ' ] sup ' ' ' w ' l w ' 0 sup ' w '

7 Concave-Conve Proceure Suppose we can copue he sub-raen o v he Concave-Conve Proceure CCCP solves a D.C. Prora b a seres o conve proble: [Yulle an Ranarajan 2003]: s.. ar n j u v 0... p 0 j... q [Yulle an Ranarajan 2003] shows uaranees escen o he D.C. Prora. [B. Srperubuur e al. 2009] prove Global Converence o va Zanwll s hoer. However he pone ou he Local Converence Rae o s an open proble. Goal: Show ha has a leas Lnear Converence Rae va he connecon o ore eneral Block Coornae Descen BCD alorh.

8 CCCP as Majorzaon Mnzaon MM CCCP s a specal case o Majorzaon Mnzaon MM where we consruc a ajorzaon uncon o objecve uncon =u-v: where Ω s he easble oan. hen he MM alorh solves: ar n 2

9 CCCP as Majorzaon Mnzaon MM CCCP s a specal case o Majorzaon Mnzaon MM where we consruc a ajorzaon uncon o objecve uncon =u-v: where Ω s he easble oan. hen he MM alorh solves: ar n In CCCP s consruce b s orer alor Approaon o v a pon : v v u v u or or 2 v u ar n ar n hereore

10 CCCP as Majorzaon Mnzaon MM CCCP s a specal case o Majorzaon Mnzaon MM where we consruc a ajorzaon uncon o objecve uncon =u-v: where Ω s he easble oan. hen he MM alorh solves: ar n 2 [R. Salakhunov 2003] analze local converence rae o eneral MM alorh b akn 2 as a erenable ap + = ψ. However ψ s no erenable when here are consrans or non-sooh uncon. Here we ook anoher vew o 2 o analze converence.

11 MM as Block Coornae Descen Snce he nu o occurs a = we can vew MM alorh as Block Coornae Descen over an : ar n ar n However when v s pecewse-lnear he aser proble n s sconnuous an har o analze. u v v

12 MM as Block Coornae Descen Snce he nu o occurs a = we can vew MM alorh as Block Coornae Descen over an : ar n ar n We can ake an alernave orulaon b observn: b a ar n where a v b a a k k v Block Coornae Descen over an on he alernave orulaon:... 0 an.. n s b a u R els he sae CCCP alorh.

13 Lea Block Coornae Descen or Non-conve Non-sooh Proble Conser he proble: n F cp where s sooh an P s nonsooh conve lower se-connuous an separable or an. he Block Coornae Descen 3 ar n ar n F F 4 5 Converes o a saonar pon o 3 wh a leas lnear rae he sooh par o 4 5 are srcl conve quarac s quarac an P s polheral. Proo. Snce 4 5 are srcl conve quarac he BCD correspon o Coornae Graen Descen CGD n [Paul sen eal. 2009] wh eac Hessan ar an lne search. he resul hols b heore 2 4 o her paper.

14 Converence heore o CCCP... 0 an.. n s b a u R heore he CCCP converes o saonar pon o D.C. Prora wh a leas lnear rae he non-sooh par o u an v are pecewse-lnear he sooh par o u s srcl conve quarac an he oan Ω s polheral. Proo. P P P n u b a u R R n he CCCP can be nerpree as BCD over an o Whch can be also wren as Where sooh par s quarac an P s polheral separable. Mnzn over he proble srcl conve quarac. Mnzn over here s equvalen srcl conve quarac proble Lea 2 n paper.

15 Reerence [] Paul sen an Sanwoon Yun A coornae raen escen eho or nonsooh separable nzaon. Maheacal Proran. [2] A. L. Yulle an A. Ranarajan. he concave-conve proceure. Neural Copuaon 5:95V [3] L. Wan X. Shen an W. Pan. On ransucve suppor vecor achnes. In J. Verucc X. Shen an J. Laer eors Precon an Dscover. Aercan Maheacal Soce [4] B. K. Srperubuur D. A. orres an G. R. G. Lanckre. Sparse een ehos b.c. proran. In Proc. o he 24h Annual Inernaonal Conerence on Machne Learnn [5] J. Neuann C. Schnorr an G. Sel. Cobne SVM-base eaure selecon an classcaon. Machne Learnn 6:29V [6] X.-L. Men. Dscusson on opzaon ranser usn surroae objecve uncons. Journal o Copuaonal an Graphcal Sascs 9:35V [7] R. Salakhunov S. Rowes an Z. Ghahraan. On he converence o boun opzaon alorhs. In Proc. 9h Conerence n Unceran n Arcal Inellence paes 509V [8] D. D. Lee an H. S. Seun. Alorhs or non-neave ar acorzaon. In.K. Leen.G. Deerch an V. resp eors Avances n Neural Inoraon Processn Sses 3 paes 556V562. MI Press Cabre 200. [9] C. B. Do Q. V. Le C. H. eo O. Chapelle an A. J. Sola. her bouns or srucure esaon. In Avances n Neural Inoraon Processn Sses o appear. [0]. Pha Dnh an L.. Hoa An. Conve analss approach o.c. proran: heor alorhs an applcaons. Aca Maheaca Venaca 22:289V [] R. Collober F. Snz J. Weson an L. Boou. Lare scale ransucve SVMs. Journal o Machne Learnn Research 7:687V [2] Collober R. Weson J. Boou L ran conve or scalabl. ICML06 23r Inernaonal Conerence on Machne Learnn. Psburh USA. [3] C.-N. J. Yu an. Joachs. Learnn srucural svs wh laen varables. In Inernaonal Conerence on Machne Learnn ICML 2009 [4] B. Srperubuur an G. Lanckre On he converence o he concave-conve proceure n Neural Inoraon Processn Sses 2009.