Large-scale Distance Metric Learning with Uncertainty

Transcription

1 i Large-scale Disance Meric Learning wih Uncerainy Qi Qian Jiasheng Tang Ha Li Shenghu Zhu Rng Jin Alibaba Grup, Bellevue, WA, 98004, USA {qi.qian, jiasheng.js, liha.lh, shenghu.zhu, Opimizing he meric wih a se f riple cnsrains is challenging since he number f riple cnsrains can be up O(n 3 ), where n is he number f he riginal raining examples. I makes DML cmpuainally inracable fr he large-scale prblems. Many sraegies have been develped deal wih his challenge and ms f hem fall in w caegries, learning by schasic gradien descen (SGD) and learning wih he acive se. Wih he sraegy f SGD, DML mehds can sample jus ne cnsrain r a mini-bach f cnsrains a each ierain bserve an unbiased esimain f he full gradien and avid cmpuarxiv: v1 [cs.lg] 25 May 2018 Absrac Disance meric learning (DML) has been sudied exensively in he pas decades fr is superir perfrmance wih disance-based algrihms. Ms f he exising mehds prpse learn a disance meric wih pairwise r riple cnsrains. Hwever, he number f cnsrains is quadraic r even cubic in he number f he riginal examples, which makes i challenging fr DML handle he large-scale daa se. Besides, he real-wrld daa may cnain varius uncerainy, especially fr he image daa. The uncerainy can mislead he learning prcedure and cause he perfrmance degradain. By invesigaing he image daa, we find ha he riginal daa can be bserved frm a small se f clean laen examples wih differen disrins. In his wrk, we prpse he margin preserving meric learning framewrk learn he disance meric and laen examples simulaneusly. By leveraging he ideal prperies f laen examples, he raining efficiency can be imprved significanly while he learned meric als becmes rbus he uncerainy in he riginal daa. Furhermre, we can shw ha he meric is learned frm laen examples nly, bu i can preserve he large margin prpery even fr he riginal daa. The empirical sudy n he benchmark image daa ses demnsraes he efficacy and efficiency f he prpsed mehd. 1. Inrducin Disance meric learning (DML) aims learn a disance meric where examples frm he same class are well separaed frm examples f differen classes. I is an essenial ask fr disance-based algrihms, such as k-means clusering [18], k-neares neighbr classificain [17] and infrmain rerieval [2]. Given a disance meric M, he squared Mahalanbis disance beween examples x i and x j can be cmpued as D 2 M (x i, x j ) = (x i x j ) M(x i x j ) Ms f exising DML mehds prpse learn he meric by minimizing he number f vilains in he se f xjx xi xk Figure 1. Illusrain f he prpsed mehd. Le rund and square pins dene he arge daa and impsrs, respecively. Le riangle pins dene he crrespnding laen examples. Daa pins wih he same clr are frm he same class. I demnsraes ha he meric learned wih laen examples n nly separaes he dissimilar laen daa wih a large margin bu als preserves he large margin fr he riginal daa. pairwise r riple cnsrains. Given a se f pairwise cnsrains, DML ries learn a meric such ha he disances beween examples frm he same class are sufficienly small (e.g., smaller han a predefined hreshld) while hse beween differen nes are large enugh [3, 18]. Differen frm pairwise cnsrains, each riple cnsrain cnsiss f hree examples (x i, x j, x k ), where x i and x j have he same label and x k is frm a differen class. An ideal meric can push away x k frm x i and x j by a large margin [17]. Learning wih riple cnsrains pimizes he lcal psiins f examples and is mre flexible fr real-wrld applicains, where defining he apprpriae hreshlds is hard fr pairwise cnsrains. In his wrk, we will fcus n DML wih riple cnsrains. xj x xi xk

2 ing he gradien frm he whle se [2, 10]. Oher mehds learn he meric wih a se f acive cnsrains (i.e., vilaed by he curren meric), where he size can be significanly smaller han he riginal se [17]. I is a cnveninal sraegy applied by cuing plane mehds [1]. Bh f hese sraegies can alleviae he large-scale challenge bu have inheren drawbacks. Appraches based n SGD have search hrugh he whle se f riple cnsrains, which resuls in he slw cnvergence, especially when he number f acive cnsrains is small. On he her hand, he mehds relying n he acive se have idenify he se a each ierain. Unfrunaely, his perain requires cmpuing pairwise disances wih he curren meric, where he cs is O(n 2 ) and is expensive fr large-scale prblems. Besides he challenge frm he size f daa se, he uncerainy in he daa is als an issue, especially fr he image daa, where he uncerainy can cme frm he differences beween individual examples and disrins, e.g., pse, illuminain and nise. Direcly learning wih he riginal daa will lead a pr generalizain perfrmance since he meric ends verfi he uncerainy in he daa. By furher invesigaing he image daa, we find ha ms f riginal images can be bserved frm a much smaller se f clean laen examples wih differen disrins. The phenmenn is illusraed in Fig. 5. This bservain inspires us learn he meric wih laen examples in lieu f he riginal daa. The challenge is ha laen examples are unknwn and nly images wih uncerainies are available. In his wrk, we prpse a framewrk learn he disance meric and laen examples simulaneusly. I sufficienly explres he prperies f laen examples address he menined challenges. Firs, due he small size f laen examples, he sraegy f idenifying he acive se becmes affrdable when learning he meric. We adp i accelerae he learning prcedure via aviding he aemps n inacive cnsrains. Addiinally, cmpared wih he riginal daa, he uncerainy in laen examples decreases significanly. Cnsequenly, he meric direcly learned frm laen examples can fcus n he naure f he daa raher han he uncerainy in he daa. T furher imprve he rbusness, we adp he large margin prpery ha laen examples frm differen classes shuld be pushed away wih a daa dependen margin. Fig. 1 illusraes ha an apprpriae margin fr laen examples can als preserve he large margin fr he riginal daa. We cnduc he empirical sudy n benchmark image daa ses, including he challenging ImageNe daa se, demnsrae he efficacy and efficiency f he prpsed mehd. The res f he paper is rganized as fllws: Secin 2 summarizes he relaed wrk f DML. Secin 3 describes he deails f he prpsed mehd and Secin 4 summarizes he hereical analysis. Secin 5 cmpares he prpsed mehd he cnveninal DML mehds n he benchmark image daa ses. Finally, Secin 6 cncludes his wrk wih fuure direcins. 2. Relaed Wrk Many DML mehds have been prpsed in he pas decades [3, 17, 18] and cmprehensive surveys can be fund in [7, 19]. The represenaive mehds include Xing s mehd [18], ITML [3] and LMNN [17]. ITML learns a meric accrding pairwise cnsrains, where he disances beween pairs frm he same class shuld be smaller han a predefined hreshld and he disances beween pairs frm differen classes shuld be larger han anher predefined hreshld. LMNN is develped wih riple cnsrains and a meric is learned make sure ha pairs frm he same class are separaed frm he examples f differen classes wih a large margin. Cmpared wih pairwise cnsrains, riple cnsrains are mre flexible depic he lcal gemery. T handle he large number f cnsrains, sme mehds adp SGD r nline learning sample ne cnsrain r a mini-bach f cnsrains a each ierain [2, 10]. OA- SIS [2] randmly samples ne riple cnsrain a each ierain and cmpues he unbiased gradien accrdingly. When he size f he acive se is small, hese mehds require exremely large number f ierains imprve he mdel. Oher mehds ry explre he cncep f he acive se. LMNN [17] prpses learn he meric effecively a each ierain by cllecing an acive se ha cnsiss f cnsrains vilaed by he curren meric wihin he k-neares neighbrs fr each example. Hwever, i requires O(n 2 ) bain he apprpriae acive se. Besides he research abu cnveninal DML, deep meric learning has araced much aenin recenly [9, 13, 15, 16]. These sudies als indicae ha sampling acive riples is essenial fr acceleraing he cnvergence. FaceNe [15] keeps a large size f mini-bach and searches hard cnsrains wihin a mini-bach. LefedSruc [16] generaes he mini-bach wih he randmly seleced psiive examples and he crrespnding hard negaive examples. Prxy-NCA [9] adps prxy examples reduce he size f riple cnsrains. Once an anchr example is given, he similar and dissimilar examples will be searched wihin he se f prxies. In his wrk we prpse learn he meric nly wih laen examples which can dramaically reduce he cmpuainal cs f baining he acive se. Besides, he riangle inequaliy dse n hld fr he squared disance, which makes ur analysis significanly differen frm he exising wrk. 3. Margin Preserving Meric Learning Given a raining se {(x i, y i ) i = 1,, n}, where x i R d is an example and y i is he crrespnding label, DML

3 aims learn a gd disance meric such ha x i, x j, x k D 2 M (x i, x k ) D 2 M (x i, x j ) 1 where x i and x j are frm he same class and x k is differen. Given he disance meric M S+ d d, he squared disance is defined as D 2 M (x i, x j ) = (x i x j ) M(x i x j ) where S d d + denes he se f d d psiive semi-definie (PSD) marices. Fr he large-scale image daa se, we assume ha each bserved example is frm a laen example wih cerain zer mean disrins, i.e., i, E[x i ] = z :f(i)= where f( ) prjecs he riginal daa is crrespnding laen example. Then, we cnsider he expeced disance [20] beween bserved daa and he bjecive is learn a meric such ha x i, x j, x k E[D 2 M (x i, x k )] E[D 2 M (x i, x j )] 1 (1) Le z, z p and z q dene laen examples f x i, x j and x k respecively. Fr he disance beween examples frm he same class, we have E[D 2 M (x i, x j )] = E[(x i z + z ) M(x i z + z )] + E[(x j z p + z p ) M(x j z p + z p )] E[2x i Mx j ] = D 2 M (z, z p ) + E[D 2 M (x i, z )] + E[D 2 M (x j, z p )] = D 2 M (z, z p ) + 2E[D 2 M (x i, z )] (2) The las equain is due he fac ha x i and x j are i.i.d, since hey are frm he same class. By applying he same analysis fr he dissimilar pair, we have E[D 2 M (x i, x k )] = D 2 M (z, z + E[D 2 M (x i, z )] + E[D 2 M (x k, z ] D 2 M (z, z + E[D 2 M (x i, z )] (3) The inequaliy is because ha M is a PSD marix. Cmbining Eqns. 2 and 3, we find ha he difference beween he disances in he riginal riple can be lwer bunded by hse in he riple cnsising f laen examples E[D 2 M (x i, x k )] E[D 2 M (x i, x j )] D 2 M (z, z D 2 M (z, z p ) E[D 2 M (x i, z )] Therefre, he meric can be learned wih he cnsrains defined n laen examples such ha z, z p, z q D 2 M (z, z D 2 M (z, z p ) 1+E[D 2 M (x i, z )] Once he meric is bserved, he margin fr he expeced disances beween riginal daa (i.e., as in Eqn. 1) is als guaraneed. Cmpared wih he riginal cnsrains, he margin beween laen examples is increased by he facr f E[DM 2 (x i, z )]. This erm indicaes he expeced disance beween he riginal daa and is crrespnding laen example. I means ha he igher a lcal cluser is, he less a margin shuld be increased. Furhermre, each class akes a differen margin, which depends n he disribuin f he riginal daa and makes i mre flexible han a glbal margin. Wih he se f riples {z, z p, z q}, he pimizain prblem can be wrien as min L(M, z) = l(z, z p, z q; M) M S d d +, M F δ,z R d m where m n is he number f laen examples. We add a cnsrain fr he Frbenius nrm f he learned meric preven i frm verfiing. l( ) is he lss funcin and he hinge lss is applied in his wrk. l(z, z p, z q; M) = [1 + E[D 2 M (x i, z )] (D 2 M (z, z q) D 2 M (z, z p))] + This prblem is hard slve since bh he meric and laen examples are he variables be pimized. Therefre, we prpse slve i in an alernaing way and he deailed seps are demnsraed belw Updae z wih Upper Bund When fixing M k 1, he subprblem a he k-h ierain becmes min L(M k 1, z) = [ 1 + E[DM 2 z k 1 (x i, z )] }{{} a ] (DM 2 k 1 (z, z q) DM 2 k 1 (z, z p)) (4) }{{} + b The variable z appears in bh he erm f margin a and he erm f he riple difference b, which makes i hard pimize direcly. Our sraegy is find an apprpriae upper bund fr he riginal prblem and slve he simple prblem insead. Therem 1. The funcin L(M k 1, z) can be upper bunded by he series f funcins r F r(z). Fr he r- h class, we have F r (z) = c 1 E[DM 2 k 1 (x i, z )]+c 2 +c 3 DM 2 k 1 (z, z k 1 ) where c 1, c 2 and c 3 are cnsans and r F r(z k 1 ) = L(M k 1, z k 1 ).

4 The deailed prf can be fund in Secin 4. Afer remving he cnsan erms and rearrange he cefficiens, pimizing F r (z) is equivalen pimizing he fllwing prblem min Fr (z) = (5) z R d mr,µ:µi, {0,1}, µi,=1 µ i, DM 2 k 1 (x i, z ) + γ DM 2 k 1 (z, z k 1 ) i:y(i)=r where µ denes he membership ha assigns a laen example fr each riginal example. Till nw, i shws ha he riginal bjecive L(M k 1, z) can be upper bunded by r F r(z). Minimizing he upper bund is similar k-means bu wih he disance defined n he meric M k 1. S we can slve i by he sandard EM algrihm. When fixing µ, laen examples can be updaed by he clsed-frm sluin 1, z = i µ i, + γ ( i µ i, x i + γz k 1 ) (6) When fixing z, µ jus assigns each riginal example is neares laen example wih he disance defined n he meric M k 1 { 1 = arg min DM 2 i, µ i, = (x k 1 i, z ) (7) 0.w. Alg. 1 summarizes he mehd fr slving F r (z). Algrihm 1 Algrihm f Updaing z Inpu: daa se {X, Y }, z k 1, M k 1, γ and S Iniialize z = z k 1 fr s = 1 S d Fix z and bain he assignmen µ as in Eqn. 7 Fix µ and updae z as in Eqn. 6 end fr reurn z k = z 3.2. Updae M wih Upper Bund When fixing z k a he k-h ierain, he subprblem becmes min M S d d + L(M, z k ) = (8) [1 + E[DM 2 (x i, z )] (DM 2 (z }{{}, z q) DM 2 (z, z p)) }{{} a b where M als appears in muliple erms. Wih he similar prcedure, an upper bund can be fund make he pimizain simpler. ] + Therem 2. The funcin L(M, z k ) can be upper bunded by he funcin H(M) which is H(M) = λ 2 M M k 1 2 F + [ 1 + E[DM 2 k 1 (x i, z )] ] (DM 2 (z, z q) DM 2 (z, z p)) + where λ is a cnsan and H(M k 1 ) = L(M k 1, z k ). Minimizing H(M) is a sandard DML prblem. Since he number f laen examples z k is small, many exising DML mehds can handle he prblem well. In his wrk we slve he prblem by SGD bu sample ne epch acive cnsrains a each sage. The acive cnsrains cnain he riples f z k ha incur he hinge lss wih he disance defined n M k 1. This sraegy enjys he efficiency f SGD and he efficacy f learning wih he acive se. T furher imprve he efficiency, ne prjecin paradigm is adped avid he expensive PSD prjecin which css O(d 3 ). I perfrms he PSD prjecin nce a he end f he learning algrihm and shws be effecive in many applicains [2, 11]. Finally, since he prblem is srngly cnvex, we apply he α-suffix averaging sraegy, which averages he sluins ver he las several ierains, bain he pimal cnvergence rae [12]. The cmplee apprach fr baining M k is shwn in Alg. 2. Algrihm 2 Algrihm f Updaing M Inpu: daa se {X, Y }, z k, M k 1, δ, λ and S Iniialize M 0 = M k 1 Sample ne epch acive cnsrains A accrding z k and M k 1 fr s = 1 S d Randmly sample ne cnsrain frm A Cmpue he schasic gradien g = H(M) Updae he meric as M s = M s 1 1 λs g Check he Frbenius nrm M s = Π δ (M s) end fr Prjec he learned marix n he PSD cne M k = Π P SD ( 2 S S s=s/2+1 M s) reurn M k Alg. 3 summarizes he prpsed margin preserving meric learning framewrk. Differen frm he sandard alernaing mehd, we nly pimize he upper bund fr each subprblem. Hwever, he mehd cnverges as shwn in he fllwing herem. Therem 3. Le (z k 1, M k 1 ) and (z k, M k ) dene he resuls bained by applying he algrihm in Alg. 3 a (k 1)-h and k-h ierains respecively. Then, we have L(z k, M k ) L(z k 1, M k 1 ) which means he prpsed mehd can cnverge.

5 Algrihm 3 Margin Preserving Meric Learning (MaPML) Inpu: daa se {X, Y }, δ, m, γ, λ and K Iniialize M 0 = I fr k = 1 K d Fix M k 1 and bain laen examples z k by Alg. 1 Fix z k and updae he meric M k by Alg. 2 end fr reurn M K and z K Cmpuainal Cmplexiy The prpsed mehd cnsiss f w pars: baining laen examples and meric learning. Fr he frmer ne, he cs is linear in he number f laen examples and riginal examples as O(mn). Fr he laer ne, he cs f sampling an acive se dminaes he learning prcedure. Since he number f ierains is fixed, he cmplexiy f sampling becmes min{o(sm), O(m 2 )}. Therefre, he whle algrihm can be linear in he number f laen examples. Ne ha he efficiency can be furher imprved wih disribued cmpuing since many cmpnens f MaPML can be implemened in parallel. Fr example, when updaing z, each class is independen and all subprblems can be slved simulaneusly. 4. Thereical Analysis 4.1. Prf f Therem 1 Prf. Firs, fr he disance f he dissimilar pair in erm b f Eqn. 4, we have DM 2 (z, z = DM 2 (z k 1, zq k 1 ) + DM 2 (z, z k 1 ) + 2(z z k 1 ) M(z k 1 + DM 2 (z q, z k 1 2(z q z k 1 M(z k 1 2(z z k 1 ) M(z q z k 1 DM 2 (z k 1, z k 1 2D M (z, z k 1 2D M (z q, z k 1 D M (z k 1, z k 1 z k 1 z k 1 )D M (z k 1, z k 1 where z k 1 are laen examples frm he las ierain. We le M dene M k 1 in his prf fr simpliciy. The inequaliy is frm ha M is a PSD marix and can be decmpsed as M = LL. Then i is bained by applying he Cauchy-Schwarz inequaliy. Wih he assumpins ha, D M (z, z k 1 ) is sufficienly large and D M (z k 1, z k 1 is bunded by a cnsan c 2, he inequaliy can be simplified as D 2 M (z, z (9) D 2 M (z k 1, z k 1 ) cdm 2 (z, z k 1 ) cdm 2 (z q, z k 1 q The assumpin is easy verify since D M (z k 1, z k 1 ) z k 1 q z k 1 q 2 2 M k 1 2 Ne ha M k 1 2 M k 1 F δ and z is in he cnvex hull f he riginal daa, and he cnsan c can be se as c = 8δ max i x i 2 2. Wih he similar prcedure, we have he bund fr he disance f he similar pair as DM 2 (z, z p ) DM 2 (z k 1, z k 1 p ) (10) +(c + 2)DM 2 (z, z k 1 ) + (c + 2)DM 2 (z p, z k 1 p ) Taking Eqns. 9 and 10 back he riginal funcin L(M k 1, z) and using he prpery f he hinge lss, he riginal ne can be upper bunded by G(z) = [1 + E[DM 2 (x i, z )] (DM 2 (z :k 1, z :k 1 DM 2 (z :k 1, z :k 1 p ))] + + c 3 m D 2 M (z, z k 1 ) where c 3 = O(T c) is a cnsan. By invesigaing he srucure f his prblem, we find ha each class is independen in he pimizain prblem and he subprblem fr he r-h class can be wrien as min z R d mr G r (z) = + c 3 :y(z )=r :y(z )=r [E[D 2 M (x i, z )] + c ] + D 2 M (z, z k 1 ) where m r is he number f laen examples fr he r-h class and c is a cnsan as c = 1 (D 2 M (z :k 1, z :k 1 DM 2 (z :k 1, z :k 1 p )) Nex we ry upper bund he hinge lss in G r (z) wih a linear funcin in he inerval f [c, E[DM 2 (x i, z k 1 )]+c ], where he hinge lss incurred by he pimal sluin z k is guaraneed be in i. Figure 2. Illusrain f bunding he hinge lss. The hinge lss beween [c, α + c ] is upper bunded by he linear funcin dened by he red line. Le α = E[DM 2 (x i, z k 1 )], which is he expeced disance beween he riginal daa f he r-h class and he crrespnding laen examples frm he las ierain, and β be a cnsan sufficienly large as β min c

6 Then, fr each acive hinge lss (i.e., α + c > 0), if we have E[D 2 M (x i, z )] α (11) [E[DM 2 (x i, z )] + c ] + α + c α + c + β (E[D2 M (x i, z )] + c + β) Fig. 2 illusraes he linear funcin ha can bund he hinge lss and he prf is sraighfrward. We will shw ha he cndiin in Eqn. 11 can be saisfied hrughu he algrihm laer. Wih he upper bund f he hinge lss, G r (z) can be bunded by F r (z) = c 1 E[D 2 M (x i, z )] + c 2 + c 3 where and c 2 = c 1 = :y(z )=r :y(z )=r α + c α + c + β I(α + c ) D 2 M (z, z k 1 ) α + c α + c + β (c + β)i(α + c ) { 1 ν > 0 I( ) is an indicar funcin as I(ν) = 0.w. Finally, we check he cndiin in Eqn. 11. Le z k dene laen examples bained by pimizing F(z) wih Alg. 1. Since we use z k 1 as he saring pin pimize F r (z), i is bvius ha F r (z k ) F r (z k 1 ) A he same ime, we have DM 2 (z k, z k 1 ) DM 2 (z k 1, z k 1 ) = 0 I is bserved ha Eqn. 11 is saisfied by cmbining hese inequaliies Prf f Therem 2 Prf. Fr he erm a in Eqn. 8, we have E[D 2 M (x i, z )] = E[D 2 M k 1 (x i, z ) + (x i z ) (M M k 1 )(x i z )] E[DM 2 k 1 (x i, z )] + max x i z 2 2 M M k 1 F i E[DM 2 k 1 (x i, z )] + c M M k 1 2 F where we assume ha M M k 1 F is sufficienly large and c is a cnsan which has max i x i z 2 2 c and can be se as c = 4 max i x i 2 2. Therefre, he riginal funcin L(M, z k ) can be upper bunded by H(M) = λ 2 M M k 1 2 F + [ 1 + E[DM 2 k 1 (x i, z )] ] (DM 2 (z, z q) DM 2 (z, z p)) + where λ = O(T c) Prf f Therem 3 Prf. When fixing M k 1 a he k-h ierain, we have L(M k 1, z k ) r r When fixing z k, we have G r (z k ) r F r (z k 1 ) = L(M k 1, z k 1 ) F r (z k ) L(M k, z k ) H(M k ) H(M k 1 ) = L(M k 1, z k ) Therefre, afer each ierain, we have L(M k, z k ) L(M k 1, z k 1 ) Since he value f L( ) is bunded, he sequence will cnverge afer a finie number f ierains. 5. Experimens We cnduc he empirical sudy n fur benchmark image daa ses. 3-neares neighbr classifier is applied verify he efficacy f he learned merics frm differen mehds. The mehds in he cmparisn are summarized as fllws. Euclid: 3-NN wih Euclidean disance. LMNN [17]: he sae-f-he-ar DML mehd ha idenifies a se f acive riples wih he curren meric a each ierain. The acive riples are searched wihin 3-neares neighbrs fr each example. OASIS [2]: an nline DML mehd ha receives ne randm riple a each ierain. I nly updaes he meric when he riple cnsrain is acive. HR-SGD [10]: ne f he ms efficien DML mehds wih SGD. We adp he versin ha randmly samples a mini-bach f riples a each ierain in he cmparisn. Afer sampling, a Bernulli randm variable is generaed decide if updaing he curren meric r n. Wih he PSD prjecin, i guaranees ha he learned meric is in he PSD cne a each ierain.

7 MaPML τ : he prpsed mehd ha learns he meric and laen examples simulaneusly, where τ denes he rai beween he number f laen examples and he number f riginal nes τ% = m n Differen frm her mehds, 3-NN is implemened wih laen examples as reference pins. The mehd ha akes 3-NN wih riginal daa is referred as MaPML τ -O. The parameers f OASIS, HR-SGD and MaPML are searched in {10 i : i = 3,, 3}. The size f mini-bach in HR-SGD is se be 10 as suggesed [10]. T rain he mdel sufficienly, he number f ierains fr LMNN is se be 10 3 while he number f randmly sampled riples is 10 5 fr OASIS and HR-SGD. The number f ierains fr MaPML is se as K = 10 while he number f maximal ierains fr slving M k in he subprblem is se as S = 10 4, which rughly has he same number f riples as OASIS and HR-SGD. All experimens are implemened n a server wih 96 GB memry and 2 Inel Xen E CPUs. Average resuls wih sandard deviain ver 5 rails are repred MNIST Firs, we evaluae he perfrmance f differen algrihms n MNIST [8]. I cnsiss f 60, 000 handwrien digi images fr raining and 10, 000 images fr es. There are 10 classes in he daa se, which are crrespnding he digis 0-9. Each example is a grayscale image which leads he 784-dimensinal feaures and hey are nrmalized he range f [0, 1]. Tes errr (%) Euclid LMNN OASIS HR-SGD MaPML MaPML-O 5% 10% 15% 20% 25% Rai f #laen examples = Tes errr (%) Sandard deviain f Gaussian nise (a) Cmparisn f errr rae (b) Cmparisn wih Gaussian nise Figure 3. Cmparisns n MNIST Euclid LMNN MaPML10-O MaPML10 Fig. 3 (a) cmpares he perfrmance f differen merics n he es se. Fr MaPML, we vary he rai f laen examples frm 5% 25%. Firs f all, I is bvius ha he merics learned wih he acive se uperfrm hse frm randm riples. I cnfirms ha he sraegy f sampling riples randmly can n explre he daa se sufficienly due he exremely large number f riples. Secndly, he perfrmance f MaPML 10 -O is cmparable wih LMNN, which shws ha he prpsed mehd can learn a gd meric wih nly a small amun f laen examples (i.e., 10%). Finally, bh MaPML and MaPML-O wrk well wih he meric bained by MaPML, which verifies ha he learned meric can preserve he large margin prpery fr bh he riginal and laen daa. Ne ha when he number f laen examples is small, he perfrmance f k- NN wih laen examples is slighly wrse han ha wih he whle raining se. Hwever, k-nn wih laen examples can be mre rbus in real-wrld applicains. T demnsrae he rbusness, we cnduc anher experimen ha randmly inrduces he zer mean Gaussian nise (i.e., N (0, σ 2 )) each pixel f he riginal raining images. The sandard deviain f he Gaussian nise is varied in he range f [50/255, 250/255] and τ is fixed as 10. Fig. 3 (b) summarizes he resuls. I shws ha MaPML 10 has he cmparable perfrmance as MaPML 10 - O and LMNN when he nise level is lw. Hwever, wih he increasing f he nise, he perfrmance f LMNN drps dramaically. This can be inerpreed by he fac ha he meric learned wih he riginal daa has been misled by he nisy infrmain. In cnras, he errrs made by MaPML and MaPML-O increase mildly and i demnsraes ha he learned meric is mre rbus han he ne learned frm he riginal daa. MaPML perfrms bes amng all mehds and i is due he reasn ha he uncerainy in laen examples are much less han ha in he riginal nes. I implies ha k-nn wih laen examples is mre apprpriae fr real-wrld applicains wih large uncerainy. CPU Time (secnds) OASIS HR-SGD MaPML10 LMNN MNIST CIFAR-10 CIFAR #Ierains: k (a) CPU ime fr raining (b) Cnvergence curve f MaPML Figure 4. Illusrain f he efficiency f he prpsed mehd. Then, we cmpare he CPU ime cs by differen algrihms evaluae he efficiency. The resuls can be fund in Fig. 4 (a). Firs, as expeced, all algrihms wih SGD are mre efficien han LMNN, which has cmpue he full gradien frm he redefined acive se a each ierain. Mrever, he running ime f MaPML 10 is cmparable ha f HR-SGD, which shws he efficiency f MaPML wih he small se f laen examples. Ne ha OASIS has he exremely lw cs, since i allws he inernal meric be u f he PSD cne. Fig. 4 (b) illusraes he cnvergence curve f MaPML and shws ha he prpsed mehd cnverges fas in pracice. Finally, since we apply he prpsed mehd he rig- kzk! zk!1k 2 F

8 inal pixel feaures direcly, he learned laen examples can be recvered as images. Fig. 5 illusraes he learned laen examples and he crrespnding examples in he riginal raining se. I is bvius ha he riginal examples are frm laen examples wih differen disrins as claimed. Figure 5. Illusrain f he learned laen examples and crrespnding riginal examples frm MNIST. The lef clumn indicaes laen examples while five riginal images frm each crrespnding cluser are n he righ CIFAR-10 & CIFAR-100 CIFAR-10 cnains 10 classes wih 50, 000 clr images f size fr raining and 10, 000 images fr es. CIFAR-100 has he same number f images in raining and es bu fr 100 classes [6]. Since deep learning algrihms shw he verwhelming perfrmance n hese daa ses, we adp ResNe18 [4] in Caffe [5], which is pre-rained n ImageNe ILSVRC 2012 daa se [14], as he feaure exracr and each image is represened by a 512-dimensinal feaure vecr. Table 1. Cmparisn f errr rae (%) n CIFAR-10 and CIFAR Mehds CIFAR-10 CIFAR-100 Euclid OASIS ± ± 0.21 HR-SGD ± ± 0.19 LMNN ± ± 0.13 MaPML 10 -O ± ± 0.15 MaPML ± ± 0.16 Table 1 summarizes errr raes f mehds in he cmparisn. Firs, we have he same bservain as n MNIST, where he perfrmance f mehds adping acive riples is much beer han ha f he mehds wih randmly sampled riples. Differen frm MNIST, MaPML 10 uperfrms LMNN n bh f he daa ses. I is because ha images in hese daa ses describe naural bjecs which cnain much mre uncerainy han digis in MNIST. Finally, he perfrmance f MaPML 10 -O is superir ver OASIS and HR-SGD, which shws he learned meric can wrk well wih he riginal daa represened by deep feaures. I cnfirms ha he large margin prpery is preserved even fr he riginal daa ImageNe Finally, we demnsrae ha he prpsed mehd can handle he large-scale daa se wih ImageNe. ImageNe ILSVRC 2012 cnsiss f 1, 281, 167 raining images and 50, 000 validain daa. The same feaure exracin prcedure as abve is applied fr each image. Given he large number f raining daa, we increase he number f riples fr OASIS and HR-SGD Crrespndingly, he number f maximal ierains fr slving he subprblem in MaPML is als raised Table 2. Cmparisn f errr rae (%) n ImageNe. Mehds Tes errr (%) Euclid OASIS ± 0.08 HR-SGD ± 0.08 MaPML 5 -O ± 0.03 MaPML ± 0.09 LMNN des n finish he raining afer 24 hurs s he resul is n repred fr i. In cnras, MaPML bains he meric wihin abu ne hur. The perfrmance f available mehds can be fund in Table 2. Since ResNe18 is rained n ImageNe, he exraced feaures are pimized fr his daa se and i is hard furher imprve he perfrmance. Hwever, wih laen examples, MaPML can furher reduce he errr rae by 1.7%. I indicaes ha laen examples wih lw uncerainy are mre apprpriae fr he large-scale daa se as he reference pins. Ne ha he small number f reference pins will als accelerae he es phase. Fr example, i css 0.15s predic he label f an image wih he riginal se while he cs is nly 0.007s if evaluaing wih laen examples. I makes MaPML wih laen examples a penial mehd fr real-ime applicains. 6. Cnclusin In his wrk, we prpse a framewrk learn he disance meric and laen examples simulaneusly. By learning frm a small se f clean laen examples, MaPML can sample he acive riples efficienly and he learning prcedure is rbus he uncerainy in he real-wrld daa. Mrever, MaPML can preserve he large margin prpery fr he riginal daa when learning merely wih laen examples. The empirical sudy cnfirms he efficacy and efficiency f MaPML. In he fuure, we plan evaluae MaPML n differen asks (e.g., infrmain rerieval) and differen ypes f daa. Besides, incrpraing he prpsed sraegy deep meric learning is als an aracive direcin. I can accelerae he learning fr deep embedding and he resuling laen examples may furher imprve he perfrmance.

9 References [1] S. Byd and L. Vandenberghe. Cnvex Opimizain. Cambridge Universiy Press, New Yrk, NY, USA, [2] G. Chechik, V. Sharma, U. Shali, and S. Bengi. Large scale nline learning f image similariy hrugh ranking. JMLR, 11: , , 2, 4, 6 [3] J. V. Davis, B. Kulis, P. Jain, S. Sra, and I. S. Dhilln. Infrmain-hereic meric learning. In ICML, pages , , 2 [4] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning fr image recgniin. In CVPR, pages , [5] Y. Jia, E. Shelhamer, J. Dnahue, S. Karayev, J. Lng, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Cnvluinal archiecure fr fas feaure embedding. In ACM MM, pages , [6] A. Krizhevsky. Learning muliple layers f feaures frm iny images [7] B. Kulis. Meric learning: A survey. Fundains and Trends in Machine Learning, 5(4): , [8] Y. LeCun, L. Bu, Y. Bengi, and P. Haffner. Gradien-based learning applied dcumen recgniin. Prceedings f he IEEE, 86(11): , [9] Y. Mvshviz-Aias, A. Tshev, T. K. Leung, S. Iffe, and S. Singh. N fuss disance meric learning using prxies. In ICCV, pages , [10] Q. Qian, R. Jin, J. Yi, L. Zhang, and S. Zhu. Efficien disance meric learning by adapive sampling and mini-bach schasic gradien descen (SGD). ML, 99(3): , , 6, 7 [11] Q. Qian, R. Jin, S. Zhu, and Y. Lin. Fine-grained visual caegrizain via muli-sage meric learning. In CVPR, pages , [12] A. Rakhlin, O. Shamir, and K. Sridharan. Making gradien descen pimal fr srngly cnvex schasic pimizain. In ICML, [13] O. Rippel, M. Paluri, P. Dllár, and L. D. Burdev. Meric learning wih adapive densiy discriminain. In ICLR, [14] O. Russakvsky, J. Deng, H. Su, J. Krause, S. Saheesh, S. Ma, Z. Huang, A. Karpahy, A. Khsla, M. S. Bernsein, A. C. Berg, and F. Li. Imagene large scale visual recgniin challenge. IJCV, 115(3): , [15] F. Schrff, D. Kalenichenk, and J. Philbin. Facene: A unified embedding fr face recgniin and clusering. In CVPR, pages , [16] H. O. Sng, Y. Xiang, S. Jegelka, and S. Savarese. Deep meric learning via lifed srucured feaure embedding. In CVPR, pages , [17] K. Q. Weinberger and L. K. Saul. Disance meric learning fr large margin neares neighbr classificain. JMLR, 10: , , 2, 6 [18] E. P. Xing, A. Y. Ng, M. I. Jrdan, and S. J. Russell. Disance meric learning wih applicain clusering wih side-infrmain. In NIPS, pages , , 2 [19] L. Yang and R. Jin. Disance meric learning: a cmprehensive survery [20] H. Ye, D. Zhan, X. Si, and Y. Jiang. Learning mahalanbis disance meric: Cnsidering insance disurbance helps. In IJCAI, pages ,