APPENDIX A SMO ALGORITHM

Transcription

1 AENDIX A SMO ALGORITHM Sequetial Miimal Optimizatio SMO) is a simple algorithm that ca quickly solve the SVM Q problem without ay extra matrix storage ad without usig time-cosumig umerical Q optimizatio steps []. SMO decomposes the overall Q problem ito the smallest possible optimizatio problem. This sub-problem ca be solved aalytically. A appropriate variat of SMO to solve 7) is detailed below followig []. Give α, the algorithm optimizes two variables of α with other variables fixed. Two variables to be optimized should be chose from α i i I or α i i I +. Otherwise, the variables which we are tryig to optimize caot chage sice the other variables are fixed ad due to the costraits i I α i =ad α i =. Suppose that we choose two variables from α i i I +. For otatioal coveiece, assume the two variables are α ad α ad, I +. The, 7) reduces to mi α,α i= j= s.t α,α 0, α i α j Q ij + d i α i + D i= α i =Δ where D = i=3 j=3 α iα j Q ij i=3 c iα i ad d i = α j Q ij c i, Δ= j=3 i= \, We discard D, which is idepedet of α ad α, ad elimiate α to obtai α i. mi α Δ α ) Q + α Δ α ) Q ) + α Q +Δ α ) d + α d s.t 0 α Δ. Sice the objective fuctio is quadratic ad covex i oe variable α, we ca take the derivative of ) ad set it equal to zero. The, α = ΔQ Q )+d d. 3) Q Q + Q Let α deote the value before the optimizatio step. If we defie O i := Q i α + Q i α + d i = j= α i Q ij c i, the 3) ca be expressed as the update equatio α = α O O +. 4) Q Q + Q If α is outside [0, Δ], we trucate it so that it is withi [0, Δ]. After fidig α, α ca be recovered from α =Δ α. The optimality coditio ad the choice of α i s ca be foud i the followig way. There are three cases whe choosig α ad α : a) Both are zero, b) Oe is positive ad the other is zero, c) Both are positive. Case a): α ad α are ot updated because of oegativity costraits. Case b): Assume that α is zero. From 4), α is updated oly whe O O > 0 ad so is α Case c): α ad α are updated oly whe O O. The objective value will strictly decrease if ad oly if α ad α are updated after optimizatio step. Therefore, the optimal solutio should satisfy O i O j for α i =0,α j > 0 5) O i = O j for α i,α j > 0. 6) The covergece to the global miimum is thus guarateed by choosig two α i s which do ot satisfy 5) or 6) for each optimizatio step. The optimizatio procedure for two variables from α i I is similar.

2 AENDIX B ROOF OF LEMMA Note that for ay give i, k σ X j, X i )) j i are idepedet ad bouded by M =/ πσ ) d. For radom vectors Z f + x) ad W f x), h X i ) i 6) ca be expressed as h X i )=E [k σ Z, X i ) X i ] γe [k σ W, X i ) X i ]. Sice X i f + x) for i I + ad X i f x) for i I, it ca be easily show that ] E [ĥi X i = h X i ). For i I +, ĥi h X i ) >ɛ X i = x,e k σ X j, X i ) E [k σ Z, X i ) X i ] + > ɛ γ + k σ X j, X i ) γe [k σ W, X i ) X i ] > γɛ j I Sice we are coditioig o E, the first term i 7) is k σ X j, X i ) + )E [k σ Z, X i ) X i ] > + ) ɛ ] = k σ X j, X i ) E k σ X j, X i ) X i > + ) ɛ + γ) X i = x ] = k σ X j, X i ) E k σ X j, X i ) X i > + ) ɛ + γ) X i = x e + )ɛ /) M. where the last iequality holds by Hoeffdig s iequality [3]. The secod term i 7) is k σ X j, X i ) E [k σ W, X i ) X i ] > ɛ j I ] k σ X j, X i ) E k σ X j, X i ) X i > ɛ j I j I Therefore, ĥ i h X i ) >ɛ e ɛ /) M e )ɛ /) M. = x x X i = x X i = x ĥ i h X i ) >ɛ X i = x e + )ɛ /) M +e )ɛ /) M ) 7) = e + )ɛ /) M +e )ɛ /) M. I a similar way, it ca be show that for i I, ĥ i h X i ) >ɛ e + )ɛ /) M +e )ɛ /) M.

3 3 The, sup sup α i Y i ĥi h X i )) >ɛ i= α i Y i ĥi h X i ) >ɛ H α) H α) >ɛ = sup = sup i= α i ĥi h X i ) + α i γ ĥi h X i ) >ɛ i I ĥi h X i ) > ɛ + sup sup α i α i γ ĥi h X i ) > i I = max ĥ i h X i ) > ɛ + max ĥ i h X i ) > ɛ i I = ĥ i h X i ) > ɛ + ĥ i h X i ) > ɛ i I ĥ i h X i ) > ɛ + ĥ i h X i ) > ɛ i I + e + )ɛ /) 4 M +e )ɛ /) 4 M ) + e + )ɛ /) 4 M +e )ɛ /) 4 M ) = e + )ɛ /) 4 M +e )ɛ /) 4 M ). γɛ AENDIX C ROOF OF THEOREM Defie u =u,...,u ) such that u i =/ + for i I + ad u i =/ for i I. By the similar argumet for the covergece of MISE of kerel desity estimate [4], it ca be show, usig a multivariate Taylor series, that MISEu; +, )=E[ISE u)] ) = Var ) dγ x; u) + bias dγ x; u) dx = + σ d + γ σ d R k)+ 4 σ4 R tr ) H dγ + o + σ d + σ d + σ 4) where R f) = f x) dx ad H f represet the Hessia matrix of f. Therefore, ISE u) coverges to 0 i probability sice σ 0, + σ d ad + σ d as. Furthermore, ISE α) >ɛ = ISE α) >ɛ,iseu) > ɛ + ISE α) >ɛ,iseu) ɛ ISE u) > ɛ + ISE α) >ISEu)+ ɛ. From the cosistecy of ISE u) ad the oracle iequality stated i Theorem, ISE α) coverges to 0 i probability. AENDIX D ROOF OF THEOREM 3 First ote that i the previous aalyses we treat N +,N ad γ as determiistic variables but ow we tur to the case where these variables are radom. Thus, some of the previous results should be restated cosiderig this. Lemma : γ coverges to γ with probability. roof: Note that N + ad N are biomial radom variables with, p) ad, q) where q = p. From the Hoeffdig s iequality, we kow that for ɛ >0 N+ p>ɛ e ɛ, N q>ɛ e ɛ, N+ p< ɛ e ɛ N q< ɛ e ɛ.

4 4 The, for ay ɛ>0 N ɛ) q N + p >ɛ = pn qn + > ɛpn + = pn qn + > ɛpn +,N + p + pn qn + > ɛpn +,N + < p pn qn + >ɛp p + N + < p pn pq + pq qn + > ɛp + N + p < p pn pq > ɛp3 + qn + pq > ɛp q + N + p < p N = q > ɛp N + + p > ɛp N+ + p< p 4 exp ɛ p 4 ) ) + exp p. Sice = ɛ) < for all ɛ>0, γ coverges to γ with probability. Lemma 3: Suppose the assumptios i Theorem 3 are satisfied. For ay ɛ > 0, ISE α) > if ISE α)+ɛ coverges to 0. roof: We eed to restate Theorem as follows. For ay δ>0, l /δ) ISE α) > if ISE α)+4 c[mi N +,N ) ] N + = +,N = δ sice l /δ) c[mi +, ) ] ɛ l /δ) c[max +, ) ]. N + p,n p) Let us defie c = πσ ) d / ) 4 ad a evet D = l/δ) ISE α) > if ISE α)+4 c [mi p, p)) ] D c + D ISE α) > if ISE α)+4,γ γ. The, l/δ) c [mi p, p)) ]. D The first term coverges to 0 from the strog law of large umbers ad Lemma. The secod term becomes l/δ) ISE α) > if ISE α)+4 c [mi p, p)) ] D l /δ) ISE α) > if ISE α)+4 c[mi N +,N ) ] D = l /δ) ISE α) > if ISE α)+4 c[mi N +,N ) ] D, N + = +,N = N + = +,N = δ N + = +,N = = δ. l/δ) For ay δ>0, we ca make 4 c [mip, p)) ] smaller tha ɛ as, provided that l /σ d 0 as 0. Therefore, ISE α) > if ISE α)+ɛ coverges to 0. Lemma 4: Suppose the assumptios i Theorem 3 are satisfied. The, ISE u) coverges to 0 i probability. roof: Defie a evet D = N + p,n p),γ γ. For ay ɛ>0, ISE u) >ɛ D c + ISE u) >ɛ,d.

5 5 The first term coverges to 0 from the strog law of large umbers ad Lemma. Let defie a set S = +, ) + p, p), + γ. The, ISE u) >ɛ,d = ISE u) >ɛ,d N + = +,N = N + = +,N = = ISE u) >ɛ N + = +,N = N + = +,N = +, ) S +, ) S ɛ +, ) S E [ ISE u) N + = +,N = ] ɛ N + = +,N = [ ) σ d p + 8γ R k)+ 4 p σ4 R tr ) H dγ + o σ d + σ 4)] N + = +,N = ɛ σ d p + γ R k)+ 4 p σ4 R tr ) H dγ + o σ d + σ 4)) where the secod to the last step, we used MISEu; +, ) formula i explaied i Appedix C ad the fact that for +, ) S, + σ d + σ d pσ d + p)σ d = σ d p + ) p Therefore, ISE u) coverges to 0 sice σ 0 ad σ d as. Now let s prove Theorem 3. From Theorem 3 i [5], it suffices to show that ) dγ x; α) d γ x) dx 0 i probability. Note that d γ x; α) d γ x) L = d γ x; α) d γ x)+γ γ ) f x) L d γ x; α) d γ x) L + γ γ ) f x) L = ISE α)+ γ γ f x) L. 8) For the first term i 8), ISE α) >ɛ coverges to 0 i probability sice SE α) >ɛ ISE α) >ISEu)+ ɛ + ISE u) > ɛ ad from Lemma 3 ad 4,. The secod term i 8) also coverges to 0 i probability from Lemma. This proves the theorem. REFERENCES [] Joh C.latt, Sequetial miimal optimizatio: A fast algorithm for traiig support vector machies, Techical Report MSR-TR-98-4, April 00. [] Mark Girolami ad Chao He, robability desity estimatio from optimally codesed data samples, IEEE Trasactios o atter Aalysis ad Machie Itelligece, vol. 5, o. 0, pp , OCT 003. [3] L. Devroye ad G. Lugosi, Combiatorial methods i desity estimatio, 00. [4] D. W. Scott, Multivariate Desity Estimatio, Wiley, New York, 99. [5] Charles T. Wolverto ad Terry J. Wager, Asymptotically optimal discrimiat fuctios for patter classificatio, IEEE Tras. Ifo. Theory, vol. 5, o., pp , Mar 969.