Spectral Regularization for Max-Margin Sequence Tagging

Transcription

1 Spectrl Regulriztion for Mx-Mrgin Sequence Tgging Aridn Quttoni Borj Blle Xvier Crrers Amir Globerson pq Universitt Politècnic de Ctluny now t p q McGill University p q The Hebrew University of Jeruslem Xerox Reserch Centre Europe ICML 2014 June 2014, Beijing Supported by: XLike EU Project, VISEN EU Project, ISF Centers of Excellence, NSERC nd Jmes McGill Reserch Fund

2 Sequence Tgging output: h I p - x p t x m x s input: h i p p o p o t m u s Fully Observble Models Ltent-vrible Models y 1 y 2 y 3... y T y 1 y 2 y 3 y T x 1 x 2 x 3 x T h 1 h 2 h 3... h T x 1 x 2 x 3 x T ` Mking predictions is trctble ` Lerning is convex Performnce crucilly depends on fetures ` Hidden lyer provides more expressivity Mking predictions is not trctble Lerning is non-convex (this pper)

3 Lerning Structured Predictors with Ltent Vribles Desidert: Expressive scoring functions Trctble prediction function Effective regulrizer Convex trining procedure

4 Min Ide: Chnge of Representtion + Relxtion Problem Formultion Scoring functions re Input-Output OOM (generliztion of HMM) Piecewise Prediction nd Loss Function Solving the Lerning Problem Spectrl trick: optimize over prmeters of f Ñ optimize low-rnk mtrix H Relx low-rnk constrint using nucler norm of H Recover prmeters of f from H using the spectrl method

5 Outline IO-OOM for Sequence Tgging A Convex Formultion for IO-OOM Lerning Experiments

6 Scoring Functions Computed by IO-OOM Ltent Score θpx, y, hq: αph 0 q Tź t 1 A x t y t ph t 1, h t q βph T q Finl Weights: β P R n Scoring Function F A px, yq: ÿ Observble Opertors A θpx, y, hq α T A x 1 y 1... A x b P Rnˆn T y T β h Model: A : xα, β, ta b uy Number of sttes: n Inititl Weights: α P R n Expressive Function Fmily Ñ e.g. it includes HMM Mking Predictions (i.e. mximizing F A px, yq) Ñ NP-hrd

7 Piecewise Prediction nd Loss for IO-OOM Approximtion: F k Apx, yq: T pk 1q ÿ t 1 F A px t:t`k 1, y t:t`k 1 q Loss L k px, y, F A q: Fctor size: k Sum k grms Tsk loss: lpy, zq e.g. hmming distnce mx z rfk A px, zq Fk Apx, yq ` lpy, zqq Prediction nd Loss Function Ñ computed in OpT Y k q using the Viterbi Algorithm

8 Discrete Regulrizer for IO-OOM rgmin APF Lerning Problem: mÿ L k px i, y i, F A q ` τ A q i 1 Function clss (IO-OOM): F Trining Exmples: xx i, y i y Loss Function: L k Regulrizer Ñ number of sttes: A Trde-off constnt: τ k ě 2 Ñ Non-convex dependence of L k on prmeters of A L k involves polynomils of order k ` 3

9 Optimiztion Strtegy L k convex on vlues of A Ñ optimiztion over px ˆ Yq k vlues Three chllenges 1. Tble of vlues Ñ must correspond to vlid IO-OOM 2. Regulrizer over tble Ñ must correspond to #sttes of IO-OOM 3. Recover prmeters of A from this tble

10 Optimiztion Strtegy L k convex on vlues of A Ñ optimiztion over px ˆ Yq k vlues Three chllenges 1. Tble of vlues Ñ must correspond to vlid IO-OOM 2. Regulrizer over tble Ñ must correspond to #sttes of IO-OOM 3. Recover prmeters of A from this tble Solution: the Spectrl Trick loss function trining set Low rnk Mtrix Estimtion Hnkel mtrix Spectrl Method IO-OOM

11 IO-OOM nd Hnkel Mtrices X t, b, cu Y t, u b b c c b b c c b b b b b b b b c c b b c c b b c c... Fp b q

12 IO-OOM nd Hnkel Mtrices X t, b, cu Y t, u b b c c b b c c b b b b b b b b c c b b c c b b c c... F k 3 p c b c q Fpc q ` Fp b q ` Fp b c q

13 IO-OOM nd Hnkel Mtrices X t, b, cu Y t, u b b c c b b c c b b b b b b b b c c b b c c b b c c... Hnkel Structure: Equlity constrints Low-rnk constrints Fundmentl Theorem: F is relized by n n-stte IO-OOM ðñ H hs rnk t most n for every bsis

14 Mx-Mrgin Completion of Hnkel Mtrices Optimiztion with rnk regulriztion: mÿ rgmin L k px i, y i, Hq ` τ rnkphq HPHpP,Sq i 1 Convex relxtion: mÿ rgmin L k px i, y i, Hq ` τ H HPHpP,Sq i 1 Set of Hnkel Mtrices over some bsis: HpP, Sq Rnk regulrizer: rnkphq Nucler norm relxtion: H Optimiztion lmost equivlent Ñ we serch over IO-OOM tht cn be recovered from H P HpP, Sq Once we solve for H we cn recover prmeters using the spectrl technique

15 Estimtion of Hnkel Mtrices vi Convex Optimiztion FOBOS Algorithm: Minimiztion of LpHq ` τ H Initilize: H 0 0 while t ď MxIter do Set Gt to subgrdient of LpHq t H t Set Ht`0.5 H t c? t G t Clculte the SVD of Ht`0.5 UΣV J Define digonl mtrix Σ 1 such tht σ 1 i mxrσ i ν t τ, 0s set H t`1 UΣ 1 V J end while

16 Spectrl Recovery using the method by (Hsu et l. 2009) Spectrl Algorithm for IO-OOM Assume F is relized by miniml n-stte IO-OOM A We re given bsis pp, Sq such tht H hs rnk n We re given corresponding H b To recover prmeters of A: Perform SVD to get H UΣV J Define A b phvq`h b V Typicl spectrl lgorithms ssume tht we cn estimte H In contrst, we regrd H s n optimiztion vrible in loss minimiztion procedure

17 Experiments Tsk: Phonetic Trnscription (UCI Nettlk p - L - h I p - x p t x m x s p p l e h i p p o p o t m u s Regulriztion Pth Lerning Curve Hmming Accurcy (development) No Regulriztion Spectrl Mx Mrgin L2 Mx Mrgin Hmming Accurcy (test) No Regulriztion Avg. Perceptron CRF Spectrl IO HMM L2 Mx Mrgin Spectrl Mx Mrgin 500 1K 2K 5K 10K 15K Trining Smples

18 Conclusion Convex formultion for lerning structured prediction models with ltent vribles nd mx-sum predictions The spectrl trick seen s lineriztion Polynomil optimiztion Ñ liner optimiztion over low-rnk Hnkel mtrices Generlizble to other losses nd structured prediction settings Tke-home messge: Fundmentl ides behind spectrl lerning hve wide rnge of pplicbility for structured prediction

19 Conclusion Convex formultion for lerning structured prediction models with ltent vribles nd mx-sum predictions The spectrl trick seen s lineriztion Polynomil optimiztion Ñ liner optimiztion over low-rnk Hnkel mtrices Generlizble to other losses nd structured prediction settings Tke-home messge: Fundmentl ides behind spectrl lerning hve wide rnge of pplicbility for structured prediction For more informtion: Ñ Come tonight to our poster S63 Ñ On wednesdy, Workshop on Method of Moments nd Spectrl Lerning