Lagrangian Relaxation Algorithms for Inference in Natural Language Processing

Transcription

1 Lagrangian Relaxation Algorithms for Inference in Natural Language Processing Alexander M. Rush and Michael Collins (based on joint work with Yin-Wen Chang, Tommi Jaakkola, Terry Koo, Roi Reichart, David Sontag)

2 Decoding in NLP focus: structured prediction for natural language processing decoding as a combinatorial optimization problem y = arg max y Y f (y) where f is a scoring function and Y is a set of structures for some problems, use simple combinatorial algorithms dynamic programming minimum spanning tree min cut

3 Structured prediction: Tagging United flies some large jet N V D A N United 1 flies 2 some 3 large 4 jet 5

4 Structured prediction: Parsing United flies some large jet United flies some large jet S NP VP N V NP * 0 United 1 flies 2 some 3 large 4 jet 5 United flies D A N some large jet

5 Decoding complexity issue: simple combinatorial algorithms do not scale to richer models y = arg max y Y f (y) need decoding algorithms for complex natural language tasks motivation: richer model structure often leads to improved accuracy exact decoding for complex models tends to be intractable

6 Structured prediction: Phrase-based translation das muss unsere sorge gleichermaßen sein das muss unsere sorge gleichermaßen sein this must also be our concern

7 Structured prediction: Word alignment the ugly dog has red fur le chien laid a fourrure rouge

8 high complexity Decoding tasks combined parsing and part-of-speech tagging (Rush et al., 2010) loopy HMM part-of-speech tagging syntactic machine translation (Rush and Collins, 2011) NP-Hard symmetric HMM alignment (DeNero and Macherey, 2011) phrase-based translation (Chang and Collins, 2011) higher-order non-projective dependency parsing (Koo et al., 2010) in practice: approximate decoding methods (coarse-to-fine, beam search, cube pruning, gibbs sampling, belief propagation) approximate models (mean field, variational models)

9 Lagrangian relaxation a general technique for constructing decoding algorithms solve complicated models y = arg max f (y) y by decomposing into smaller problems. upshot: can utilize a toolbox of combinatorial algorithms. dynamic programming minimum spanning tree shortest path min cut...

10 Lagrangian relaxation algorithms Simple - uses basic combinatorial algorithms Efficient - faster than solving exact decoding problems Strong guarantees gives a certificate of optimality when exact direct connections to linear programming relaxations

11 MAP problem in Markov random fields given: binary variables x 1... x n goal: MAP problem arg max x 1...x n (i,j) E f i,j (x i, x j ) where each f i,j (x i, x j ) is a local potential for variables x i, x j

12 Dual decomposition for MRFs (Komodakis et al., 2010) + goal: arg max equivalent formulation: arg max x 1...x n (i,j) E f x 1...x n,y 1...y n (i,j) T 1 f i,j (x i, x j ) i,j(x i, x j ) + such that for i = 1... n, x i = y i (i,j) T 2 f i,j(y i, y j ) Lagrangian: L(u, x, y) = (i,j) T 1 f i,j(x i, x j ) + (i,j) T 2 f i,j(y i, y j ) + i u i (x i y i )

13 Related work belief propagation using combinatorial algorithms (Duchi et al., 2007; Smith and Eisner, 2008) factored A* search (Klein and Manning, 2003)

14 Tutorial outline 1. worked algorithm for combined parsing and tagging 2. important theorems and formal derivation 3. more examples from parsing and alignment 4. relationship to linear programming relaxations 5. practical considerations for implemention 6. further example from machine translation

15 1. Worked example aim: walk through a Lagrangian relaxation algorithm for combined parsing and part-of-speech tagging introduce formal notation for parsing and tagging give assumptions necessary for decoding step through a run of the Lagrangian relaxation algorithm

16 Combined parsing and part-of-speech tagging S NP VP N V NP United flies D A N some large jet goal: find parse tree that optimizes score(s NP VP) + score(vp V NP) score(n V) + score(n United) +...

17 notation: Constituency parsing Y is set of constituency parses for input y Y is a valid parse f (y) scores a parse tree goal: arg max y Y f (y) example: a context-free grammar for constituency parsing S NP VP N V NP United flies D A N some large jet

18 notation: goal: Part-of-speech tagging Z is set of tag sequences for input z Z is a valid tag sequence g(z) scores of a tag sequence arg max z Z g(z) example: an HMM for part-of speech tagging N V D A N United 1 flies 2 some 3 large 4 jet 5

19 Identifying tags notation: identify the tag labels selected by each model y(i, t) = 1 when parse y selects tag t at position i z(i, t) = 1 when tag sequence z selects tag t at position i example: a parse and tagging with y(4, A) = 1 and z(4, A) = 1 S NP VP N V NP N V D A N United flies D A N United 1 flies 2 some 3 large 4 jet 5 some large jet z y

20 Combined optimization goal: arg max y Y,z Z such that for all i = 1... n, t T, f (y) + g(z) y(i, t) = z(i, t) i.e. find the best parse and tagging pair that agree on tag labels equivalent formulation: arg max f (y) + g(l(y)) y Y where l : Y Z extracts the tag sequence from a parse tree

21 Exact method: Dynamic programming intersection can solve by solving the product of the two models example: parsing model is a context-free grammar tagging model is a first-order HMM can solve as CFG and finite-state automata intersection S replace VP V NP with VP N,V V N,V NP V,N NP N V VP NP United flies D A N some large jet

22 Intersected parsing and tagging complexity let G be the number of grammar non-terminals parsing CFG require O(G 3 n 3 ) time with rules VP V NP S NP,N VP N,N N V N,V NP V,N United flies D V,D A N A,N some large jet with intersection O(G 3 n 3 T 3 ) with rules VP N,V V N,V NP V,N becomes O(G 3 n 3 T 6 ) time for second-order HMM

23 Parsing assumption assumption: optimization with u can be solved efficiently arg max y Y f (y) + i,t u(i, t)y(i, t) example: CFG with rule scoring function h f (y) = h(x Y Z) + h(x w i ) where X Y Z y (i,x ) y arg max y Y f (y) + i,t u(i, t)y(i, t) = arg max y Y h(x Y Z) + X Y Z y (i,x ) y (h(x w i ) + u(i, X ))

24 Tagging assumption assumption: optimization with u can be solved efficiently arg max z Z g(z) i,t u(i, t)z(i, t) example: HMM with scores for transitions T and observations O g(z) = T (t t ) + O(t w i ) where t t z (i,t) z arg max z Z g(z) i,t u(i, t)z(i, t) = arg max z Z T (t t ) + (O(t w i ) u(i, t)) t t z (i,t) z

25 Lagrangian relaxation algorithm Set u (1) (i, t) = 0 for all i, t T For k = 1 to K y (k) arg max y Y f (y) + i,t u (k) (i, t)y(i, t) [Parsing] z (k) arg max z Z g(z) i,t u (k) (i, t)z(i, t) [Tagging] If y (k) (i, t) = z (k) (i, t) for all i, t Return (y (k), z (k) ) Else u (k+1) (i, t) u (k) (i, t) α k (y (k) (i, t) z (k) (i, t))

26 CKY Parsing Penalties u(i, t) = 0 for all i,t Viterbi Decoding y = arg max y Y (f (y) + i,t u(i, t)y(i, t)) United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

27 CKY Parsing NP S VP Penalties u(i, t) = 0 for all i,t A N D A V United flies some large jet Viterbi Decoding y = arg max y Y (f (y) + i,t u(i, t)y(i, t)) United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

28 CKY Parsing NP S VP Penalties u(i, t) = 0 for all i,t A N D A V United flies some large jet Viterbi Decoding y = arg max y Y (f (y) + i,t u(i, t)y(i, t)) N V D A N United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

29 CKY Parsing NP S VP Penalties u(i, t) = 0 for all i,t A N D A V United flies some large jet Viterbi Decoding y = arg max y Y (f (y) + i,t u(i, t)y(i, t)) N V D A N United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

30 CKY Parsing A United N flies Viterbi Decoding NP D some S A large y = arg max y Y (f (y) + i,t VP V jet u(i, t)y(i, t)) N V D A N Penalties u(i, t) = 0 for all i,t Iteration 1 u(1, A) -1 u(1, N) 1 u(2, N) -1 u(2, V ) 1 u(5, V ) -1 u(5, N) 1 United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

31 CKY Parsing Viterbi Decoding y = arg max y Y (f (y) + i,t u(i, t)y(i, t)) Penalties u(i, t) = 0 for all i,t Iteration 1 u(1, A) -1 u(1, N) 1 u(2, N) -1 u(2, V ) 1 u(5, V ) -1 u(5, N) 1 United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

32 CKY Parsing Viterbi Decoding NP N United S V flies y = arg max y Y (f (y) + i,t VP D some NP A large u(i, t)y(i, t)) N jet Penalties u(i, t) = 0 for all i,t Iteration 1 u(1, A) -1 u(1, N) 1 u(2, N) -1 u(2, V ) 1 u(5, V ) -1 u(5, N) 1 United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

33 CKY Parsing Viterbi Decoding NP N United S V flies y = arg max y Y (f (y) + i,t VP D some NP A large u(i, t)y(i, t)) A N D A N N jet Penalties u(i, t) = 0 for all i,t Iteration 1 u(1, A) -1 u(1, N) 1 u(2, N) -1 u(2, V ) 1 u(5, V ) -1 u(5, N) 1 United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

34 CKY Parsing Viterbi Decoding NP N United S V flies y = arg max y Y (f (y) + i,t VP D some NP A large u(i, t)y(i, t)) A N D A N United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) N jet Penalties u(i, t) = 0 for all i,t Iteration 1 u(1, A) -1 u(1, N) 1 u(2, N) -1 u(2, V ) 1 u(5, V ) -1 u(5, N) 1 Iteration 2 u(5, V ) -1 u(5, N) 1 Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

35 CKY Parsing Viterbi Decoding y = arg max y Y (f (y) + i,t u(i, t)y(i, t)) United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) Penalties u(i, t) = 0 for all i,t Iteration 1 u(1, A) -1 u(1, N) 1 u(2, N) -1 u(2, V ) 1 u(5, V ) -1 u(5, N) 1 Iteration 2 u(5, V ) -1 u(5, N) 1 Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

36 CKY Parsing Viterbi Decoding NP N United S V flies y = arg max y Y (f (y) + i,t VP D some NP A large u(i, t)y(i, t)) United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) N jet Penalties u(i, t) = 0 for all i,t Iteration 1 u(1, A) -1 u(1, N) 1 u(2, N) -1 u(2, V ) 1 u(5, V ) -1 u(5, N) 1 Iteration 2 u(5, V ) -1 u(5, N) 1 Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

37 CKY Parsing Viterbi Decoding NP N United S V flies y = arg max y Y (f (y) + i,t VP D some NP A large u(i, t)y(i, t)) N V D A N United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) N jet Penalties u(i, t) = 0 for all i,t Iteration 1 u(1, A) -1 u(1, N) 1 u(2, N) -1 u(2, V ) 1 u(5, V ) -1 u(5, N) 1 Iteration 2 u(5, V ) -1 u(5, N) 1 Key f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

38 CKY Parsing Viterbi Decoding Key NP N United S V flies y = arg max y Y (f (y) + i,t VP D some NP A large u(i, t)y(i, t)) N V D A N United 1 flies 2 some 3 large 4 jet 5 z = arg max z Z (g(z) i,t u(i, t)z(i, t)) N jet Penalties u(i, t) = 0 for all i,t Iteration 1 u(1, A) -1 u(1, N) 1 u(2, N) -1 u(2, V ) 1 u(5, V ) -1 u(5, N) 1 Iteration 2 u(5, V ) -1 u(5, N) 1 Converged y = arg max f (y) + g(y) y Y f (y) CFG g(z) HMM Y Parse Trees Z Taggings y(i, t) = 1 if y contains tag t at position i

39 Main theorem theorem: if at any iteration, for all i, t T y (k) (i, t) = z (k) (i, t) then (y (k), z (k) ) is the global optimum proof: focus of the next section

40 Convergence % examples converged <=1 <=2 <=3 <=4 <=10 <=20 <=50 number of iterations

41 2. Formal properties aim: formal derivation of the algorithm given in the previous section derive Lagrangian dual prove three properties upper bound convergence optimality describe subgradient method

42 goal: Lagrangian arg Lagrangian: max y Y,z Z f (y) + g(z) such that y(i, t) = z(i, t) L(u, y, z) = f (y) + g(z) + i,t u(i, t) (y(i, t) z(i, t)) redistribute terms L(u, y, z) = f (y) + i,t u(i, t)y(i, t) + g(z) i,t u(i, t)z(i, t)

43 Lagrangian: L(u, y, z) = Lagrangian dual f (y) + u(i, t)y(i, t) + g(z) i,t i,t u(i, t)z(i, t) Lagrangian dual: L(u) = max y Y,z Z = max y Y max z Z L(u, y, z) f (y) + i,t g(z) i,t u(i, t)y(i, t) + u(i, t)z(i, t)

44 Theorem 1. Upper bound define: y, z is the optimal combined parsing and tagging solution with y (i, t) = z (i, t) for all i, t theorem: for any value of u L(u) f (y ) + g(z ) L(u) provides an upper bound on the score of the optimal solution note: upper bound may be useful as input to branch and bound or A* search

45 Theorem 1. Upper bound (proof) theorem: for any value of u, L(u) f (y ) + g(z ) proof: L(u) = max L(u, y, z) y Y,z Z (1) max L(u, y, z) y Y,z Z:y=z (2) = max f (y) + g(z) y Y,z Z:y=z (3) = f (y ) + g(z ) (4)

46 Formal algorithm (reminder) Set u (1) (i, t) = 0 for all i, t T For k = 1 to K y (k) arg max y Y f (y) + i,t u (k) (i, t)y(i, t) [Parsing] z (k) arg max z Z g(z) i,t u (k) (i, t)z(i, t) [Tagging] If y (k) (i, t) = z (k) (i, t) for all i, t Return (y (k), z (k) ) Else u (k+1) (i, t) u (k) (i, t) α k (y (k) (i, t) z (k) (i, t))

47 Theorem 2. Convergence notation: u (k+1) (i, t) u (k) (i, t) + α k (y (k) (i, t) z (k) (i, t)) is update u (k) is the penalty vector at iteration k α k > 0 is the update rate at iteration k theorem: for any sequence α 1, α 2, α 3,... such that lim t αt = 0 and α t =, t=1 we have lim t L(ut ) = min L(u) u i.e. the algorithm converges to the tightest possible upper bound proof: by subgradient convergence (next section)

48 Dual solutions define: for any value of u y u = arg max y Y f (y) + i,t u(i, t)y(i, t) and z u = arg max z Z g(z) i,t u(i, t)z(i, t) y u and z u are the dual solutions for a given u

49 Theorem 3. Optimality theorem: if there exists u such that y u (i, t) = z u (i, t) for all i, t then f (y u ) + g(z u ) = f (y ) + g(z ) i.e. if the dual solutions agree, we have an optimal solution (y u, z u )

50 Theorem 3. Optimality (proof) theorem: if u such that y u (i, t) = z u (i, t) for all i, t then f (y u ) + g(z u ) = f (y ) + g(z ) proof: by the definitions of y u and z u L(u) = f (y u ) + g(z u ) + i,t u(i, t)(y u (i, t) z u (i, t)) = f (y u ) + g(z u ) since L(u) f (y ) + g(z ) for all values of u f (y u ) + g(z u ) f (y ) + g(z ) but y and z are optimal f (y u ) + g(z u ) f (y ) + g(z )

51 Dual optimization Lagrangian dual: L(u) = max y Y,z Z = max y Y max z Z L(u, y, z) f (y) + i,t g(z) i,t u(i, t)y(i, t) + u(i, t)z(i, t) goal: dual problem is to find the tightest upper bound min L(u) u

52 L(u) = max y Y f (y) + i,t Dual subgradient u(i, t)y(i, t) + max g(z) z Z i,t u(i, t)z(i, t) properties: L(u) is convex in u (no local minima) L(u) is not differentiable (because of max operator) handle non-differentiability by using subgradient descent define: a subgradient of L(u) at u is a vector g u such that for all v L(v) L(u) + g u (v u)

53 Subgradient algorithm L(u) = max y Y f (y) + i,t u(i, t)y(i, t) + max g(z) z Z i,j u(i, t)z(i, t) recall, y u and z u are the argmax s of the two terms subgradient: g u (i, t) = y u (i, t) z u (i, t) subgradient descent: move along the subgradient u (i, t) = u(i, t) α (y u (i, t) z u (i, t)) guaranteed to find a minimum with conditions given earlier for α

54 3. More examples aim: demonstrate similar algorithms that can be applied to other decoding applications context-free parsing combined with dependency parsing combined translation alignment

55 Combined constituency and dependency parsing (Rush et al., 2010) setup: assume separate models trained for constituency and dependency parsing problem: find constituency parse that maximizes the sum of the two models example: combine lexicalized CFG with second-order dependency parser

56 notation: Lexicalized constituency parsing Y is set of lexicalized constituency parses for input y Y is a valid parse f (y) scores a parse tree goal: arg max y Y f (y) example: a lexicalized context-free grammar S(flies) NP(United) VP(flies) N V NP(jet) United flies D A N some large jet

57 Dependency parsing define: Z is set of dependency parses for input z Z is a valid dependency parse g(z) scores a dependency parse example: * 0 United 1 flies 2 some 3 large 4 jet 5

58 Identifying dependencies notation: identify the dependencies selected by each model y(i, j) = 1 when word i modifies of word j in constituency parse y z(i, j) = 1 when word i modifies of word j in dependency parse z example: a constituency and dependency parse with y(3, 5) = 1 and z(3, 5) = 1 S(flies) NP(United) N V VP(flies) NP(jet) United flies D A N * 0 United 1 flies 2 some 3 large 4 jet 5 some large jet z y

59 Combined optimization goal: arg max y Y,z Z f (y) + g(z) such that for all i = 1... n, j = 0... n, y(i, j) = z(i, j)

60 CKY Parsing Penalties u(i, j) = 0 for all i,j Dependency Parsing y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) * 0 United 1 flies 2 some 3 large 4 jet 5 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) CFG g(z) Dependency Model Y Parse Trees Z Dependency Trees y(i, j) = 1 if y contains dependency i, j

61 CKY Parsing NP S(flies) VP(flies) Penalties u(i, j) = 0 for all i,j N V D NP(jet) United flies some A N large jet Dependency Parsing y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) * 0 United 1 flies 2 some 3 large 4 jet 5 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) CFG g(z) Dependency Model Y Parse Trees Z Dependency Trees y(i, j) = 1 if y contains dependency i, j

62 CKY Parsing NP S(flies) VP(flies) Penalties u(i, j) = 0 for all i,j N V D NP(jet) United flies some A N large jet Dependency Parsing y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) * 0 United 1 flies 2 some 3 large 4 jet 5 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) CFG g(z) Dependency Model Y Parse Trees Z Dependency Trees y(i, j) = 1 if y contains dependency i, j

63 CKY Parsing NP S(flies) VP(flies) Penalties u(i, j) = 0 for all i,j N V D NP(jet) United flies some A N large jet Dependency Parsing y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) * 0 United 1 flies 2 some 3 large 4 jet 5 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) CFG g(z) Dependency Model Y Parse Trees Z Dependency Trees y(i, j) = 1 if y contains dependency i, j

64 CKY Parsing NP N United S(flies) V flies VP(flies) D NP(jet) some A N large jet Penalties u(i, j) = 0 for all i,j Iteration 1 u(2, 3) -1 u(5, 3) 1 Dependency Parsing y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) * 0 United 1 flies 2 some 3 large 4 jet 5 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) CFG g(z) Dependency Model Y Parse Trees Z Dependency Trees y(i, j) = 1 if y contains dependency i, j

65 CKY Parsing Penalties u(i, j) = 0 for all i,j Iteration 1 u(2, 3) -1 u(5, 3) 1 Dependency Parsing y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) * 0 United 1 flies 2 some 3 large 4 jet 5 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) CFG g(z) Dependency Model Y Parse Trees Z Dependency Trees y(i, j) = 1 if y contains dependency i, j

66 CKY Parsing NP N United S(flies) V flies VP(flies) D some NP(jet) A large N jet Penalties u(i, j) = 0 for all i,j Iteration 1 u(2, 3) -1 u(5, 3) 1 Dependency Parsing y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) * 0 United 1 flies 2 some 3 large 4 jet 5 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) CFG g(z) Dependency Model Y Parse Trees Z Dependency Trees y(i, j) = 1 if y contains dependency i, j

67 CKY Parsing NP N United S(flies) V flies VP(flies) D some NP(jet) A large N jet Penalties u(i, j) = 0 for all i,j Iteration 1 u(2, 3) -1 u(5, 3) 1 Dependency Parsing y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) * 0 United 1 flies 2 some 3 large 4 jet 5 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) CFG g(z) Dependency Model Y Parse Trees Z Dependency Trees y(i, j) = 1 if y contains dependency i, j

68 CKY Parsing NP N United S(flies) V flies VP(flies) D some NP(jet) A large N jet Penalties u(i, j) = 0 for all i,j Iteration 1 u(2, 3) -1 u(5, 3) 1 Dependency Parsing y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) Converged y = arg max f (y) + g(y) y Y * 0 United 1 flies 2 some 3 large 4 jet 5 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) CFG g(z) Dependency Model Y Parse Trees Z Dependency Trees y(i, j) = 1 if y contains dependency i, j

69 Convergence % examples converged <=1 <=2 <=3 <=4 <=10 <=20 <=50 number of iterations

70 Integrated Constituency and Dependency Parsing: Accuracy Collins Dep Dual F 1 Score Collins (1997) Model 1 Fixed, First-best Dependencies from Koo (2008) Dual Decomposition

71 Combined alignment (DeNero and Macherey, 2011) setup: assume separate models trained for English-to-French and French-to-English alignment problem: find an alignment that maximizes the score of both models example: HMM models for both directional alignments (assume correct alignment is one-to-one for simplicity)

72 define: English-to-French alignment Y is set of all possible English-to-French alignments y Y is a valid alignment f (y) scores of the alignment example: HMM alignment Le 1 laid 3 chien 2 a 4 rouge 6 fourrure 5 The 1 ugly 2 dog 3 has 4 red 5 fur 6

73 define: French-to-English alignment Z is set of all possible French-to-English alignments z Z is a valid alignment g(z) scores of an alignment example: HMM alignment The 1 ugly 2 dog 3 has 4 fur 6 red 5 Le 1 chien 2 laid 3 a 4 fourrure 5 rouge 6

74 Identifying word alignments notation: identify the tag labels selected by each model y(i, j) = 1 when e-to-f alignment y selects French word i to align with English word j z(i, j) = 1 when f-to-e alignment z selects French word i to align with English word j example: two HMM alignment models with y(6, 5) = 1 and z(6, 5) = 1

75 Combined optimization goal: arg max y Y,z Z f (y) + g(z) such that for all i = 1... n, j = 1... n, y(i, j) = z(i, j)

76 English-to-French Penalties u(i, j) = 0 for all i,j French-to-English y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) HMM Alignment g(z) HMM Alignment Y English-to-French model Z French-to-English model y(i, j) = 1 if French word i aligns to English word j

77 English-to-French Penalties u(i, j) = 0 for all i,j French-to-English y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) HMM Alignment g(z) HMM Alignment Y English-to-French model Z French-to-English model y(i, j) = 1 if French word i aligns to English word j

78 English-to-French Penalties u(i, j) = 0 for all i,j French-to-English y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) HMM Alignment g(z) HMM Alignment Y English-to-French model Z French-to-English model y(i, j) = 1 if French word i aligns to English word j

79 English-to-French Penalties u(i, j) = 0 for all i,j French-to-English y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) HMM Alignment g(z) HMM Alignment Y English-to-French model Z French-to-English model y(i, j) = 1 if French word i aligns to English word j

80 English-to-French French-to-English y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) Penalties u(i, j) = 0 for all i,j Iteration 1 u(3, 2) -1 u(2, 2) 1 u(2, 3) -1 u(3, 3) 1 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) HMM Alignment g(z) HMM Alignment Y English-to-French model Z French-to-English model y(i, j) = 1 if French word i aligns to English word j

81 English-to-French French-to-English y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) Penalties u(i, j) = 0 for all i,j Iteration 1 u(3, 2) -1 u(2, 2) 1 u(2, 3) -1 u(3, 3) 1 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) HMM Alignment g(z) HMM Alignment Y English-to-French model Z French-to-English model y(i, j) = 1 if French word i aligns to English word j

82 English-to-French French-to-English y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) Penalties u(i, j) = 0 for all i,j Iteration 1 u(3, 2) -1 u(2, 2) 1 u(2, 3) -1 u(3, 3) 1 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) HMM Alignment g(z) HMM Alignment Y English-to-French model Z French-to-English model y(i, j) = 1 if French word i aligns to English word j

83 English-to-French French-to-English y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) Penalties u(i, j) = 0 for all i,j Iteration 1 u(3, 2) -1 u(2, 2) 1 u(2, 3) -1 u(3, 3) 1 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) HMM Alignment g(z) HMM Alignment Y English-to-French model Z French-to-English model y(i, j) = 1 if French word i aligns to English word j

84 English-to-French French-to-English y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) Penalties u(i, j) = 0 for all i,j Iteration 1 u(3, 2) -1 u(2, 2) 1 u(2, 3) -1 u(3, 3) 1 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) HMM Alignment g(z) HMM Alignment Y English-to-French model Z French-to-English model y(i, j) = 1 if French word i aligns to English word j

85 English-to-French French-to-English y = arg max y Y (f (y) + i,j u(i, j)y(i, j)) Penalties u(i, j) = 0 for all i,j Iteration 1 u(3, 2) -1 u(2, 2) 1 u(2, 3) -1 u(3, 3) 1 Key z = arg max z Z (g(z) i,j u(i, j)z(i, j)) f (y) HMM Alignment g(z) HMM Alignment Y English-to-French model Z French-to-English model y(i, j) = 1 if French word i aligns to English word j

86 4. Linear programming aim: explore the connections between Lagrangian relaxation and linear programming basic optimization over the simplex formal properties of linear programming full example with fractional optimal solutions

87 define: Simplex y R Y is the simplex over Y where α y implies α y 0 and y α y = 1 α is distribution over Y z is the simplex over Z δ y : Y y maps elements to the simplex example: Y = {y 1, y 2, y 3 } vertices δ y (y 1 ) = (1, 0, 0) δ y (y 2 ) = (0, 1, 0) δ y (y 3 ) = (0, 0, 1) δ y (y 2 ) δ y (y 1 ) y δ y (y3)

88 Theorem 1. Simplex linear program optimize over the simplex y instead of the discrete sets Y goal: optimize linear program theorem: max α y y max f (y) = max y Y α y f (y) α y y α y f (y) proof: points in Y correspond to the exteme points of simplex {δ y (y) : y Y} linear program has optimum at extreme point proof shows that best distribution chooses a single parse

89 Combined linear program optimize over the simplices y and z instead of the discrete sets Y and Z goal: optimize linear program max α y f (y) + α y,β z y z β z g(z) such that for all i, t α y y(i, t) = y z β z z(i, t) note: the two distributions must match in expectation of POS tags the best distributions α,β are possibly no longer a single parse tree or tag sequence

90 Lagrangian Lagrangian: M(u, α, β) = α y f (y) + β zg(z) + ( u(i, t) α y y(i, t) y z i,t y z ( = α y f (y) + u(i, t) ) α y y(i, t) + y i,t y ( β zg(z) u(i, t) ) β zz(i, t) z i,t z Lagrangian dual: β zz(i, t) ) M(u) = max M(u, α, β) α y,β z

91 Theorem 2. Strong duality define: α, β is the optimal assignment to α, β in the linear program theorem: min M(u) = u y α y f (y) + z β z g(z) proof: by linear programming duality

92 Theorem 3. Dual relationship theorem: for any value of u, M(u) = L(u) note: solving the original Lagrangian dual also solves dual of the linear program

93 Theorem 3. Dual relationship (proof sketch) focus on Y term in Lagrangian L(u) = max y Y f (y) + i,t u(i, t)y(i, t) +... M(u) = max α y y α y f (y) + i,t u(i, t) y α y y(i, t) +... by theorem 1. optimization over Y and y have the same max similar argument for Z gives L(u) = M(u)

94 Summary f (y) + g(z) L(u) y α y f (y) + z β zg(z) M(u) original primal objective original dual LP primal objective LP dual relationship between LP dual, original dual, and LP primal objective min u M(u) = min L(u) = u y α y f (y) + z β z g(z)

95 Concrete example Y = {y 1, y 2, y 3 } Z = {z 1, z 2, z 3 } y R 3, z R 3 y 1 y 2 y 3 x x x Y a a b b c c He is He is He is z 1 z 2 z 3 Z a b b a c c He is He is He is

96 Simple solution y 1 y 2 y 3 Y a x a b x b c x c He is He is He is z 1 z 2 z 3 Z a b b a c c He is He is He is choose: α (1) = (0, 0, 1) y is representation of y 3 β (1) = (0, 0, 1) z is representation of z 3 confirm: α y (1) y(i, t) = β z (1) z(i, t) y z α (1) and β (1) satisfy agreement constraint

97 Fractional solution y 1 y 2 y 3 Y a x a b x b c x c He is He is He is z 1 z 2 z 3 Z a b b a c c He is He is He is choose: α (2) = (0.5, 0.5, 0) y is combination of y 1 and y 2 β (2) = (0.5, 0.5, 0) z is combination of z 1 and z 2 confirm: y α y (2) y(i, t) = z β (2) z z(i, t) α (2) and β (2) satisfy agreement constraint, but not integral

98 Optimal solution weights: the choice of f and g determines the optimal solution if (f, g) favors (α (2), β (2) ), the optimal solution is fractional example: f = [1 1 2] and g = [1 1 2] f α (1) + g β (1) = 0 vs f α (2) + g β (2) = 2 α (2), β (2) is optimal, even though it is fractional summary: dual and LP primal optimal: min u M(u) = min L(u) = u y α y (2) f (y) + z β (2) z g(z) = 2 original primal optimal: f (y ) + g(z ) = 0

99 round 1 dual solutions: y 3 z 2 dual values: y (1) 2.00 z (1) 1.00 L(u (1) ) 3.00 x c c He is b He a is previous solutions: y 3 z Round

100 round 2 dual solutions: y 2 z 1 dual values: y (2) 2.00 z (2) 1.00 L(u (2) ) 3.00 x b b He is a He b is previous solutions: y 3 z 2 y 2 z Round

101 round 3 dual solutions: y 1 z 1 dual values: y (3) 2.50 z (3) 0.50 L(u (3) ) 3.00 x a a He is a He b is previous solutions: y 3 z 2 y 2 z 1 y 1 z Round

102 round 4 dual solutions: y 1 z 1 dual values: y (4) 2.17 z (4) 0.17 L(u (4) ) x a a He is a He b is previous solutions: y 3 z 2 y 2 z 1 y 1 z 1 y 1 z Round

103 round 5 dual solutions: y 2 z 2 dual values: y (5) 2.08 z (5) 0.08 L(u (5) ) x b b He is b He a is previous solutions: y 3 z 2 y 2 z 1 y 1 z 1 y 1 z 1 y 2 z Round

104 round 6 dual solutions: y 1 z 1 dual values: y (6) 2.12 z (6) 0.12 L(u (6) ) x a a He is a He b is previous solutions: y 3 z 2 y 2 z 1 y 1 z 1 y 1 z 1 y 2 z 2 y 1 z Round

105 round 7 dual solutions: y 2 z 2 dual values: y (7) 2.05 z (7) 0.05 L(u (7) ) x b b He is b He a is previous solutions: y 3 z 2 y 2 z 1 y 1 z 1 y 1 z 1 y 2 z 2 y 1 z 1 y 2 z Round

106 round 8 dual solutions: y 1 z 1 dual values: y (8) 2.09 z (8) 0.09 L(u (8) ) x a a He is a He b is previous solutions: y 3 z 2 y 2 z 1 y 1 z 1 y 1 z 1 y 2 z 2 y 1 z 1 y 2 z 2 y 1 z Round

107 round 9 dual solutions: y 2 z 2 dual values: y (9) 2.03 z (9) 0.03 L(u (9) ) x b b He is b He a is previous solutions: y 3 z 2 y 2 z 1 y 1 z 1 y 1 z 1 y 2 z 2 y 1 z 1 y 2 z 2 y 1 z 1 y 2 z Round

108 5. Practical issues tracking the progress of the algorithm know current dual value and (possibly) primal value choice of update rate α k various strategies; success with rate based on dual progress lazy update of dual solutions if updates are sparse, can avoid dynamically update soltuions extracting solutions if algorithm does not converge best primal feasible solution; average solutions

109 Phrase-Based Translation define: source-language sentence words x 1,..., x N phrase translation p = (s, e, t) translation derivation y = p 1,..., p L example: x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein p 1 p 2 p 3 p 4 y = {(1, 2, this must), (5, 5, also), (6, 6, be), (3, 4, our concern)}

110 Phrase-Based Translation define: source-language sentence words x 1,..., x N phrase translation p = (s, e, t) translation derivation y = p 1,..., p L example: x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must p 1 y = {(1, 2, this must), (5, 5, also), (6, 6, be), (3, 4, our concern)}

111 Phrase-Based Translation define: source-language sentence words x 1,..., x N phrase translation p = (s, e, t) translation derivation y = p 1,..., p L example: x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also p 1 p 2 y = {(1, 2, this must), (5, 5, also), (6, 6, be), (3, 4, our concern)}

112 Phrase-Based Translation define: source-language sentence words x 1,..., x N phrase translation p = (s, e, t) translation derivation y = p 1,..., p L example: x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also be p 1 p 2 p 3 y = {(1, 2, this must), (5, 5, also), (6, 6, be), (3, 4, our concern)}

113 Phrase-Based Translation define: source-language sentence words x 1,..., x N phrase translation p = (s, e, t) translation derivation y = p 1,..., p L example: x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also be our concern p 1 p 2 p 3 p 4 y = {(1, 2, this must), (5, 5, also), (6, 6, be), (3, 4, our concern)}

114 Phrase-Based Translation define: source-language sentence words x 1,..., x N phrase translation p = (s, e, t) translation derivation y = p 1,..., p L example: x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also be our concern p 1 p 2 p 3 p 4 y = {(1, 2, this must), (5, 5, also), (6, 6, be), (3, 4, our concern)}

115 Phrase-Based Translation define: source-language sentence words x 1,..., x N phrase translation p = (s, e, t) translation derivation y = p 1,..., p L example: x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also be our concern p 1 p 2 p 3 p 4 y = {(1, 2, this must), (5, 5, also), (6, 6, be), (3, 4, our concern)}

116 Scoring Derivations derivation: y = {(1, 2, this must), (5, 5, 2also), (6, 6, be), (3, 4, our concern)} x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also be our concern objective: f (y) = h(e(y)) + L g(p k ) + k=1 L 1 η t(p k ) + 1 s(p k+1 ) k=1 language model score h phrase translation score g distortion penalty η

117 Scoring Derivations derivation: y = {(1, 2, this must), (5, 5, 2also), (6, 6, be), (3, 4, our concern)} x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also be our concern objective: f (y) = h(e(y)) + L g(p k ) + k=1 L 1 η t(p k ) + 1 s(p k+1 ) k=1 language model score h phrase translation score g distortion penalty η

118 Scoring Derivations derivation: y = {(1, 2, this must), (5, 5, 2also), (6, 6, be), (3, 4, our concern)} x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also be our concern objective: f (y) = h(e(y)) + L g(p k ) + k=1 L 1 η t(p k ) + 1 s(p k+1 ) k=1 language model score h phrase translation score g distortion penalty η

119 Scoring Derivations derivation: y = {(1, 2, this must), (5, 5, 2also), (6, 6, be), (3, 4, our concern)} x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also be our concern objective: f (y) = h(e(y)) + L g(p k ) + k=1 L 1 η t(p k ) + 1 s(p k+1 ) k=1 language model score h phrase translation score g distortion penalty η

120 Relaxed Problem Y : only requires the total number of words translated to be N Y ={y : N y(i) = N and the distortion limit d is satisfied} i=1 example: y(i) sum 6 x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein our concern must be our concern (3, 4, our concern)(2, 2, must)(6, 6, be)(3, 4, our concern)

121 Relaxed Problem Y : only requires the total number of words translated to be N Y ={y : N y(i) = N and the distortion limit d is satisfied} i=1 example: y(i) sum 6 x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein our concern must be our concern (3, 4, our concern)(2, 2, must)(6, 6, be)(3, 4, our concern)

122 Relaxed Problem Y : only requires the total number of words translated to be N Y ={y : N y(i) = N and the distortion limit d is satisfied} i=1 example: y(i) sum 6 x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein our concern must be our concern (3, 4, our concern)(2, 2, must)(6, 6, be)(3, 4, our concern)

123 Relaxed Problem Y : only requires the total number of words translated to be N Y ={y : N y(i) = N and the distortion limit d is satisfied} i=1 example: y(i) sum 6 x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein our concern must be our concern (3, 4, our concern)(2, 2, must)(6, 6, be)(3, 4, our concern)

124 Lagrangian Relaxation Method original: arg max f (y) y Y }{{} exact DP is NP-hard rewrite: Y = {y : y(i) = 1 i = 1... N} arg max f (y) such that y Y }{{ } can be solved efficiently by DP y(i) = 1 i = 1... N }{{} using Lagrangian relaxation Y = {y : N i=1 y(i) = N} }{{} sum to N

125 Lagrangian Relaxation Method original: arg max f (y) y Y }{{} exact DP is NP-hard rewrite: Y = {y : y(i) = 1 i = 1... N} arg max f (y) such that y Y }{{ } can be solved efficiently by DP y(i) = 1 i = 1... N }{{} using Lagrangian relaxation Y = {y : N i=1 y(i) = N} }{{} sum to N

126 Lagrangian Relaxation Method original: arg max f (y) y Y }{{} exact DP is NP-hard rewrite: Y = {y : y(i) = 1 i = 1... N} arg max f (y) such that y Y }{{ } can be solved efficiently by DP y(i) = 1 i = 1... N }{{} using Lagrangian relaxation Y = {y : N i=1 y(i) = N} }{{} sum to N

127 Lagrangian Relaxation Method original: arg max f (y) y Y }{{} exact DP is NP-hard rewrite: Y = {y : y(i) = 1 i = 1... N} arg max f (y) such that y Y }{{ } can be solved efficiently by DP y(i) = 1 i = 1... N }{{} using Lagrangian relaxation Y = {y : N i=1 y(i) = N} }{{} sum to N

128 Lagrangian Relaxation Method original: arg max f (y) y Y }{{} exact DP is NP-hard rewrite: Y = {y : y(i) = 1 i = 1... N} arg max f (y) such that y Y }{{ } can be solved efficiently by DP y(i) = 1 i = 1... N }{{} using Lagrangian relaxation Y = {y : N i=1 y(i) = N} }{{} sum to N

129 Algorithm Iteration 1: update u(i): u(i) u(i) α(y(i) 1) α = 1 u(i) y(i) update x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein

130 Algorithm Iteration 1: update u(i): u(i) u(i) α(y(i) 1) α = 1 u(i) y(i) update x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein our concern must be our concern

131 Algorithm Iteration 1: update u(i): u(i) u(i) α(y(i) 1) α = 1 u(i) y(i) update x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein our concern must be our concern

132 Algorithm Iteration 2: update u(i): u(i) u(i) α(y(i) 1) α = 0.5 u(i) y(i) update x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein

133 Algorithm Iteration 2: update u(i): u(i) u(i) α(y(i) 1) α = 0.5 u(i) y(i) update x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must be equally must equally

134 Algorithm Iteration 2: update u(i): u(i) u(i) α(y(i) 1) α = 0.5 u(i) y(i) update x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must be equally must equally

135 Algorithm Iteration 3: update u(i): u(i) u(i) α(y(i) 1) α = 0.5 u(i) y(i) update x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein

136 Algorithm Iteration 3: update u(i): u(i) u(i) α(y(i) 1) α = 0.5 u(i) y(i) update x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also be our concern

137 Tightening the Relaxation In some cases, we never reach y(i) = 1 for i = 1... N If dual L(u) is not decreasing fast enough run for 10 more iterations count number of times each constraint is violated add 3 most often violated constraints

138 Tightening the Relaxation Iteration 41: count(i) y(i) x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also our concern equally

139 Tightening the Relaxation Iteration 42: count(i) y(i) x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must be our concern is

140 Tightening the Relaxation Iteration 43: count(i) y(i) x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also our concern equally

141 Tightening the Relaxation Iteration 44: count(i) y(i) x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must be our concern is

142 Tightening the Relaxation Iteration 50: count(i) y(i) x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also our concern equally

143 Tightening the Relaxation Iteration 51: count(i) 0 y(i) 1 1 x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also be our concern Add 2 hard constraints (x 5, x 6 ) to the dynamic program

144 Tightening the Relaxation Iteration 51: count(i) 0 y(i) x 1 x 2 x 3 x 4 x 5 x 6 das muss unsere sorge gleichermaßen sein this must also be our concern Add 2 hard constraints (x 5, x 6 ) to the dynamic program

145 Experiments: German to English Europarl data: German to English Test on 1,824 sentences with length 1-50 words Converged: 1,818 sentences (99.67%)

146 Experiments: Number of Iterations Percentage words words words words words all Maximum Number of Lagrangian Relexation Iterations

147 Experiments: Number of Hard Constraints Required Percentage words words words words words all Number of Hard Constraints Added

148 Experiments: Mean Time in Seconds # words All mean median

149 Comparison to ILP Decoding (sec.) (sec.) , , , ,692.6

150 Summary presented Lagrangian relaxation as a method for decoding in NLP formal guarantees gives certificate or approximate solution can improve approximate solutions by tightening relaxation efficient algorithms uses fast combinatorial algorithms can improve speed with lazy decoding widely applicable demonstrated algorithms for a wide range of NLP tasks (parsing, tagging, alignment, mt decoding)

151 Higher-order non-projective dependency parsing setup: given a model for higher-order non-projective dependency parsing (sibling features) problem: find non-projective dependency parse that maximizes the score of this model difficulty: model is NP-hard to decode complexity of the model comes from enforcing combinatorial constraints strategy: design a decomposition that separates combinatorial constraints from direct implementation of the scoring function

152 Non-Projective Dependency Parsing * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 Important problem in many languages. Problem is NP-Hard for all but the simplest models.

153 Dual Decomposition A classical technique for constructing decoding algorithms. Solve complicated models y = arg max f (y) y by decomposing into smaller problems. Upshot: Can utilize a toolbox of combinatorial algorithms. Dynamic programming Minimum spanning tree Shortest path Min-Cut...

154 Non-Projective Dependency Parsing * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 Starts at the root symbol * Each word has a exactly one parent word Produces a tree structure (no cycles) Dependencies can cross

155 Arc-Factored * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) =

156 Arc-Factored * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, mod =saw 2 )

157 Arc-Factored * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, mod =saw 2 ) +score(saw 2, John 1 )

158 Arc-Factored * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, mod =saw 2 ) +score(saw 2, John 1 ) +score(saw 2, movie 4 )

159 Arc-Factored * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, mod =saw 2 ) +score(saw 2, John 1 ) +score(saw 2, movie 4 ) +score(saw 2, today 5 )

160 Arc-Factored * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, mod =saw 2 ) +score(saw 2, John 1 ) +score(saw 2, movie 4 ) +score(saw 2, today 5 ) +score(movie 4, a 3 ) +...

161 Arc-Factored * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, mod =saw 2 ) +score(saw 2, John 1 ) +score(saw 2, movie 4 ) +score(saw 2, today 5 ) +score(movie 4, a 3 ) +... e.g. score( 0, saw 2 ) = log p(saw 2 0 ) (generative model)

162 Arc-Factored * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, mod =saw 2 ) +score(saw 2, John 1 ) +score(saw 2, movie 4 ) +score(saw 2, today 5 ) +score(movie 4, a 3 ) +... e.g. score( 0, saw 2 ) = log p(saw 2 0 ) or score( 0, saw 2 ) = w φ(saw 2, 0 ) (generative model) (CRF/perceptron model)

163 Arc-Factored * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, mod =saw 2 ) +score(saw 2, John 1 ) +score(saw 2, movie 4 ) +score(saw 2, today 5 ) +score(movie 4, a 3 ) +... e.g. score( 0, saw 2 ) = log p(saw 2 0 ) or score( 0, saw 2 ) = w φ(saw 2, 0 ) (generative model) (CRF/perceptron model) y = arg max f (y) Minimum Spanning Tree Algorithm y

164 Sibling Models * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) =

165 Sibling Models * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, prev = NULL, mod = saw 2 )

166 Sibling Models * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, prev = NULL, mod = saw 2 ) +score(saw 2, NULL, John 1 )

167 Sibling Models * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, prev = NULL, mod = saw 2 ) +score(saw 2, NULL, John 1 ) +score(saw 2, NULL, movie 4 )

168 Sibling Models * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, prev = NULL, mod = saw 2 ) +score(saw 2, NULL, John 1 ) +score(saw 2, NULL, movie 4 ) +score(saw 2,movie 4, today 5 ) +...

169 Sibling Models * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, prev = NULL, mod = saw 2 ) +score(saw 2, NULL, John 1 ) +score(saw 2, NULL, movie 4 ) +score(saw 2,movie 4, today 5 ) +... e.g. score(saw 2, movie 4, today 5 ) = log p(today 5 saw 2, movie 4 )

170 Sibling Models * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, prev = NULL, mod = saw 2 ) +score(saw 2, NULL, John 1 ) +score(saw 2, NULL, movie 4 ) +score(saw 2,movie 4, today 5 ) +... e.g. score(saw 2, movie 4, today 5 ) = log p(today 5 saw 2, movie 4 ) or score(saw 2, movie 4, today 5 ) = w φ(saw 2, movie 4, today 5 )

171 Sibling Models * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 f (y) = score(head = 0, prev = NULL, mod = saw 2 ) +score(saw 2, NULL, John 1 ) +score(saw 2, NULL, movie 4 ) +score(saw 2,movie 4, today 5 ) +... e.g. score(saw 2, movie 4, today 5 ) = log p(today 5 saw 2, movie 4 ) or score(saw 2, movie 4, today 5 ) = w φ(saw 2, movie 4, today 5 ) y = arg max f (y) NP-Hard y

172 Thought Experiment: Individual Decoding * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8

173 Thought Experiment: Individual Decoding * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 score(saw 2, NULL, John 1 ) + score(saw 2, NULL, movie 4 ) +score(saw 2, movie 4, today 5 )

174 Thought Experiment: Individual Decoding * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 score(saw 2, NULL, John 1 ) + score(saw 2, NULL, movie 4 ) +score(saw 2, movie 4, today 5 ) score(saw 2, NULL, John 1 ) + score(saw 2, NULL, that 6 )

175 Thought Experiment: Individual Decoding * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 score(saw 2, NULL, John 1 ) + score(saw 2, NULL, movie 4 ) +score(saw 2, movie 4, today 5 ) score(saw 2, NULL, John 1 ) + score(saw 2, NULL, that 6 ) score(saw 2, NULL, a 3 ) + score(saw 2, a 3, he 7 )

176 Thought Experiment: Individual Decoding * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 score(saw 2, NULL, John 1 ) + score(saw 2, NULL, movie 4 ) +score(saw 2, movie 4, today 5 ) 2 n 1 possibilities score(saw 2, NULL, John 1 ) + score(saw 2, NULL, that 6 ) score(saw 2, NULL, a 3 ) + score(saw 2, a 3, he 7 )

177 Thought Experiment: Individual Decoding * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 score(saw 2, NULL, John 1 ) + score(saw 2, NULL, movie 4 ) +score(saw 2, movie 4, today 5 ) 2 n 1 possibilities score(saw 2, NULL, John 1 ) + score(saw 2, NULL, that 6 ) score(saw 2, NULL, a 3 ) + score(saw 2, a 3, he 7 ) Under Sibling Model, can solve for each word with Viterbi decoding.

178 Thought Experiment Continued * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 Idea: Do individual decoding for each head word using dynamic programming. If we re lucky, we ll end up with a valid final tree.

179 Thought Experiment Continued * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 Idea: Do individual decoding for each head word using dynamic programming. If we re lucky, we ll end up with a valid final tree.

180 Thought Experiment Continued * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 Idea: Do individual decoding for each head word using dynamic programming. If we re lucky, we ll end up with a valid final tree.

181 Thought Experiment Continued * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 Idea: Do individual decoding for each head word using dynamic programming. If we re lucky, we ll end up with a valid final tree.

182 Thought Experiment Continued * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 Idea: Do individual decoding for each head word using dynamic programming. If we re lucky, we ll end up with a valid final tree.

183 Thought Experiment Continued * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 Idea: Do individual decoding for each head word using dynamic programming. If we re lucky, we ll end up with a valid final tree.

184 Thought Experiment Continued * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 Idea: Do individual decoding for each head word using dynamic programming. If we re lucky, we ll end up with a valid final tree.

185 Thought Experiment Continued * 0 John 1 saw 2 a 3 movie 4 today 5 that 6 he 7 liked 8 Idea: Do individual decoding for each head word using dynamic programming. If we re lucky, we ll end up with a valid final tree.