Spectral Probabilistic Modeling and Applications to Natural Language Processing

Transcription

1 School of Computer Science Spectral Probabilistic Modeling and Applications to Natural Language Processing Doctoral Thesis Proposal Ankur Parikh Thesis Committee: Eric Xing, Geoff Gordon, Noah Smith, Le Song, Ben Taskar Ankur CMU,

2 Probabilistic Modeling Modeling uncertainty among a set of random variables. Key to advances many aspects of AI: natural language processing, bioinformatics, computer vision Ankur CMU,

3 Probabilistic Graphical Models Sophisticated formalism for probabilistic modeling Ankur CMU,

4 Key Aspects of Graphical Models Structure Learning Ankur CMU,

5 Key Aspects of Graphical Models Structure Learning Chow Liu Algorithm can find optimal tree Ankur CMU,

6 Key Aspects of Graphical Models Structure Learning Chow Liu Algorithm can find optimal tree Parameter Learning Max Likelihood Ankur CMU,

7 Key Aspects of Graphical Models Structure Learning Chow Liu Algorithm can find optimal tree Parameter Learning Max Likelihood Inference Belief Propagation Ankur CMU,

8 Key Aspects of Graphical Models Structure Learning Chow Liu Algorithm can find optimal tree stuffy nose fever Parameter Learning Max Likelihood body aches Inference Belief Propagation sore throat Modeling How to use these tools to model real world phenomena fatigue Ankur CMU,

9 Is This A Good Model? stuffy nose fever body aches sore throat fatigue Not really, they are all dependent on one another regardless of what we condition on. Ankur CMU,

10 But This Model Seems Overly Complex stuffy nose fever body aches Sore throat fatigue Ankur CMU,

11 Latent Variables Can Help! There is a simpler explanation underlying the data. All the symptoms are conditionally independent given this explanation. stuffy nose fever Intuitively represents the flu body aches sore throat fatigue Ankur CMU,

12 Latent Variable Models... H 1 H 2 H 3 H 4 X 1 X 2 X 3 X 4 Sequence models Parsing Ho. et al Mixed membership models Cohen et al Ankur CMU,

13 But Learning And Inference Become Harder Structure Learning Cannot compute mutual information among latents Parameter Learning Likelihood is no longer convex Inference marginal MAP is hard even for trees Ankur CMU,

14 Existing Approaches Most existing approaches formulate problem as nonconvex optimization (e.g. Expectation Maximization) From Computational Perspective Can get trapped in local optima Slow to converge Hard to parallelize From Modeling Perspective Unclear what the relationship is between model and difficulty of learning/inference problem Ankur CMU,

15 Linear Algebra and Graphical Models Approach these problems through the lens of linear algebra Latent variable models correspond to low rank factorizations of the observed variables Leverage linear algebra tools to derive new solutions Rank SVD / Matrix Factorization Eigenvalues Tensors Ankur CMU,

16 Previous Work Our work is inspired by recent theoretical results from different communities Theoretical Computer Science (Dasgupta, 1999) Dynamical Systems (Katayama, 2005) Phylogenetics (Mossel and Roch 2005) Statistical Machine Learning (Hsu et al. 2009, Bailly 2009) Spectral Learning Algorithms for HMMs Hsu et al. 2009, Bailly 2009 (Spectral HMMs) Siddiqi et al (Reduced Rank HMMs) Song et al (Kernel HMMs) Ankur CMU,

17 (Informal) Thesis Statement Take a more general graphical models point of view. Aim to approach the key problems of structure learning, parameter learning, and inference from the linear algebra perspective. Use these insights to design new models and solutions for problems in Natural Language Processing Unsupervised Parsing Language Models Ankur CMU,

18 Proposed Contributions (ML) Spectral Parameter Learning for Latent Trees / Junction Trees Key Theme: Parameter Learning Kernel Embeddings for Latent Trees Key Themes: Parameter/Structure Learning Spectral Approximations for Inference (e.g. marginal MAP / collapsed sampling) Key Theme: Inference Ankur CMU,

19 Proposed Contributions (NLP) A Conditional Latent Tree Model for Unsupervised Parsing Key Themes: Structure Learning / Modeling N-gram language modeling with non-integer n Key Theme: Modeling Ankur CMU,

20 Other Concurrent Work to this Thesis Predictive state representations (Boots et al., 2010/2013) Topic models (Anandkumar et al., 2012/2013) Latent variable supervised parsing (Luque et al., 2012, Cohen et al., 2012, Dhillon et al. 2012, Cohen and Collins, 2013) Spectral Learning via Local Loss Optimization (Balle et al. 2012) Word embeddings (Dhillon et al. 2011/2012) Ankur CMU,

21 Outline Linear Algebra View of Latent Variable Models Spectral Parameter Learning Spectral Approximations for Inference Language Modeling (Kernels and unsupervised parsing omitted due to time) Ankur CMU,

22 School of Computer Science Linear Algebra View of Latent Variable Models Ankur CMU,

23 Important Notation Probabilities P X, Y = P Y X) P(X) Probability Vectors/Matrices/Tensors P Y = P Y X)P(X) Ankur CMU,

24 Traditional View Of Graphical Models Complexity of mostly characterized by structure or treewidth Ankur CMU,

25 Shortcoming Of This View Marginalizing out latent variable leads to a clique H But aren t these models different? Ankur CMU,

26 It Depends on The Number Of States If H takes on only one state then the variables are independent. H = Ankur CMU,

27 It Depends on the Number of States Assume observed variables take m states If H takes on m 5 states then model can be equivalent to clique H = But what about all the other cases? Ankur CMU,

28 Graphical Models: The Linear Algebra View X 1 X 2 X 1 and X 2 have m states each. P(X 1, X 2 ) In general, nothing we can say about the nature of this matrix. Ankur CMU,

29 Independence: The Linear Algebra View What if we know X 1 and X 2 are independent? P(X 1, X 2 ) X 1 X 2 [ P X 1 = 1, X 2 = 1,, P X 1 = 1, X 2 = m ] = [ P X 1 = 1) (P(X 2 = 1,, P X 2 = m )] Matrix is rank one! Ankur CMU,

30 Independence and Rank X 1 X 2 P(X 1, X 2 ) has rank m (at most) X 1 X 2 P(X 1, X 2 ) has rank 1 What about rank in between 1 and m? Ankur Parikh@ CMU,

31 Low Rank Structure X 1 and X 2 are not marginally independent (They are only conditionally independent given H). X 1 H X 2 Assume H has k states Then, rank P X 1, X 2 k Why? Ankur Parikh,@ CMU

32 Low Rank Structure X 1 H X 2 P X 1, X 2 P X 1 H P H P X 2 H T = rank k rank k rank k rank k Ankur Parikh,@ CMU

33 The Linear Algebra View Latent variable models encode low rank dependencies among variables (both marginal and conditional) Use tools from linear algebra to exploit this structure. Rank Eigenvalues SVD / Matrix Factorization Tensors Ankur CMU,

34 School of Computer Science Spectral Parameter Learning Ankur CMU,

35 Related Papers A.P. Parikh, L. Song, E.P. Xing, A Spectral Algorithm for Latent Tree Graphical Models (ICML 2011) L.Song, A.P. Parikh, E.P. Xing, Kernel Embeddings of Latent Tree Graphical Models (NIPS 2011) A.P. Parikh, L. Song, M. Ishteva, G. Teodoru, E.P. Xing, A Spectral Algorithm for Latent Junction Trees (UAI 2012) L. Song, M. Ishteva, A.P. Parikh, H. Park, and E.P. Xing, Hierarchical Tensor Decomposition for Latent Tree Graphical Models (ICML 2013) Ankur CMU,

36 Tensors Multidimensional arrays A Tensor of order N has N modes (N indices): Each mode is associated with a dimension. In the example, Dimension of mode 1 is 4 Dimension of mode 2 is 3 Dimension of mode 3 is 4 T(i 1,, i n ) 4 3 Ankur CMU,

37 Tensors 1 st order tensor (vector) 3 rd order tensor (cube) 2 nd order tensor (matrix) Higher order tensors Ankur CMU, 2013

38 Tensor Tensor Multiplication Order 4 = 3,2 R i, j, k, l = T 1 i, j, m T 2 k, m, l m Ankur CMU, 2013

39 Tensor Tensor Multiplication Matrix multiplication is closed (i.e. product of matrices is a matrix). Tensor multiplication is not closed (e.g. product of two tensors of order n is a tensor of order 2n 2) Ankur CMU,

40 Parameter Learning in Latent Trees A B C D X 1 X 2 X 3 X 4 X 5 X 6 Conditional probability tables depend on latent variables Common approach is Expectation Maximization (EM) Local optima Slow to converge Ankur CMU,

41 {X 1, X 2, X 3, X 4 } Low Rank Perspective A k states B C D X 1 X 2 X 3 X 4 X 5 X 6 m states {X 5, X 6 } P X {1,2,3,4}, X {5,6} has rank k Ankur CMU,

42 Low Rank Matrices Factorize M = LR If M has rank k m by n m by k k by n We already know one factorization!!! P X {1,2,3,4}, X {5,6} = P X {1,2,3,4} A P A P X 5,6 A T Factor of 6 variables Factor of 5 variables Factor of 3 variables Factor of 1 variable Ankur CMU,

43 Alternate Factorizations The key insight is that this factorization is not unique. Consider Matrix Factorization. Can add any invertible transformation: M = LR M = LS S 1 R The magic of spectral learning is that there exists an alternative factorization that only depends on observed variables! Ankur CMU,

44 An Alternate Factorization Let us say we only want to factorize this matrix of 6 variables P X {1,2,3,4}, X {5,6} such that it is product of matrices that contain at most five observed variables e.g. P X {1,2,3,4}, X {5} P X {4}, X {5,6} Ankur CMU,

45 An Alternate Factorization Note that P X {1,2,3,4}, X {5} = P X {1,2,3,4} A P A P X 5 A T P X {4}, X {5,6} = P X {4} A P A P X 5,6 A T Product of green terms (in some order) is P X {1,2,3,4}, X {5,6} Product of red terms (in some order) is P X {4}, X {5} Ankur CMU,

46 An Alternate Factorization P X {1,2,3,4}, X {5,6} = P X {1,2,3,4}, X {5} P X 4, X 5 1 P X {4}, X {5,6} factor of 6 variables factor of 5 variables factor of 3 variables Advantage: Factors are only functions of observed variables! Can be directly computed from data without EM!!!! Caveat: Factors are no longer probability tables Ankur CMU,

47 What Low Rank Relationship Did We Use P X {1,2,3,4}, X {5} P X {4}, X {5,6} = P X {1,2,3,4} A P A P X {5} A = P X {4} A P A P X {5,6} A A B C D X 1 X 2 X 3 X 4 X 5 X 6 Ankur CMU,

48 The Tensor Point Of View P X {1,2,3,4}, X {5,6} = P X {1,2,3,4}, X {5} P X 4, X 5 1 P X {4}, X {5,6} P X 1, X 2, X 3, X 4, X 5, X 6 = P X 1, X 2, X 3, X 4, X 5 5 P X 4, X P X 4, X 5, X 6 Order 6 Order 5 Decompose recursively!! Ankur CMU,

49 Observable Factorization P X 1, X 2, X 3, X 4, X 5, X 6 P X 1, X 3, X 5 P X 1, X 2, X 3 3 P X 1, X 3 1 P X 3, X 4, X 5 5 P X 3, X 5 1 P X 4, X 5, X 6 4 P X 4, X 5 1 Ankur CMU,

50 Also Works For Junction Trees Need to use higher order tensor operations since separator sets have size > 1 Multi-mode multiplication Tensor inversion I H AC AC ACE A H C E D BCDE F B G BC F BD G Ankur CMU,

51 Empirical Results Some example results for junction trees Ankur CMU,

52 Runtime(s) Error 3 rd Order NonHomogeneous HMM (Spectral/EM/Online EM) Runtime vs. Sample Size Error vs. Sample Size Online EM Online EM EM EM Spectral Spectral Training Samples (x10 3 ) Training Samples (x10 3 ) Results from Parikh et al Ankur CMU,

53 Runtime(s) Error Synthetic Junction Tree Runtime vs. Sample Size Error vs. Sample Size EM online EM online EM spectral EM spectral Training Samples (x10 3 ) Training Samples (x10 3 ) Results from Parikh et al Ankur CMU,

54 Error Error Real Data Stock data Splice dataset CL EM EM- EM+ spectral online EM spectral Number of query variables Training Samples (x10 3 ) Results from Parikh et al. 2011/2012 Ankur CMU,

55 Kernel Embeddings of Latent Trees Algorithm generalizes to non-gaussian, continuous variables with Hilbert Space Embeddings [Smola et al. 2007, Song et al. 2010] Ankur CMU,

56 Error Kernel Embeddings of Latent Trees Gaussian NPN Kernel query size Ankur CMU,

57 School of Computer Science Spectral Approximations for Inference Ankur CMU,

58 Inference from the Spectral Perspective Traditionally, inference methods designed and analyzed from the perspective of the structure of the graphical model Difficulty of inference characterized by treewidth However, this paradigm can be limiting. Ankur CMU,

59 Example 1: Collapsed Sampling X 1 X 1 X 8 X 2 X 8 X 2 X 7 H X 3 X 7 X 3 X 6 X 5 X 4 X 6 X 5 X 4 x i P X i h i ) h i P H x 1,, x 8 ) Slow mixing Easy parallelizable x i P X i x 1,, x i 1, x i+1, x 8 ) Faster mixing Hard to parallelize Ankur CMU,

60 Parallel Collapsed Sampling x 1 P X 1 x 2,, x 8 ) x 2 P X 2 x 1, x 3,, x 8 ) Technically, must be in sequential. However, (partially) asynchronous sampling works well in practice Fully Asynchronous (Symth et al. 2008) Stale Synchronous (Ho et al. 2013) But a theoretical understanding is lacking One exception is Ihler et al Ankur CMU,

61 Treewidth Perspective Only Gives Worst Case Assume H takes on m 8 states. Then model is equivalent to the clique. X 1 X 8 X 2 X 7 X 3 X 6 X 5 X 4 In this case, there are O m 8 parameters and thus P X i x 1,, x i 1, x i+1, x 8 ) P X i x 1,, x i 1, x i+1, x 8 ) Ankur CMU,

62 But If Model Is Low Rank If H takes on fewer states, then model is low rank. X 1 X 8 X 2 Acts as bottleneck X 7 H X 3 X 6 X 5 X 4 It is now likely that P X i x 1,, x i 1, x i+1, x 8 ) P X i x 1,, x i 1, x i+1, x 8 ) How can this help us justify approximate parallel sampling methods? Ankur CMU,

63 Example 2: Marginal MAP H 1 H 2 H 3 H 4 H 5 X 1 X 2 X 3 X 4 X 5... Find most likely assignment of a set of variables (max nodes) while marginalizing out the rest (sum nodes). Ankur CMU,

64 Sometimes Easy Sum out X 1,, X 5 while finding the most likely assignment to most likely assignment to H 1,, H 5 max H 1,,H 5 X1,,X 5 i P X i H i ) P(H i H i 1 ) First run sum-product to marginalize out X 1,, X 5 H 1 H 2 H 4 H 5... H 1 H 3 Then run on max-product on remaining nodes H 1,, H 5 Ankur CMU,

65 But Sometimes Hard Sum out H 1,, H 5 while finding the most likely assignment to most likely assignment to X 1,, X 5 max X 1,,X 5 H1,,H 5 i P X i H i ) P(H i H i 1 ) First run sum-product to marginalize out H 1,, H 5 X 1 X 2 But now I have a clique X 3 X 4 Ankur CMU, X 5

66 But Again The Clique Is Low Rank... Can we somehow take advantage of this? X 1 X 2 X 3 X 4 X 5 A work in progress! Ankur CMU,

67 School of Computer Science A Low Rank Framework For Language Modeling Ankur CMU,

68 Language Modeling Evaluate probabilities of sentences P w 1,.., w 4 = Spectral methods are boring P w 1,.., w 4 = Spectral methods are awesome Very useful in downstream applications such as machine translation and speech recognition. Ankur CMU,

69 N-Gram Language Model Predominant language modeling technique for two decades Assume joint distribution of words follows n th order Markov chain P w 1,.., w l l = i=1 P w i w 1,, w i 1 ) l i=1 P w i w i n+1,, w i 1 ) Ankur CMU,

70 But This Doesn t Solve the Problem Vocabulary is often very large 10 4 (for relatively small datasets) parameters 10 8 parameters 10 4 parameters Extreme bias-variance tradeoff Ankur CMU,

71 N-gram Smoothing Essentially combine these as needed. P w i w i 1, w i 2 ) P w i w i 1 ) P(w i ) Subtract probability out of higher order n-grams to make sure probability sums to one Ankur CMU,

72 Discounting P d w i w i 1 ) = c w i, w i 1 d w c w, w i 1 count P sm w i w i 1 ) = P d w i w i 1 ) if P d w i w i 1 ) = 0 γ w i 1 P w i otherwise Where γ w i 1 is the leftover probability Ankur CMU,

73 Shortcoming P w i = w i 1 P w i w i 1 ) P w i P sm w i = w i 1 P sm w i w i 1 ) P w i Lower order marginal doesn t align!! P w i P sm (w i ) Intuition: Lower order distribution needs to be adjusted. Ankur CMU,

74 Kneser Ney - Intuition Consider first a unigram model Consider two words York and door York may have higher marginal probability than door P York > P(door) Ankur CMU,

75 Kneser Ney - Intuition Consider bigram model Diversity of history York only follows very few words i.e. New York Door can follow many words i.e. the door, red door, my door etc. Therefore if we back off on w i 1, w i = door is more likely P w i = door backed off on w i 1 ) > P(w i = York backed off on w i 1 ) Ankur CMU,

76 Kneser Ney Use altered lower order distributions to take this into account MLE trigram altered bigram altered unigram Ankur CMU,

77 Kneser Ney N w i = w c w i, w > 0 Diversity of w i s history P kn uni (w i ) = N w i w N w P kney w i w i 1 ) = P d w i w i 1 ) if P d w i w i 1 ) = 0 γ w i 1 P kn uni w i otherwise Ankur CMU,

78 Lower Order Marginal Aligns! P w i = w i 1 P kney w i w i 1 ) P w i Ankur CMU,

79 Low Rank Approaches Consider ML problems with sparse matrices e.g. Collaborate filtering Matrix completion Many solutions involve low rank approximation These solutions have been attempted in NLP Saul and Periera 1997 Hutchinson et al Unfortunately, not generally competitive Kneser Ney Ankur CMU,

80 Our Contribution We present a generalization of existing language models such as Kneser Ney to allow for non-integer n Key Insights: Treat Kneser Ney as a special case of a low rank framework Take low rank approximations of alternate quantities instead of MLE Generalize discounting for this framework Ankur CMU,

81 Rank Rank distinguishes bigram and unigram P w i w i 1 ) = P(w i, w i 1 ) P(w i 1 ) Full rank Ankur CMU,

82 Rank Let B be the matrix such that B w i, w i 1 = c w i, w i 1 Let M 1 = min M:M 0,rank M =1 B M KL = Then M 1 w i, w i 1 P w i P w i 1 Ankur CMU,

83 Rank MLE unigram is normalized rank 1 approx. of MLE bigram under KL: P w i = M 1 w i, w i 1 w M 1 (w i, w i 1 ) Vary rank to obtain quantities between bigram and unigram Ankur CMU,

84 Generalizing The Alteration Kneser Ney uses alternate lower order matrices. Need continuous generalization of this alteration Take low rank approximations of these alternate quantities Ankur CMU,

85 Consider Elementwise Power B B B row sum row sum row sum emphasis on diversity Ankur CMU,

86 Consider Elementwise Power M 1 0 = min M:M 0,rank M =1 B 0 M KL P kn uni (w i ) = M 1 0 w i, w i 1 w M 1 0 w, w i 1 Ankur CMU,

87 Low Rank Ensembles Intermediate matrices/tensors are both low rank and different power r 1, p 1 r 2, p 2 r 3, p 3 r 4, p 4 r 5, p 5 r 6, p 6 r 7, p 7 Ankur CMU,

88 What about the marginal constraint?? P w i = w i 1 P low rank w i w i 1 ) P w i Not if fixed discounting is used But we have developed a generalized discounting scheme such that the constraint holds Ankur CMU,

89 Empirical results coming soon! Ankur CMU,

90 Outline Linear Algebra View of Latent Variable Models Spectral Parameter Learning Spectral Approximations for Inference Language Modeling (Kernels and unsupervised parsing omitted due to time) Ankur CMU,

91 Conclusion Linear algebra can provide a different perspective on graphical models This viewpoint allows us to design solutions for probabilistic modeling that have both theoretical and practical value. Results on spectral parameter learning show promise Working on inference, language modeling, unsupervised parsing Ankur CMU,

92 (Tentative) Timeline Contribution Status and Schedule Spectral Learning of Latent Trees [ICML 2011, ICML 2013], journal (Summer 2014) Spectral Learning of Latent Junction Trees [UAI 2012], journal (Summer 2014) Kernel Embeddings of Latent Trees [NIPS 2011] Power Low Rank Language Models Fall 2013 Unsupervised Parsing Fall 2013/Early Spring 2014 Spectral Approximations of Inference Spring/Summer 2014 Scalable Low Rank Language Models Fall 2014 Writing Thesis Fall/Winter 2014 Ankur CMU,

93 Acknowledgements Thesis Committee Members Eric Xing Geoff Gordon Noah Smith Le Song Ben Taskar Ankur CMU,

94 Acknowledgements Thesis-Related Collaborators Eric Xing Le Song Mariya Ishteva Chris Dyer Shay Cohen Avneesh Saluja Other Collaborators Wei Wu, Qirong Ho, Mladen Kolar, Hoifung Poon, Kristina Toutanova, Asela Gunawardana, Chris Meek Ankur CMU,

95 School of Computer Science Thanks! Ankur CMU,