Bipartite spectral graph partitioning to co-cluster varieties and sound correspondences

Transcription

1 Bipartite spectral graph partitioning to co-cluster varieties and sound correspondences Martijn Wieling Department of Computational Linguistics, University of Groningen Seminar in Methodology and Statistics - May 20, 2009 Martijn Wieling 1/32

2 Goal Making the title of this presentation understandable! Bipartite spectral graph partitioning to co-cluster varieties and sound correspondences Martijn Wieling 2/32

3 Overview Why co-clustering? Method Introduction to eigenvalues and eigenvectors Simple clustering Co-clustering Complete dataset Results Conclusions Martijn Wieling 3/32

4 Why co-clustering? Research interest: language and dialectal variation Important method: cluster similar (dialectal) varieties together Problem: clustering varieties does not yield a linguistic basis Previous solutions: investigate sound correspondences post hoc (e.g., Heeringa, 2004) Co-clustering: clusters varieties and sound correspondences simultaneously Eigenvalues and eigenvectors are central in this approach Martijn Wieling 4/32

5 Graphs and matrices A graph is a set of vertices connected with edges: A graph can also be represented by its adjacency matrix A A B C D A B C D Martijn Wieling 5/32

6 Eigenvalues and eigenvectors The eigenvalues λ and the eigenvectors x of a square matrix A are defined as follows: Ax = λx [ (A λi)x = 0] In matrix-form: [ a11 λ a 12 ] [ x1 a 22 λ a 21 x 2 ] = [ 0 0 ] This is solved when: (a 11 λ)x 1 + a 12 x 2 = 0 a 21 x 1 + (a 22 λ)x 2 = 0 Martijn Wieling 6/32

7 Eigenvalues and eigenvectors The eigenvalues λ and the eigenvectors x of a square matrix A are defined as follows: Ax = λx [ (A λi)x = 0] In matrix-form: [ a11 λ a 12 ] [ x1 a 22 λ a 21 x 2 ] = [ 0 0 ] This is solved when: (a 11 λ)x 1 + a 12 x 2 = 0 a 21 x 1 + (a 22 λ)x 2 = 0 Martijn Wieling 6/32

8 Eigenvalues and eigenvectors The eigenvalues λ and the eigenvectors x of a square matrix A are defined as follows: Ax = λx [ (A λi)x = 0] In matrix-form: [ a11 λ a 12 ] [ x1 a 22 λ a 21 x 2 ] = [ 0 0 ] This is solved when: (a 11 λ)x 1 + a 12 x 2 = 0 a 21 x 1 + (a 22 λ)x 2 = 0 Martijn Wieling 6/32

9 Example of calculating eigenvalues and eigenvectors Consider the following example: A = Using (A λi)x = 0 we get: [ 1 λ 2 ] [ x1 2 1 λ x 2 [ ] ] = [ 0 0 ] Solved when det(a) = 0: (1 λ) 2 4 = 0 Using λ 1 = 3 and λ 2 = 1 we obtain x = [ ] [ ] 1 1 and x = 1 1 Martijn Wieling 7/32

10 Example of calculating eigenvalues and eigenvectors Consider the following example: A = Using (A λi)x = 0 we get: [ 1 λ 2 ] [ x1 2 1 λ x 2 [ ] ] = [ 0 0 ] Solved when det(a) = 0: (1 λ) 2 4 = 0 Using λ 1 = 3 and λ 2 = 1 we obtain x = [ ] [ ] 1 1 and x = 1 1 Martijn Wieling 7/32

11 Example of calculating eigenvalues and eigenvectors Consider the following example: A = Using (A λi)x = 0 we get: [ 1 λ 2 ] [ x1 2 1 λ x 2 [ ] ] = [ 0 0 ] Solved when det(a) = 0: (1 λ) 2 4 = 0 Using λ 1 = 3 and λ 2 = 1 we obtain x = [ ] [ ] 1 1 and x = 1 1 Martijn Wieling 7/32

12 Example of calculating eigenvalues and eigenvectors Consider the following example: A = Using (A λi)x = 0 we get: [ 1 λ 2 ] [ x1 2 1 λ x 2 [ ] ] = [ 0 0 ] Solved when det(a) = 0: (1 λ) 2 4 = 0 Using λ 1 = 3 and λ 2 = 1 we obtain x = [ ] [ ] 1 1 and x = 1 1 Martijn Wieling 7/32

13 Spectrum of a graph The spectrum of a graph are the eigenvalues of the adjacency matrix A of the graph The spectrum is considered to capture important structural properties of a graph (Chung, 1997) Some interesting applications of eigenvalues and eigenvectors: Principal Component Analysis (PCA; Duda et al., 2001: ) Pagerank (Google; Brin and Page, 1998) Partitioning (i.e. clustering; Von Luxburg, 2007) Martijn Wieling 8/32

14 Example of spectral graph clustering (1/8) Consider the matrix A with sound correspondences: In graph-form: [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] Martijn Wieling 9/32

15 Example of spectral graph clustering (2/8) To partition this graph, we have to determine the optimal cut: The optimal cut yielding balanced clusters is obtained by finding the eigenvectors of the normalized Laplacian: L n = D 1 L, with L = D A and D the degree matrix of A (Shi and Malik, 2000; Von Luxburg, 2007). Martijn Wieling 10/32

16 Example of spectral graph clustering (3/8) The adjacency matrix A: [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] Martijn Wieling 11/32

17 Example of spectral graph clustering (4/8) The Laplacian matrix L: [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] Martijn Wieling 12/32

18 Example of spectral graph clustering (5/8) The normalized Laplacian matrix L n : [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] Martijn Wieling 13/32

19 Example of spectral graph clustering (6/8) The eigenvalues λ and eigenvectors x of L n (i.e. L n x = λx): λ 1 = 0 with x = [ ] T λ 2 = 0.21 with x = [ ] T λ 3 = 1.17 with x = [ ] T... The first (smallest) eigenvector does not yield clustering information. Does the second? Martijn Wieling 14/32

20 Example of spectral graph clustering (6/8) The eigenvalues λ and eigenvectors x of L n (i.e. L n x = λx): λ 1 = 0 with x = [ ] T λ 2 = 0.21 with x = [ ] T λ 3 = 1.17 with x = [ ] T... The first (smallest) eigenvector does not yield clustering information. Does the second? Yes! Martijn Wieling 14/32

21 Example of spectral graph clustering (7/8) If we use the k-means algorithm (i.e. minimize the within-cluster sum of squares; Lloyd, 1982) to cluster the eigenvector in two groups we obtain the following partitioning: To cluster in k > 2 groups we use the second to k (smallest) eigenvectors Martijn Wieling 15/32

22 Example of spectral graph clustering (8/8) To cluster in k = 3 groups, we use: λ 2 = 0.21 with x = [ ] T λ 3 = 1.17 with x = [ ] T We obtain the following clustering: Martijn Wieling 16/32

23 Example of spectral graph clustering (8/8) To cluster in k = 3 groups, we use: λ 2 = 0.21 with x = [ ] T λ 3 = 1.17 with x = [ ] T We obtain the following clustering: Martijn Wieling 16/32

24 Bipartite graphs A bipartite graph is a graph whose vertices can be divided in two disjoint sets where every edge connects a vertex from one set to a vertex in another set. Vertices within a set are not connected. A matrix representation of a bipartite graph: [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] Appelscha Oudega Zoutkamp Kerkrade Appelscha Martijn Wieling 17/32

25 Example of co-clustering a biparte graph (1/6) The (naive) co-clustering procedure is equal to clustering in one dimension (i.e. cluster eigenvector(s) of normalized Laplacian) Consider the following graph: Martijn Wieling 18/32

26 Example of co-clustering a biparte graph (2/6) The adjacency matrix A: appelscha oudega zoutkamp kerkrade vaals [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] appelscha oudega zoutkamp kerkrade appelscha [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] Martijn Wieling 19/32

27 Example of co-clustering a biparte graph (3/6) The Laplacian matrix L: appelscha oudega zoutkamp kerkrade vaals [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] appelscha oudega zoutkamp kerkrade appelscha [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] Martijn Wieling 20/32

28 Example of co-clustering a biparte graph (4/6) The normalized Laplacian matrix L n : appelscha oudega zoutkamp kerkrade vaals [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] appelscha oudega zoutkamp kerkrade appelscha [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] Martijn Wieling 21/32

29 Example of co-clustering a biparte graph (5/6) To cluster in k = 2 groups, we use: λ 2 =.057, x = [ ] T Martijn Wieling 22/32

30 Example of co-clustering a biparte graph (5/6) To cluster in k = 2 groups, we use: λ2 =.057, x = [ ]T We obtain the following co-clustering: Martijn Wieling 22/32

31 Example of co-clustering a biparte graph (6/6) To cluster in k = 3 groups, we use: λ 2 =.057, x = [ ] T λ 3 =.53, x = [ ] T Martijn Wieling 23/32

32 Example of co-clustering a biparte graph (6/6) To cluster in k = 3 groups, we use: λ2 =.057, x = [ ]T λ3 =.53, x = [ ]T We obtain the following co-clustering: ( 0. 34,0. 25) ( 0. 32,0. 12) ( 0. 34,0. 25) ( 0. 32,0. 12) ( 0. 23,0, 33) ( 0,0. 7) ( 0. 23,0. 33) ( 0. 32,0. 12) ( 0. 34,0. 25) ( 0. 32,0. 12) ( 0. 34,0. 25) Martijn Wieling 23/32

33 A faster method The previous method is relatively slow due to the use of the large (sparse) matrix A of size (n + m) (n + m) The matrix A contains the same information, but is more dense (size: n m): [a]/[i] [2]/[i] [r]/[x] [k]/[x] [r]/[ö] [r]/[k] Appelscha Oudega Zoutkamp Kerkrade Appelscha The singular value decomposition (SVD) of A n also results in equivalent clustering information and is quicker to compute (Dhillon, 2001) Martijn Wieling 24/32

34 Complete dataset Alignments of pronunciations of 562 words for 424 varieties in the Netherlands against a reference pronunciation Pronunciations originate from the GTRP (Goeman and Taeldeman, 1996; Van den Berg, 2003; Wieling et al., 2007) The pronunciations of Delft were used as the reference Alignments were obtained using the Levenshtein algorithm using a Pointwise Mutual Information procedure (Wieling et al., 2009) We generated a bipartite graph of varieties v and sound correspondences s There is an edge between v i and s j iff freq(s j in v i ) > 0 All the following co-clustering results are obtained applying the fast SVD method Martijn Wieling 25/32

35 Distribution of sites Martijn Wieling 26/32

36 Results: {2,3,4} co-clusters in the data Martijn Wieling 27/32

37 Results: {2,3,4} clusters of varieties Martijn Wieling 28/32

38 Results: {2,3,4} clusters of sound correspondences Sound correspondences specific for the Frisian area Reference [2] [2] [a] [o] [u] [x] [x] [r] Frisian [I] [i] [i] [E] [E] [j] [z] [x] Sound correspondences specific for the Limburg area Reference [r] [r] [k] [n] [n] [w] Limburg [ö] [K] [x] [ö] [K] [f] Sound correspondences specific for the Low Saxon area Reference [-] [a] Low Saxon [m] [N] [ð] [P] [e] Martijn Wieling 29/32

39 To conclude Bipartite spectral graph partitioning is a very useful method to simultaneously cluster varieties and sound correspondences words and documents genes and conditions... and... Do you now understand the title? Bipartite Spectral graph partitioning to co-cluster varieties and sound correspondences Martijn Wieling 30/32

40 To conclude Bipartite spectral graph partitioning is a very useful method to simultaneously cluster varieties and sound correspondences words and documents genes and conditions... and... Do you now understand the title? Bipartite Spectral graph partitioning to co-cluster varieties and sound correspondences Martijn Wieling 30/32

41 Any questions? Thank You! Martijn Wieling 31/32

42 References Boudewijn van den Berg Phonology & Morphology of Dutch & Frisian Dialects in 1.1 million transcriptions. Goeman-Taeldeman-Van Reenen project , Meertens Instituut Electronic Publications in Linguistics 3. Meertens Instituut (CD-ROM), Amsterdam. Sergey Brin, and Lawrence Page The anatomy of a large-scale hypertextual Web search engine. Computers Networks and ISDN Systems, 30(1 7): Fan Chung Spectral Graph Theory. American Mathematical Society. Inderjit Dhillon Co-clustering documents and words using bipartite spectral graph partitioning. Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pp ACM New York. Richard Duda, Peter Hart, and David Stork Pattern Classification. Wiley New York. Ton Goeman, and Johan Taeldeman Fonologie en morfologie van de Nederlandse dialecten. Een nieuwe materiaalverzameling en twee nieuwe atlasprojecten. Taal en Tongval, 48: Wilbert Heeringa Measuring Dialect Pronunciation Differences using Levenshtein Distance. Ph.D. thesis, Rijksuniversiteit Groningen. Stuart Lloyd Least squares quantization in PCM. IEEE Transactions on Information Theory, 28(2): Ulrike von Luxburg A tutorial on spectral clustering. Statistics and Computing, 17(4): Jianbo Shi, and Jitendra Malik Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8): Martijn Wieling, Wilbert Heeringa, and John Nerbonne An aggregate analysis of pronunciation in the Goeman-Taeldeman-Van Reenen-Project data. Taal en Tongval, 59(1): Martijn Wieling, Jelena Prokić, and John Nerbonne Evaluating the pairwise string alignment of pronunciations. In: Lars Borin and Piroska Lendvai (eds.) Language Technology and Resources for Cultural Heritage, Social Sciences, Humanities, and Education (LaTeCH - SHELT&R 2009) Workshop at the 12th Meeting of the European Chapter of the Association for Computational Linguistics. Athens, 30 March 2009, pp Martijn Wieling 32/32