Generalizing Convolutional Neural Networks to Graphstructured. Ben Walker Department of Mathematics, UNC-Chapel Hill 5/4/2018

Transcription

1 Generalizing Convolutional Neural Networks to Graphstructured Data Ben Walker Department of Mathematics, UNC-Chapel Hill 5/4/2018

2 Overview Relational structure in data and how to approach it Defferrard, Bresson, Vandergheynst 2016: Fast spectral filter method Kipf, Welling 2017: A first-order simplification for improved performance Discussion

3 Unstructured Data

4 Unstructured Data Name Alice Bob Age Gender F M Smokes? N Y Gender M F Smokes? Y N Age Name Bob Alice

5 Unstructured Data Name Alice Bob Age Gender F M Smokes? N Y Gender M F Smokes? Y N Age Name Bob Alice The order is irrelevant to processing - there is no prescribed relationship between the variables

6 Unstructured Data Name Alice Bob Age Gender F M Smokes? N Y Gender M F Smokes? Y N Age Name Bob Alice The order is irrelevant to processing - there is no prescribed relationship between the variables Use a fully-connected network to learn the relationships

7 Grid-structured Data

8 Grid-structured Data A kitten

9 Grid-structured Data A kitten Google Vision Results

10 Grid-structured Data A kitten Google Vision Results Same Kitten, Different Order

11 Grid-structured Data A kitten Google Vision Results Same Kitten, Different Order Reordered kitten picture is unintelligible Use a convolutional neural network to reduce parameters

12 Graph-structured Data Graph Convolutional Network, (Kipf and Welling 2017) There is some relationship between data, which is given on an input-specific basis, not known a priori What can you use here?

13 Defferrard et al 2016

14 Defferrard et al 2016 Spectral method allows for robust application to the neighborhood of a node.

15 Defferrard et al 2016 Spectral method allows for robust application to the neighborhood of a node. L = D W

16 Defferrard et al 2016 Spectral method allows for robust application to the neighborhood of a node. L = D W y = K 1 X k=0 k L k x

17 Defferrard et al 2016 Spectral method allows for robust application to the neighborhood of a node. L = D W y = L = 2 max L I n y = K 1 X k=0 K 1 X k=0 k L k x k T k ( L)x

18 Defferrard et al 2016 Spectral method allows for robust application to the neighborhood of a node. L = D W y = L = 2 max L I n y = K 1 X k=0 K 1 X k=0 k L k x k T k ( L)x This filtering that maps x to y is the equivalent of the convolution step in a standard convolutional network - K parameters to learn.

19 Defferrard et al 2016 y = K 1 X k=0 k T k ( L)x

20 Defferrard et al 2016 y = K 1 X k=0 k T k ( L)x Localized - k th term in sum includes contribution up to k hops from the node

21 Defferrard et al 2016 y = K 1 X k=0 k T k ( L)x Localized - k th term in sum includes contribution up to k hops from the node Recursive definition, allowing for efficient computation T k+1 ( L)x =2 LT k ( L)x T k 1 ( L)x

22 Defferrard et al 2016 y = K 1 X k=0 k T k ( L)x Localized - k th term in sum includes contribution up to k hops from the node Recursive definition, allowing for efficient computation T k+1 ( L)x =2 LT k ( L)x T k 1 ( L)x This filter is something we can apply machine learning techniques to

23 Validation Chebyshev filter Graph CNN tested on MNIST Graph created to represent grid structure Comparable performance to classical CNN Also validated on 20NEWS text categorization dataset.

24 Kipf, Welling 2017

25 Kipf, Welling 2017 Aim to improve the approach from Defferrard Linearize the previous filter equation y = 0 0x 0 1D 1 2 AD 1 2 x

26 Kipf, Welling 2017 Aim to improve the approach from Defferrard Linearize the previous filter equation y = 0 0x 0 1D 1 2 AD 1 2 x Simplify and renormalize for improved numerical stability, and generalize to multiple feature maps to get an equation Z = D 1 2 Ã D 1 2 X

27 Kipf, Welling 2017 Aim to improve the approach from Defferrard Linearize the previous filter equation y = 0 0x 0 1D 1 2 AD 1 2 x Simplify and renormalize for improved numerical stability, and generalize to multiple feature maps to get an equation Z = D 1 2 Ã D 1 2 X X k+1 = (MX k k )

28 Validation Validation Datasets Citeseer, Cora, and Pubmed citation networks NELL knowledge graph Comparison of classification accuracy percentage of different methods. (Kipf Welling 2017)

29 Discussion

30 Discussion Graph-structured data is an interesting new frontier for machine-learning methods

31 Discussion Graph-structured data is an interesting new frontier for machine-learning methods Kipf and Welling GCN is very similar to standard neural network formulations X k+1 = (MX k k )

32 Discussion Graph-structured data is an interesting new frontier for machine-learning methods Kipf and Welling GCN is very similar to standard neural network formulations X k+1 = (MX k k ) By nature of linearization, it is localized at a distance of 1.

33 References Defferrard, Michaël, Xavier Bresson, and Pierre Vandergheynst. "Convolutional neural networks on graphs with fast localized spectral filtering." Advances in Neural Information Processing Systems Kipf, Thomas N., and Max Welling. "Semi-supervised classification with graph convolutional networks." arxiv preprint arxiv: (2016).