Machine Learning with Tensor Networks

Transcription

1 Machine Learning with Tensor Networks E.M. Stoudenmire and David J. Schwab Advances in Neural Information Processing 29 arxiv: Beijing Jun 2017

2 Machine learning has physics in its DNA # " # # " " # " Boltzmann Machines Disordered Ising Model Deep Belief Networks The "Renormalization Group" P. Mehta and D.J. Schwab, arxiv:

3 Convolutional neural network "MERA" tensor network

4 Impact of tensor networks in physics Critical Phenomena Quantum Gravity High Precision Quantum Chemistry

5 Are tensor networks useful for machine learning? This Talk Tensor networks can compress weights of powerful machine learning models Benefits include Linear scaling Adaptive optimization Feature sharing

6 Prior tensor networks + machine learning Markov random field models Novikov et al., Proceedings of 31st ICML (2014) Large scale PCA Lee, Cichocki, arxiv: (2014) Feature extraction of tensor data Bengua et al., IEEE Congress on Big Data (2015) Compressing weights of neural nets Novikov et al., Advances in Neural Information Processing (2015)

7 What are Tensor Networks?

8 Original setting is quantum mechanics Spin model (Transverse field Ising model): z z h x Ĥ = X j ( z j z j+1 h x j ) " " " " " " " h<<1 " " " " " " "h>>1

9 Wavefunction just a rule to map spin configurations to numbers s 1 s 2 s 3 s 4 s 5 s 6 s 7 s 8 " # " " " " " " " # " " " " " " " # " # " " " " " " " " # " # " " # " # " " " " " " " " # " # " " # # # # " " " " # # # # " " " Simplest rule: store every amplitude separately

10 [ [ Let's make a different rule Introduce matrices, one for each spin " M " = [ # M # = [

11 Compute amplitude by multiplying matrices together (with boundary vectors v L and v R ) " # " " # v L

12 Compute amplitude by multiplying matrices together (with boundary vectors v L and v R ) " # " " # v M " L

13 Compute amplitude by multiplying matrices together (with boundary vectors v L and v R ) " # " " # v M " M # L

14 Compute amplitude by multiplying matrices together (with boundary vectors v L and v R ) " # " " # v M " M # M " L

15 Compute amplitude by multiplying matrices together (with boundary vectors v L and v R ) " # " " # v M " M # M " M " L

16 Compute amplitude by multiplying matrices together (with boundary vectors v L and v R ) " # " " # v M " M # M " M " M # L

17 Compute amplitude by multiplying matrices together (with boundary vectors v L and v R ) " # " " # M " M # M " M " M # v L v R

18 Compute amplitude by multiplying matrices together (with boundary vectors v L and v R ) " # " " # M " M # M " M " M # v L v R " " # # # v L M " M " M # M # M # v R " # # " " v L M " M # M # M " M " v R

19 This rule is called a matrix product state (MPS) s 1 s 2 s 3 s 4 = v L M s 1 M s 2 M s 3 M s 4 v R Matrices can vary from site to site Size of matrices called m (the "bond dimension") For m = 2 N/2 can represent any state of N spins Really just a way of compressing a big tensor

20 What have we gained? By representing a wavefunction as an MPS with small matrices (small bond dimension m) Then we've represented 2 N amplitudes using only (2 N m 2 ) parameters Efficient to compute properties of an MPS, or to optimize an MPS (DMRG algorithm)

21 MPS = matrix product state MPS come with powerful optimization techniques (DMRG algorithm) White, PRL 69, 2863 (1992) Stoudenmire, White, PRB 87, (2013)

22 Tensor Diagrams (Briefly)

23 Helpful to draw N-index tensor as blob with N lines s 1 s 2 s 3 s 4 s N s 1 s 2 s 3 s N =

24 Diagrams for simple tensors j v j i j M ij i j k T ijk

25 Joining lines implies contraction, can omit names X i j j M ij v j A ij B jk = AB A ij B ji =Tr[AB]

26 Matrix product state in diagram notation s 1 s 2 s 3 s 4 s 5 s 6 = X M s 1 1 M s M s M s M s M s 6 5 s 1 s 2 s 3 s 4 s 5 s 6 = Can suppress index names, very convenient

27 Matrix product state in diagram notation s 1 s 2 s 3 s 4 s 5 s 6 = X M s 1 1 M s M s M s M s M s 6 5 = Can suppress index names, very convenient

28 Besides MPS, other successful tensor are PEPS and MERA PEPS (2D systems) MERA (critical systems) Evenbly, Vidal, PRB 79, (2009) Verstraete, Cirac, cond-mat/ (2004) Orus, Ann. Phys. 349, 117 (2014)

29 Learning with Tensor Networks

30 Proposal: 1. Lift data to exponentially higher space (feature space = Hilbert space) 2. Apply linear classifier in feature space 3. Compress weights using a tensor network Following slides use feature map of Novikov et al. Novikov, Trofimov, Oseledets, "Exponential Machines", arxiv: Stoudenmire, Schwab, "Supervised Learning with Tensor Networks", arxiv:

31 Original / raw data vectors x =(x 1,x 2,x 3,...,x N ) Example of grayscale images, components of x are pixels x j 2 [0, 1]

32 1. Lift data to exponentially higher space (feature space = Hilbert space) x =(x 1,x 2,x 3,...,x N ) lift (x) =(1,x 1,x 2,x 3,... x 1 x 2, x 1 x 3, x 2 x 3,... x 1 x 2 x 3, x 1 x 2 x 4, x 1 x 2 x 3 x N ) singles pairs triples... N-tuple

33 2. Apply linear classifier in feature space f(x) =W (x) = X s W s1 s 2 s 3 s N x s 1 1 xs 2 2 xs 3 3 xs N N s j =0, 1 Weights are an N-index tensor Just like an N-site wavefunction

34 N=3 example: f(x) =W (x) = X s W s1 s 2 s x s xs 2 2 xs 3 3 = W W 100 x 1 + W 010 x 2 + W 001 x 3 + W 110 x 1 x 2 + W 101 x 1 x 3 + W 011 x 2 x 3 + W 111 x 1 x 2 x 3 Contains linear classifier, and various poly. kernels

35 3. Compress weights as a tensor network W s1 s 2 s 3 s N M s1 M s2 M s3 M sn Could also use MERA or PEPS instead of MPS

36 [ Tensor diagrams of the approach s 1 s 2 s 3 s 4 s 5 s 6 s N x (x) = s j = [ 1 x j

37 [ [ Tensor diagrams of the approach s 1 s 2 s 3 s 4 s 5 s 6 s N x (x) = s j = [ 1 x j Other choices include: [ 1 x j [ or x 2 j [ cos 2 x j sin 2 x j

38 Tensor diagrams of the approach f(x) = W (x) = W (x) (M s1 M s2 M sn ) s 1s 2 s N (x)

39 Linear scaling Can use algorithm similar to DMRG to optimize Scaling is N N T m 3 N = size of input N T = size of training set m = MPS bond dimension f(x) = W (x)

40 Linear scaling Can use algorithm similar to DMRG to optimize Scaling is N N T m 3 N = size of input N T = size of training set m = MPS bond dimension f(x) = W (x)

41 Linear scaling Can use algorithm similar to DMRG to optimize Scaling is N N T m 3 N = size of input N T = size of training set m = MPS bond dimension f(x) = W (x)

42 Linear scaling Can use algorithm similar to DMRG to optimize Scaling is N N T m 3 N = size of input N T = size of training set m = MPS bond dimension f(x) = W (x)

43 Linear scaling Can use algorithm similar to DMRG to optimize Scaling is N N T m 3 N = size of input N T = size of training set m = MPS bond dimension f(x) = W (x)

44 Linear scaling Can use algorithm similar to DMRG to optimize Scaling is N N T m 3 N = size of input N T = size of training set m = MPS bond dimension f(x) = W (x) Could improve with stochastic gradient

45 Linear scaling Model similar to kernel learning Can train without "kernel trick", avoiding its N 2 T scaling problem

46 Does the weight tensor obey an "area law"? More entangled than a ground-state wavefunction? Less entangled? Do an experiment to find out...

47 MNIST Experiment MNIST is a benchmark data set of grayscale handwritten digits (labels ` = 0,1,2,...,9) 60,000 labeled training images 10,000 labeled test images

48 MNIST Experiment Details: Shrink images from 28x28 14x14 Trained 10 models * to distinguish each digit, largest output is prediction Minimize quadratic cost function C = 1 N T N T X n=1 (W ` (x n ) yǹ) 2 + W 2

49 MNIST Experiment One-dimensional mapping

50 MNIST Experiment Results: Bond dimension m = 10 m = 20 m = 120 Test Set Error ~5% (500/10,000 incorrect) ~2% (200/10,000 incorrect) 0.97% (97/10,000 incorrect)

51 MNIST Experiment Demo

52 MNIST in friendly neighborhood of Hilbert space (feature space) W MNIST MPS

53 Situation for other data sets? Optimistic = typical W machine learning Pessimistic = tensor networks

54 Benefits of Tensor Network Models f(x) = W (x)

55 f(x) = W (x) Many interesting benefits of using tensor network weights. Two benefits: 1. Adaptive training 2. Feature sharing

56 { 1. Tensor networks are adaptive boundary pixels not useful for learning grayscale training data

57 2. Feature sharing Multi-class decision function f `(x) =W ` (x) Index ` runs over possible labels Predicted label is argmax` f `(x) ` f `(x) = W ` (x)

58 2. Feature sharing ` f `(x) = W ` (x) ` = Different central tensors "Wings" shared between models Regularizes models

59 2. Feature sharing ` f `(x) = Progressively learn shared features

60 2. Feature sharing ` f `(x) = Progressively learn shared features

61 2. Feature sharing ` f `(x) = Progressively learn shared features

62 2. Feature sharing ` f `(x) = Progressively learn shared features Deliver to central tensor `

63 Implications for quantum computing? Weights formally inhabit same space as quantum Hilbert space (space of wavefunctions) Negative Outlook! Weights have low entanglement, quantum computer not needed Positive Outlook " Quantum computer could train extremely expressive models Tensor networks equivalent to finite-depth quantum circuits...

64 Connections to Other Approaches Graphical models: like tensor networks but with positive weights (Rolfe, "Multifactor Expectation Maximization...") Weighted Finite Automata: like translation invariant MPS, trained with spectral method (Balle, "Spectral Learning..." Mach Learn (2014) 96:33-63) Neural Networks: "ConvAC" neural networks with linear activation and product pooling equivalent to tensor networks (Levine, "Deep Learning and Quantum Entanglement..." arxiv: )

65