Some Statistical Properties of Deep Networks

Transcription

1 Some Statistical Properties of Deep Networks Peter Bartlett UC Berkeley August 2, / 22

2 Deep Networks Deep compositions of nonlinear functions h = h m h m 1 h 1 2 / 22

3 Deep Networks Deep compositions of nonlinear functions h = h m h m 1 h 1 e.g., h i : x σ(w i x) 1 σ(v) i = 1 + exp( v i ), 2 / 22

4 Deep Networks Deep compositions of nonlinear functions h = h m h m 1 h 1 e.g., h i : x σ(w i x) 1 σ(v) i = 1 + exp( v i ), 2 / 22

5 Deep Networks Deep compositions of nonlinear functions h = h m h m 1 h 1 e.g., h i : x σ(w i x) h i : x r(w i x) 1 σ(v) i = 1 + exp( v i ), r(v) i = max{0, v i } 2 / 22

6 Deep Networks Deep compositions of nonlinear functions h = h m h m 1 h 1 e.g., h i : x σ(w i x) h i : x r(w i x) 1 σ(v) i = 1 + exp( v i ), r(v) i = max{0, v i } 2 / 22

7 Deep Networks Representation learning Rich non-parametric family 3 / 22

8 Deep Networks Representation learning Depth provides an effective way of representing useful features. Rich non-parametric family 3 / 22

9 Deep Networks Representation learning Depth provides an effective way of representing useful features. Rich non-parametric family Depth provides parsimonious representions. Nonlinear parameterizations provide better rates of approximation. (Birman & Solomjak, 1967), (DeVore et al, 1991) 3 / 22

10 Deep Networks Representation learning Depth provides an effective way of representing useful features. Rich non-parametric family Depth provides parsimonious representions. Nonlinear parameterizations provide better rates of approximation. (Birman & Solomjak, 1967), (DeVore et al, 1991) Some functions require much more complexity for a shallow representation. (Telgarsky, 2015), (Eldan & Shamir, 2015) 3 / 22

11 Deep Networks Representation learning Depth provides an effective way of representing useful features. Rich non-parametric family Depth provides parsimonious representions. Nonlinear parameterizations provide better rates of approximation. (Birman & Solomjak, 1967), (DeVore et al, 1991) Some functions require much more complexity for a shallow representation. (Telgarsky, 2015), (Eldan & Shamir, 2015) Statistical complexity? 3 / 22

12 Outline VC theory: Number of parameters Margins analysis: Size of parameters Understanding generalization failures 4 / 22

13 VC Theory 5 / 22

14 VC Theory Assume network maps to { 1, 1}. (Threshold its output) 5 / 22

15 VC Theory Assume network maps to { 1, 1}. (Threshold its output) Data generated by a probability distribution P on X { 1, 1}. 5 / 22

16 VC Theory Assume network maps to { 1, 1}. (Threshold its output) Data generated by a probability distribution P on X { 1, 1}. Want to choose a function f such that P(f (x) y) is small (near optimal). 5 / 22

17 VC Theory Theorem (Vapnik and Chervonenkis) Suppose F { 1, 1} X. For every prob distribution P on X { 1, 1}, with probability 1 δ over n iid examples (x 1, y 1 ),..., (x n, y n ), every f in F satisfies P(f (x) y) 1 n {i : f (x i) y i } + ( c n (VCdim(F) + log(1/δ)) ) 1/2. 6 / 22

18 VC Theory Theorem (Vapnik and Chervonenkis) Suppose F { 1, 1} X. For every prob distribution P on X { 1, 1}, with probability 1 δ over n iid examples (x 1, y 1 ),..., (x n, y n ), every f in F satisfies P(f (x) y) 1 n {i : f (x i) y i } + ( c n (VCdim(F) + log(1/δ)) ) 1/2. For uniform bounds (that is, for all distributions and all f F, proportions are close to probabilities), this inequality is tight within a constant factor. 6 / 22

19 VC Theory Theorem (Vapnik and Chervonenkis) Suppose F { 1, 1} X. For every prob distribution P on X { 1, 1}, with probability 1 δ over n iid examples (x 1, y 1 ),..., (x n, y n ), every f in F satisfies P(f (x) y) 1 n {i : f (x i) y i } + ( c n (VCdim(F) + log(1/δ)) ) 1/2. For uniform bounds (that is, for all distributions and all f F, proportions are close to probabilities), this inequality is tight within a constant factor. For neural networks, VC-dimension: 6 / 22

20 VC Theory Theorem (Vapnik and Chervonenkis) Suppose F { 1, 1} X. For every prob distribution P on X { 1, 1}, with probability 1 δ over n iid examples (x 1, y 1 ),..., (x n, y n ), every f in F satisfies P(f (x) y) 1 n {i : f (x i) y i } + ( c n (VCdim(F) + log(1/δ)) ) 1/2. For uniform bounds (that is, for all distributions and all f F, proportions are close to probabilities), this inequality is tight within a constant factor. For neural networks, VC-dimension: increases with number of parameters 6 / 22

21 VC Theory Theorem (Vapnik and Chervonenkis) Suppose F { 1, 1} X. For every prob distribution P on X { 1, 1}, with probability 1 δ over n iid examples (x 1, y 1 ),..., (x n, y n ), every f in F satisfies P(f (x) y) 1 n {i : f (x i) y i } + ( c n (VCdim(F) + log(1/δ)) ) 1/2. For uniform bounds (that is, for all distributions and all f F, proportions are close to probabilities), this inequality is tight within a constant factor. For neural networks, VC-dimension: increases with number of parameters depends on nonlinearity and depth 6 / 22

22 VC-Dimension of Neural Networks Theorem Consider the class F of { 1, 1}-valued functions computed by a network with L layers, p parameters, and k computation units with the following nonlinearities: 7 / 22

23 VC-Dimension of Neural Networks Theorem Consider the class F of { 1, 1}-valued functions computed by a network with L layers, p parameters, and k computation units with the following nonlinearities: 1 Piecewise constant (linear threshold units): VCdim(F) = Θ (p). (Baum and Haussler, 1989) 7 / 22

24 VC-Dimension of Neural Networks Theorem Consider the class F of { 1, 1}-valued functions computed by a network with L layers, p parameters, and k computation units with the following nonlinearities: 1 Piecewise constant (linear threshold units): VCdim(F) = Θ (p). (Baum and Haussler, 1989) 2 Piecewise linear (ReLUs): VCdim(F) = Θ (pl). (B., Harvey, Liaw, Mehrabian, 2017) 7 / 22

25 VC-Dimension of Neural Networks Theorem Consider the class F of { 1, 1}-valued functions computed by a network with L layers, p parameters, and k computation units with the following nonlinearities: 1 Piecewise constant (linear threshold units): VCdim(F) = Θ (p). (Baum and Haussler, 1989) 2 Piecewise linear (ReLUs): VCdim(F) = Θ (pl). (B., Harvey, Liaw, Mehrabian, 2017) 3 Piecewise polynomial: VCdim(F) = Õ ( pl 2). (B., Maiorov, Meir, 1998) 7 / 22

26 VC-Dimension of Neural Networks Theorem Consider the class F of { 1, 1}-valued functions computed by a network with L layers, p parameters, and k computation units with the following nonlinearities: 1 Piecewise constant (linear threshold units): VCdim(F) = Θ (p). (Baum and Haussler, 1989) 2 Piecewise linear (ReLUs): VCdim(F) = Θ (pl). (B., Harvey, Liaw, Mehrabian, 2017) 3 Piecewise polynomial: VCdim(F) = Õ ( pl 2). (B., Maiorov, Meir, 1998) 4 Sigmoid: VCdim(F) = Õ ( p 2 k 2). (Karpinsky and MacIntyre, 1994) 7 / 22

27 Outline VC theory: Number of parameters Margins analysis: Size of parameters Understanding generalization failures 8 / 22

28 Generalization in Deep Networks Spectrally-normalized margin bounds for neural networks. B., Dylan J. Foster, Matus Telgarsky, NIPS arxiv: Dylan Foster Cornell Matus Telgarsky UIUC 9 / 22

29 Large-Margin Classifiers Consider a vector-valued function f : X R m used for classification, y {1,..., m}. 10 / 22

30 Large-Margin Classifiers Consider a vector-valued function f : X R m used for classification, y {1,..., m}. The prediction on x X is arg max y f (x) y. 10 / 22

31 Large-Margin Classifiers Consider a vector-valued function f : X R m used for classification, y {1,..., m}. The prediction on x X is arg max y f (x) y. For a pattern-label pair (x, y) X {1,..., m}, define the margin M(f (x), y) = f (x) y max i y f (x) i. 10 / 22

32 Large-Margin Classifiers Consider a vector-valued function f : X R m used for classification, y {1,..., m}. The prediction on x X is arg max y f (x) y. For a pattern-label pair (x, y) X {1,..., m}, define the margin M(f (x), y) = f (x) y max i y f (x) i. If M(f (x), y) > 0 then f classifies x correctly. 10 / 22

33 Large-Margin Classifiers Consider a vector-valued function f : X R m used for classification, y {1,..., m}. The prediction on x X is arg max y f (x) y. For a pattern-label pair (x, y) X {1,..., m}, define the margin M(f (x), y) = f (x) y max i y f (x) i. If M(f (x), y) > 0 then f classifies x correctly. We can view a larger margin as a more confident correct classification. 10 / 22

34 Large-Margin Classifiers Consider a vector-valued function f : X R m used for classification, y {1,..., m}. The prediction on x X is arg max y f (x) y. For a pattern-label pair (x, y) X {1,..., m}, define the margin M(f (x), y) = f (x) y max i y f (x) i. If M(f (x), y) > 0 then f classifies x correctly. We can view a larger margin as a more confident correct classification. Minimizing a continuous loss, such as encourages large margins. n f (X i ) Y i 2, i=1 10 / 22

35 Large-Margin Classifiers Consider a vector-valued function f : X R m used for classification, y {1,..., m}. The prediction on x X is arg max y f (x) y. For a pattern-label pair (x, y) X {1,..., m}, define the margin M(f (x), y) = f (x) y max i y f (x) i. If M(f (x), y) > 0 then f classifies x correctly. We can view a larger margin as a more confident correct classification. Minimizing a continuous loss, such as encourages large margins. n f (X i ) Y i 2, i=1 For large-margin classifiers, we should expect the fine-grained details of f to be less important. 10 / 22

36 Generalization in Deep Networks New results for generalization in deep ReLU networks Measure the size of functions computed by a network of ReLUs via operator norms. 11 / 22

37 Generalization in Deep Networks New results for generalization in deep ReLU networks Measure the size of functions computed by a network of ReLUs via operator norms. Large multiclass versus binary classification. 11 / 22

38 Generalization in Deep Networks New results for generalization in deep ReLU networks Measure the size of functions computed by a network of ReLUs via operator norms. Large multiclass versus binary classification. Definitions Consider operator norms: For a matrix A i, A i := sup x 1 A i x. 11 / 22

39 Generalization in Deep Networks New results for generalization in deep ReLU networks Measure the size of functions computed by a network of ReLUs via operator norms. Large multiclass versus binary classification. Definitions Consider operator norms: For a matrix A i, A i := sup x 1 A i x. Recall: Multiclass margin function for f : X R m, y {1,..., m}, is M(f (x), y) = f (x) y max i y f (x) i. 11 / 22

40 Generalization in Deep Networks Theorem With high probability, every f A 12 / 22

41 Generalization in Deep Networks Theorem With high probability, every f A Definitions Network with L layers, parameters A 1,..., A L : f A (x) := σ L (A L σ L 1 (A L 1 σ 1 (A 1 x) )). 12 / 22

42 Generalization in Deep Networks Theorem With high probability, every f A satisfies Pr(M(f A (X ), Y ) 0) Definitions Network with L layers, parameters A 1,..., A L : f A (x) := σ L (A L σ L 1 (A L 1 σ 1 (A 1 x) )). 12 / 22

43 Generalization in Deep Networks Theorem With high probability, every f A Pr(M(f A (X ), Y ) 0) 1 n satisfies n 1[M(f A (X i ), Y i ) γ] i=1 Definitions Network with L layers, parameters A 1,..., A L : f A (x) := σ L (A L σ L 1 (A L 1 σ 1 (A 1 x) )). 12 / 22

44 Generalization in Deep Networks Theorem With high probability, every f A Pr(M(f A (X ), Y ) 0) 1 n satisfies n 1[M(f A (X i ), Y i ) γ] + Õ i=1 ( ) rl γ. n Definitions Network with L layers, parameters A 1,..., A L : f A (x) := σ L (A L σ L 1 (A L 1 σ 1 (A 1 x) )). 12 / 22

45 Generalization in Deep Networks Theorem With high probability, every f A with R A r Pr(M(f A (X ), Y ) 0) 1 n satisfies n 1[M(f A (X i ), Y i ) γ] + Õ i=1 ( ) rl γ. n Definitions Network with L layers, parameters A 1,..., A L : f A (x) := σ L (A L σ L 1 (A L 1 σ 1 (A 1 x) )). Scale of f A : R A := L i=1 A i. 12 / 22

46 Generalization in Deep Networks Theorem With high probability, every f A with R A r Pr(M(f A (X ), Y ) 0) 1 n satisfies n 1[M(f A (X i ), Y i ) γ] + Õ i=1 ( ) rl γ. n Definitions Network with L layers, parameters A 1,..., A L : f A (x) := σ L (A L σ L 1 (A L 1 σ 1 (A 1 x) )). Scale of f A : R A := L i=1 A i (Assume σ i is 1-Lipschitz, inputs normalized.) L i=1 A i 2/3 2,1 A i 2/3 3/2. 12 / 22

47 Outline VC theory: Number of parameters Margins analysis: Size of parameters Understanding generalization failures 13 / 22

48 Understanding Generalization Failures CIFAR / 22

49 Understanding Generalization Failures Stochastic Gradient Training Error on CIFAR10 (Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals, 2017) 15 / 22

50 Understanding Generalization Failures Training margins on CIFAR10 with true and random labels How does this match the large margin explanation? 16 / 22

51 Understanding Generalization Failures If we rescale the margins by R A (the scale parameter): Rescaled margins on CIFAR10 17 / 22

52 Understanding Generalization Failures If we rescale the margins by R A (the scale parameter): Rescaled cumulative margins on MNIST 18 / 22

53 Generalization in Deep Networks Theorem With high probability, every f A with R A r satisfies Pr(M(f A (X ), Y ) 0) 1 n n 1[M(f A (X i ), Y i ) γ] + Õ i=1 ( ) rl γ. n Network with L layers, parameters A 1,..., A L : f A (x) := σ(a L σ L 1 (A L 1 σ 1 (A 1 x) )). Scale of f A : R A := L i=1 A i L i=1 A i 2/3 2,1 A i 2/3 3/2. 19 / 22

54 Understanding Generalization Failures cifar Lipschitz cifar [random] Lipschitz epoch 10 epoch / 22

55 Understanding Generalization Failures excess risk 0.9 cifar excess risk cifar Lipschitz cifar [random] excess risk cifar [random] Lipschitz excess risk 0.3 epoch 10 epoch / 22

56 Generalization in Neural Networks With appropriate normalization, the margins analysis is qualitatively consistent with the generalization performance. 22 / 22

57 Generalization in Neural Networks With appropriate normalization, the margins analysis is qualitatively consistent with the generalization performance. Recent work by Golowich, Rakhlin, and Shamir give bounds with improved dependence on depth. 22 / 22

58 Generalization in Neural Networks With appropriate normalization, the margins analysis is qualitatively consistent with the generalization performance. Recent work by Golowich, Rakhlin, and Shamir give bounds with improved dependence on depth. Regularization and optimization: explicit control of operator norms? 22 / 22