Principles of Riemannian Geometry in Neural Networks

Transcription

1 Principles of Riemannian Geometry in Neural Networks Michael Hauser, Asok Ray Pennsylvania State University Presented by Chenyang Tao Nov 16, 2018

2 Brief Summary Goal This study deals with neural networks in the sense of geometric transformations acting on the coordinate representation of the underlying data manifold which the data is sampled from. Contributions C k neural net and ResNet Understanding Riemannian metric tensor learned by NN Alternative Lie Group theory for the metric tensor Backpropagation as Lie Group actions

3 NIPS Reviewer s Comments Pros: R3: The paper presents a very interesting framework for working with neural network representations. Cons (that were taken out of context): R1: there need more discussions and comparisons in the derivations of the theory and in the experiments. R3: The paper is not always completely clear; experiments are a bit disappointing; does not make practical application easy. Why it got in? R2: Although my understanding of the technical arguments are limited, I believe that it could be of great interest to the NIPS community. (Heavy notations. So if you got confused, don t feel bad.)

4 Getting Prepared Notations x (l), φ (l) : l-th layer, coordinate transformation, etc. A a b : tensor index (just treat it as A ab for matrices) Some concepts/assumptions M is a topological manifold locally homeomorphic to R dim M The homeoorphism x U x(u) R dim M is called a coordinate system on U M A feedforward network is considered as coordinate transformations: φ (l) x (l) (M) (φ (l) x (l) )(M) Some activations functions like ReLU are not bijection therefore not a proper coordinate transformation

5 Continued Einstein notation Just put back the summation Einstein α i z i = i α i z i Regular Metric tensor g(x) Think of it as a localized Mahalanobis distance ds = ij g ij (x) dx i dx j

6 C k Differentiable Neural Networks ResNet as discretized dynamical systems x (l+1) = x (l) + f l (x (l) ) l δx (l) = x (l+1) x (l) = f l (x (l) ) l k th order differentiable smoothness δ 2 x (l) = x (l+1) 2x (l) + x (l 1) = f l (x (l) ) 2 Autonomous dx dl = f (x(l)), non-autonomous dx dl = f (x(l), l) C k neural networks C 0 (MLP): x (l) = f l (x l 1) ; C 1 (ResNet): x (l) = x (l 1) + f l (x (l) ) l C 2 (Momentum ResNet): x (l+1) = x (l) + f l (x (l) ) l 2 + δx (l 1) δx (l 1) = x (l) x (l 1)

7 A Riemannian manifold (M, g) is a real smooth manifold M with an inner product, defined by the psd metric tensor g, varying smoothly on the tangent space of M. The metric in the output coordinates is assumed to be Euclidean: g(x (L) ) al b L = η al b L Metric tensor transforms: g(x (l) ) al b l = ( x(l+1) x (l) g(x (l) ) al b l = L 1 l =l ) a l+1 a l x(l +1) [( x (l ) ( x(l+1) ) b l+1 g(x (l+1) ) x (l) al+1 b b l+1 l ) a l +1 a l ( x(l +1) x (l ) b ) l +1 ] η al b L b l

8 A concrete example with ResNet If the network is taken to be the residual net ( x(l+1) a l+1 x (l) ) = δ a l+1 a l + ( f l(x (l) a l+1 ) a l x (l) ) l a l P a L 1 L a l = l =l δ a a l +1 l + ( f l (z(l +1) a ) l +1 ) ( z(l +1) z l +1 e x (l ) ) l +1 where z (l+1) = W (l) x (l) + b (l) ds 2 = η ab P a a l P b b l dx a l dx b l e l +1 a l

9 Metric Tensor Visualization

10 Additional comments As L, the infinite product converges in the limit. For varying layerwsie dimension, manifold can be submersed and immersed into lower and higher dimensional spaces so long as the rank of the pushforward Jacobian is constant for every p M The work is to show the coordinate representation of the metric tensor can be backpropagated through to the input so that distance can be measured in the input coordinates

11 My remarks Using the metric induced, we can define the geodesics d (M,g0 )(x, y) to define similarities in the input space; This differs from measuring similarity in the feature space, e.g., d (M,g0 )(x, y) x (L) y (L). Yet it is not computationally feasible to compute the geodesic distance with the induced metric.

12 Lie Group Actions on the Metric Fibre Bundle Definitions (Principle fiber bundle) A bundle (E, π, M) is called a principle G-bundle if 1 E is equipped with a right G-action E G E 2 The right G-action is free (no fixed point) 3 (E, π, M) is (bundle) isomorphic to (E, ρ, E/G) where the surjective projection map ρ E E/G is defined by ρ(ɛ) = [ɛ] as the equivalent class of points of ɛ Definitions (Associated fibre bundle) Given a G-principle bundle and a smooth manifold F on which exists a left G-action G F F, the associated fibre bundle (P F, π F, M) is defined as follows: 1 Let G be the relation on P F defined as follows: (p, f ) G (p, f ) h G p = p h and f = h 1 f and thus P F = (P F)/ G Define π F = P F M by π F ([(p, f )]) = π(p)

13 Neural network actions on the manifold M are a (layerwise) sequence of left G-actions on the associated (metric space) fibre bundle. Let d = dim M The structure group G is the general linear group of dimension d over R: G = GL(d, R) = {φ R d R d det φ 0} The principle bundle P is raken to be the frame bundle: P = LM = p M L p M The right G-action LM GL(d, R) LM is defined by : e h = (e 1,, e d ) h = (h a l 1 e a l,, h a l d e a l ) The fibre F in the associated bundle is the metric tensor space F = (R d ) (R d ), where denotes the cospace, and the left G-action GL(d, R) F F is defined as the inverse of the left (??? right) (h 1 g) al b l = (g h) al b l = g al+1 b l+1 h a l+1 a l h b l+1 b l (h 1 0 h l =L 1 (ha l +1 a l h 1 L g) a 0 b 0 = (h 1 0 h 1 1 h 1 L ) g a L b L = h b l +1 b ) g l al b L

14 Backpropagation as Sequential Right Lie Group Actions Error backpropagation as a sequence of right Lie Group actions Standard backpropagation for error E wrt layer weight W (l 1) l +1) a E E l +1 = ( W (l 1) x (L) ) ( x(l ) ( x(l) a l a L l =L 1 x l a W (l 1) ) (1) l As such, error backpropagation is a sequence of right G-actions l l =L 1 ( x(l +1) ( x (L) ) al a l +1 ) x l a l on the output frame bundle

15 Only experiments on spiral disentanglement is considered C k networks Effect of batch-size Effect of network depth

16 C k Neural Networks

17 Effect of Batch-size

18 Effect of Network Depth