Convolutional networks. Sebastian Seung

Transcription

1 Convolutional networks Sebastian Seung

2 Convolutional network Neural network with spatial organization every neuron has a location usually on a grid Translation invariance synaptic strength depends on locations only through spatial separation. Locality synapses only between nearby neurons.

3 Locality and translation invariance in 1d r = f ( w s + w s + w s ) i 1 i 1 0 i 1 i+1

4 Regular vs. convolutional network stimulus s and response r in 1d r i = f # & W s % ij j ( $ j ' r i = f $ & % j ' w s i j j ) ( constrained weight matrix w has local support W ij = w i j

5 Toeplitz matrix For each diagonal, the elements are identical. #!!! & % (! w w w % ( %! w w w!( % ( w w w! % ( $!!!'

6 Convolution + nonlinearity linear filtering operation j ( w s) = w s = i i j j nonlinear transformation j w j s i j r i = f (( w s) ) i

7 Translation invariance: Operand definition does not change when the translation operator is applied Operator commutes with the translation operator.

8 Any translation invariant linear operator is a convolution.

9 Convolutional network (2d) r ij = f (( w s) ) ij kl ( w s) = w s = ij i k, j l kl kl w kl s i k, j l

10 Convolution in d dimensions Sum over multi-indices points on hypercubic lattice i = ( i,i,,i ) j = ( j, j,, j ) 1 2 d 1 2 d j w s ( ) i = w i j s j

11 Convolution + nonlinearity Can represent basic operations in image processing. Can be used to model the first stages of the visual system.

12 Noise suppression Gaussian filtering

13 Edge detection Laplacian of Gaussian

14 Convolutional network I α = f # % $ β & w αβ I β + b α ( '

15 Feature map array of detectors of the same feature multiple feature maps in single layer each neuron has a location and a feature index. I α i = f # % $ β j & i ji β j + b α ( ' w αβ

16 Conjunction neurons as feature detectors An image is composed of features. Logical conjunction can detect features. synaptic weights trigger feature

17 Divergent convolution creates multiple feature maps

18 Convergent convolution builds detectors for complex features

19

20 Depth-size tradeoff revisited Any circuit with two layers of AND gates can be collapsed to a single layer. When is the collapsed circuit larger?

21 The deeper network is more efficient when: A. Disjunctive normal form is compact. B. Conjunctive normal form is compact. C. The fan-out of the hidden layer is large. D. The fan-in of the hidden layer is large.

22 Generalize layered structure to directed acyclic graph Nodes are feature maps Edges are convolutions

23 Forward pass u = w x + b 1 x = f(u )

24 Backward pass ˆx j = X i i w i j ˆx = X w j = f 0 (u j )ˆx j

25 Backward pass Convolution is multiplication by a matrix of the form W ij = w i The matrix transpose is W ji = w j Therefore, use spatial inversion of filters i.e., flip the filter about each axis j i

26 Weight update The weights of neurons in the same feature map are tied together. How to perform gradient learning when parameters are shared?

27 Gradient sharing x(t) =t y(t) =t d dt f(x(t),y(t)) = f = x f dx dt + x + f y f y dy dt

28 Gradient learning with parameter sharing Find the derivatives with respect to each of the individual parameters, treating them as independent. Then sum all of these to find the derivative with respect to the shared parameter.

29 For Whoever Shares, to Him More Gradient Will Be Given corruption of Mark 4:25

30 Weight update for a convolution is cross-correlation W ij = i x j w k = X i,j i x j i = X j j=k j+kx j

31 Receptive field r = f ( w s + w s + w s ) i 1 i 1 0 i 1 i+1 ( w w w ) 1 0 1

32 Receptive field (structural definition) The locations of the neurons in the input layer that are connected to a neuron Paths of one or more synapses.

33 Receptive field size increases

34 After L convolutions with a k k filter, how large is the receptive field? A. (Lk L+1) (Lk L+1) B. (Lk 1) (Lk 1) C. (2k 1) (2k 1) D. (L+k) (L+k) E. (Lk k+2) (Lk k+2)

35 Projective field r = f ( w s + w s + w s ) i 1 i 1 0 i 1 i+1 ( w w w ) 1 0 1