Ângelo Cardoso 27 May, Symbolic and Sub-Symbolic Learning Course Instituto Superior Técnico

Transcription

1 BIOLOGICALLY INSPIRED COMPUTER MODELS FOR VISUAL RECOGNITION Ângelo Cardoso 27 May, 2010 Symbolic and Sub-Symbolic Learning Course Instituto Superior Técnico

2 Index Human Vision Retinal Ganglion Cells Simple and Complex Cells Cell Columns and Layers Computational Models Neocognitron LISSOM HMAX

3 Human Vision Is it just raw sensory information? Our brain creates a mental picture of what it is seeing Tries to make sense of the raw inputs Optical Illusions illustrate some ways in which our brain makes interpretatations

4 Some Optical Illusions

5 Some Optical Illusions

6 Some Optical Illusions

7 Some Optical Illusions

8 Visual System Fig. D. Hubel and T. Wiesel 1981 Nobel Prize in Physiology or Medicine For their discoveries concerning information processing in the visual system Fig. Primary visual cortex location in humans Fig. Visual processing pathway *Section figures from Eye, Brain and Vision by D. Hubel

9 Retina Retina translates light into nerve signals Descriminates wavelength so we can see colors Retina is connected to the rest of the brain trough the optic nerve Retina s output is produced by the ganglion cells outputs which bundle in the optic nerve Two main types of ganglion cells on-center and off-center Fig. Left: on-center ganglion cells response and Right: off-center Fig. A cross section detail of the retina

10 Experiments Fig. Monkey visual cortex (1mm² area)

11 Retinal Ganglion Cells Receptive Fields Receptive field refers to an area on which the response of a cell depends and how that area should be stimulated to get a response. Deeper into the visual cortex these descriptions become more complex Neighbour ganglion cells have overlapping receptive fields Topographical organization is also present throughout the visual cortex Fig. Retinal Ganglion Cells Receptive Fields

12 Lateral Geniculate Body Lateral Geniculate Cells Receive input from retinal ganglion cells Have backward connections from the cerebral cortex Do not seem to make a profund transformation of information received from retinal ganglion cells It is hypotethized that they play a role in attention

13 Simple Cells Simple Cells Recognize patterns with a specific orientation and position Their receptive field size depends specially on their position regarding the retina area to which they react Even for a particular part of the retina their size can vary Fig. Simple Cell Response Fig. Three typical receptive fields for simple cells

14 Complex Cells Complex Cells Recognize patterns with a specific orientation but allow a positional shift. Most common type of cells in the primary visual cortex Some exhibit directional selectivity Fig. Complex Cell Fig. Complex Cell with Directional Selectivity

15 End-Stopping Usually simple and complex cells exihbit lenght summation Prolonguing their prefered stimulus (e.g. a line) over their receptive field makes no change in their response Some simple and complex cells are end-stopped Prolonguing their prefered stimulus over their receptive field diminishs their response Fig. End-Stopped Complex Cell

16 Orientation and Ocular-Dominance Columns Cells are topologically organized according to ocular dominance and orientation preference Fig. Recordings from an electroded moving Fig. Prefered orientation map in primary approximatelly parallel to the cortical surface visual cortex [Blasdel 86]

17 Hierarchy of layers The visual cortex is composed essentially as an hierarchy of cells Layers of simple and complex cells are arranged in a hierachical way The input of a layer is the output of the previous layer Fig. Visual pathway [nips.ac.jp]

18 Increasing Complexity Throughout the visual cortex there is a gradual increase in the complexity of the preferred stimulus The receptive field sizes and invariance properties also increase gradually Fig. Increasing Complexity in prefered stimulus Fig. Receptive fields from a region including V4 [Kobatake et al. 94] and IT [Kobatake et al. 94]

19 Overcomplete dictionaries of features Complete dictionaries (Small and Independent) e.g. represent a sentence with several words Overcomplete dictionaries (Large and Dependent) e.g. represent a sentence with one or very few words Neurons in the infero temporal cortex may be tuned to overcomplete dictionaries [Tanaka et al. 96]

20 Topological Data Non-topological data The position of the parts of a pattern does not contain information e.g information about one person [1,80m, 75 kg, 40 years] is the same as [75kg, 40 years, 1,80m]. Meaning doesn t depend on order Topological data The position of the parts of pattern contains information e.g. a word building is different from gnidbuil Problems of fully-connected networks for topological data The information implicit in the data topology is ignored No built-in invariance to shifts and distortion Overfitting High computational Costs

21 Invariance vs. Specifitiy We can recognize a specific face despice changes in viewpoint, scale, ilumination or expression Our vision is invariant to all these image variations recognizing objects indepently of these conditions Our vision has high specifity being face recognition the utterly example of such How do we achieve these two rather opposite properties? Fig. Samples from a Face Dataset [AT&T Laboratories Cambridge]

22 Hierarchical Neural Networks Biological Inspiration Mammalian Visual System Layers are not fully connected Each neuron has connections to a small and localized part of the neurons of the previous layers The global vision is created as we ascend in the layers towards the final layer They work with less units and connections Some intrinsic invariance to shifts and distortions

23 Hierarchical Neural Networks Local receptive fields Each neuron does not have a connection with all the neurons of the previous layers, making it s view local Deals only with a little and localized part of the information Shared-Weights A set of units represent the same template in different positions by sharing the same weight vector Reduces overfitting problems Reduces computational costs Subsampling The information is reduced from the input layer trought the output layer The information is reduced from the input layer trought the output layer improving generatilization

24 Neocognitron A biologically inspired Hierarchical Neural Network for visual recognition [Fukushima 80] Invariance to shifts and distortions Reduces the information progressively troughout the several layers from the input to the output. It allows the interpretation of the network operation Unlike Fully-Connected Networks

25 Neocognitron Network Architecture Cell represents a biological cell, has specific number of connections in a certain position Cell-plane a group of cells of the same type, which all recognize the same feature, like a feature map Layer set of cell-planes of the same type of cells Stage an ordered pair of layers in which the first is an S-cell Layer (simple) and the second a C-cell Layer (complex) cell cell-plane layer stage adapted from [Fukushima 03]

26 Neocognitron S-cells e C-cells S-cells represent simple cells in the visual cortex Extract features Learn to form a template of particular feature in particular position Share a weight-vector with all cells in their cell-plane C-cells In a cell-plane all cells extract the same feature in different positions Represent complex cells in the visual cortex Allow positional shifts in features It s output is a blurred version of their input

27 Learning in Neocognitron Sequential Learning Each stage is trained separately S-cells connections are changed by learning C-cells connections are fixed Starting from the stages close the input layer The training of a stage only starts after all the preceding one s is finished Unsupervised learning Except the final stage which corresponds to a classifier Assign a label to each cell-plane in the last stage if the winner cell has the same label as the input stimulus then reinforce connections If the winner cell has a different label from the input stimulus create a new cellplane for the presented input stimulus Other variations exist

28 Learning in Neocognitron Competitive learning in S-cells For each input only the cell-plane of the winner cell gets it s excitatory connections reinforced Inhibitory connections make each cell-plane speacialize in only one feature A selectivity threshold controls the number of cell-planes A small threshold makes the cells less selectives making the A small threshold makes the cells less selectives making the number of cell-planes small and vice-versa

29 Neocognitron Example Fig. Pattern examples Fig. Model operation figures from [Fukushima 88]

30 Neocognitron S-cell response selectivity constant variable excitatory connection weight excitatory input position cell-plane inhibitory input radius of the variable inhibitory root mean-square of connectable area connection weight the preceding signals decreasing function of v Fig. S-cell response [Fukushima 88] representing fixed excitatory connections

31 Neocognitron C-cell response fixed excitatory connection weight excitatory input position cell-plane radius of the connectable area representing saturation characteristic of C-cell

32 Parametrization Problem Highly susceptibe to parametrization unlike fully-connected networks Large number of parameters Number of layers Number of cell-planes in each layer Size of the receptive fields in each layer Overlap of the receptive fields Weight of the inhibitory connections... Solution Trial and Error Parameter Optimization e.g. Hill Climbing, Genetic Algorithm Use biologically plausible parameters Uncertainty in experimental data

33 Convolutional Neural Network (CNN) Inspired by the same biological principles as the Neocognitron but more loosely Local receptive fields, shared weights and subsampling Essentially an engineering model Back-propagation learning Network topology is hierarchical and not fully-connected Fig. Model operation [yann.lecun.com]

34 LISSOM Inspired in Self-Organizing Map but uses only local rules Uses Hebbian-Learning and includes lateral connections between neurons Biologically plausible learning mechanism Model gives to nontopographically organized lateral connectivity similar to that observed in the neocortex Proposed initially to explain only the organization of the primary visual cortex (RF-LISSOM) [Sirosh et al. 94] Later the model was extend to deal with natural images by modeling retina and LGN (CRF-LISSOM) [Bednar et al. 99] The model was further extend to deal with the entire visual cortex by adding a hierarchy of layers (HLISSOM) [Bednar et al. 01]

35 RF-LISSOM relation to SOM Self-Organizing Maps [Kohonen 82] Are a simple and efficient model for self-organization as an alternative to lateral connections winner neuron stimulus Will converge to globally topologically organized if training is appropriate update weights according to neuron learning neighborhood winner prefered rate amplitude stimulus Fig. SOM neighborhood example Fig. Color Map SOM [pymvpa.org]

36 RF-LISSOM RF-LISSOM Network Afferent Connections Lateral Connections Short-range excitatory lateral connections Long-range inhibitory lateral connections Weight Adaption Activity-dependent Unsupervised Local Fig. LISSOM Network Architecture [Bednar et al. 04]

37 RF-LISSOM Afferent, Excitatory Lateral and Inhibitory Lateral Connections learning is simultaneous Afferent connections are all excitatory Each neuron has reciprocal excitatory and inhibitory lateral connections with other neurons Neurons initial response is based on afferent connection weights Primary effect of lateral connections is to sharpen the Primary effect of lateral connections is to sharpen the contrast between areas of high and low activity

38 RF-LISSOM Ignoring lateral connections A neuron output is simply the value of the activation function for the scalar product of afferent input and the corresponding weights of the neuron Initial response does not have lateral connections Activation function makes the neuron selective and non-linear Only responds above the low-threshold Saturates at the upper-threshold Fig. Neuron Activation Function [Sirosh et al. 94]

39 RF-LISSOM Learning Neuron output is the value of the activation function for the weighted sum of Afferent input Excitatory lateral Inhibitory lateral Lateral weights develop through a version of the Hebb rule Excitatory and inhibitory weight vectors are update according pre and postsynaptic correlation between neurons Synaptic strength of lateral excitatory and inhibitory connections is kept constant Afferent connections update is analogous

40 RF-LISSOM Learning Activation thresholds are modified during learning to increase selectivity The lower and upper threshold come closer Depending on the activity of neurons Until they reach a prescribed limit Non-significant connections are eliminated according to a threshold

41 RF-LISSOM Model replicates orientation preference patterns of primary visual cortex Map is not globally topologically organized Locally organized only Fig. LISSOM development of orientation in V1 [Plebe et al. 07]

42 How to achieve invariant object recognition? Objected-centered representation hypothesis Objects represented as descriptions of spatial arrangements among parts in 3-D coordinates centered on the object itself View-based representation hypothesis Objects represented as collections of view-specific features Some psychophysical and physiological evidence View-invariant output is explicitly represented by a small number of neurons

43 Object-centered representation Object-centered model Objects represented as descriptions of spatial arrangements among parts in 3-D coordinates centered on the object itself Recognition by Components (RBC) Theory [Biederman et al. 87] Extract a view-invariant structural description of the object using volumetric primitives and their spatial relationships Compare the extracted description with stored object descriptions

44 View-based representation View-based model Invariance is achieved by pooling over afferents tuned to various transformations of the same stimulus [Perrett et al. 93] Computational Model [Poggio et al. 90] Train a set of view-tuned units Fed these units into another into a view-invariant unit New views are interpolated over the learnt views Some psychophysical and physiological evidence View-invariant output is explicitly represented by a small number of neurons Fig. Feedforward view-based model Fig. Feedforward view-based model [Riesenhuber 00]

45 HMAX Two pooling mechanisms Linear summation for simple cells Non-linear maximum for complex cells More robust response Clutter Multiple stimulli Linear summation for complex cells problems Unable to achieve size invariance Fig. Model Sketch [Riesenhuber et al. 99] Several units which react to the same stimulus on different scales are added up increasing the reaction to larger stimulus

46 HMAX Fig. HMAX Schematic [Serre et al. 07]

47 Feedfoward accounts for rapid categorization Rapid categorization is likely mostly feed-forward Given the number of processing stages and typical neural latency Humans were asked to detect if an animal was present in an image shown for 20 ms The results were similar to the HMAX model using only feed-forward connections Fig. Experiment Outline [Serre et al. 2007] Fig. Experiment Results [Serre et al. 2007]

48 Feedback EEG studies show that the human visual system can detect an object within 150 ms This implies that the role of feedback in such tasks is limited In clutter situations and multiple stimuli attention could play a major role

49 Future Directions Parametrization Attention Feedback Learning Role in Connections Population Codes

50 Questions