Deep learning in the visual cortex

Transcription

1 Deep learning in the visual cortex Thomas Serre Brown University. Fundamentals of primate vision. Computational mechanisms of rapid recognition and feedforward processing. Beyond feedforward processing: Attentional mechanisms and cortical feedback

2 Ventral visual stream Feedforward processing

3 Coarse initial base representation - Enables rapid object detection/recognition ( what is there? ) - nsufficient for object localization - Sensitive to presence of clutter Feedforward processing

4 The crowding problem isolated objects L E F F L EE L E F F EE F see also Dayan 08; Pelli & Tillman 08; and many others

5 The crowding problem isolated objects L E F F EE L L E F F EE F see also Dayan 08; Pelli & Tillman 08; and many others

6 The crowding problem isolated objects L E F F EE L clutter L E F F EE F see also Dayan 08; Pelli & Tillman 08; and many others L E F F FL EE

7 The crowding problem Prediction: recognition in clutter requires attentional mechanisms via cortical feedback isolated objects L E F F EE L clutter L E F F EE F see also Dayan 08; Pelli & Tillman 08; and many others spatial attention L E F L E F F FL EE F FL EE

8 The crowding problem Prediction: recognition in clutter requires attentional mechanisms via cortical feedback isolated objects L E F F EE L clutter L E F F EE F see also Dayan 08; Pelli & Tillman 08; and many others spatial attention L E F L E F E E E E X XX F FL FL F

9 1. Monkey electrophysiology Read-out of inferior temporal cortex population activity: Spatial attention eliminates clutter with Zhang, Meyers, Bichot, Poggio & Desimone 2. Computational model of integrated attention and recognition What and where: a Bayesian attention theory of attention with Chikkerur, Tan & Poggio

10 1. Monkey electrophysiology Read-out of inferior temporal cortex population activity: Spatial attention eliminates clutter with Zhang, Meyers, Bichot, Poggio & Desimone 2. Computational model of integrated attention and recognition What and where: a Bayesian attention theory of attention with Chikkerur, Tan & Poggio

11 The readout approach neuron 1 neuron 2 neuron 3 neuron n Prediction Pattern Classifier Zhang Meyers Bichot Serre Poggio Desimone PNAS 11

12 The experiment Zhang Meyers Bichot Serre Poggio Desimone unpublished data

13 The experiment train readout classifier on isolated object test generalization in clutter + + Zhang* Meyers* Bichot Serre Poggio Desimone in sub

14 The experiment train readout classifier on isolated object test generalization in clutter isolated + + Zhang* Meyers* Bichot Serre Poggio Desimone in sub

15 The experiment train readout classifier on isolated object test generalization in clutter isolated + + Zhang* Meyers* Bichot Serre Poggio Desimone in sub

16 The experiment train readout classifier on isolated object test generalization in clutter isolated + + attended Zhang* Meyers* Bichot Serre Poggio Desimone in sub

17 The experiment train readout classifier on isolated object test generalization in clutter isolated + + attended Non-attended Zhang* Meyers* Bichot Serre Poggio Desimone in sub

18 Effects of attention on decoding accuracy: Decoding category Decoding object identity Zhang* Meyers* Bichot Serre Poggio Desimone in sub

19 Effects of attention on decoding accuracy: Decoding category Decoding object identity + Zhang* Meyers* Bichot Serre Poggio Desimone in sub

20 Effects of attention on decoding accuracy: Decoding category Decoding object identity + Zhang* Meyers* Bichot Serre Poggio Desimone in sub

21 Effects of attention on decoding accuracy: Decoding category Decoding object identity + Zhang* Meyers* Bichot Serre Poggio Desimone in sub

22 Effects of attention on decoding accuracy: Decoding category Decoding object identity + Zhang* Meyers* Bichot Serre Poggio Desimone in sub

23 Effects of attention on decoding accuracy: Decoding category Decoding object identity + Zhang* Meyers* Bichot Serre Poggio Desimone in sub

24 Effects of attention on decoding accuracy: Decoding category Decoding target location + Zhang* Meyers* Bichot Serre Poggio Desimone in sub

25 Changes in the salience of distractor stimuli dominate over attention related enhancements Aligned to the time when one of the distractors underwent a change Zhang* Meyers* Bichot Serre Poggio Desimone in sub

26 Summary

27 Summary Consistent with feedforward hierarchical models: n the absence of attention information about the identity of individual objects (and position) in clutter is greatly reduced relative to when objects are shown in isolation

28 Summary Consistent with feedforward hierarchical models: n the absence of attention information about the identity of individual objects (and position) in clutter is greatly reduced relative to when objects are shown in isolation Attention seems to restore the pattern of neural activity toward the vector representing the isolated object

29 Summary Consistent with feedforward hierarchical models: n the absence of attention information about the identity of individual objects (and position) in clutter is greatly reduced relative to when objects are shown in isolation Attention seems to restore the pattern of neural activity toward the vector representing the isolated object n spite of this nearly exclusive representation of the attended object, an increase in the salience of non-attended objects overrode these attentional enhancements

30 Summary Consistent with feedforward hierarchical models: n the absence of attention information about the identity of individual objects (and position) in clutter is greatly reduced relative to when objects are shown in isolation Attention seems to restore the pattern of neural activity toward the vector representing the isolated object n spite of this nearly exclusive representation of the attended object, an increase in the salience of non-attended objects overrode these attentional enhancements Results provide computational level explanation for how attention operates on neural representations to solve the problem of invariant recognition in clutter

31 1. Monkey electrophysiology Read-out of inferior temporal cortex population activity: Spatial attention eliminates clutter with Zhang, Meyers, Bichot, Poggio & Desimone 2. Computational model of integrated attention and recognition What and where: a Bayesian attention theory of attention with Chikkerur, Tan & Poggio

32 image measurements Perception as Bayesian inference Mumford 92; Knill & Richards 96; Dayan & Zemel 99; Rao 02 04; Kersten & Yuille 03; Kersten et al 04; Lee & Mumford 03; Dean 05; George & Hawkins 05 09; Yu & Dayan 05; Hinton 07; Epshtein et al 08; Murray & Kreutz-Delgado 07

33 image measurements Perception as Bayesian inference Mumford 92; Knill & Richards 96; Dayan & Zemel 99; Rao 02 04; Kersten & Yuille 03; Kersten et al 04; Lee & Mumford 03; Dean 05; George & Hawkins 05 09; Yu & Dayan 05; Hinton 07; Epshtein et al 08; Murray & Kreutz-Delgado 07

34 S visual scene description image measurements Perception as Bayesian inference Mumford 92; Knill & Richards 96; Dayan & Zemel 99; Rao 02 04; Kersten & Yuille 03; Kersten et al 04; Lee & Mumford 03; Dean 05; George & Hawkins 05 09; Yu & Dayan 05; Hinton 07; Epshtein et al 08; Murray & Kreutz-Delgado 07

35 P (S ) / P ( S)P (S) S visual scene description image measurements Perception as Bayesian inference Mumford 92; Knill & Richards 96; Dayan & Zemel 99; Rao 02 04; Kersten & Yuille 03; Kersten et al 04; Lee & Mumford 03; Dean 05; George & Hawkins 05 09; Yu & Dayan 05; Hinton 07; Epshtein et al 08; Murray & Kreutz-Delgado 07

36 Hypothesis #1 To recognize and localize objects in a scene, the visual system selects objects one at a time P (O, L, ) O,L visual scene description image measurements Chikkerur Serre Tan & Poggio 10;

37 Two independent streams of processing for object and location Hypothesis #2 Explicit tuning for position (shape implicit) Dorsal where stream P (O)P (L)P ( L, O) O object L PFC location image measurements Ventral what stream Explicit tuning for shape (position implicit) Chikkerur Serre Tan & Poggio 10;

38 Hypothesis #3 Objects encoded by collections of generic features (cond. ind. given an object and its location) Explicit tuning for position (shape implicit) Dorsal where stream F i PFC X i... Ventral what stream Explicit tuning for shape (position implicit) Chikkerur Serre Tan & Poggio 10;

39 Perception as O object Bayesian inference location F i position and scale tolerant features (T) L L X i N retinotopic features (V4) X i F i O image measurements Chikkerur Serre Tan & Poggio 10;

40 Perception as Bayesian inference p(l ) saliency map L X i F i O Chikkerur Serre Tan & Poggio 10;

41 Perception as O object Bayesian inference Goal of visual perception is to estimate posterior probabilities of visual features, objects and their locations in an image Attention corresponds to conditioning on high-level latent variables representing particular objects or locations (as well as on sensory input), and doing inference over the other latent variables location L F i X i N position and scale tolerant features (T) retinotopic features (V4) image measurements Here we used belief propagation to solve the inference problem Biologically-plausible implementations of belief propagation: Zemel et al. 98; Beck & Pouget 07; Deneve 08; George 08; Litvak & Ullman 09; Rao 04; Steimer et al. 09

42 L dorsal / O ventral / X i O where what F i F i L X i N P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) Bayesian inference and attention Special case of the normalization model of attention by Reynolds & Heeger 09

43 L dorsal / O ventral / X i O where what F i F i L feedforward input X i N P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) Bayesian inference and attention feedforward (bottom-up) sweep Special case of the normalization model of attention by Reynolds & Heeger 09

44 L dorsal / O ventral / X i O where what F i F i L X i X k feedforward input N P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) suppressive drive Bayesian inference and attention feedforward (bottom-up) sweep Special case of the normalization model of attention by Reynolds & Heeger 09

45 L dorsal / O ventral / X i O where what F i top-down spatial attention L F i top-down featurebased attention X i X k feedforward input attentional modulation N P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) suppressive drive Bayesian inference and attention feedforward (bottom-up) sweep Special case of the normalization model of attention by Reynolds & Heeger 09

46 dorsal / where O ventral / what F i P(L ) L X i X k feedforward input P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) suppressive drive Bayesian inference and feedforward (bottom-up) sweep attention

47 (parallel) dorsal / where O ventral / what feature-based attention Visual search F i L X i feedforward input attentional modulation N P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) suppressive drive Bayesian inference and attention

48 dorsal / where O ventral / what T F i F i V4 X i L X i feedforward input attentional modulation N P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) suppressive drive Bayesian inference and attention Consistent with data from V4 by Bichot et al 05

49 dorsal / where O ventral / what T F i F i V4 X i L X i feedforward input attentional modulation N P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) suppressive drive Bayesian inference and attention Consistent with data from V4 by Bichot et al 05

50 P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) O Visual search F i L Experimental data a b c Model data X i N Bayesian inference and attention Consistent with data from V4 by Bichot et al 05

51 (serial) spatial dorsal / where O ventral / what attention Spatial cueing F i L X i feedforward input attentional modulation N P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) suppressive drive Bayesian inference and attention

52 P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) McAdams and Maunsell 99 Model W attention P (L = x) 1 P (L = x) =1/ L Multiplicative scaling of tuning curves by spatial attention

53 P (X i ) = P ( X i ) P F i,l P (Xi F i,l)p(l)p(f i ) n PX P ( X i ) P o i F i,l P (Xi F i,l)p(l)p(f i ) Trujillo and Treue 02 Mc Adams and Maunsell 99 Contrast vs. response gain Predicted by Reynolds & Heeger 09

54 Learning to localize cars and pedestrians in street scenes O F i L X i N

55 Learning to localize cars and pedestrians in street scenes learning object priors O F i L X i N

56 Learning to localize cars and pedestrians in street scenes learning object priors O Context F i L X i N

57 Learning to localize cars and pedestrians in street scenes learning location priors (global contextual cues) learning object priors O Context F i L X i N (see Torralba et al)

58 The experiment Dataset: street-scenes images with cars & pedestrians and 20 without Experiment - 8 participants asked to count the number of cars/ pedestrians - Blocks/randomized presentations - Each image presented twice Eye movements recorded using an infra-red eye tracker Eye movements as proxy for attention

59 Predicting eye movements during searches for cars and pedestrians O P (L ) Context F i L X i Saliency map

60 Predicting eye movements during searches for cars and pedestrians O Context F i L X i N

61 Predicting eye movements during searches for cars and pedestrians O Context F i L X i N

62 Predicting eye movements during searches for cars and pedestrians O Context F i L X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

63 Predicting eye movements during searches for cars and pedestrians 1st three fixations O Context F i L X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

64 Predicting eye movements during searches for cars and pedestrians 1st three fixations O 1 Context 0.75 F i L 0.5 cars pedestrians X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

65 Predicting eye movements during searches for cars and pedestrians 1st three fixations O 1 Context 0.75 F i L 0.5 cars pedestrians X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

66 Predicting eye movements during searches for cars and pedestrians 1st three fixations O 1 Context 0.75 F i L 0.5 cars pedestrians X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

67 Predicting eye movements during searches for cars and pedestrians 1st three fixations O 1 Context 0.75 F i L 0.5 cars pedestrians X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

68 Predicting eye movements during searches for cars and pedestrians 1st three fixations O 1 Context 0.75 F i L 0.5 cars pedestrians X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

69 Predicting eye movements during searches for cars and pedestrians 1st three fixations O 1 Context 0.75 F i L 0.5 cars pedestrians X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

70 Predicting eye movements during searches for cars and pedestrians 1st three fixations O 1 Context 0.75 F i L 0.5 cars pedestrians X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

71 Predicting eye movements during searches for cars and pedestrians 1st three fixations O 1 Context 0.75 F i L 0.5 cars pedestrians X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

72 Predicting eye movements during searches for cars and pedestrians 1st three fixations O 1 Context 0.75 F i L 0.5 cars pedestrians X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

73 Predicting eye movements during searches for cars and pedestrians Overall model accounts for 92% of inter-subject agreement! 1 1st three fixations Context O 0.75 F i L 0.5 cars pedestrians X i Uniform priors (bottom-up) Feature priors Feature + contextual (spatial) priors Humans *similar (independent) results by Ehinger Hidalgo Torralba & Oliva (in press)

74 Predicting eye movements during free viewing Method ROC area O Bruce and Tsotos % tti et al % Proposed 77.9% F i L X i N human eye data from Bruce & Tsotsos

75 Summary

76 Summary Attention as part of the inference process that solves the visual recognition problem of what is where

77 Summary Attention as part of the inference process that solves the visual recognition problem of what is where Main goal of the visual system is to infer the identity and the position of objects in visual scenes: - Spatial attention emerges as a strategy to reduce the uncertainty in shape information while feature-based attention reduces the uncertainty in spatial information - Featural and spatial attention represent two distinct modes of a computational process solving the problem of recognizing and localizing objects, especially in difficult recognition tasks such as in cluttered natural scenes

78 Summary Attention as part of the inference process that solves the visual recognition problem of what is where Main goal of the visual system is to infer the identity and the position of objects in visual scenes: - Spatial attention emerges as a strategy to reduce the uncertainty in shape information while feature-based attention reduces the uncertainty in spatial information - Featural and spatial attention represent two distinct modes of a computational process solving the problem of recognizing and localizing objects, especially in difficult recognition tasks such as in cluttered natural scenes Model agnostic about the specific algorithm for the inference process (i.e., no claim made about the brain computing probabilities explicitly)

79 Two modes of vision

80 Two modes of vision Rapid bottom-up / feedforward processing during first ms of visual processing: - Coarse/initial base representation - Enables rapid object detection/recognition ( what is there? )

81 Two modes of vision Rapid bottom-up / feedforward processing during first ms of visual processing: - Coarse/initial base representation - Enables rapid object detection/recognition ( what is there? ) Top-down / re-entrant attentional processing - Enables recognition in clutter - Enables object localization

82 Collaborators (MT): Narcisse Bichot Sharat Chikkerur Bob Desimone Ethan Meyers Tomaso Poggio Cheston Tan Ying Zhang Robert J. and Nancy D. Carney Fund for Scientific nnovation Acknowledgments