Flexible Gating of Contextual Influences in Natural Vision. Odelia Schwartz University of Miami Oct 2015

Transcription

1 Flexible Gating of Contextual Influences in Natural Vision Odelia Schwartz University of Miami Oct 05

2

3 Contextual influences Perceptual illusions: no man is an island.. Review paper on context: Schwartz, Hsu, Dayan, Nature Reviews Neuroscience 007

4 Contextual influences Perceptual illusions: no man is an island.. Review paper on context: Schwartz, Hsu, Dayan, Nature Reviews Neuroscience 007

5 Contextual influences Perceptual illusions

6 What about neurons? Cortical neural processing Record From neuron

7 What about neurons? Computer science / Engineering: visual receptive field or filter

8 Contextual influences Cortical visual neurons (V)??

9 Motivation Spatial context plays critical role in object grouping and recognition, and in segmentation. It is key to everyday behavior; deficits have been implicated in neurological and developmental disorders and aging Range of existing experimental data on spatial context (neural; perceptual). Lacking principled explanation Poor understanding for how we (and our cortical neurons) process complex, natural images

10 Outline Experimental data on cortical responses to natural images Computational neural model that captures contextual regularities in natural images Interplay of modeling with biological neural and psychology data (focus on natural images data)

11 Suppression index (SI) Cortical Neurons Response A 30 0 Suppression index (SI) Spatial context and natural scenes 0 0 deg B Data: Adam Kohn lab (Coen-Cagli, Kohn, Schwartz, 05; in press)

12 Cortical Neurons Modulation Ratio (MR) Supp. index (SI) Spatial context and natural scenes facilitation Data suppression Surround inference: Image ID OFF ON deg B Data: Adam Kohn lab (Coen-Cagli, Kohn, Schwartz, 05; in press)

13 Cortical Neurons Spatial context and natural scenes Can we capture data with canonical divisive normalization? (descriptive model)

14 Divisive normalization a Standard normalization RF Response surround Descriptive model Canonical computation (Carandini, Heeger, Nature Reviews Neuro, 0) Has been applied to visual cortex, as well as other systems and c modalities, multimodal processing, value encoding, etc

15 Cortical Neurons x c Canonical divisive normalization: x s R c x c x c + x s V Data: Kohn lab

16 Cortical responses to natural images Modulation Ratio (MR) Supp. index (SI ) facilitation suppression Image ID Image ID Data Standard model We fit the standard normalization model to neural data Poor prediction quality Data: Adam Kohn lab Coen-Cagli, Kohn, Schwartz, 05 (in press)

17 Cortical responses to natural images Modulation Ratio (MR) Supp. index (SI ) facilitation suppression Image ID Image ID Data Standard model Can we explain as strategy to encode natural images optimally based on expected contextual regularities? Data: Adam Kohn lab Coen-Cagli, Kohn, Schwartz, 05 (in press)

18 Outline Experimental data on cortical responses to natural images (standard descriptive model can t explain) Computational neural model that captures contextual regularities in natural images A Interplay of modeling with biological neural and psychology data (focus on natural images data)

19 Two overarching computational principles Sensory processing as inference of properties of the input (can be formalized via probabilistic Bayesian inference) Sensory systems aim to form an efficient code by reducing redundancies of the input (Barlow; also Attneave); influenced by information theory in the 950s

20 Contextual dependencies across space

21 Contextual dependencies across space

22 Contextual dependencies across space

23 Contextual dependencies across space

24 Contextual dependencies across space

25 Contextual dependencies across space Schwartz, Simoncelli, Nature Neuroscience 00

26 Generative model framework x x 0 Hypothesize that cortical neurons aim to reduce statistical dependencies (so as to highlight what is salient) Schwartz, Simoncelli 00 (for salience: Zhaoping Li, 00) Formally, we build a generative model of the dependencies and invert the model (Bayesian inference) richer representation! Andrews, Mallows, 974; Wainwright, Simoncelli, 000; Schwartz, Sejnowski, Dayan 006 Generating the dependencies is a multiplicative process and to undo the dependencies we divide 0

27 Modeling Statistical dependencies: Gaussian Scale Mixture (GSM) x x Filter activations x x Andrews & Mallows, 974; Wainwright & Simoncelli, 000

28 Modeling Statistical dependencies: Gaussian Scale Mixture (GSM) x = vg x = vg Gaussians local Global (shared) mixer g g x x Filter activations x 0g g 0 x Andrews & Mallows, 974; Wainwright & Simoncelli, 000

29 Modeling Statistical dependencies: Gaussian Scale Mixture (GSM) Gaussians local Global (shared) mixer x = vg x = vg g g x x Filter activations Infer local Gaussian E(g x,x ) = Model neuron activity

30 Modeling Statistical dependencies: Gaussian Scale Mixture (GSM) x = vg x = vg Gaussians local Global (shared) mixer g g x x Filter activations x 0g g 0 x EFFICIENT CODING

31 Modeling Statistical dependencies: Gaussian Scale Mixture (GSM) Gaussians local Global (shared) mixer g g x x Filter activations Computed via Bayes rule E(g x, x ) x l ; l = x + x DIVISIVE NORMALIZATION

32 Divisive Normalization Canonical Model Divisive normalization descriptive models have been applied in many neural systems. Here we provide a principled explanation. We will next show that it also leads to a richer model based on image statistics and makes predictions

33 Non-homogeneity of images g c v c g s x c x s homogenous image patches Center and surround dependent Schwartz, Sejnowski, Dayan, 009; Coen-Cagli, Dayan, Schwartz, PLoS Comp Biology 0

34 Non-homogeneity of images g c v c v s g s non-homogenous image patches x c xs Center and surround independent Schwartz, Sejnowski, Dayan, 009; Coen-Cagli, Dayan, Schwartz, PLoS Comp Biology 0

35 x = vg () THExAUTHOR = vg x B( x sign(x ) AUTHOR E[g x] THE = x x B( n(), l ) l B () E[g x] = sign(x ) () () ne of images Non-homogeneity n l B( x x l B( x =,vg) x = vg () E[g x] = sign(x ) () l = x n l = x = vgx = vg i i x = vgx l= vg B( x l =x = vg x x l =x = vg c c i i x i i l = xc xc sxcx l = xs i xi l = x i xi B x v xs v x c s xc() c x xc() ns E[g x] = sign(x ) B( x x vc vc x v l B xs () E[g x] = sign(x ) () c s n l B( gc gc v vc v s c l= x gs gc gs gc l= x i i i v s x xicxi gs l = gs l = xic i xc xs xs vc homogenous vc xc xs xs vc heterogeneous vc Schwartz, Sejnowski, Dayan, 009; Coen-Cagli, Dayan, Schwartz, PLoS Comp Biology 0

36 x = vg () THE xauthor = vg x() x B( THE AUTHOR E[g x] = sign(x ) B( n, l ) x x B () l E Non-homogeneity () () E[g of x] images = sign(x ) n nx x= l B( x l vg B(, ) x = vg () () E[g x] = sign(x ) x = vg l = l = x n x l= vg x = vgx = vg i i B( l =x = vg xc l =x = vg x i x i xcl = i i xc sxcx xc l = x l = xs i xi i i v x xs c xs x B v xs x c c() xc() n E[g x] = sign(x ) B( x x v v sv l B c x vc c s xs () E[g x] = sign(x ) () n gc v l g B( c v s c vc l= gs g x l= c x gs gc i i i v s l = x g x s l = x ic i xic i gs xc xs xs vc divisive vc normalization ON xc xs xs vc divisive vc normalization OFF Schwartz, Sejnowski, Dayan, 009; Coen-Cagli, Dayan, Schwartz, PLoS Comp Biology 0

37 () THE xauthor = vg x() x B( THE AUTHOR E[g x] = sign(x ) B( n, l ) x x xs vc xc xs vc x = vg B () l E Non-homogeneity () () E[g of x] images = sign(x ) n nx x= l B( x l vg B(, ) x = vg () () E[g x] = sign(x ) x = vg l = l = x n x l= vg x = vgx = vg i i B( l =x = vg xc l =x = vg x i x i xcl = i i xc sxcx xc l = x l = xs i xi i i v x xs c xs x B v xs x c c() xc() n E[g x] = sign(x ) B( x x v v sv l B c x vc c s xs () E[g x] = sign(x ) () n gc v l g B( c v s c vc l= gs g x l= c x gs gc i i i v s l = x g x s l = x ic i xic i gs xc xs vc xs vc Schwartz, Sejnowski, Dayan, 009; Coen-Cagli, Dayan, Schwartz, PLoS Comp Biology 0

38 Model: xcl = xc l =x = vg l =x = vg i x i x i xi i i xc l = xsxicxi l = xs i xxic x xs () xs () xc() vce[g xc() vc xs E[g x] x=e sb x] = sign(x ) x Optimizing Image Ensemble v v v vc c s xs x]sv= () E[g ) x] = sign(x ) xs () E[g c sign(x g l B v g cv s c v vc s c l =x l= l = gs gs gc i xi l = x gc i xi i i i i xc l = l = x g xic i gs xc xicxi s xc xc xs xs xs vc - 3x3 spatial positions, 6px separation vc vc - 4 orientations in the center xs vc xs vc xs vc - 4 orientations in the surround - phases (quadrature) - model parameters (prior probability for and also linear covariance matrices) optimized to maximize the likelihood of a database of natural images using Expectation Maximization Coen-Cagli, Dayan, Schwartz, PLoS Comp Biology 0; Schwartz, Sejnowski, Dayan, 006

39 Outline Experimental data on cortical responses to natural images Computational neural model that captures contextual regularities in natural images Interplay of modeling with biological neural and psychology data (focus on natural images data)

40 Cortical predictions for natural images In the past, we have tested modeling with simple stimuli (e.g., Coen-Cagli, Dayan, Schwartz, 0; Schwartz, Sejnowski, Dayan, 009) Here, we make predictions for natural images (Coen-Cagli, Kohn, Schwartz, 05, in press)

41 Flexible Divisive Normalization

42 Model predictions for natural images Homogeneous and heterogeneous determined by model! Expect more suppression in neurons for homogeneous Related to salience (eg, Zhaoping) Coen-Cagli, Kohn, Schwartz, 05; in press

43 Model summary RF Response surround ON OFF surround inference Inference determined by model

44 Model Predictions for Natural Scenes homogeneous versus heterogeneous determined by the model

45 Model Predictions for Natural Scenes Cortical V data: a.5 MR heterogeneous n=6 neurons MR homogeneous Coen-Cagli, Kohn, Schwartz, 05, in press

46 Model Predictions for Natural Scenes Cortical V data: a.5 MR heterogeneous n=6 neurons MR homogeneous Not explained by: firing rate with small frames surround energy Coen-Cagli, Kohn, Schwartz, 05, in press

47 Model predictions for natural images Per image, across neurons homogeneous heterogeneous Coen-Cagli, Kohn, Schwartz, 05; in press

48 Model predictions for natural images Testing predictions with cortical data Homogeneo d NMR when heterogeneous Coen-Cagli, Kohn, Schwartz, 05; in press n=39 images NMR when homogeneous

49 Modulation Ratio (MR) Supp. index (SI) Natural scenes data facilitation Data suppression Surround inference: Image ID OFF ON deg B Coen-Cagli, Kohn, Schwartz, 05, in press

50 Natural scenes data Modulation Ratio (MR) facilitation suppression Image ID Data Standard model Flexible model Surround inference: OFF ON Coen-Cagli, Kohn, Schwartz, 05, in press

51 Model predictions for natural images Comparing model performance for cortical data Standard divisive normalization E c,φ R i = α pref ε + βe c + γe s Flexible divisive normalization: E c,φ R i = α pref ε + βe c + q(c,s)γe s n n Determined by the model (not fit!) if p(ξ c,s) otherwise (similar results if non binary)

52 Natural scenes data Cross-validated prediction quality There are many standard model versions Prediction quality: = oracle (observed mean for each image) 0 = null (mean response across all images) Coen-Cagli, Kohn, Schwartz, 05, in press

53 Divisive normalization: Model Mechanisms Feedback inhibition Distal dendrite inhibition Depressing synapses Internal biochemical adjustments Non-Poisson spike generation

54 Flexible Normalization Mechanism? Adjusting gain by circuit or postsynaptic mechanisms? Distinct classes of inhibitory interneurons? (eg, Adesnik, Scanziani et al. 0; Pfeffer, Scanziani et al. 03; Pi, Kepecs et al. 03; Lee, Rudy et al. 03) Output Surround suppression Gating Pyr SOM VIP Input Normalization Pool

55 Key take-home points New approach to understanding cortical processing of natural images. Rather than fitting more complicated models, use insights from scene statistics Connects to neural computations that are ubiquitous, but enriches the standard model Our results suggest flexibility of contextual influences in natural vision, depending on whether center and surround are deemed statistically homogeneous Next/currently: hierarchical representations; adaptation

56 Acknowledgments Albert Einstein Ruben Coen Cagli Adam Kohn Gatsby, UCL Peter Dayan Salk Institute Terry Sejnowski Funding NIH (Collaborative Research in Computational Neuroscience) Army Research Office Alfred P. Sloan Foundation Google