Model- based fmri analysis. Shih- Wei Wu fmri workshop, NCCU, Jan. 19, 2014

Similar documents
General linear model: basic

Reinforcement learning

Decision Making in Robots and Autonomous Agents

Experimental design of fmri studies

Experimental design of fmri studies

Experimental design of fmri studies & Resting-State fmri

Summary of part I: prediction and RL

Experimental design of fmri studies

Experimental design of fmri studies

Animal learning theory

Data Analysis I: Single Subject

Contents. Introduction The General Linear Model. General Linear Linear Model Model. The General Linear Model, Part I. «Take home» message

Reinforcement Learning. Odelia Schwartz 2016

CO6: Introduction to Computational Neuroscience

Randomization-Based Inference With Complex Data Need Not Be Complex!

Lecture 23: Reinforcement Learning

6 Extending a biologically inspired model of choice: multi-alternatives, nonlinearity, and value-based multidimensional choice

The homogeneous Poisson process

Rewards and Value Systems: Conditioning and Reinforcement Learning

Reinforcement Learning In Continuous Time and Space

+ + ( + ) = Linear recurrent networks. Simpler, much more amenable to analytic treatment E.g. by choosing

Key concepts: Reward: external «cons umable» s5mulus Value: internal representa5on of u5lity of a state, ac5on, s5mulus W the learned weight associate

Reinforcement Learning

Piotr Majer Risk Patterns and Correlated Brain Activities

Optimization of Designs for fmri

Extracting fmri features

The General Linear Model. Guillaume Flandin Wellcome Trust Centre for Neuroimaging University College London

Extending a biologically inspired model of choice: multialternatives, nonlinearity and value-based multidimensional choice

The Perceptron. Volker Tresp Summer 2014

Glimcher Decision Making

Reinforcement Learning II

Assessing Time-Varying Causal Effect. Moderation

Exercise Sheet 4: Covariance and Correlation, Bayes theorem, and Linear discriminant analysis

Encoding or decoding

Neuroimaging for Machine Learners Validation and inference

Neural networks: Associative memory

CSE/NB 528 Final Lecture: All Good Things Must. CSE/NB 528: Final Lecture

Reinforcement Learning and Time Perception - a Model of Animal Experiments

Classical Conditioning IV: TD learning in the brain

Reinforcement Learning for Continuous. Action using Stochastic Gradient Ascent. Hajime KIMURA, Shigenobu KOBAYASHI JAPAN

Course 16:198:520: Introduction To Artificial Intelligence Lecture 13. Decision Making. Abdeslam Boularias. Wednesday, December 7, 2016

Themes for today Why did the Earth- centered Ptolemaic system last so long?

Reinforcement Learning and NLP

Chapter 6: Temporal Difference Learning

Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm

Visual motion processing and perceptual decision making

Machine Learning. Machine Learning: Jordan Boyd-Graber University of Maryland REINFORCEMENT LEARNING. Slides adapted from Tom Mitchell and Peter Abeel

The Markov Decision Process Extraction Network

Temporal Difference Learning & Policy Iteration

What is the neural code? Sekuler lab, Brandeis

Jan 16: The Visual System

Functional Causal Mediation Analysis with an Application to Brain Connectivity. Martin Lindquist Department of Biostatistics Johns Hopkins University

Time-Varying Causal. Treatment Effects

Decoding Cognitive Processes from Functional MRI

The Perceptron. Volker Tresp Summer 2016

The convergence limit of the temporal difference learning

This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer.

Introduction of Reinforcement Learning

Modelling temporal structure (in noise and signal)

Physics and Brain Imaging

Monte Carlo is important in practice. CSE 190: Reinforcement Learning: An Introduction. Chapter 6: Temporal Difference Learning.

The Temporal Dynamics of Cortical Normalization Models of Decision-making

Beyond Univariate Analyses: Multivariate Modeling of Functional Neuroimaging Data

16.410/413 Principles of Autonomy and Decision Making

MALLOY PSYCH 3000 Hypothesis Testing PAGE 1. HYPOTHESIS TESTING Psychology 3000 Tom Malloy

Part 2: Multivariate fmri analysis using a sparsifying spatio-temporal prior

The Reinforcement Learning Problem

Signal detection theory

Estimating representational dissimilarity measures

Reinforcement Learning

Strength-based and time-based supervised networks: Theory and comparisons. Denis Cousineau Vision, Integration, Cognition lab. Université de Montréal

SRNDNA Model Fitting in RL Workshop

Introduction to Reinforcement Learning. CMPT 882 Mar. 18

Machine learning strategies for fmri analysis

Outline. Superconducting magnet. Magnetic properties of blood. Physiology BOLD-MRI signal. Magnetic properties of blood

Resource Allocation in the Brain

MOL410/510 Problem Set 1 - Linear Algebra - Due Friday Sept. 30

2015 Todd Neller. A.I.M.A. text figures 1995 Prentice Hall. Used by permission. Neural Networks. Todd W. Neller

CS599 Lecture 1 Introduction To RL

ARTIFICIAL INTELLIGENCE. Reinforcement learning

Optimal Control. McGill COMP 765 Oct 3 rd, 2017

6 Reinforcement Learning

Statistical Analysis Aspects of Resting State Functional Connectivity

When is an Integrate-and-fire Neuron like a Poisson Neuron?

How Behavioral Constraints May Determine Optimal Sensory Representations

Grundlagen der Künstlichen Intelligenz

The General Linear Model (GLM)

The General Linear Model (GLM)

Deep Reinforcement Learning. Scratching the surface

Daniel Mess Cornell University May 17,1999

State Space Abstractions for Reinforcement Learning

A Spiking Independent Accumulator Model for Winner-Take-All Computation

Reinforcement Learning as Variational Inference: Two Recent Approaches

Introduction to MRI Acquisition

Generalization and Function Approximation

Squeezing Every Ounce of Information from An Experiment: Adaptive Design Optimization

New Procedures for False Discovery Control

Contents. design. Experimental design Introduction & recap Experimental design «Take home» message. N εˆ. DISCOS SPM course, CRC, Liège, 2009

Brains and Computation

Reinforcement Learning

Transcription:

Model- based fmri analysis Shih- Wei Wu fmri workshop, NCCU, Jan. 19, 2014

Outline: model- based fmri analysis I: General linear model: basic concepts II: Modeling how the brain makes decisions: Decision- making models III: Modeling how the brain learns: Reinforcement learning models IV: Modeling response dynamics: DriL diffusion model

Model- based fmri Problem: Characterizing mental operaqons SQmulus Response Neural acqvity How do we characterize the mental operaqons involved in producing behavioral response given the sqmulus?

Model- based fmri Problem: Characterizing mental operaqons 1. How do we characterize the mental operaqons involved in producing behavioral response given the sqmulus? SQmulus Response Mental operaqons cannot be simply represented by the observable: sqmulus and response

Model- based fmri Problem: Characterizing mental operaqons 1. How do we characterize the mental operaqons involved in producing behavioral response given the sqmulus? 2. Mental operaqons cannot be simply represented by the observable: sqmulus and response SQmulus Response

Model- based fmri Problem: Characterizing mental operaqons One opqon: Build or apply some computaqonal model that SQmulus Response quanqtaqvely describes the mental operaqons describes the behavioral performance

I. General linear model: basic concepts Univariate analysis - Each voxel in the brain is analyzed separately - Each voxel presents a - series data Time

I. General linear model: basic concepts Time- series data - Suppose you have the following experiment 20 40 60 80 100 120 Time (sec)

I. General linear model: basic concepts Time- series data - This is the data you get (from a single voxel) BOLD signal (arbitrary units) 2 1 0 20 40 60 80 100 120 Time (sec)

I. General linear model: basic concepts Time- series data - When you compare predicqon (based on your design) and data, you realize that there is somewhat a match, but not close BOLD signal (arbitrary units) 2 1 0 20 40 60 80 100 120 Time (sec)

I. General linear model: basic concepts Time- series data - What about this one? Which aspect of the comparison is the same, which aspect might be different? BOLD signal (arbitrary units) 2 1 0 20 40 60 80 100 120 Time (sec)

I. General linear model: basic concepts BOLD signal Design matrix =! X + noise Parameter esqmate: this is what we are interested in

I. General linear model: basic concepts!" = #! + """! $#$"# % & m n n x m BOLD s series Design matrix Parameter vector (BOLD: Blood OxygenaQon Level Dependent)

Modeling how the brain makes decisions: Decision- making models

II. Modeling how the brain makes decisions Losses loom larger than gains - The psychological impact of a loss (or potenqal loss) is greater than the same- sized gain (or potenqal gain)

II. Modeling how the brain makes decisions Prospect theory - Value func4on # V(x) = x!,x! 0' % % $ &% "! ("x)! ( )% reference point! : Controls the degree of loss aversion Loss aversion: Steeper slope in the loss domain compared with gains

II. Modeling how the brain makes decisions ImplicaQon of loss aversion on choice behavior Example: Is (gain $2000,50%; Lose $1000,50%) an agracqve gamble? Suppose! = 1," = 1,# = 2 Then the value of the gamble is V($2000)! 0.5 +V(!$1000) " 0.5 = 2000"0.5! 2"1000"0.5 = 0 This gamble is not agracqve at all to the decision maker and hence it is not likely that s/he is going to bet on it

II. Modeling how the brain makes decisions Prospect theory in the brain? 1. How does the brain represent gains and losses? 2. Is there a way to explain loss aversion from a neurobiological perspecqve?

II. Modeling how the brain makes decisions Neural basis of loss aversion - Tom et al. (2007, Science): A decision- making experiment involving monetary gains and losses Russell Poldrack - Subjects in each trial had to decide whether to accept a gamble (Gain,50%; Loss,50%) - Gain and loss in each trial were decided independently

II. Modeling how the brain makes decisions Measuring loss aversion by choice behavior - Suppose this is the data from a subject Trial Gain Loss Choice (yes/no) 1 2 3 4 5 6 7... $1000 $2000 $350 $500 $1200 $60 $290... $800 $1000 $450 $100 $380 $55 $148... 0 0 0 1 1 0 0...

- We can esqmate how much loss averse a subject is based on his /her choice data 1 2 3 4 5 6 7... II. Modeling how the brain makes decisions Measuring loss aversion by choice behavior Trial Gain Loss $1000 $2000 $350 $500 $1200 $60 $290... $800 $1000 $450 $100 $380 $55 $148... Choice (yes/no) 0 0 0 1 1 0 0... Method: LogisQc regression (a staqsqcal method) choice =! G Gain +! L Loss! G : how strong gains contribute to choice data! L : how strong losses contribute to choice data Degree of loss aversion! behavior =!! L! G

II. Modeling how the brain makes decisions Measuring loss aversion by neural acqvity BOLD Design matrix Gains Losses Beta $&'(")!!"#$% Time = X %$&'()!!"##$# Neural measure of loss aversion: neural! neural =!" Losses neural! " Gains

II. Modeling how the brain makes decisions Analysis focus 1. How does the brain represent gains and losses? - Prospect theory indicates posiqve correlaqon with gains, negaqve correlaqon with losses; if this is the case, then! $&'(")!"#$%! *+%#,#-&! %$&'()!"##$#! %$*(+,-$

II. Modeling how the brain makes decisions Analysis focus 2. Neural basis of loss aversion - An area driving (contribuqng to) loss aversion should exhibit a close match between loss aversion measured in behavior and loss aversion measured according to its neural acqvity (psychometric - neurometric match)!!"#$%&'(!! )"*($+

II. Modeling how the brain makes decisions Neural representaqon of gains and losses vmpfc and ventral striatum posiqvely correlated with gains and negaqvely correlated with losses

II. Modeling how the brain makes decisions Neural basis of loss aversion Neural measure of loss aversion in ventral striatum strongly correlated with behavioral measure of loss aversion

III. Modeling how the brain learns: Reinforcement learning models

III. Modeling how the brain learns Example: O Doherty et al. (2003) Pavlovian condiqoning task.... 3 sec 3 sec 3 sec 3 sec Taste delivery Taste delivery

III. Modeling how the brain learns Example: O Doherty et al. (2003) Pavlovian condiqoning task SQmulus- reward associaqons # trials delivery # trials no delivery 80 20 80 20 # trials = 80 *RandomizaQon on sqmulus order and delivery/no delivery

III. Modeling how the brain learns Example: O Doherty et al. (2003) Pavlovian condiqoning task SQmulus- reward associaqons # trials delivery # trials no delivery 80 20 80 20 # trials = 80 *RandomizaQon on sqmulus order and delivery/no delivery

III. Modeling how the brain learns Example: O Doherty et al. (2003) Pavlovian condiqoning task Behavioral results: The animals exhibit condiqoned response (salivate when seeing the sqmulus) aler experiencing the sqmulus- reward pairing

III. Modeling how the brain learns AcQvity of midbrain dopamine neurons (Schultz et al. 1997) Burst of firing aler reward delivery Burst of firing aler sqmulus; acqvity remained at baseline at reward delivery Burst of firing aler sqmulus; acqvity dipped down when no reward was delivered

III. Modeling how the brain learns QuesQon: How do we characterize the learning process (learning the associaqon between sqmulus and reward) that takes place in the brain?

III. Modeling how the brain learns ObservaQons: 1. Midbrain DA neurons first showed an increase in firing in response to reward delivery

III. Modeling how the brain learns ObservaQons: 2. When a sqmulus is paired with a reward, midbrain DA neurons gradually (over the course of experiment) shil their responses to the the sqmulus is presented

III. Modeling how the brain learns ObservaQons: 3. When a reward is expected aler a sqmulus is presented and when the reward is indeed delivered, no change in DA response

III. Modeling how the brain learns ObservaQons: 4. When a reward is expected aler a sqmulus is presented and when the reward is NOT delivered, there is a decrease in DA response

III. Modeling how the brain learns Example: O Doherty et al. (2003) Pavlovian condiqoning task QuesQons: How could we explain the 4 observaqons we just made about neural acqvity in midbrain DA neurons? How could we quanqtaqvely describe or even predict the neuronal response profiles?

III. Modeling how the brain learns Example: O Doherty et al. (2003) Pavlovian condiqoning task A soluqon: Apply a computaqonal learning model: Temporal difference (TD) model (Sugon & Barto, 1990)

III. Modeling how the brain learns Temporal difference (TD) learning Define a value for each moment in separately, V(t i ) v(t i ) = E " # r(t i ) +! r(t i + 1) +! 2 r(t i + 2) +... $ % 0! "! 1 discount parameter The value at t i is the sum of expected future rewards

III. Modeling how the brain learns Temporal difference (TD) learning V(t) = E " # r(t) +! r(t + 1) +! 2 r(t + 2) +... $ % can be expressed as V(t) = E "# r(t) +!V(t + 1) $ % Hence E!" r(t) # $ = V(t) ˆ % & V(t ˆ + 1)

III. Modeling how the brain learns Temporal difference (TD) learning UpdaQng occurs by comparing the difference between r(t) E!" r(t) # $ What actually occurs What is expected to occur V new (t) = V new (t) +! #$ r(t) " E[r(t)] % & PredicQon error!

III. Modeling how the brain learns Temporal difference (TD) learning UpdaQng equaqon V new (t) = V new (t) +! #$ r(t) " E[r(t)] % & Given E!" r(t) # $ = V(t) ˆ % & V(t ˆ + 1) We get V new (t) = V old (t) +! $% r(t) + "V old (t + 1) #V old (t)& ' learning rate 0! "! 1

III. Modeling how the brain learns Temporal difference (TD) learning UpdaQng equaqon V new (t) = V new (t) +! #$ r(t) " E[r(t)] % & V new (t) = V old (t) +! $% r(t) + "V old (t + 1) #V old (t)& ' PredicQon error!

Temporal difference (TD) learning Trial 1: III. Modeling how the brain learns Suppose at t 1 you see V trial1 (t 1 ) = V trial0 (t 1 ) +! $% r trial1 (t 1 ) + "V trial0 (t 2 ) #V trial0 (t 1 )& ' = 0 1 V 0 t 1 t 2 t 3 assume! = 0.5," = 0.99

Temporal difference (TD) learning Trial 1: III. Modeling how the brain learns t 2 FixaQon (nothing happened) For t 2 V trial1 (t 2 ) = V trial0 (t 2 ) +! $% r trial1 (t 2 ) + "V trial0 (t 3 ) #V trial0 (t 2 )& ' = 0 1 V 0 t 1 t 2 t 3

Temporal difference (TD) learning Trial 1: III. Modeling how the brain learns t 3 A reward is delivered For t 3 V trial1 (t 3 ) = V trial0 (t 3 ) +! $% r trial1 (t 3 ) + "V trial0 (t 4 ) #V trial0 (t 3 )& ' =! 1 V 0 t 1 t 2 t 3

Temporal difference (TD) learning Trial 1: III. Modeling how the brain learns t 4 fixaqon For t 4 V trial1 (t 4 ) = V trial0 (t 4 ) +! $% r trial1 (t 4 ) + "V trial0 (t 5 ) #V trial0 (t 4 )& ' = 0 1 V 0 t 1 t 2 t 3 t 4

Temporal difference (TD) learning Trial 2: III. Modeling how the brain learns t 1 V trial 2 (t 1 ) = V trial1 (t 1 ) +! $% r trial 2 (t 1 ) + "V trial1 (t 2 ) #V trial1 (t 1 )& ' = 0 1 V 0 t 1 t 2 t 3

Temporal difference (TD) learning Trial 2: III. Modeling how the brain learns t 2 fixaqon V trial 2 (t 2 ) = V trial1 (t 2 ) +! $% r trial 2 (t 2 ) + "V trial1 (t 3 ) #V trial1 (t 2 )& ' = "! 2 1 V 0 t 1 t 2 t 3

Temporal difference (TD) learning Trial 2: III. Modeling how the brain learns t 3 A reward is delivered V trial 2 (t 3 ) =V trial1 (t 3 ) +! $% r trial 2 (t 3 ) + "V trial1 (t 4 ) #V trial1 (t 3 )& ' = 2! #! 2 1 V 0 t 1 t 2 t 3

Temporal difference (TD) learning Trial 2: III. Modeling how the brain learns t 4 fixaqon V trial 2 (t 4 ) = V trial1 (t 4 ) +! $% r trial 2 (t 4 ) + "V trial1 (t 5 ) #V trial1 (t 4 )& ' = 0 1 V 0 t 1 t 2 t 3 t 4

Temporal difference (TD) learning Trial 3: III. Modeling how the brain learns t 1 V trial3 (t 1 ) = V trial 2 (t 1 ) +! $% r trial3 (t 1 ) + "V trial 2 (t 2 ) #V trial 2 (t 1 )& ' = " 2! 3 1 V 0 t 1 t 2 t 3

III. Modeling how the brain learns Temporal difference (TD) learning Trial 3: t 2 fixaqon V trial3 (t 2 ) = V trial 2 (t 2 ) +! $% r trial3 (t 2 ) + "V trial 2 (t 3 ) #V trial 2 (t 2 )& ' = ((3! 2 # 2! 3 ) 1 V 0 t 1 t 2 t 3

III. Modeling how the brain learns Temporal difference (TD) learning Trial 3: t 3 A reward is delivered V trial3 (t 3 ) = V trial 2 (t 3 ) +! $% r trial3 (t 3 ) + "V trial 2 (t 4 ) #V trial 2 (t 3 )& ' = 3! # 3! 2 +! 3 1 V 0 t 1 t 2 t 3

Temporal difference (TD) learning Trial 3: III. Modeling how the brain learns t 4 fixaqon V trial3 (t 4 ) = V trial 2 (t 4 ) +! $% r trial3 (t 4 ) + "V trial 2 (t 5 ) #V trial 2 (t 4 )& ' = 0 1 V 0 t 1 t 2 t 3 t 4

III. Modeling how the brain learns Temporal difference (TD) learning Trial 1: 1 V 0 t 1 t 2 t 3 t 4 Trial 2: 1 V 0 t 1 t 2 t 3 t 4 Trial 3: 1 V 0 t 1 t 2 t 3 t 4

III. Modeling how the brain learns Temporal difference (TD) learning Trial 4: 1 V 0 t 1 t 2 t 3 t 4 Trial 5: 1 V 0 t 1 t 2 t 3 t 4 Trial 6: 1 V 0 t 1 t 2 t 3 t 4

Temporal difference (TD) learning Let s look at predicqon error Recall that III. Modeling how the brain learns! V trial( X +1) (t) = V trialx (t) +! $ % r trial( X +1) (t) + "V trialx (t + 1) #V trialx (t)& '!

III. Modeling how the brain learns Just looking at Trial 1: t 3 1! 0 t 1 t 2 t 3 t 4 Trial 2: 1! 0 t 1 t 2 t 3 t 4 Trial 3: 1! 0 t 1 t 2 t 3 t 4

III. Modeling how the brain learns Just looking at t 3 Trial 4: 1! 0 t 1 t 2 t 3 t 4 Trial 5: 1! 0 t 1 t 2 t 3 t 4 Trial 6: 1! 0 t 1 t 2 t 3 t 4

III. Modeling how the brain learns Temporal difference (TD) learning 1 V 0 t 1 t 2 t 3 1! 0 t 4 t 1 t 2 t 3 t 4

III. Modeling how the brain learns Based on TD model, we can - Construct a General Linear Model (GLM) to analyze data ""##! =! $ +! % $ % "##+! & &"##+! " ""## Indicator variable Value at t PredicQon error TD model provides a quanqtaqve predicqon on the course of data

Modeling response dynamics: DriL diffusion model

IV. Modeling response dynamics AcQon selecqon is a dynamic process - MulQple alternaqves compete during this process 30% moqon coherence 5% moqon coherence QuesQon: how do we model the dynamics of neural acqvity during this process? Courtesy to Bill Newsome

IV. Modeling response dynamics Dynamics of neural acqvity during sqmulus presentaqon - AcQvity in area LIP rises up faster as moqon coherence level increases - Prior to eye movement, acqvity does not differ between different coherence trials Golad & Shadlen (2007, ARN)

IV. Modeling response dynamics Modeling response dynamics as an evidence accumulaqon process Firing rates behave as if neurons integrate momentary evidence over Each moment in : loglr(t i ) = log p(e(!,t i ) L) p(e(!,t i ) R)! = motion coherence Over :! i loglr(t 1,...t k ) = loglr(t i ) Golad & Shadlen (2007, ARN)

IV. Modeling response dynamics Use DriL diffusion model to characterize evidence accumulaqon and acqon selecqon Decision bound (e.g. lelward moqon) The point process drils as goes by Decision bound (e.g. rightward moqon) StarQng point (unbiased) Gold & Shadlen (2007, ARN)

IV. Modeling response dynamics Use DriL diffusion model to characterize evidence accumulaqon and acqon selecqon What determines the dril? Ans: Momentary evidence is sampled from a Gaussian distribuqon to determine the next step Gold & Shadlen (2007, ARN)

2024 © sciencedocbox.com Privacy Policy | Terms of Service | Feedback