THE MAN WHO MISTOOK HIS COMPUTER FOR A HAND

Transcription

1 THE MAN WHO MISTOOK HIS COMPUTER FOR A HAND Neural Control of Robotic Devices ELIE BIENENSTOCK, MICHAEL J. BLACK, JOHN DONOGHUE, YUN GAO, MIJAIL SERRUYA BROWN UNIVERSITY Applied Mathematics Computer Science Neuroscience

2 XEROX ALTO 1973 Dell workstation, 2001

3 BRAIN-COMPUTER INTERFACES Mad scientist Nancy Reagan Brain If I could find a code which translates the relation between the reading of the encephalograph and the mental image the brain could communicate with me. Curt Siodmak, 1942

4 NEURAL PROSTHETICS Sensation Action

5 NEURAL PROSTHETICS Sensation Action

6 NEURAL PROSTHETICS Signals? Representation, inference, and algorithms? Robot control? < 200 ms

7 VISUALIZING THE PLAYERS Single cells of the nervous system NEURON

8 BRAIN VERSUS COMPUTER 100,000,000,000 Neurons Computational Elements 50,000,000 Transistors (P IV) Speed (operations/second/element) * 10 9

9 MASSIVE CONNECTIVITY SYNAPSES

10 LONG DISTANCE CALLS Conturo et al., 1999

11

12

13

14 THE EEG

15 SINGLE UNIT ACTIVITY Spikes 2/1000 s second 1/10 mm

16 CELL ENSEBLES B Central Sulcus MI SMA PMA PMA Arcuate 5 mm 80µV 1 ms 100 electrodes, 400µm separation 4x4 mm Implanted in the MI arm area of motor cortex - lacks strict somatopy

17 Neural Implant Acyrlic Silicone Bone Connector Arachnoid space Dura I 500 µm 400 µm Cortex III V VI White Matter Chronically implanted. Stable recording for 2-3 years (but not necessarily the same cells every day)

18 PATTERNS IN NEURAL ACTIVITY

19 LANGUAGE OF THE BRAIN Language of the brain. Language of the computer. Interpretation Translation

20 AMBIGUOUS SIGNALS

21 INFERENCE Ambiguous measurements Prior knowledge about how brains work. Prior knowledge about the environment History of measurements and interpretations Inference?

22 GOALS * Model neural activity in motor cortex. * Explore how the brain codes information. * Model the statistical relationship between neural activity and action. * Develop new statistical methods for analyzing neural codes. * Build prosthetic devices to assist the severely disabled. * Explore new output devices that can be controlled by brains.

23 MODELING NEURAL FUNCTION Simultaneously record hand position, velocity, and neural activity in motor cortex.

24 ARM MOTION Distribution of training motions: Position Velocity

25 MODELING NEURAL FUNCTION Spike trains time direction speed Cell 3 speed, r, (cm/s) direction θ (radians)

26 MODELING NEURAL FUNCTION Spike trains time direction speed Cell 3 speed, r, (cm/s) direction θ (radians)

27 NEURAL ACTIVITY speed, r, (cm/s) Cell 3 Cell 16 Cell π π direction θ (radians) Is there some true underlying response function?

28 NON-PARAMETRIC MODEL f v g v : Observed mean firing rate for velocity v : True mean firing rate for velocity v v = [ r, θ ] T f v is a noisy realization of the model g v g v f v Infer from using Bayesian inference. η p( g f) = v ( κ p( fv gv) i= 1 p( gv gv i )) likelihood spatial prior

29 LIKELIHOOD Observed firing rate modeled a sample from Poisson: p P ( f g) 1 = f! g f e g

30 SPATIAL PRIOR Markov Random Field assumption p( g v g) = η i= 1 p( g v g v i ) T-distribution. p( g) = π ( σ 3 2σ + g 2 2 ) 2

31 OPTIMIZATION Many ways to maximize over g v p ( g f) = κ p( f g ) p( g g) v Simulated annealing, etc. We exploit an approximate deterministic regularization method. Take the negative log of p( g f ) Minimize using gradient descent Not ideal (loopy propagation, see Yedidia, Freeman, & Weiss, NIPS 00). v v v

32 Poisson+Robust 12 Cell 3 Cell 16 Cell 19 0 π angle (radians) π

33 INFERENCE FROM ACTIVITY t direction? Inference speed Neural Data Prediction

34 INFERENCE FROM ACTIVITY t direction? speed prior knowledge Neural Data Prediction

35 GOALS sound probabilistic inference causal estimate over short time intervals to reduce lag cope with non-linear dynamics of hand motion cope with ambiguities (multi-modal distributions) more realistic firing models (Poisson or Poisson+refractory period [Kass&Venture, Neural Comp. 01]) support higher level analysis of activities

36 BAYESIAN INFERENCE p( st Ct ) = κ 2 p( ct st ) p( st Ct 1) 1 = ci, p( c t st ) gst c! i= cells i, t likelihood t e g st Want to infer state of hand s t = [ r, θ ] given the activity, C t, of (~25 cells) up to time t.

37 prior = ) ( ) ( ) ( t t t t t t t d p p p s C s s s C s Temporal dynamics (constant velocity) ) ( ) ( ) ( 1 2 = t t t t t t p p p C s s c C s κ BAYESIAN INFERENCE

38 PARTICLE FILTER Represent posterior with a discrete set of N states and their normalized likelihood. Posterior p( s t 1 C t 1) sample Temporal dynamics p( st st 1) sample p( s t C t) Posterior normalize p( c s ) t t Likelihood

39 PARTICLE FILTER Isard & Blake 96

40 1000 SYNTHETIC YNTHETIC CELLSC Particle filtering in 50 ms: v x : r 2 = v y : r 2 =

41 A NEURAL PROSTHETIC mathematical algorithm Voluntar y control signal EEG single and multineuron activity Computer cursor and keyboard entry Robotic arm Stimulation of Muscles, Spinal Cord, and Brain

42 NEURAL-PROSTHETIC LIMBS

43 HYBRID SYSTEMS Sensory input Motor control Connecting Brains to Robots, Reger et al, Artificial Life, 2000.

44 BRAIN/MACHINE HYBRIDS Explore biological sensory/control systems with artificial systems. Develop computational models of biological control. Re-map input modalities. Opportunity for robotic prostheses. Augment limited neural control with autonomy (eg. obstacle avoidance).

45 OUR BODIES OURSELVES? Service robots under neural control. Sensation and action at a distance. Stimulating the brain. Ethics, liminality, fear, and the uncanny. Probotics/Jim Judkis Honda robot

46 THANKS D. Sheinberg, Neuroscience N. Hatsopoulos, Neuroscience W. Patterson, Engineering A. Nurmikko, Engineering G. Friehs, Brown Medical School S. Geman, Division of Applied Mathematics M. Fellows, Neuroscience L. Paninski, NYU, Center for Neural Science N.K. Logothetis, Max Planck Institute, Tuebingen