Model of Neurons. CS 416 Artificial Intelligence. Early History of Neural Nets. Cybernetics. McCulloch-Pitts Neurons. Hebbian Modification.

Transcription

1 Page 1 Model of Neurons CS 416 Artfcal Intellgence Lecture 18 Neural Nets Chapter 20 Multple nputs/dendrtes (~10,000!!!) Cell body/soma performs computaton Sngle output/axon Computaton s typcally modeled as lnear Early Hstory of Neural Nets Eons ago: Neurons are nvented 1868: J. C. Maxwell studes feedback mechansms 1942: Wener et al. formulate cybernetcs 1943: McCulloch-Ptts Neurons 1949: Hebb ndcates bologcal mechansm 1962: Rosenblatt s Perceptron 1969: Mnsky and Papert decompose perceptrons Cybernetcs he theoretcal study of communcaton and control processes n bologcal, mechancal, and electronc systems, especally the comparson of these processes n bologcal and artfcal systems. ( McCulloch-Ptts Neurons One or two nputs to neuron Inputs are multpled by weghts If product exceeds a threshold, the neuron fres How would we create xor? Hebban Modfcaton When an axon of cell A s near enough to excte cell B and repeatedly or persstently takes part n frng t, some growth process or metabolc change takes place n one or both cells such that A s effcency, as one of the cells frng B, s ncreased from Hebb s 1949 he Organzaton of Behavor, p. 62

2 Each nput s bnary and has assocated wth t a weght Not gates are allowed he sum of the nner product of the nput and weghts s calculated If ths sum exceeds a threshold, the perceptron fres Perceptrons w Error Correcton = εx ( c Θ( x w )) Only updates weghts for non-zero nputs For postve nputs If the perceptron should have fred but dd not, the weght s ncreased If the perceptron fred but should not have, the weght s decreased For negatve nputs Behavor s opposte Example modfed from he Essence of Artfcal Intellgence by Alson Cawsey Intalze all weghts to 0.2 Let epslon = 0.05 and threshold = 0.5 Weghts Frst output s 1 snce >0.5 Should be 0, so weghts wth actve connectons are decremented by 0.05 Old w New w Next output s 0 snce <=0.5 Should be 1, so weghts wth actve connectons are ncremented by 0.05 New weghts work for Alson, Jeff, and Gal Old w New w Output for Smon s 1 ( >0.5) Should be 0, so weghts wth actve connectons are decremented by 0.05 Are we fnshed? Old w New w Page 2

3 Page 3 Class Exercse After processng all the examples agan we get weghts that work for all examples What do these weghts mean? In general, how often should we reprocess? Weghts Fnd w1, w2, and theta such that heta(x1*w1+x2*w 2)= x1 xor x2 Or, prove that t 2 nd Class Exercse 3 rd Class Exercse x3 = ~x1, x4 = ~x2 Fnd w1, w2, w3, w4, and theta such that heta(x1*w1+x2*w 2)= x1 xor x2 Or, prove that t Fnd w1, w2, and f() such that f(x1*w1+x2*w2) = x1 xor x2 Or, prove that t Mult-layered Perceptrons 4 th Class Exercse Input layer, output layer, and hdden layers Elmnates some concerns of Mnsky and Papert Modfcaton rules are more complcated! Fnd w1, w2, w3, w4, w5, theta1, and theta2 such that output s x1 xor x2 Or, prove that t

4 Page 4 Recent Hstory of Neural Nets Lmtatons of Perceptrons 1969 Mnsky & Papert kll neural nets 1974 Werbos descrbes back-propagaton 1982 Hopfeld renvgorates neural nets 1986 Parallel Dstrbuted Processng (Here s some source code: 624/) he report of my death s greatly exaggerated. Mark wan Mnsky & Papert publshed Perceptrons stressng the lmtatons of perceptrons Sngle-layer perceptrons cannot solve problems that are lnearly nseparable (e.g., xor) Most nterestng problems are lnearly nseparable Klls fundng for neural nets for years Back-Propagaton he concept of local error s requred We ll examne our smple 3-layer perceptron wth xor Intal weghts are random hreshold s now sgmodal (functon should have dervatves) Intal weghts: w1=0.90, w2=-0.54 w3=0.21, w4=-0.03 w5 = f ( x w) = 1+ e x w Cypher: It means, buckle your seatbelt, Dorothy, because Kansas s gong bye-bye. Input layer two unt Hdden layer one unt Output layer one unt Output s related to nput by r F w, x = f f x w Performance s defned as P = 1 F ( w r, x ) ( ) ( ( ) w ) ( c ) x, c 2 Error at last layer (hdden output) s defned as: δ 1 = ( F ( w, x ) c ) Error at prevous layer (nput hdden) s defned as: δ = w k o k ( 1 o ) δ k k Change n weght: x, c w = β x, c 2 Where: x, c = o o ( 1 o ) δ 2 I hate math... so lttle room to make small errors. Caleb Schaefer, UGA student

5 Page 5 (0,0) 0 1st example Input to hdden unt s 0, sgmod(0)=0.5 Input to output unt s (0.5)(-0.03)= Sgmod(-0.015)= error= δ So, o = = (0.5)(0.4963)( )( ) = Example s contrbuton to w 4 s Why are we gnorng the other weght changes? (0,1) 1 2nd example h =-0.54 o h = o =(0.3862)(-0.03)+0.78=0.769 o o = δ = = o 4 5 = ( )( )( )( ) = = (1)( )( )( ) = δ = ( 0.03)( )( )( ) = h 2 = (1)( )( )( ) = &c Hopfeld Nets Intal performance = After 100 teratons we have: w=(0.913, , 0.036, , 0.288) Performance = After 100K teratons we have: w=(15.75, , 7.146, , ) Performance = After 1M teratons we have: w=(21.38, , 9.798, , ) Performance = Created neural nets that have contentaddressable memory Can reconstruct a learned sgnal from a fracton of t as an nput Provded a bologcal nterpretaton What s the Purpose of NN? Quck Lst of erms o create an Artfcal Intellgence, or Although not an nvald purpose, many people n the AI communty thnk neural networks do not provde anythng that cannot be obtaned through other technques o study how the human bran works? Ironcally, those studyng neural networks wth ths n mnd are more lkely to contrbute to the prevous purpose Presynaptc Modfcaton: Synapse weghts are only modfed when ncomng (afferent) neuron fres Postsynaptc Modfcaton: Synapse weghts are only modfed when outgong (efferent) neuron fres Error Correcton: Synapse weghts are modfed relatve to an error can be pre- or postsynaptc; requres some form of feedback Self-supervsed: Synapse weghts are modfed relatve to nternal exctaton of neuron can be pre- or postsynaptc

6 Page 6 Self-supervsed Neurons More Self-Supervson One example s a neuron that has the followng synaptc modfcaton rule: w = εy E E ( x w ) y = x w = x w 0 = E[ εx y ] E[ εy w ] [ xx w ] = E[ y w ] [ xx ] w = E[ y ] w Internal exctaton Convergence of weghts Egenvalue equaton! Prevous rule could not learn to dstngush between dfferent classes of data However, f the rule s modfed to: w = εθ( y )( x w ) he neuron wll learn to only respond to a certan class of nputs Dfferent neurons respond to dfferent classes Some Bran Facts Contans ~100,000,000,000 neurons Hppocampus CA3 regon contans ~3,000,000 neurons Each neurons s connected to ~10,000 other neurons ~1,000,000,000,000,000 (10 15 ) connectons! Contrary to a BranPlace.com, ths s consderably less than number of stars n the unverse to Consumes ~20-30% of the body s energy Contans about 2% of the body s mass