CS 256: Neural Computation Lecture Notes

Transcription

1 CS 56: Neural Computation Lecture Notes Chapter : McCulloch-Pitts Neural Networks A logical calculus of the ideas immanent in nervous activity M. Minsky s adaptation (Computation: Finite and Infinite Machines, 967). Implementing Boolean functions. Networks with feedback. The binary scaler: a network that counts. Sequence and parallel decoders Serial and parallel encoders Finite state machines Kleene s Theorem: Neural networks can simulate any finite state machine. Copyright Robert R. Snapp 5, 8. Compiled with LAT E X at 9:34 PM on January 3, 8.

2 McCulloch-Pitts Neural Networks Warren S. McCulloch and Walter H. Pitts, A logical calculus of the ideas immanent in nervous activity, Bulletin of Mathematical Biophysics, 5, 943, pp ABSTRACT Because of the all-or-none character of nervous activity, neural events and the relations among them can be treated by means of propositional logic. It is found that the behavior of every net can be described in these terms, with the addition of more complicated logical means for nets containing circles; and that for any logical expression satisfying certain conditions, one can find a net behaving in the fashion it describes. It is shown that many particular choices among possible neurophysiological assumptions are equivalent, in the sense that for every net behaving under one assumption, there exists another net which behaves under the other and gives the same results, although perhaps not in the same time. Various applications of the calculus are discussed. [The original paper is reprinted in J. Anderson and E. Rosenfeld ed., Neurocomputing: Foundations of Research, vol., MIT Press, Cambridge, MA, 988, on reserve for CS 56 in the Bailey-Howe Library. Some biographical information about Walter Pitts can be found in J. A. Anderson and E. Rosenfeld, ed., Talking Nets, MIT Press, Cambridge, MA, 998, pp..]

3 Fundamental Assumptions We shall make the following physical assumptions for our calculus.. The activity of the neuron is an all-or-none process.. A certain fixed number of synapses must be excited within the period of latent addition in order to excite a neuron at any time, and this number is independent of previous activity and position on the neuron. 3. The only significant delay within the nervous system is synaptic delay. 4. The activity of any inhibitory synapse absolutely prevents excitation of the neuron at that time. 5. The structure of the net does not change with time. From W. McCulloch and W. Pitts (943). McCulloch-Pitts (943) influenced both theoretical neuroscience as well as computer science. The McCulloch-Pitts neuron became the basic unit for the earliest formal studies of finite automata and regular expressions. (See, for example, C. E. Shannon and J. McCarthy, ed. Automata Studies, Princeton University Press, Princeton, NJ, 956). We will follow the more accessible exposition found in M. Minsky, Computation: Finite and Infinite Machines, Prentice-Hall, NJ, 967, on which the following is based.

4 Minsky s adaption Assume a simple McCulloch-Pitts unit (MPU), with r excitatory and s inhibitory synapses, and threshold k Z +, x (t) x r (t) x r+ (t) x r+s (t) k y(t) Assume that x i {, }, for i =,..., r + s. Define if u Θ(u) = otherwise. Then, the output y is y(t + ) = Θ r i= x i (t) k s j= ( ) x r +j (t). M. Minsky, Computation: Finite and Infinite Machines, 967, Chapter 3.

5 Boolean Functions: Conjunctions x (t) x (t) y(t) x (t) x r (t) r y(t) y(t + ) = x (t) x (t) y(t + ) = x (t) x r (t) AND elements.

6 Boolean Functions: Disjunctions x (t) x (t) y(t) x (t) x r (t) y(t) y(t + ) = x (t) x (t) y(t + ) = x (t) x r (t) OR elements.

7 Boolean Functions: Negation x(t) y(t) x (t) x r (t) y(t) y(t + ) = x(t) NOT element. y(t + ) = x (t) x r (t) = x (t) x r (t) Multichannel NOR element.

8 Other gates x x y x x x 3 y y(t + ) = x (t) x (t) AND NOT element. y(t + ) = (x (t) x (t)) (x (t) x 3 (t)) (x (t) x 3 (t)) = majority (x (t), x (t), x 3 (t)) MAJORITY element.

9 Identity Function and Delays Lines x(t) y(t)=x(t )

10 Identity Function and Delay Lines x(t) y(t)=x(t ) x(t) x(t l) l units

11 Networks with feedback y(t) set reset output bit register

12 Networks with feedback y(t) if t is odd y(t) = if t is even set reset bit register output

13 Gating networks control line control line

14 A multi-throw switch (router) Receiver Transmitter Receiver

15 Binary scaler A Start AB x(t) B y(t) C A C Neural Network State Transition Diagram

16 Binary scaler Start A AB x(t) B y(t) C A C Neural Network State Transition Diagram x(t) A B C

17 Binary Counter A A A 3 x(t) B B B 3 y(t) C C C 3

18 Binary Counter A A A 3 x(t) B B B 3 y(t) C C C 3 x(t) B B B 3

19 Sequence Decoder x(t) start

20 Parallel Decoder x 3 (t) x (t) x (t) 3

21 Serial Encoder x(t) y(t)

22 Parallel Encoder x(t) y (t) y (t) y 3 (t) y 4 (t)

23 Number of States? Question: What is the maximum number of states that can be represented by a neural network that contains N units?

24 Number of States? Question: What is the maximum number of states that can be represented by a neural network that contains N units? N

25 Finite State Machines M is a machine with a finite number of internal states, Q, Q,..., Q p, that is coupled to an environment. S Environment At time t, M receives a stimulus signal S(t), and emits a response signal R(t). S and R can only assume a finite number of distinguishable values. Assume that all transitions occur at discrete moments of time (epochs), t =,,.... Let Q(t) denote the state of machine M at time t. R(t + ) = F(Q(t), S(t)) Q(t + ) = G(Q(t), S(t)) M R

26 Theorem 3.5, Minsky, page 55 Theorem: Every finite-state machine is equivalent to, and can be simulated by, some finite neural network. Proof: For a given finite-state machine, let Q, Q,..., Q p denote each distinguishable internal state, R, R,..., R n, each distinguishable output (response) value, S, S,..., S m, each distinguishable input (stimulus) value. An equivalent neural network is constructed with p columns on m neurons with threshold, plus n neurons with threshold, and n output channels, and m input channels.

27 Finite-state Neural Network G(Q,S ) = Q S Q Q Q 3 S S 3 F(Q,S ) = R R R

28 Remarks All connections in this realization are excitatory. Are inhibitory connections necessary? Fan-in and threshold requirements: But interesting functions usually have features that permit enormous reductions in the sizes of the nets required to realize them. (p. 55) AND and OR gates have a monotonic stimulus-response property: It is impossible to make the output smaller by making the input larger. Universal sets of cells: NOR with DELAY, NAND with DELAY, etc.

29 NOR with DELAY form a universal set x x x x x x x x x x x x x x x x x x x x x x x x Extra delays can be appended to ensure a processing time of three time steps for each function. Can you construct the remaining eight Boolean functions of x and x using only NORs and DELAYs?)

30 von Neumann s double-line trick a a a a b b b b AND OR a a a b b a AND NOT NOT If every signal a is always accompanied by its complement, a then von Neumann showed that every Boolean operations can be implemented using a basis of ANDs and ORs, as illustrated above. (See, C. E. Shannon and J. McCarthy, ed., Automata Studies, Princeton University Press, Princeton, NJ, 956.)

31 Regular Expressions a a b c d R b c d R S S

32 Regular Expressions a a b c d R b c d R S S Recognizes any sequence of the form a, aa, aaa,..., aa a, which we will denote as a+. Recognizes any sequence that ends in a, which we might denote as [a z] a.

33 Regular Expressions a b c d S R

34 Regular Expressions a b c d S R Recognizes any sequence of the form ab, abab, ababab,..., which we can denote as (ab)+.

35 Regular Expressions: Definition (Minsky, p. 7) DEFINITION OF THE CLASS OF REGULAR EXPRESSIONS. Any letter symbol x, alone, is a regular expression.. If E and F are regular expressions, then so is (EF). 3. If E, F,..., G are regular expressions, then so is (E F G). 4. If E is a regular expression, then so is E. [Here, E denotes a sequence of zero or more consecutive occurrences of expression E, i.e., E ɛ E EE EEE, where ɛ denotes null. We also let E+ denote a sequence of one or more occurrences of E, i.e., E+ EE.] 5. The regular expressions are all of those defined by the above rules, and no others [with the proviso that pairs of matching parentheses can be removed if there is no ambiguity]. 6. The regular sets of sequences are all sets of sequences defined by the above representation rules. Kleene s theorem (956): Any set of expressions recognized by a finite-state machine (i.e., a McCulloch-Pitts network) is regular; and any regular set can be recognized by some finite-state machine.