IST 4 Information and Logic

T = today x= hw#x out x= hw#x due mon tue wed thr fri 30 M 6 oh M oh 3 oh oh 2M2M 20 oh oh 2 27 oh M2 oh midterms Students MQ oh = office hours Mx= MQx out 4 3 oh 3 4 oh oh oh presentations Mx= MQx due 8 oh oh 4 5 25 T oh 5 oh oh oh

MQs. Everyone has a gift! (Tuesday) 2. Memory (Thursday)

Tuesday, 6/2, 2:30pm. Christopher Haack: The gift of resilience 2. Joon Lee: Settling is not an option 3. Spencer Strumwasser: The gift of dyslexia 4. Richard Zhu: The gift of memory 5. Ah Ashwin Hari: The gift of musical composition 6. Jessica Nassimi: Evolution a gift in disguise 7. Serena Delgadillo: The gift of self-expression 8. Megan Keehan: Gift of motherliness 9. Zane Murphy: Grandmother and the piano

Thursday, 6/4, 2:30pm. Connor Lee: Memory is a fickle thing blessing or curse 2. Pallavi Aggarwal: The wonders of human memory 3. Peter Kundzicz and Anshul Ramachandran: Muscle memories 4. Siva Gangavarapu: A cultural retrospection 5. Philip Liu: The light of other days 6. Jason Simon: Math and Broadway 7. Yujie Xu: Memory v.s. ESL 8. Celia Zhang: When memory sours

Last Lecture Gates and circuits AON: AND, OR, Not LT: Linear Threshold LT: Linear Threshold > > a b c a b > > a b c a >b > > > b c a b c a b c > > > > b c a b c a b c

Last Lecture AON: AND, OR, Not LT: Linear Threshold General construction for symmetric functions AON 5 LT-l 4 LT-nl 2 * Exponential gap in size What are the symmetric functions that can be computed by a single LT gate? * = it is optimal * *

Linear Threshold and SYM

LT: Linear Threshold

Symmetric Functions and LT Circuits Q: Which class has more functions? Q: How is SYM related to LT?? Definitions: () SYM = the class of Boolean symmetric functions (2) LT = the class of Boolean functions that can be (2) LT the class of Boolean functions that can be realized by a single LT gate.

AND, OR, XOR and MAJ are symmetric functions Q: Which h symmetric functions are in LT?? X AND OR XOR MAJ 0 0 0 0 0 0 0 2 0 0 3 LT not LT LT LT LT = the class of Boolean functions that can be realized by a single LT gate.

Definition: A symmetric ti Boolean function is in TH if it has at most a single transition in the symmetric function table = a transition X AND OR XOR MAJ 0 0 0 0 0 0 0 2 0 0 3 In TH Not in TH

The Class TH is in LT X TH0 TH TH2 TH3 TH0 TH TH2 TH3 0 0 0 0 0 0 0 0 0 2 0 0 0 0 3 0 0 0 0

Q: How is TH related to SYM and LT?? We know that: We Proved that: SYM TH LT

TH is exactly in the intersection of SYM and LT Theorem: Proof: Not today... you might want to try and prove it... Q: What are the 4 functions? SYM TH LT???

LT Function that is not Symmetric - - 0 0 0 0 - -2 0-0 0 0

Linear Threshold l Circuits for symmetric functions

AON 5 LT-l 4 LT-nl 2 General construction for symmetric functions

X XOR Q: compute XOR 0 0 2 0 with TH gates? X TH TH2 TH+TH2-0 0 0 2 0 0

LT Depth-2 Circuits TH - + TH2 X TH TH2 TH+TH2-0 0 0 2 0 0

Generalization X f(x) 0 0 2 3 0 4 0

Generalization X f(x) TH 0 0 0 2 3 0 4 0

Generalization X f(x) TH TH3 0 0 0 2 3 0 0 4 0 0

Generalization X f(x) TH TH3 Σ - 0 0 0 0 2 3 0 0 0 4 0 0 0

X f(x) TH TH3 Σ - 0 0 0 0 2 3 0 0 0 4 0 0 0

Generalization to EQ 0 0 n 0 0 0 0 0

Generalization to EQ 0 0 2 0 0

Generalization to SYM - + Q: What is the generalization to arbitrary symmetric functions?

Generalization to SYM Q: What is the generalization to arbitrary symmetric functions? A: Consider the symmetric function table, it is a sum of non-overlapping -intervals 0 0 Sum of two TH functions

Back to XOR n TH gates for XOR of n variables 0 0 2 0 3 4 0 5

LT-l Circuit Design Algorithm for SYM f(x) 0 2 0 3 4 5 0 6 7 Subtract for every isolated -block

The Layered Construction for SYM -Some History Saburo Muroga 925-2009 959 Was born in Japan Majority Decision PhD in 958 from Tokyo U, Japan 960-964: Researcher at IBM Research, NY 964-2002: professor at the University of Illinois, Urbana-Champaign

Saburo Muroga 925-2009 HW#5 problem 2a

neural circuits and logic some more history...

Being Homeless and Interdisciplinary Research Warren McCulloch 899-969 Walter Pitts 923-969 Neurophysiologist, MD Logician, Autodidact Warren McCulloch arrived in early 942 to the University of Chicago, invited Pitts, who was homeless, to live with his family In the evenings McCulloch and Pitts collaborated. Pitts was familiar with the work of Leibniz on computing. They considered the question of whether the nervous system is a kind of universal computing device as described by Leibniz This led to their 943 seminal neural networks paper: A Logical Calculus of Ideas Immanent in Nervous Activity

Impact Warren McCulloch Walter Pitts 899-969 923-969 Neurophysiologist, MD Logician, Autodidact This led to their 943 seminal neural networks paper: p A Logical Calculus of Ideas Immanent in Nervous Activity Neural networks and Logic Time Memory Threshold Logic and Learning State Machines

neural circuits and memory m computing with dynamics

Linear Threshold Some Adjustments Linear Threshold (LT) gate -t threshold -t -

AND Function with {0,} -2 0 0 0-2 - 0 0 0-0 0

AND Function with {-,} The AND function of two variables with {-, }:??? -- - - -3- - - - - +

AND Function with {-,} The AND function of two variables with {-, }:??? -- - - -3 - -

Linear Threshold with Memory Elephants are symbols of wisdom in Asian cultures and are famed for their exceptional memory A memory nose Remembers the last f(x)

Feedback Networks Example - 0-0 weights thresholds Th t t f th t k th t th t d The state of the network: the vector that corresponds to the states (noses ) of the gates

Feedback Networks Example Label the gates 2-0 - 0

Feedback Networks Example 2-0 - 0

Feedback Networks Example 2-0 - - 0

Feedback Networks Example 2 - - - 0-0 - is a stable state

Feedback Networks Example 2-0 - 0

Feedback Networks Example 2-0 - 0 -

Feedback Networks Example - 2-0 - 0 - - is a stable state

Feedback Networks Example 2-0 - 0 state The node that computes State transition diagram (state space) - 2 - -- Q: Is -- a stable state?

Feedback Networks Example - 2-0 - - 0 - Answer: No Q: Is -- a stable state?

Feedback Networks Example - 2-0 - 0 - - 2 - --

Feedback Networks Example 2 - - 0 - - 0 - - 2 - --

Feedback Networks Example 2 - - - 0-0 - 2 2 - --

Feedback Networks Example 2-0 - - 0 stable states - 2 2 - --

neural circuits and memory m associative memory

Feedback Networks Computing with Dynamics stable states - 2 - -- 2 Input: initial state Feedback Network Output: stable state -

Input: initial state Feedback Networks Computing with Dynamics Associative Memory The Leibniz-Boole Machine Output: stable state Feedback Network

Who is this person?????

John Hopfield Feedback Networks Hopfield Model (Caltech 982)

John Hopfield Feedback Networks Hopfield Model (Caltech 982) 2-0 - 0 i = node i - 2 0 0-0 = threshold ti = state vi - = weight of edge (i,j)

The matrix description

Feedback Networks The Vector/Matrix Description An n node feedback network can be specified by: W an nxn matrix of weights T an n vector of thresholds V an n vector of states 2 3 4 5

The Matrix Description Example An n node feedback network can be specified by: W an nxn matrix of weights T an n vector of thresholds V an n vector of states - 2 0 0 - - - 2

The Matrix Description Computation Computation ti in N= (W,T) 2 3 4 by column 5

Order of computation serial and parallel

Modes of Operation Q: when do the nodes compute? Serial mode: one node at a time (arbitrary order) - - 2

Modes of Operation Q: when do the nodes compute? Serial mode: one node at a time (arbitrary order) - - 2 Fully-Parallel mode: all nodes at the same time - - 2

Three examples

Example Serial Mode Symmetric Weight Matrix - - 2 The state space: stable states - 2 - -- 2

Example 2 Fully-Parallel (FP) Mode Symmetric Weight Matrix Q: how does the state space look? start with - - 2 It s a cycle!

Example 2 Fully-Parallel (FP) Mode Symmetric Weight Matrix - - 2 The state space: stable states - - -- cycle of length 2

Example 23 Fully-Parallel Mode Antisymmetric Symmetric Weight Matrix W T = WW - - 2 Q: how does the state space look?

Example 3 Fully-Parallel Mode Antisymmetric Weight Matrix - Q: how does the state space look? 2 cycle of length 4

Example 3 Fully-Parallel Mode Antisymmetric Weight Matrix - 2 The state space: - cycle of length 4 - --

The Three Cases - - - - -- - -- - -- 2 3 Cycle lengths mode W symmetric antisymmetric Example # serial? fully-parallel,2 4 2 3

The Three Cases Cycle lengths Example # mode W symmetric antisymmetric serial? fully-parallel,2 4 2 3 Hopfield 982 2 Goles 985 3 Goles 986

Proof Ideas Cycle lengths W symmetric antisymmetric mode Example # serial? fully-parallel,2 4 2 3 The proofs of these three results use the concept of an energy function For the serial mode: Show that: t Namely, stable states are local max of the energy E

Questions on Convergence Posted on the class web site Cycle lengths mode W symmetric antisymmetric Example # serial? Hopfield 982 fully-parallel,2 4 2 3 2 Goles 985 Q: Are the three cases distinct? 3 Goles 986 Q2: Elementary proof? (wo/energy)