Neural Computation, Analogical Promiscuity, and the Induction of Semantic Roles: A Preliminary Sketch Simon D. Levy Washington & Lee University Lexington, Virginia, USA Ross Gayler Melbourne, Australia
Inspiration(s) [I]t turns out that we don t think the way we think we think!... The scientific evidence coming in all around us is clear: Symbolic conscious reasoning, which is extracted through protocol analysis from serial verbal introspection, is a myth. J. Pollack [W]hat kinds of things suggested by the architecture of the brain, if we modeled them mathematically, could give some properties that we associate with mind? P. Kanerva
Two Dogs of Empiricism* 1. The Short Circuit (Localist) Approach i) Traditional models of phenomenon X (language) use entities A, B, C,... (Noun Phrase, Phoneme,...) ii) We wish to model X in a more biologically realistic way. iii) Therefore our model of X will have a neuron (pool) for A, one for B, one for C, etc. * with apologies to W.V.O Quine
E.g. Neural Blackboard Model (van der Velde & de Kamps 2006)
Benefits of Localism (Page 2000) Transparent (one node, one concept) Supports lateral inhibition / winner-takes all
Problems with Localism Philosophical problem: fresh coat of paint on old rotting theories (MacLennan 1991): what new insights does neuro-x provide? Engineering problem: need to recruit new hardware for each new concept/ combination leads to combinatorial explosion (Stewart & Eliasmith 2008)
The Appeal of Distributed Representations (Rumelhart & McClelland & al. 1986)
WALKED WALK
ROARED ROAR
SPOKE SPEAK
WENT GO
Two Dogs of Empiricism 2. The Homunculus problem, a.k.a. Ghost in the Machine(Ryle 1949) In cognitive modeling, the homunculus is the researcher: supervises learning, hand-builds representations, etc.
Beyond Associationism Mary won t give John the time of day. ignores(mary, john)
The Binding Problem +????
The Problem of Two +?
The Problem of Variables ignores(x, Y) X won t give Y the time of day.
Vector Symbolic Architectures (Plate 1991; Kanerva 1994; Gayler 1998)
Tensor Product Binding (Smolensky 1990)
Binding
Bundling + =
Unbinding (query)
Lossy
Lossy
Cleanup Hebbian / Hopfield / Attractor Net
Reduction (HRR)
Reduction (MAP/BSC)
Composition / Recursion
Variables john X
Recent Applications Modeling Surface and Structural Properties in Analogy Processing (Eliasmith & Thagard 2001) Variables & Quantification / Wason Task (Eliasmith 2005) Representing Word Order in a Holographic Lexicon (Jones & Mewhort 2007)
Banishing the Homunculus
Step I: Automatic Variable Substiution If A is a vector over {+1,-1}, then A*A = vector of 1 s (multiplicative identity) Supports substitution of anything for anything: everything (names, individuals, structures, propositions) can be a variable)!
What is the Dollar of Mexico? (Kanerva, to appear) Let X = <country>, Y = <currency>, A = <USA>, B = <Mexico> Then A = X*U + Y*D, D*A*B = D*(X*U + Y*D) * (X*M + Y*P) = (D*X*U + D*Y*D) * (X*M + Y*P) = (D*X*U + Y) * (X*M + Y*P) = D*X*U*X*M + D*X*U*Y*P + Y*X*M + Y*Y*P = P + noise B = X*M + Y*P
Learning Grammatical Constructions from a Single Example (Levy, to appear) Given Meaning: KISS(MARY, JOHN) Form: Mary kissed John Lexicon: KISS/kiss, MARY/Mary,... What is the form for HIT(BILL, FRED)?
Learning Grammatical Constructions from a Single Example (Levy, to appear) (ACTION*KISS + AGENT*MARY + PATIENT*JOHN) * (P1*Mary + P2*kissed + P3*John) * (KISS*kissed + MAY*Mary + JOHN*John + BILL*Bill + FRED*Fred + HIT*hit) * (ACTION*HIT + AGENT*BILL + PATIENT*FRED) =... = (P1*Bill + P2*hit + P3*Fred) + noise
Step II: Distributed Lateral Inhibition Analogical mapping as holistic graph isomorphsm (Gayler & Levy, in progress) A P B Q C D R S cf. Pelillo (1999)
A B P Q C D R S Possibilities x: A*P + A*Q + A*R + A*S +... + D*S Evidence w: A*B*P*Q + A*B*P*R +...+ B*C*Q*R +.. + C*D*R*S X*W = A*Q + B*R +... + A*P +... + D*S
xt w * cleanup / πt xt+1 c c
Step III: Automatic (De)composition of Entities MSC (Arathorn 2002)