logic is everywhere Logik ist überall Hikmat har Jaga Hai Mantık her yerde la logica è dappertutto lógica está em toda parte

Transcription

1 Simple Connectionist Systems Steffen Hölldobler logika je všude International Center for Computational Logic Technische Universität Dresden Germany logic is everywhere Connectionist Networks la lógica está por todas partes Connections Activation Functions Output Functions Units Logika ada di mana-mana Updates Winner-Take-All Networks lógica está em toda parte la logique est partout Hikmat har Jaga Hai logika je svuda Mantık her yerde la logica è dappertutto Logik ist überall Logica este peste tot Simple Connectionist Systems 1

2 Connectionist Networks A connectionist network consists of a finite set U of units and a finite set of connections W U U. Each connection is labelled by a weight (W R). A unit consists of an input (vector) (i 1,..., i m ) with i j R for all 1 j m, an activation function φ : R m R, a potential p = φ(i 1,..., i m ), an output function ψ : R R, and an (output) value v = ψ(p). Sometimes a linear time t N is added. Let U = {u i 1 i n} be a connectionist network. The state of U at time t is (v 1 (t),..., v n (t)), where v i (t) is the output of unit u i at time t for all 1 i n. Simple Connectionist Systems 2

3 Example u 1 v 1 u k u j. w k1 i k1 v j wkj w kj v j = i kj p k = Φ k (i k ) v k = Ψ k (p k ). i km um vm w km Simple Connectionist Systems 3

4 Connections Directed and weighted connection from u j to u k with weight w kj R generating input i k = w kj v j. Higher order connection from units u j1,..., u j m to unit u k with weight w kj1...jm R (wrt some ordering j 1 <... < j m ) Q m generating input i k = w kj1...jm l=1 v j l. A undirected and weighted connection between u j and u k consists of a directed and weighted connection from u j to u k and a directed and weighted connection from u k to u j, where w kj = w kj. Simple Connectionist Systems 4

5 Activation Functions Let u k U be a unit with inputs (i 1,..., i m ). Weighted sum activation function φ(i 1,..., i m, t + 1) = Sigma-pi activation function mx i j (t), where i j = w kj v j (t), j=1 φ(i 1,..., i m, t + 1) = mx i j (t), where i j = w kj1...jm j j=1 m j Y l=1 v jl (t). Simple Connectionist Systems 5

6 Some Output Functions Let u k U be a unit with potential p k. Let θ k R; θ k is called threshold. Binary threshold output function Ψ(p k ) = j 1 if pk θ k 0 otherwise. Binary bipolar threshold output function Ψ(p k, t) = j 1 1 if pk θ k otherwise. Linear output function: Φ(p k ) is linear. Simple Connectionist Systems 6

7 More Output Functions Let u k U be a unit with potential p k. Let θ k R be a threshold and β > 0 a steepness parameter. A function is a squashing function if it is non-constant, bounded, monotone increasing and continuous Examples Sigmoidal output function Ψ(p k ) = e β(p k θ k ). Bipolar Sigmoidal or hyperbolic tangent output function Ψ(p k ) = e 2p k 1 = tanh(p k). Simple Connectionist Systems 7

8 Threshold Units Binary threshold unit activation function: weighted sum, output function: binary threshold. Bipolar binary threshold unit activation function: weighted sum, output function: bipolar binary threshold. Sigma-pi unit activation function: sigma-pi, output function: binary threshold. A binary threshold unit is said to be active if its output is 1; it is said to be passive if its output is 0. A binary threshold unit flips if its output changes from 1 to 0 or from 0 to 1 between two consecutive time points; Flips for bipolar binary threshold units are defined likewise. Similar notions can be defined for bipolar threshold units. Simple Connectionist Systems 8

9 Squashing Units Sigmoidal unit activation function: weighted sum, output function: sigmoidal. Bipolar sigmoidal unit activation function: weighted sum, output function: bipolar sigmoidal. Simple Connectionist Systems 9

10 Linear Units Linear unit activation function: weighted sum, output function: linear. Simple Connectionist Systems 10

11 Input and Output Units Connectionist networks may be embedded in an environment. In this case, some of its units may receive additional input from the environment. These units are called input units. Inputs from the environment are real numbers. They are simply added to the input vector of input units, i.e., the input vector of an input unit consists of inputs received from other units and inputs received from the environment. If connectionist networks are embedded in an environment, then the external activation must be specified accordingly for each input unit. Likewise, output units are defined as units which send their output to the environment. Simple Connectionist Systems 11

12 Updates If a unit is updated then its potential and value are computed given the current inputs. In networks with linear time, updates yield potentials and values at t + 1 given inputs at time t. We will consider networks where either all units are updated synchronously or units are updated asynchronously. Simple Connectionist Systems 12

13 Example Consider the following connectionist network, where i 1 (t) = i 2 (t) = j 6 if t = 0 2 otherwise j 5 if t = 0 2 otherwise v j (t) = round(p j (t)) (j = 1, 2, 3, 4) j 0 if t = 0 p j (t) = p j (t 1) + P 4 k=1 w jkv k (t 1) otherwise j ff 0 if t = 0 p j (t) = (j = 1, 2) i j (t 1) otherwise i 2 i 1 u 2 u 1 ff 1 1 u 4 u (j = 3, 4) v 4 v 3 w 32 = w 41 = w 33 = w 44 = 0; all other weights are given in the figure. What happens if the network is synchronously updated? Simple Connectionist Systems 13

14 Winner-Take-All Networks A winner-take-all network is a synchronously updated connectionist network of n units such that after each unit receives an initial input at t = 0 (all other inputs are 0) eventually only the unit with the highest initial input outputs a value greater than 0 whereas the value of all other units is 0. Exercise Construct a winner-take-all network of 3 units (ignoring input units). Simple Connectionist Systems 14