Learning Multiplicative Interactions

Transcription

1 CSC Lecture 6a Learning Multiplicative Interaction Geoffrey Hinton

2 Two different meaning of multiplicative If we take two denity model and multiply together their probability ditribution at each point in data-pace, we get a product of expert. The product of two Gauian expert i a Gauian. If we take two variable and we multiply them together to provide input to a third variable we get a multiplicative interaction. The ditribution of the product of two Gauianditributed variable i NOT Gauian ditributed. It i a heavy-tailed ditribution. One Gauian determine the tandard deviation of the other Gauian. Heavy-tailed ditribution are the ignature of multiplicative interaction between latent variable.

3 The heavy-tailed world The prediction error for financial time-erie are typically heavy-tailed. Thi i mainly becaue the variance i much higher in time of uncertainty. The prediction error made by a linear dynamical ytem are uually heavy-tailed on real data. Occaional very weird thing happen. Thi violate the condition of the central limit theorem. The output of linear filter applied to image are heavytailed. Gabor filter nearly alway output almot exactly zero. But occaionally they have large output.

4 Learning multiplicative interaction It i fairly eay to learn multiplicative interaction if all of the variable are oberved. Thi i poible if we control the variable ued to create a training et (e.g. poe, lighting, identity ) It i alo eay to learn energy-baed model in which all but one of the term in each multiplicative interaction are oberved. Inference i till eay. If more than one of the term in each multiplicative interaction are unoberved, the interaction between hidden variable make inference difficult. Alternating Gibb can be ued if the latent variable form a bi-partite graph.

5 Higher order Boltzmann machine (Sejnowki, ~1986) The uual energy function i quadratic in the tate: E bia term i j But we could ue higher order interaction: i j w ij E bia term i, j, h i j h w ijh Hidden unit h act a a witch. When h i on, it witche in the pairwie interaction between unit i and unit j. Unit i and j can alo be viewed a witche that control the pairwie interaction between j and h or between i and h.

6 Learning how tyle and content interact Tenenbaum and Freeman (2000) decribe a model in which a tyle vector and a content vector interact multiplicatively to determine a datavector (e.g. and image). The outer-product of the tyle and content vector determine a et of coefficient for bai function. Thi i not at all like the way a uer vector and a movie vector interact to determine a rating. The rating i the inner-product.

7 It i an unfortunate coincidence that the number of component in each poe vector i equal to the number of different poe vector. The model i only really intereting if we have le component per tyle or content vector than tyle or content vector

8 A higher-order Boltzmann machine with one viible group and two hidden group We can view it a a Boltzmann machine in which the input create interaction between the other variable. Thi type of model i now called a conditional random field. Inference can be hard in thi model. Inference i much eaier with two viible group and one hidden group viewing tranform retina-baed feature object-baed feature I thi an I or an H?

9 Uing higher-order Boltzmann machine to model image tranformation (Memievic and Hinton, 2007) A global tranformation pecifie which pixel goe to which other pixel. Converely, each pair of imilar intenity pixel, one in each image, vote for a particular global tranformation. image tranformation image(t) image(t+1)

10 Making the recontruction eaier Condition on the firt image o that only one viible group need to be recontructed. Given the hidden tate and the previou image, the pixel in the econd image are conditionally independent. image tranformation image(t) image(t+1)

11 The main problem with 3-way interaction There are far too many of them. We can reduce the number in everal traightforward way: Do dimenionality reduction on each group before the three way interaction. Ue patial locality to limit the range of the threeway interaction. A much more intereting approach (which can be combined with the other two) i to factor the interaction o that they can be pecified with fewer parameter. Thi lead to a novel type of learning module.

12 Factoring three-way interaction If three-way interaction are being ued to model a nice regular multi-linear tructure, we may not need cubically many degree of freedom. For modelling effect like viewpoint and illumination many fewer degree of freedom may be ufficient. There are many way to factor 3-D interaction tenor. We ue factor that correpond to 3-way outerproduct. Each factor only ha 3N parameter. By uing about N/3 factor we get quadratically many parameter which i the ame a a imple weight matrix.

13 Factoring the three-way interaction h hf h if i i jf j f j f j jf j if i i hf h f h f f hf jf if h j h j i i ijh h j h j i i w w w E E w w w E E w w w E w E ) ( ) ( ) ( ) ( ,,,, factored unfactored How changing the binary tate of unit j change the energy contributed by factor f. What unit j need to know in order to do Gibb ampling

14 A picture of the rank 1 tenor contributed by factor f w jf w hf w if It a 3-way outer product. Each layer i a caled verion of the ame rank 1 matrix.

15 The dynamic The viible and hidden unit get weighted input from the factor and ue thi input in the uual tochatic way. They have tochatic binary tate (or a mean-field approximation to tochatic binary tate). The factor are determinitic and implement a type of belief propagation. They do not have tate. Each factor compute three eparate um by adding up the input it get from each eparate group of unit. Then it end the product of the ummed input from two group to the third group.

16 Belief propagation h wif f w hf w jf The outgoing meage at each vertex of the factor i the product of the weighted um at the other two vertice. i j

17 A naty numerical problem In a tandard Boltzmann machine the gradient of a weight on a training cae alway lie between 1 and -1. With factored three-way interaction, the gradient contain the product of two um each of which can be large, o the gradient can explode. We can keep a running average of each um over many training cae and divide the gradient by thi average (or it quare). Thi help. For any particular weight, we mut divide the gradient by the ame quantity on all training cae to guarantee a poitive correlation with the true gradient. Updating the weight on every training cae may alo help becaue we get feedback fater when weight are blowing up.

18 Showing what a factor learn by alternating between it pre- and pot- field receptive field in pre-image receptive field in pot-image pre-image pot-image

19 The factor receptive field The network i trained on tranlated random dot pattern.

20 The factor receptive field The network i trained on tranlated random dot pattern.

21 The network i trained on rotated random dot pattern.

22 The network i trained on rotated random dot pattern.

23 How doe it perceive two overlaid pare dot pattern moving in different direction? Firt we train a econd hidden layer. Each of thee unit prefer motion in a different direction. Then we compute the perceived motion by adding up the preference of the active unit in the econd hidden layer. If the two motion are within about 30 degree it ee a ingle average motion. If they are further apart it ee two eparate motion. The eparate motion are lightly further apart than the real one. Thi i jut like human perception and it wa not trained on tranparent motion. The training i entirely unupervied.

24 Time erie model Inference i difficult in directed model of time erie if we ue non-linear, ditributed repreentation in the hidden unit. It i hard to fit directed graphical model to highdimenional equence (e.g motion capture data). So people tend to ue method with much le repreentational power HMM give up on ditributed repreentation Linear Dynamical Sytem give up on nonlinearity.

25 The conditional RBM model (a partially oberved bipartite CRF) Start with a generic RBM. Add two type of conditioning connection. Given the data, the hidden unit at time t are conditionally independent. The autoregreive weight can model mot hort-term temporal tructure very well, leaving the hidden unit to model nonlinear irregularitie. t-2 t-1 t j i h v

26 Caual generation from a learned model Keep the previou viible tate fixed. They provide a time-dependent bia for the hidden unit. Perform alternating Gibb ampling for a few iteration between the hidden unit and the mot recent viible unit. Thi pick new hidden and viible tate that are compatible with each other and with the recent hitory. j i

27 Higher level model Once we have trained the model, we can add more layer. Treat the hidden activitie of the firt CRBM a data for training the next CRBM. Add autoregreive connection to a layer when it become the viible layer. Adding a econd layer make it generate more realitic equence. k j i t-2 t-1 t

28 An application to modeling motion capture data Human motion can be captured by placing reflective marker on the joint Ue lot of infrared camera to track the 3-D poition of the marker Given a keletal model, the 3-D poition of the marker can be converted into The joint angle The 3-D tranlation of the pelvi The roll, pitch and delta yaw of the pelvi

29 Uing a tyle variable to modulate the interaction (there i additional weight haring: Taylor&Hinton, ICML 2009) tyle: 1-of-N 600 hidden unit 100 tyle feature 200 factor 6 earlier viible frame current viible frame

30 Show demo of multiple tyle of walking Thee can be found at