Variational Bayes and Variational Message Passing

Transcription

1 Variational Bayes and Variational Message Passing Mohammad Emtiyaz Khan CS,UBC Variational Bayes and Variational Message Passing p.1/16

2 Variational Inference Find a tractable distribution Q(H) that closely approximates the true posterior distribution P(H V ). log P(V ) = H = H = H Q(H) log P(V ) Q(H) log P(H,V ) P(H V ) [ P(H,V ) Q(H) log Q(H) ] Q(H) P(H V ) = Q(H) log P(H,V ) + Q(H) log P(H V ) Q(H) Q(H) H H L(Q) KL(Q P) Variational Bayes and Variational Message Passing p.2/16

3 Variational Inference log P(V ) = L(Q) + KL(Q P) (1) L(Q) = H Q(H) log P(H,V ) Q(H) (2) KL(Q P) = H Q(H) log P(H V ) Q(H) (3) Find Q(H) that maximizes lower bound L(Q) (and hence minimizes KL divergence). For Q(H) = P(H V ), KL vanishes to zero, but P(H V ) is intractable (that s why variational approach). Trick : Consider a restricted class of Q(H), and then find the member which minimizes the KL divergence. Variational Bayes and Variational Message Passing p.3/16

4 Factorized Distributions Q(H) = i Q i (H i ) (4) Substituting this in the expression for lower bound, L(Q) = H = H = H = H i i i i Q i (H i ) log P(H,V ) i Q i(h i ) Q i (H i ) log P(H,V ) H Q i (H i ) log P(H,V ) i Q i (H i ) log P(H,V ) + i (Outline) Q i (H i ) log Q i (H i ) i i Q i (H i ) log Q i (H i ) H i i H(Q i ) Variational Bayes and Variational Message Passing p.4/16

5 Factorized Distributions Now separate out all the terms in one factor Q j. L(Q) = H j Q j (H j ) log P(H,V ) Qj (H j ) log Q j (H j) + H(Q i ) + i j H(Q i ) = KL(Q j Q j) + terms not in Q j (5) This bound is maximized wrt Q j when log Q j (H j ) = log Q j(h j ) = log P(H,V ) Qj (H j ) + c (6) Now iterate, guaranteed convergence... Variational Bayes and Variational Message Passing p.5/16

6 Variational Bayes for Bayesian Networks log Q j(h j ) = log P(H,V ) Qj (H j ) + c = i log P(X i pa i ) Qj (H j ) + c = log P(H j pa j ) Qj (H j ) + k ch j log P(X k pa j ) Qj (H j ) + c Variational Bayes and Variational Message Passing p.6/16

7 Exponential-Conjugate Models P(Y θ) = exp[φ T Y (θ)u(y ) + f(y ) + g(θ)] (7) u(y ) = Natural statistics (8) φ Y (θ) = Natural Parameter vector (9) g(θ) = Constant of integration (10) Example I: Bernoulli Distribution p(x µ) = µ x (1 µ) 1 x (11) log p(x µ) = x log µ + (1 x) log(1 µ) (12) µ = log (1 µ) (13) } {{ } φ(µ) x + log(1 µ) u(x) g(µ) Variational Bayes and Variational Message Passing p.7/16

8 Exponential-Conjugate Models P(Y θ) = exp[φ T Y (θ)u(y ) + f(y ) + g(θ)] (14) P(Y φ) = exp[φ T u(y ) + f(y ) + g(φ)](re-parametrization) Property I: u(y ) P(Y θ) = d g(φ) dφ log p(x µ) = log φ = log µ (1 µ) } {{ } φ(µ) x + log(1 µ) u(x) g(µ) (15) µ (1 µ) µ = eφ 1 + e φ (16) g(µ) = log(1 µ) = log(1 + e φ ) = g(φ) (17) E(x) = u(y ) = e φ (1 + e φ ) 1 = µ (18) Variational Bayes and Variational Message Passing p.8/16

9 Exponential-Conjugate Models P(Y θ) = exp[φ T Y (θ)u(y ) + f(y ) + g(θ)] (19) Example II: Gaussian Distribution θ Y X β p(y θ) = (2π) 1/2 exp 1 (Y θ)2 2 [ ] log p(y θ) = [θ, 1/2] φ Y (θ) Y Y 2 u Y (Y ) p(x Y,β) = (2π) 1/2 β 1/2 exp β 2 [ ] X log p(x Y, β) = [βy, β/2] X 2 φ X (Y,β) u X (X) 1 2 θ2 g Y (θ) (X Y )2 1 2 log(2π) f Y (Y ) (βy 2 + log β) g X (Y,β) 1 2 log(2π) f X (X) Variational Bayes and Variational Message Passing p.9/16

10 Exponential-Conjugate Models Property II: Multi-linearity θ Y X β [ ] X log p(x Y, β) = [βy, β/2] X 2 φ X (Y,β) = [βx, β/2] φ XY (X,β) log p(y θ) = [θ, 1/2] φ Y (θ) [ u X (X) [ Y Y 2 ] Y Y 2 u Y (Y ) ] u Y (Y ) (βy 2 + log β) g X (Y,β) (βx2 + log β) g XY (X,β) 1 2 θ2 g Y (θ) 1 2 log(2π) f Y (Y ) 1 2 log(2π) f X (X) 1 2 log(2π) f Y (Y ) Variational Bayes and Variational Message Passing p.10/16

11 Exponential-Conjugate Models Consider Y node and it s children in θ Y X β, log P(Y θ) = φ T Y (θ)u Y (Y ) + f Y (Y ) + g Y (θ) log P(X Y,β) = φ T X (Y,β)u X(X) + f X (X) + g X (Y,β) = φ T XY (X,β)u Y (Y ) + g XY (Y,β) Recall that, log Q Y (Y ) = log P(Y θ) Q Y (Y ) + log P(X Y,β) QY (Y ) + c = φ T Y (θ)u Y (Y ) + f Y (Y ) + g Y (θ) QY (Y ) + φ T XY (X,β)u Y (Y ) + g XY (Y,β) QY (Y ) + c = φ T Y (θ) + φt XY (X,β) Q Y (Y )u Y (Y ) + f Y (Y ) + c 1 Variational Bayes and Variational Message Passing p.11/16

12 Exponential-Conjugate Models log Q Y (Y ) = φt Y (θ) + φt XY (X,β) Q Y (Y )u Y (Y ) + f Y (Y ) + c 1 Finally, φ T Y (θ) = [θ, 1/2] φ T XY (X,β) = [βx, β/2] Later is found using the property I (explain). Variational Bayes and Variational Message Passing p.12/16

13 Back to Bayesian Networks Take each node, write the expression as a function of natural statistics of that node. log Q Y (Y ) = log P(Y pa Y ) QY (Y ) + = log P(X k pa j ) QY (Y ) + c k ch j φ T Y (θ) + φ T XY (X,β) Q Y (Y ) u Y (Y ) + f Y (Y ) + c 1 k ch j The compute the expectation of natural statistics of each children node, and use that to find the quantity in bracket. Variational Bayes and Variational Message Passing p.13/16

14 Variational Message Passing Message from a parent node Y to a child node X: m Y X = u Y (20) Message from a child node X to a parent node Y: m X Y = φ XY ( u X, {m i X } i cpy ) (21) Node Y update it s posterior Q Y : φ Y = φ Y ({m i Y } i pay ) + j ch Y m j Y (22) Variational Bayes and Variational Message Passing p.14/16

15 Variational Message Passing Variational Bayes and Variational Message Passing p.15/16

16 Discussion Initialization and message passing schedule. Calculation of Lower Bound Allowable Model VIBES Variational Bayes and Variational Message Passing p.16/16