6.434J/16.391J Statistics for Engineers and Scientists May 4 MIT, Spring 2006 Handout #17. Solution 7

Transcription

1 6.434J/16.391J Statistics for Engineers and Scientists May 4 MIT, Spring 2006 Handout #17 Soution 7 Probem 1: Generating Random Variabes Each part of this probem requires impementation in MATLAB. For the resuts, you shoud submit your code, expanation of the parameters seected and correcty abeed resuts where needed. (a) Given a probabiity mass function (pmf) of a discrete random variabe, write an agorithm to generate N sampes from the given pmf. Test your agorithm for some arbitrary pmf and observe the histogram of sampes drawn by your agorithm. You may need a arge N for smoother histogram. (b) Impement the Box-Muer agorithm that was discussed in cass to generate Gaussian random variabe. Are there any numerica stabiity issues? (og 0?) Can you modify the agorithm to avoid them? Watch your agorithm for its computation time you may compare this with MATLAB s randn. On a computer, operations ike sin, cos and og are expensive to impement. We can remove sin and cos very easiy. Impement the version without them. (c) Another popuar method of generating random variabes is via ratio of uniforms. In simpe terms, the idea is this: if U 1 and U 2 are i.i.d., uniformy distributed between 0 and 1, and R = U 2 /U 1. Then conditioned on the event A = U 1 f(u 2 /U 1 )}, the random variabe R has the density function f. Soution Impement this method to obtain sampes from exponentia distribution λ =2. (a) N = 10000; % generate a random pmf, with 10 vaues: OMEGA = 0:10:90; pmf = rand(1,10); pmf = pmf/sum(pmf); X = []; for :N, 1

2 Xi = discreterv(pmf); % this function is defined beow X = [X Xi]; end % X contains the indices to random variabe vaues, corresponding to pmf % random variabe can be obtained from the indices: Xva = OMEGA(X); % See how good the historgram ooks: for index = 1:ength(pmf), pmf_e(index) = sum(x==index)/n; end stem(pmf); hod on; stem(pmf_e, r ); % function x = discreterv(pmf) % function x = discreterv(pmf) % x is the index for the discrete vaue in pmf. % This function generates a sampe of random variabe, distributed % according to the given pmf. The support of the discrete pmf is % assumed to be [0,M-1] where M is the ength of the pmf vector. % u = rand; x = 1; whie (u > sum(pmf(1:x))) x = x+1; end (b) The Box-Muer agorithm is a nice way to generate Gaussians. However, in its basic form, it can get trapped into the probems of og 0, and have increased computations because of sin and cos. The way to avoid these probems is to start the so-caed hit and miss approach that is common in Monte Caro methods. The idea is to generate sampes of two uniforms U 1 and U 2 in [ 1, +1], instead of [0, 1], and then chooses ony those which beong to the unit circe or its inside. This gives us direct way of computing sin and cos by taking ratio, and aso avoids the og 0. Code is given beow. 2

3 0.16 comparison of the given and generated sampes pmf Figure 1: Pot of the desired (bue) and the obtained (red) pmf. Computation time in MATLAB is much higher than that of randn. First the basic Box-Muer: function X = BoxMuer1(N); % Box-Muer agorithm for generating Gaussian U1 = rand(1,n); U2 = rand(1,n); Phi = 2*pi*U1; R = sqrt(-2*og(u2)); X = R.*cos(Phi); Modified Box-Muer: function X = BoxMuer2(N); % Box-Muer agorithm for generating Gaussian, modified for computation X = []; count = 0; whie (count < N) U1 = 2*rand(1,2*N) - 1; U2 = 2*rand(1,2*N) - 1; R = U1.^2 + U2.^2; indices = find(r <= 1); 3

4 R = R(indices); U1 = U1(indices); X = [X sqrt(-2*og(r)).* U1./sqrt(R)]; count = count + ength(indices); end % throw away the extra sampes X = X(1:N); Figure 2: Pot of the histograms for Gaussian variabe, N(0,1): Box-Muer Agorithm 1 (red) and 2 (yeow). (c) function X = ratioofuniforms(n) % % Ratio of Uniforms method to generate desired Random Variabe % X = []; count = 0; whie (count < N) U1 = rand(1,2*n); U2 = rand(1,2*n); R = U2./U1; indices = find(u1 < sqrt(fx_exp(r))); X = [X R(indices)]; count = count + ength(indices); 4

5 end % throw away the extra sampes X = X(1:N); function y=fx_exp(x); ambda = 2; y = ambda*exp(-ambda*x); 6000 Histogram for MATLAB exponentia RV and the one by Ratio of Uniforms Ratio of Uniforms MATLAB Figure 3: Sampe histograms: MATLAB s exponentia random variabe (bue) and the one via Ratio of Uniforms (red). Probem 2: Gibbs Samper Background: In Monte Caro based soutions, a very common requirement is to sampe from a desired distribution. There are various schemes that are generay avaiabe. One of them is Gibbs Samping, which is reasonaby simpe to impement as we as efficient when the sampes to be drawn are mutidimensiona. The basic idea of Gibbs samper is to draw one component at a time, of a K-dimensiona sampe, from the conditiona distribution of the specific component; conditioned upon the rest of the components. For exampe, consider a K-component vector x, and we want to draw N s sampes of this. Gibbs samper runs as foows: 1. Start n = N b. Initiaize x (n) =[x (n) 1,x(n) 2,...x(n) K ] from random sampes or any reasonabe vaues if known. 2. For n = N b : N s, - draw sampe x (n) 1 from P (x 1 x (n 1) 2,...x (n 1) K ) 5

6 - draw sampe x (n) 2 from P (x 2 x (n) 1,x(n 1) 3...x (n 1) K )... - draw sampe x (n) K 1 from P (x K x (n) 1,x(n) 2,...x(n) K 1 ) Sampes from n =1toN s are the desired sampes. They can then be used in Monte Caro based cacuations afterwards for exampe, in our case, we compute sampe means of the ast 500 sampes to estimate the desired parameters. N b is the so-caed burn-in period of the Gibbs samper, that is the time required for the samper to sette down. Note that, to draw the nth sampe vector, x n =[x (n) 1,x(n) 2,...x(n) K ], Gibbs samper draws x (n) 1 from p(x 1 x (n 1) 2,x (n 1) 3,...x (n 1) K ). Then for the next component x (n) 2, the distribution is conditioned on the current sampe component 1, x (n) 1 and the other components from the previous sampe, x (n 1) 2, x (n 1) 3...,x (n 1) K. Probem Description: In this probem, we want to impement a Gibbs Samper for the estimation of mean and variance of a Gaussian distribution. Assume we have L i.i.d.sampes Y 1,Y 2,...,Y L from N (µ, v), with unknown mean µ and variance v. Thatis, p(y µ, v) = 1 exp( 1 2πv 2v (y µ)2 ) (1) We want to use the given L sampes of Y (in hw7p2.mat) and estimate the unknowns; (note that in some practica systems, it is very expensive to obtain a sampe because of destructive samping, for exampe so the sampe size L may be very sma). Impement a Gibbs Samper to draw sampes reated to the µ and v respectivey. The idea is that after the samper has setted we (we are not going to expore the convergence of the samper here, instead we choose to run it for iterations (N b + N s = 10000), the drawn sampes wi be cose to the actua distributions. Take the sampe averages of ast 500 sampes as estimates of the required parameters. In Gibbs samper, we need to derive the conditiona densities p(µ Y,v) and p(v Y,µ) to draw sampes from them. We can use Bayes rue for this derivation from (1), but it aso needs some prior distribution of the unknown parameter. From the knowedge about the Gaussian distribution, a good seection of prior for µ is N (0, 1), and prior for the v is Inverse Gamma distribution. Or it is instructive to estimate 1/v, with its prior as the Gamma distribution. That is, we estimate v inv =1/v and take v inv G(2, 1). You can initiaize the first sampes to any reasonabe guess. A simpe choice may be µ =0andv inv =1. 6

7 (1) can be re-written as: p(y µ, v inv )= vinv 2π exp( v inv 2 (y µ)2 ) (2) In MATLAB, N (0, 1) distribution can be generated by randn, andg(2, 1) is obtained by gamrnd(a,1/b,...), where a and b are the parameters of the Gamma distribution as: p G (x) x a 1 exp( bx) (See, density has been mentioned ony up to an unknown constant; that constant is not important in soving this probem. The purpose in mentioning the density is to point out the difference in MATLAB s pdf and the one commony seen (and given above): MATLAB s definition has exp( x/b), so we ca its function with 1/b.) Soution The joint distribution is given using the priors: L } p(y, µ, v inv ) = p(y i µ, v inv ) π(µ)π(v inv ) = (2π) (L+1)/2 v L/2 inv exp v inv 2 } L (y i µ) 2 exp µ 2 /2 } v inv exp v inv }. where π is used to denote the distribution, as the Gibbs Samper iterates a Markov chain. To find fu conditiona for µ we seect the terms from p(y, µ, v inv ) that contains µ and normaize. π(µ v inv,y) = π(µ, v inv y) π(v inv y) = π(µ, v inv,y) π(v inv,y) π(µ, v inv,y) So, we write the conditiona distribution, dropping the constants or terms that do not invove µ: } π(µ v inv,y) exp v L inv (y i µ) 2 exp µ 2 /2 } 2 exp 1 ( 2 (1 + Lv inv) µ v ) } 2 inv yi 1+Lv inv where we have again not cared for any factors unreated to µ and have focused on getting some cose form for µ. This is ceary a Gaussian distribution, 7

8 ( N vinv P yi 1 1+Lv inv, and its sampes can be drawn easiy. Now concentrate on getting conditiona distribution of v inv. } π(v inv µ, y) v L/2 inv exp v L inv (y i µ) 2 v inv exp v inv } 2 [ v inv 1+Lv inv ) = v L/2+1 inv exp ]} L (y i µ) 2 which we identify as the Gamma distribution (unnormaized form), that is: G(2 + L/2, L (y i µ) 2 ). Note that the given sampes of Y are used in these pdf s. The Gibbs samper wi recursivey draw sampes from these distributions. MATLAB code for this is shown beow: % The mu = mean and vinv = 1/variance are the unknown parameters, % that Gibbs wants to estimate from ony 50 sampes of observations % of Gaussian y (which in this experiment is 2 mean, 9 variance). % We run NN iterations of Gibbs. % The proposed density for mu is N(0,1) and tau Gamma(2,1), % that is vinv.exp(-vinv). % From this information, we find p(y,mu,vinv) by mutipying individua % densities. % Farther, marginaize to get pi(mu vinv,y) and pi(vinv mu,y). % Which are obtained to be, respectivey: % exp(-1/2 * (1+n*vinv) (mu - vinv*sum_y / (1+n*vinv) )) % vinv^(n/2 + 1). exp ( -vinv [1+1/2 sum_1_to_n} (yi-mu)^2]) % % NN = 10000; mus = []; vinvs = []; oad hw7p2; suma = sum(y); mu = 0; % set the parameters as prior means vinv = 1; % for i = 1 : NN new_mu = sqrt(1/(1+n*vinv)) * randn + (vinv*suma)/(1+n*vinv); par = 1 + 1/2 * sum( (y - mu).^2 ); new_vinv = gamrnd(2 + n/2, 1/par,1,1); 8

9 end mus = [mus new_mu]; vinvs = [vinvs new_vinv]; mu = new_mu; vinv = new_vinv; mu_hat = mean(mus(end-499:end)) vinv_hat = mean(vinvs(end-499:end)) hist(mus) figure; hist(vinvs) % % The resut of sampe average from the ast 500 sampes of the Gibbs are: ˆµ =1.9075, v inv ˆ = which are quite cose to their actua vaues of 2 and 1/9 respectivey. Histograms beow show the overa sampes pattern histogram of µ sampes Figure 4: Histogram of sampes generated for mean via Gibbs samping. 9

10 3000 histogram of 1/σ 2 sampes Figure 5: Histogram of sampes generated for variance via Gibbs samping. Probem 3: [Gaussian Mixture] Let X 1,X 2,...,X n be an i.i.d. sampe from the mixture density, m p(x θ) = α i f(x µ i,σi 2 ), where θ (α 1,µ 1,σ 2 1, α 2,µ 2,σ 2 2,..., α m,µ m,σ 2 m) are unknown parameters: <µ i < σ i > 0 α i 0 m α i =1. for i =1, 2,...,m; for i =1, 2,...,m; for i =1, 2,...,m;and Here, function f( µ i,σi 2) is the density of a norma N(µ i,σi 2 ) random variabe: f(x µ i,σ 2 i )= 1 e 1 2σ 2 (x µ i) 2 i. 2πσ 2 i (a) Estimate θ by using the EM agorithm. (b) Interpret the M-steps. (c) How does the EM agorithm change when we know a priori that a µ i s and σi 2 s are equa? Interpret your resuts. 10

11 (d) Assume that m = 2 (two-mixture mode) and that unknown parameter α 1 takes a vaue from the set 0, 1}. How does the EM agorithm behave? Soution (a) Foowing the steps and notations of ecture on Mixture of Density, the M-step is given by: θ (n+1) = arg max θ Θ n m k=1 j=1 P Y k = j X k = x k, θ (n)} ] [n f(x k µ j,σ 2 j]+nα j, }} h(θ) where the feasibe set of parameters is Θ (α 1,µ 1,σ 2 1,α 2,µ 2,σ 2 2,...α m,µ m,σ 2 m m) α i =1; α i 0, for i =1, 2,...,m; <µ i <, for i =1, 2,...,m; } σ 2 i>0, for i =1, 2,...,m. Here, we have defined the variance σ 2 i σ 2 i. From the ecture, for each =1, 2,...,m, the next estimate of the weight α is given by α (n+1) = n k=1 P (n)} Y k = X k = x k, θ n where the probabiity in the numerator is P Y k = X k = x k, θ (n)} = Next, we obtain µ (n+1) f(x k µ (n), ν (n) m f(x k µ (n) i, (3) ) α (n), ν (n) i ) α (n) i and ν (n+1) simutaneousy for each = 1, 2,..., m. Setting the partia derivative of h(θ) with respect to µ to zero and setting the partia derivative of h(θ) with respect to σ 2 to zero yied two. 11

12 equations, 0= h(θ) µ n = P Y k = X k = x k, θ (n)} (µ x k ), k=1 0= σ 2 h(θ) n = P Y k = X k = x k, θ (n)} (1 (µ x k ) 2 ) σ 2 k=1 with two unknowns, µ and σ 2. Soving the above equations for µ and σ 2 yieds the maximizers, µ (n+1) = ν (n+1) = n k=1 P Y k = X k = x k, θ (n)} x k n α (n+1) n k=1 P Y k = X k = x k, θ (n)} (x k µ (n+1) ) 2 n α (n+1) (b) Expressions for α (n+1) and µ (n+1) are identica to those in the ecture. Hence, the interpretation for them are as given in the ecture note. An interpretation for ν (n+1) is as foows. The numerator is the weighted sum of the squared errors. The denominator is the expected number of sampes from the th machine. Hence, the ratio is the approximated variance of the data that are generated by the th machine. (c) When µ i = µ and σi 2 = σ, for a i =1, 2,...,m, we have one density as opposed to a mixture of m distinct densities: p(x θ) m α i f(x µ, σ 2 ) = f(x µ, σ 2 ). Thus, the probem reduces to finding the maximum ikeihood estimator of unknown parameters, µ and σ, from sampes of norma N(µ, σ 2 ) density. We can sove this probem by using maximum ikeihood estimation. (d) If the initia estimates is α (0) 1 =1and α (0) 2 = 0, the expression for α (n+1). 12

13 in equation (3) impies that 1= α (1) 1 = α (2) 1 = α (3) 1 =... 0= α (1) 2 = α (2) 2 = α (3) 2 =.... Simiary, if α (0) 1 =0and α (0) 2 = 1, we wi have 0= α (1) 1 = α (2) 1 = α (3) 1 =... 1= α (1) 2 = α (2) 2 = α (3) 2 =.... In concusion, the sequence of estimates, α (n),forn =1, 2, 3,...,and 1, 2}, wi be equa to the initia estimate. If the initia estimate of (α 1,α 2 ) is not equa to the ML estimate, the EM agorithm wi never converge to the ML estimate. 13