Bayesian Ingredients. Hedibert Freitas Lopes

Transcription

1 Normal Prior s Ingredients Hedibert Freitas Lopes The University of Chicago Booth School of Business 5807 South Woodlawn Avenue, Chicago, IL hlopes@chicagobooth.edu

2 Normal Prior s 1 Example i. 2 Normal Outline 3 Prior s 4 5 6

3 Normal Prior s Example i. John claims some discomfort and goes to the doctor. The doctor believes John may have the disease A. θ = 1: John has disease A; θ = 0: he does not. The doctor claims, based on his expertise (H), that P(θ = 1 H) = 0.70 Examination X is related to θ as follows { P(X = 1 θ = 0) = 0.40, positive test given no disease P(X = 1 θ = 1) = 0.95, positive test given disease

4 Normal Prior s Exam s result: X = 1 Observe X = 1 P(θ = 1 X = 1) P(X = 1 θ = 1)P(θ = 1) (0.95)(0.70) = P(θ = 0 X = 1) P(X = 1 θ = 0)P(θ = 0) Consequently (0.40)(0.30) = P(θ = 0 X = 1) = 0.120/0.785 = and P(θ = 1 X = 1) = 0.665/0.785 = The information X = 1 increases, for the doctor, the probability that John has the disease A from 70% to 84.71%.

5 Normal Prior s John undertakes the test Y, which relates to θ as follows 1 P(Y = 1 θ = 1) = 0.99 and P(Y = 1 θ = 0) = 0.04 Then, the of Y = 0 given X = 1 is given by P(Y = 0 X = 1) = P(Y = 0 X = 1, θ = 0)P(θ = 0 X = 1) + P(Y = 0 X = 1, θ = 1)P(θ = 1 X = 1) = P(Y = 0 θ = 0)P(θ = 0 X = 1) + P(Y = 0 θ = 1)P(θ = 1 X = 1) = (0.96)( ) + (0.01)( ) = 15.52% Key condition: X and Y are conditionally independent given θ. 1 Recall that P(X = 1 θ = 1) = 0.95 and P(X = 1 θ = 0) = 0.40.

6 Normal Prior s Model criticism Suppose the observed result was Y = 0. This is a reasonably unexpected result as the doctor only gave it roughly 15% chance. He should at least consider rethinking the based on this result. In particular, he might want to ask himself 1 Did 0.7 adequately reflect his P(θ = 1 H)? 2 Is test X really so unreliable? 3 Is the sample of X correct? 4 Is the test Y so powerful? 5 Have the tests been carried out properly?

7 Normal Prior s Observe Y = 0 Let H 2 = {X = 1, Y = 0}. Then, Bayes theorem leads to P(θ = 1 H 2 ) P(Y = 0 θ = 1)P(θ = 1 X = 1) (0.01)( ) = P(θ = 0 H 2 ) P(Y = 0 θ = 0)P(θ = 0 X = 1) Therefore, P(θ = 1 X = 1, Y = 0) = (0.96)( ) = P(Y = 0, θ = 1 X = 1) P(Y = 0 X = 1) = , H 0 : before X and Y P(θ = 1 H i ) = , H 1 : after X=1 and before Y , H 2 : after X=1 and Y=0

8 Normal Prior s Normal and normal Let us now consider a simple measurement error with normal for the unobserved measurement. X θ N(θ, σ 2 ) θ N(θ 0, τ 2 0 ) with σ 2, θ 0 and τ0 2 known for now. It is easy to show that the posterior of θ given X = x is also normal. More precisely, (θ X = x) N(θ 1, τ 2 1 ) where θ 1 = wθ 0 + (1 w)x τ1 2 = τ0 2 + σ 2 w = τ0 2 2 /(τ0 + σ 2 ) w measures the relative information contained in the with respect to the total information ( plus likelihood).

9 Normal Prior s Example from Box & Tiao (1973) Prior A: Physicist A (large experience): θ N(900, (20) 2 ) Prior B: Physicist B (not so experienced): θ N(800, (80) 2 ). Model: (X θ) N(θ, (40) 2 ). Observation: X = 850 (θ X = 850, H A ) N(890, (17.9) 2 ) (θ X = 850, H B ) N(840, (35.7) 2 ) Information (precision) Physicist A: from to (an increase of 25%) Physicist B: from to (an increase of 400%)

10 Normal Prior s Physicist A: Physicist B: Likelihood Physicist A: posterior Physicist B: posterior θ

11 Normal Prior s Two additional examples Observations x, parameters θ and history H. Likelihood functions/s - p(x θ, H) Example iii : x i θ, H N(θz i ; σ 2 ) i = 1,..., n Example iv : x t θ, H N(0; e θt ) t = 1,..., T Prior s - p(θ H) Example iii : θ H N(θ 0, τ 2 0 ) Example iv : θ t H N(α + βθ t 1, σ 2 ) t = 1,..., T Assume, for now, that (z 1,..., z n, σ 2, ν 0, θ 0, τ 2 0 ) and (α, β, σ 2, θ 0 ) are known and belong to H.

12 Normal Prior s (Bayes Theorem): p(θ x, H) = p(θ, x H) p(x H) = p(x θ, H)p(θ H) p(x H) p(x θ, H)p(θ H) Prior : p(x H) = p(x θ, H)p(θ H) dθ = E θ [p(x θ, H)] Θ

13 Normal Prior s Let y be a new set of observations conditionally independent of x given θ. Then, p(y x, H) = p(y, θ x, H)dθ = p(y θ, x, H)p(θ x, H)dθ Θ Θ = p(y θ, H)p(θ x, H)dθ = E θ x [p(y θ, H)] Θ Note 1: In general, but not always (time series, for example) x and y are independent given θ. Note 2: It might be more useful to concentrate on prediction rather than on estimation since the former is verifiable. In other words, x and y can be observed; not θ.

14 Normal Prior s Experimental result: x 1 p 1 (x 1 θ) p(θ x 1 ) l 1 (θ; x 1 )p(θ) Experimental result: x 2 p 2 (x 2 θ) p(θ x 2, x 1 ) l 2 (θ; x 2 )p(θ x 1 ) l 2 (θ; x 2 )l 1 (θ; x 1 )p(θ) Experimental results: x i p i (x i θ), for i = 3,, n p(θ x n,, x 1 ) l n (θ; x n )p(θ x n 1,, x 1 ) [ n ] l i (θ; x i ) p(θ) i=1

15 Normal Prior s Model, and posterior: x i θ, H N(θz i ; σ 2 ) i = 1,..., n θ H N(θ 0, τ 2 0 ) θ x, H N(θ 1, τ 2 1 ) where τ1 2 = τ0 2 + z z/σ 2 and θ 1 = τ1 2 ( θ0 τ0 2 + z x/σ 2) Note 1: As n increases, τ 1 σ 2 (z z) 1 and θ 1 (z z) 1 z x. Note 2: The same applies when τ 2 0 0, i.e. with little knowledge about θ.

16 Normal Prior s Model, and posterior: SV x t θ t, H N(0; e θt ) t = 1,..., T θ t H N(α + βθ t 1, σ 2 ) t = 1,..., T T p(θ x, H) e θt/2 exp { 12 } x te θt t=1 T t=1 { exp 1 } 2σ 2 (θ t α βθ t 1 ) 2 Unfortunately, closed form solutions are rare. How to compute E(θ 43 x, H) or V (θ 11 x, H)? How to obtain a 95% credible region for (θ 35, θ 36 x, H)? How to sample from p(θ x, H)? How to compute p(x H) or p(x T +1,..., x T +k x, H)?

17 Normal Prior s The standard approach to multiple linear is (y X, β, σ 2 ) N(X β, σ 2 I n ) where y = (y 1,..., y n ), X = (x 1,..., x n ) is the (n q), design matrix and q = p + 1. The of (β, σ 2 ) is NIG(b 0, B 0, n 0, S 0 ), i.e. β σ 2 N(b 0, σ 2 B 0 ) σ 2 IG(n 0 /2, n 0 S 0 /2) for known hyperparameters b 0, B 0, n 0 and S 0.

18 Normal Prior s Conditionals and marginals It is easy to show that (β, σ 2 ) is NIG(b 1, B 1, n 1, S 1 ), i.e. where (β σ 2, y, X ) N(b 1, σ 2 B 1 ) (σ 2 y, X ) IG(n 1 /2, n 1 S 1 /2) B1 1 = B0 1 + X X B1 1 1 = B X y n 1 = n 0 + n n 1 S 1 = n 0 S 0 + (y Xb 1 ) y + (b 0 b 1 ) B 1 0 b 0. It is also easy to derive the full conditional s, i.e. where (β y, X ) t n1 (b 1, S 1 B 1 ) (σ 2 β, y, X ) IG(n 1 /2, n 1 S 11 /2) n 1 S 11 = n 0 S 0 + (y X β) (y X β).

19 Normal Prior s It is well known that Ordinary least squares ˆβ = (X X ) 1 X y ˆσ 2 = S e n q = (y X ˆβ) (y X ˆβ) n q are the OLS estimates of β and σ 2, respectively. The conditional and unconditional sampling s of ˆβ are ( ˆβ σ 2, y, X ) N(β, σ 2 (X X ) 1 ) ( ˆβ y, X ) t n q (β, S e ) respectively, with (ˆσ 2 σ 2 ) IG ( (n q)/2, ((n q)σ 2 /2 ).

20 Normal Prior s Sufficient statistics Recall (y t x t, β, σ 2 ) N(x tβ, σ 2 ) for t = 1,..., n, with β σ 2 N(b 0, σ 2 B 0 ) and σ 2 IG(n 0 /2, n 0 S 0 /2). Then, for y t = (y 1,..., y t ) and X t = (x 1,..., x t ), it follows: (β σ 2, y t, X t ) N(b t, σ 2 B t ) (σ 2 y t, X t ) IG(n t /2, n t S t /2) where n t = n t 1 + 1, Bt 1 = B 1 t 1 + x tx t, Bt 1 b t = B 1 t 1 b t 1 + y t x t and n t S t = n t 1 S t 1 + (y t b tx t )y t + (b t 1 b t ) B 1 The only ingredients needed are: x t x t, y t x t and y 2 t. t 1 b t 1. These recursions will play an important role later on when deriving sequential Monte Carlo methods for conditionally Gaussian dynamic linear s, like many stochastic s.

21 Normal Prior s Predictive The density can be seen as the marginal likelihood, i.e. p(y X ) = p(y X, β, σ 2 )p(β σ 2 )p(σ 2 )dβdσ 2 or, by Bayes theorem, as the normalizing constant, i.e. p(y X ) = p(y X, β, σ2 )p(β σ 2 )p(σ 2 ) p(β σ 2, y, X )p(σ 2 y, X ) which is valid for all (β, σ 2 ). Closed form solution is available for the multiple normal linear : (y X ) t n0 (Xb 0, S 0 (I n + XB 0 X )).