Maximum Likelihood Parameter Estimation in State-Space Models

Size: px

Start display at page:

Download "Maximum Likelihood Parameter Estimation in State-Space Models"

Calvin York
5 years ago
Views:

1 Maximum Likelihood Parameer Esimaion in Sae-Space Models Arnaud Douce Deparmen of Saisics, Oxford Universiy Universiy College London 4 h Ocober 212 A. Douce (UCL Maserclass Oc h Ocober / 32

2 Sae-Space Models Le {X } 1 be a laen/hidden X -valued Markov process wih X 1 µ ( and X (X 1 = x f ( x. Le {Y } 1 be an Y-valued Markov observaion process such ha Y (X = x g ( x. Paricle filers esimae {p (x 1: y 1: } 1 on-line bu only esimaes of {p (x y 1: } 1 and {p (y 1: } 1 are reliable. Paricle smoohing mehods allow us o obain reliable esimaes of {p (x y 1:T } T =1. A. Douce (UCL Maserclass Oc h Ocober / 32

3 Sae-Space Models wih Unknown Parameers In mos scenarios of ineres, he sae-space model conains an unknown saic parameer θ Θ so ha X 1 µ θ ( and X (X 1 = x f θ ( x 1. The observaions {Y } 1 are condiionally independen given {X } 1 and θ Y (X = x g θ ( x. Aim: We would like o infer θ eiher on-line or off-line. A. Douce (UCL Maserclass Oc h Ocober / 32

4 Examples Sochasic Volailiy model where θ = ( φ, σ 2, β. X = φx 1 + σv, V i.i.d. N (, 1 Y = β exp (X /2 W, W i.i.d. N (, 1 Biochemical Nework model Pr ( X+d 1 =x 1 +1, X+d 2 =x 2 x 1, x 2 = α x 1 d + o (d, Pr ( X+d 1 =x 1 1, X+d 2 =x 2 +1 x 1, x 2 = β x 1 x 2 d + o (d, Pr ( X+d 1 =x 1, X+d 2 =x 2 1 x 1, x 2 = γ x 2 d + o (d, wih where θ = (α, β, γ. Y k = X 1 k T + W k wih W k i.i.d. N (, σ 2 A. Douce (UCL Maserclass Oc h Ocober / 32

5 Parameer Inference in Sae-Space Models Online Bayesian parameer inference. Offl ine Maximum Likelihood parameer inference. Online Maximum Likelihood parameer inference. A. Douce (UCL Maserclass Oc h Ocober / 32

6 Bayesian Parameer Inference in Sae-Space Models Se a prior p (θ on θ so inference relies now on p ( θ, x 1: y 1: = p (θ, x 1:, y 1: p (y 1: where wih p (θ, x 1:, y 1: = p (θ p θ (x 1:, y 1: We have p θ (x 1:, y 1: = µ θ (x 1 k=2 f θ (x k x k 1 k=1 p ( θ, x 1: y 1: = p ( θ y 1: p θ (x 1: y 1: g θ (y k x k Sandard and more sophisicaed paricle mehods o sample from {p ( θ, x 1: y 1: } 1 are ALL unreliabe. A. Douce (UCL Maserclass Oc h Ocober / 32

7 Online Bayesian Parameer Inference A ime = 1 ( θ (i 1, X (i 1 p (θ µ θ (x 1 hen p ( θ, x 1 y 1 = N W (i 1 δ ( θ (i 1,X (i (θ, x 1, W (i 1 g (i 1 θ 1 i=1 ( θ (i 1, X (i 1 p ( θ, x 1 y 1 and p ( θ, x 1 y 1 = 1 N N i=1 δ θ (i,x (i (θ, x 1. 1 A ime 2 Se θ (i ( = θ (i 1, X (i p ( θ, x 1: y 1 = θ (i N i=1 f θ (i W (i δ ( θ (i, X (i 1: p ( θ, x 1: y 1: and ( x X (i 1 and X (i p ( θ, x 1: y 1: = 1 N N i=1 δ (i θ,x (i (θ, x 1:. 1: 1: = ( ( X (i,x (i (θ, x 1:, W (i g (i 1: θ y 1 X (i 1. 1: 1, X (i ( y X (i A. Douce (UCL Maserclass Oc h Ocober / 32.

8 Online Bayesian Parameer Inference Provide consisen { esimaes } bu remarkably ineffi cien (Chopin, 22. Paricles in Θ space only sampled a ime 1: θ (i 1 degeneracy problem! Consider he exended sae Z = (X, θ hen ν (z 1 = p (θ 1 µ θ1 (x 1, f (z z 1 = δ θ 1 (θ f θ (x x 1, g (y z = g θ (y x ; i.e. θ = θ 1 for any wih θ 1 from he prior. Exponenial sabiliy assumpion on {p (z y 1: } 1 canno be saisfied. { } Use MCMC seps on θ so as o jier ; e.g. Andrieu, De Freias & D. (1999; Fearnhead (22; Gilks & Berzuini (21; Carvalho e al. (21. When p ( θ y 1:, x 1: = p ( θ s (x 1:, y 1: where s (x 1:, y 1: is a fixed-dimensional vecor, elegan bu sill implicily relies on p (x 1: y 1: so degeneracy will creep in. A. Douce (UCL Maserclass Oc h Ocober / 32 θ (i

9 Online Bayesian Parameer Inference A ime 1, we have Se θ (i X (i 1: = ( Resample ( p θ (i p ( θ, x 1: 1 y 1: 1 = 1 N = θ (i 1 ( X (i 1: 1, X (i (i, sample X and f θ (i N i=1 δ ( θ (i (i (θ, x 1,X 1: 1, 1: 1 ( X (i 1 p ( θ, x 1: y 1: = N i=1 W (i ( W (i g (i y θ X (i. θ (i, X (i 1: θ y 1:, X (i 1: p ( θ, x 1: y 1: = 1 N N i=1 δ ( θ (i δ ( θ (i, se,x (i (θ, x 1:, 1: p ( θ, x 1: y 1: hen sample o obain,x (i (θ, x 1:. 1: A. Douce (UCL Maserclass Oc h Ocober / 32

10 A Toy Example Linear Gaussian sae-space model We se p (θ 1 ( 1,1 (θ so X = θx 1 + σ V V, V i.i.d. N (, 1 Y = X + σ W W, W i.i.d. N (, 1. p ( θ y 1:, x 1: N ( θ; m, σ 2 1( 1,1 (θ where wih S 1, = σ 2 = S 1 2,, m = S 1 2, S 1, x k 1 x k, S 2, = xk 1 2 k=2 k=2 A. Douce (UCL Maserclass Oc h Ocober / 32

11 Illusraion of he Degeneracy Problem SMC esimae of E [ θ y 1: ], as increases he degeneracy creeps in. A. Douce (UCL Maserclass Oc h Ocober / 32

12 Anoher Toy Example Linear Gaussian sae-space model X = ρx 1 + V, V i.i.d. N (, 1 Y = X + σw, W i.i.d. N (, 1. We se ρ U ( 1,1 and σ 2 IG (1, 1. We use paricle filer wih perfec adapaion and Gibbs moves wih N = 1; paricle learning (Andrieu, D. & De Freias, 1999; Carvalho e al., 21 5 runs of he paricle mehod vs ground ruh obained using Kalman filer on saes and grid on parameers. A. Douce (UCL Maserclass Oc h Ocober / 32

13 pdf, n=5 pdf, n=4 pdf, n=3 pdf, n=2 pdf, n=1 Anoher Illusraion of Degeneracy for Paricle Learning σ2 y ρ Figure: Esimaes of p ( ρ y 1: and p ( σ 2 y 1: over 5 runs (red vs ground ruh (blue for = 1 3, 2.1 3,..., for N = 1 4 (Kanas e al., 212 A. Douce (UCL Maserclass Oc h Ocober / 32

14 Sepping Back For fixed θ, V [ p θ (y 1: /p θ (y 1: ] is in O (/N. In a Bayesian conex, p ( θ y 1: p θ (y 1: p (θ so we implicily need o compue p θ (y 1: a each paricle locaion θ (i. I appears impossible o obain uniformly in ime sable esimaes of {p ( θ y 1: } 1 for a fixed N. However for a given ime horizon T, we can use PF o sample effi cenly from p ( θ y 1:T ; see Lecure 3. A. Douce (UCL Maserclass Oc h Ocober / 32

15 Likelihood Funcion Esimaion Le y 1:T being given, he log-(marginal likelihood is given by l(θ = log p θ (y 1:T. For any θ Θ, one can esimae l(θ using paricle mehods, variance O (T /N. Direc maximizaion of l(θ diffi cul as esimae l(θ is no a smooh funcion of θ even for fixed random seed. For dim (X = 1, we can obain smooh esimae of log-likelihood funcion by using a smoohed resampling sep (e.g. Pi, 211; i.e. piecewise linear approximaion of Pr (X < x y 1:. For dim (X > 1, we can obain esimaes of l(θ highly posiively correlaed for neigbouring values in Θ (e.g. Lee, 28. A. Douce (UCL Maserclass Oc h Ocober / 32

16 Gradien Ascen To maximise l(θ w.r. θ, use a ieraion k + 1 θ k+1 = θ k + γ k l(θ θ=θk where l(θ θ=θk is he so-called score vecor. l(θ θ=θk can be esimaed using finie differences bu more effi cienly using Fisher s ideniy l(θ = log p θ (x 1:T, y 1:T p θ (x 1:T y 1:T dx 1:T where log p θ (x 1:T, y 1:T = log µ θ (x 1 + T =2 log f θ (x x 1 + T =1 log g θ (y x. A. Douce (UCL Maserclass Oc h Ocober / 32

17 Paricle Calculaion of he Score Vecor We have l(θ = { log µ θ (x 1 + log g θ (y 1 x 1 } p θ (x 1 y 1:T dx 1 T + { log f θ (x x 1 + log g θ (y x } p θ (x 1, x y 1:T dx 1 dx =2 To approximae l(θ, we jus need paricle approximaions of {p θ (x 1, x y 1:T } T =2. All he paricle smoohing mehods deailed before can be applied. Similar smoohed addiive funcionals have o be compued when implemening he Expecaion-Maximizaion. A. Douce (UCL Maserclass Oc h Ocober / 32

18 Comparison Direc Mehod vs FB We wan o esimae ϕ T = T =1 ϕ (x 1, x, y p (x 1, x y 1:T dx 1 dx. Mehod Direc FB # paricles N N cos O (TN O ( TN 2, O (TN Var. O ( T 2 /N O (T /N Bias O (T /N O (T /N MSE=Bias 2 +Var O ( T 2 /N O ( T 2 /N 2 Fas implemenaions FB of compuaional complexiy O (NT ouperform direc approach as MSE is O ( T 2 /N 2 whereas i is O ( T 2 /N for direc SMC. Naive implemenaions FB and TF have MSE of same order as direc mehod for fixed compuaional complexiy bu MSE is bias dominaed for FB/TF whereas i is variance dominaed for Direc SMC. A. Douce (UCL Maserclass Oc h Ocober / 32

19 Experimenal Resuls Consider a linear Gaussian model X = φx 1 + σ v V, V i.i.d. N (, 1 Y = cx + σ w W, W i.i.d. N (, 1. We simulae 1, observaions and compue paricle esimaes of ϕ T (x 1:T p (x 1:T y 1:T dx 1:T for 4 differen addiive funcionals ϕ (x 1: = ϕ 1 (x 1: 1 + ϕ (x 1, x, y including ϕ 1 (x 1, x, y = x 1 x, ϕ 2 (x 1, x, y = x 2. [Ground ruh can be compued using Kalman smooher.] We use 1 replicaions on he same daase o esimae he empirical variance. A. Douce (UCL Maserclass Oc h Ocober / 32

20 Score Score Boxplos of Direc vs FB Esimaes Algorihm 1 score esimaes for parameer σ v Algorihm score esimaes for parameer φ Algorihm 1 Algorihm Time seps Time seps Direc (lef vs FB (righ A. Douce (UCL Maserclass Oc h Ocober / 32

21 V ariance V ariance V ariance V ariance Empirical Variance for Direc vs FB Esimaes V a r ia n c e o f s c o r e e s im σ a e w.r.. v x 1 4 V a r ia n c e o f s c o r e e s im φ a e w.r.. V a r ia n c e o f s c o r e e s im σ a e w.r.. v V a r ia n c e o f s c o r e e s im φ a e w.r V a r ia n c e o f s c o r e e s im σ a e w.r.. w V a r ia n c e o f s c o r e e s im a e w.r.. c V a r ia n c e o f s c o r e e s im σ a e w.r.. w V a r ia n c e o f s c o r e e s im a e w.r.. c Time seps Time seps Time seps Time seps Direc (lef vs FB (righ; he verical scale is differen A. Douce (UCL Maserclass Oc h Ocober / 32

22 Online ML Parameer Inference Recursive maximum likelihood (Tieringon, 1984; LeGland & Mevel, 1997 proceeds as follows θ +1 = θ + γ log p θ1: (y y 1: 1 where p θ1: (y y 1: 1 is compued using θ k a ime k and γ =, γ 2 <. Under regulariy condiions, his converges owards a local maximum of he (average log-likelihood. Noe ha log p θ1: (y y 1: 1 = log p θ1: (y 1: log p θ1: 1 (y 1: 1 is given by he difference of wo pseudo-score vecors where log p θ1: (y 1: := ( k=2 log f θ (x k x k 1 θk + log g θ (y k x k θk p θ1: (x 1: y 1: dx 1:. A. Douce (UCL Maserclass Oc h Ocober / 32

23 Online ML Paricle Parameer Inference Paricle approximaion follows where θ +1 = θ + γ log p θ1: (y y 1: 1 log p θ1: (y y 1: 1 = log p θ1: (y 1: log p θ1: 1 (y 1: 1 is given by he difference of paricle esimaes of pseudo-score vecors (Poyadjis, D. & Singh, 211. Asympoic variance of log p θ1: (y y 1: 1 is uniformly bounded in O (1/N for FB esimae whereas i is O (/N for direc paricle mehod (Del Moral, D. & Singh, 211. Bias is O (1/N in boh cases. Major Problem: If we use FB, his is no an online algorihm anymore as i requires a backward pass of order O ( o approximae log p θ1: (y 1:... A. Douce (UCL Maserclass Oc h Ocober / 32

24 Variance of he Gradien Esimae for Direc vs FB Figure: Empirical variance of he gradien esimae for sandard versus FB approximaions (SV model A. Douce (UCL Maserclass Oc h Ocober / 32

25 Online Paricle ML Inference using Direc Approach x1 3 Figure: N = 1, paricles, online parameer esimaes for SV model. A. Douce (UCL Maserclass Oc h Ocober / 32

26 Online Paricle ML Inference using FB x1 3 Figure: N = 5 paricles, online parameer esimaes for SV model. A. Douce (UCL Maserclass Oc h Ocober / 32

27 Forward only Smoohing Dynamic programming allows us o compue in a single forward pass he FB esimaes of ϕ θ = ϕ (x 1: p θ (x 1: y 1: dx 1: where ϕ (x 1: = ϕ (x k 1, x k, y k k=1 Forward Backward (FB decomposiion saes T 1 p θ (x 1:T y 1:T = p θ (x T y 1:T =1 p θ (x y 1:, x +1 where p θ (x y 1:, x +1 = f θ( x +1 x p θ ( x y 1: p θ ( x +1 y 1:. Condiioned upon y 1:T, {X } T =1 is a backward Markov chain of iniial disribuion p (x T y 1:T and inhomogeneous Markov ransiions {p θ (x y 1:, x +1 } T 1 =1 independen of T. A. Douce (UCL Maserclass Oc h Ocober / 32

28 Forward only Smoohing We have ϕ θ = ϕ (x 1: p θ (x 1: 1 y 1: 1, x dx 1: 1 = ϕ (x 1: p θ (x 1: 1 y 1: 1, x dx 1: 1 p }{{} θ (x y 1: dx Forward smoohing recursion V θ (x = V θ (x [ ] V 1 θ (x 1 + ϕ (x 1:, y p θ (x 1 y 1: 1, x dx 1 Appears implicily in Ellio, Aggoun & Moore (1996, Ford (1998 and rediscovered a few imes... Presenaion follows here (Del Moral, D. & Singh, 29. A. Douce (UCL Maserclass Oc h Ocober / 32

29 Forward only Smoohing Forward smoohing recursion V θ (x = Proof is rivial [ ] V 1 θ (x 1 + ϕ (x 1:, y p θ (x 1 y 1: 1, x dx 1 V θ (x = ϕ (x 1: p θ (x 1: 1 y 1: 1, x dx 1: 1 = [ ϕ 1 (x 1: 1 + ϕ (x 1:, y ] p θ (x 1: 2 y 1: 2, x 1 p θ (x 1 y 1: 1, x dx 1: 1 = { ϕ 1 (x 1: 1 p θ (x 1: 2 y 1: 2, x 1 dx 1: 2 }{{} V 1 θ (x 1 +ϕ (x 1:, y } p θ (x 1 y 1: 1, x dx 1 Exac implemenaion possible for finie sae-space and linear Gaussian models. A. Douce (UCL Maserclass Oc h Ocober / 32

30 Paricle Forward only Smoohing V θ A ime 1, we have p θ (x 1 y 1: 1 = 1 N N i=1 δ (i X (x 1 and { ( } 1 V 1 θ X (i 1 ( 1 i N. A ime, compue p θ (x y 1: = N i=1 W (i δ (i X (x and se } ( p θ X (i = { V 1 θ (x 1 + ϕ (x 1, x, y = N j=1 f θ ϕ θ = 1 N N i=1 V θ ( ( X (i X (i X (j 1. [ V 1 θ ( N j=1 f θ ( X (j 1 X (i ( ] +ϕ X (j (i 1,X,y, X (j 1 x 1 y 1: 1, X (i dx 1 This esimae is exacly he same as he Paricle FB esimae, compuaional complexiy O ( N 2. A. Douce (UCL Maserclass Oc h Ocober / 32

31 Online Paricle ML Inference A ime 1, we have p θ1: 1 (x 1 y 1: 1, { V θ 1: 1 1 ( } X (i 1, log p θ1: 1 (y 1: 1 = V θ 1: 1 1 (x 1 p θ1: 1 (x 1 y 1: 1 dx 1 and ge θ. A ime, use your favourie PF o compue p θ1: (x y 1: and ( V θ 1: X (i = { } V θ 1: 1 1 (x 1 + ϕ (x 1, x, y ( p θ1: x 1 y 1: 1, X (i dx 1, ϕ (x 1:, y = log f θ (x x 1 θ + log g θ (y x θ and Parameer updae log p θ1: (y 1: = θ +1 = θ + γ V θ 1: (x p θ1: (x y 1: dx ( log p θ1: (y 1: log p θ1: 1 (y 1: 1 A. Douce (UCL Maserclass Oc h Ocober / 32

32 Summary Online Bayesian parameer inference using paricle mehods is ye an unsolved problem. Paricle smoohing echniques can be used o perform off-line and on-line ML parameer esimaion. Observed informaion marix can also be evaluaed online in a sable manner. For online inference, compuaional complexiy is O ( N 2 a each ime sep and requires evaluaing f θ (x x 1. A. Douce (UCL Maserclass Oc h Ocober / 32

Particle Filtering and Smoothing Methods

Particle Filtering and Smoothing Methods Paricle Filering and Smoohing Mehods Arnaud Douce Deparmen of Saisics, Oxford Universiy Universiy College London 3 rd Ocober 2012 A. Douce (UCL Maserclass Oc. 2012) 3 rd Ocober 2012 1 / 46 Sae-Space Models