Inferring State Sequences for Non-linear Systems with Embedded Hidden Markov Models

Transcription

1 Inferring Sae Sequences for Non-linear Sysems wih Embedded Hidden Markov Models Radford M. Neal, Mahew J. Beal, and Sam T. Roweis Deparmen of Compuer Science Universiy of Torono Torono, Onario, Canada M5S 3G3 Absrac We describe a Markov chain mehod for sampling from he disribuion of he hidden sae sequence in a non-linear dynamical sysem, given a sequence of observaions. This mehod updaes all saes in he sequence simulaneously using an embedded Hidden Markov Model (HMM). An updae begins wih he creaion of pools of candidae saes a each ime. We hen define an embedded HMM whose saes are indexes wihin hese pools. Using a forward-backward dynamic programming algorihm, we can efficienly choose a sae sequence wih he appropriae probabiliies from he exponenially large number of sae sequences ha pass hrough saes in hese pools. We illusrae he mehod in a simple one-dimensional example, and in an example showing how an embedded HMM can be used o in effec discreize he sae space wihou any discreizaion error. We also compare he embedded HMM o a paricle smooher on a more subsanial problem of inferring human moion from 2D races of markers. 1 Inroducion Consider a dynamical model in which a sequence of hidden saes, x = (x 0,..., x n 1 ), is generaed according o some sochasic ransiion model. We observe y = (y 0,..., y n 1 ), wih each y being generaed from he corresponding x according o some sochasic observaion process. Boh he x and he y could be mulidimensional. We wish o randomly sample hidden sae sequences from he condiional disribuion for he sae sequence given he observaions, which we can hen use o make Mone Carlo inferences abou his poserior disribuion for he sae sequence. We suppose in his paper ha we know he dynamics of hidden saes and he observaion process, bu if hese aspecs of he model are unknown, he mehod we describe will be useful as par of a maximum likelihood learning algorihm such as EM, or a Bayesian learning algorihm using Markov chain Mone Carlo. If he sae space is finie, of size K, so ha his is a Hidden Markov Model (HMM), a hidden sae sequence can be sampled by a forward-backwards dynamic programming algorihm in ime proporional o nk 2 (see [5] for a review of his and relaed algorihms). If he sae space is R p and he dynamics and observaion process are linear, wih Gaussian noise, an analogous adapaion of he Kalman filer can be used. For more general models,

2 or for finie sae space models in which K is large, one migh use Markov chain sampling (see [3] for a review). For insance, one could perform Gibbs sampling or Meropolis updaes for each x in urn. Such simple Markov chain updaes may be very slow o converge, however, if he saes a nearby imes are highly dependen. A popular recen approach is o use a paricle smooher, such as he one described by Douce, Godsill, and Wes [2], bu his approach can fail when he se of paricles doesn adequaely cover he space, or when paricles are eliminaed premaurely. In his paper, we presen a Markov chain sampling mehod for a model wih an arbirary sae space, X, in which efficien sampling is faciliaed by using updaes ha are based on emporarily embedding an HMM whose finie sae space is a subse of X, and hen applying he efficien HMM sampling procedure. We illusrae he mehod on a simple one-dimensional example. We also show how i can be used o in effec discreize he sae space wihou producing any discreizaion error. Finally, we demonsrae he embedded HMM on a problem of racking human moion in 3D based on he 2D projecions of marker posiions, and compare i wih a paricle smooher. 2 The Embedded HMM Algorihm In our descripion of he algorihm, model probabiliies will be denoed by P, which will denoe probabiliies or probabiliy densiies wihou disincion, as appropriae for he sae space, X, and observaion space, Y. The model s iniial sae disribuion is given by P (x 0 ), ransiion probabiliies are given by P (x x 1 ), and observaion probabiliies are given by P (y x. Our goal is o sample from he condiional disribuion P (x 0,..., x n 1 y 0,..., y n 1 ), which we will abbreviae o π(x 0,..., x n 1 ), or π(x). To accomplish his, we will simulae a Markov chain whose sae space is X n i.e., a sae of his chain is an enire sequence of hidden saes. We will arrange for he equilibrium disribuion of his Markov chain o be π(x 0,..., x n 1 ), so ha simulaing he chain for a suiably long ime will produce a sae sequence from he desired disribuion. The sae a ieraion i of his chain will be wrien as x (i) = (x (i) 0,..., x(i) n 1 ). The ransiion probabiliies for his Markov chain will be denoed using Q. In paricular, we will use some iniial disribuion for he sae of he chain, Q(x (0) ), and will simulae he chain according o he ransiion probabiliies Q(x (i) x (i 1) ). For validiy of he sampling mehod, we need hese ransiions o leave π invarian: π(x ) = π(x)q(x x), for all x in X n (1) x X n (If X is coninuous, he sum is replaced by an inegral.) This is implied by he deailed balance condiion: π(x)q(x x) = π(x )Q(x x ), for all x and x in X n (2) The ransiion Q(x (i) x (i 1) ) is defined in erms of pools of saes for each ime. The curren sae a ime is always par of he pool for ime. Oher saes in he pool are produced using a pool disribuion, ρ, which is designed so ha poins drawn from ρ are plausible alernaives o he curren sae a ime. The simples way o generae hese addiional pool saes is o draw poins independenly from ρ. This may no be feasible, however, or may no be desirable, in which case we can insead simulae an inner Markov chain defined by ransiion probabiliies wrien as R ( ), which leave he pool disribuion, ρ, invarian. The ransiions for he reversal of his chain wih respec o ρ will be denoed by R ( ), and are defined so as o saisfy he following condiion: ρ (x R (x x = ρ (x ) R (x x ), for all x and x in X (3)

3 If he ransiions R saisfy deailed balance wih respec o ρ, R will be he same as R. To generae pool saes by drawing from ρ independenly, we can le R (x x) = R (x x) = ρ (x ). For he proof of correcness below, we mus no choose ρ or R based on he curren sae, x (i), bu we may choose hem based on he observaions, y. To perform a ransiion Q o a new sae sequence, we begin by a each ime,, producing a pool of K saes, C. One of he saes in C is he curren sae, x (i 1) ; he ohers are produced using R and R. The new sae sequence, x (i), is hen randomly seleced from among all sequences whose saes a each ime are in C, using a form of he forwardbackward procedure. In deail, he pool of candidae saes for ime is found as follows: 1) Pick an ineger J uniformly from {0,..., K 1}. 2) Le x [0] = x (i 1). (So he curren sae is always in he pool.) 3) For j from 1 o J, randomly pick x [j] R (x [j] x [j 1]. according o he ransiion probabiliies 4) For j from 1 down o K +J +1, randomly pick x [j] according o he reversed ransiion probabiliies, R (x [j] x [j+1]. 5) Le C be he pool consising of x [j], for j { K+J +1,..., 0,..., J }. If some of he x [j] are he same, hey will be presen in he pool more han once. Once he pools of candidae saes have been found, a new sae sequence, x (i), is picked from among all sequences, x, for which every x is in C. The probabiliy of picking x (i) = x is proporional o π(x)/ n 1 =0 ρ (x, which is proporional o P (x 0 ) n 1 =1 P (x x 1 ) n 1 =0 P (y x n 1 =0 ρ (x The division by n 1 =0 ρ (x is needed o compensae for he pool saes having been drawn from he ρ disribuions. If duplicae saes occur in some of he pools, hey are reaed as if hey were disinc when picking a sequence in his way. In effec, we pick indexes of saes in hese pools, wih probabiliies as above, raher han saes hemselves. The disribuion of hese sequences of indexes can be regarded as he poserior disribuion for a hidden Markov model, wih he ransiion probabiliy from sae j a ime 1 o sae k a ime being proporional o P (x [k] x [j] 1 ), and he probabiliies of he hypoheical observed symbols being proporional o P (y x [k] /ρ (x [k]. Crucially, using he forward-backward echnique, i is possible o randomly pick a new sae sequence from his disribuion in ime growing linearly wih n, even hough he number of possible sequences grows as K n. Afer he above procedure has been used o produce he pool saes, x [j] for = 0 o n 1 and j = K +J + 1 o J, his algorihm operaes as follows (see [5]): 1) For = 0 o n 1 and for j = K +J +1 o J, le u,j = P (y x [j] /ρ (x [j]. 2) For j = K +J 0 +1 o J 0, le w 0,j = u 0,j P (X 0 = x [j] 0 ). 3) For = 1 o n 1 and for j = K +J + 1 o J, le w,j = u,j w 1,k P (x [j] x [k] 1 ) k 4) Randomly pick s n 1 from { K +J n 1 +1,..., J n 1 }, picking he value j wih probabiliy proporional o w n 1,j. (4)

4 5) For = n 1 down o 1, randomly pick s 1 from { K +J 1 +1,..., J 1 }, picking he value j wih probabiliy proporional o w 1,j P (x [s] x [j] 1 ). Noe ha when implemening his algorihm, one mus ake some measure o avoid floaingpoin underflow, such as represening he w,j by heir logarihms. Finally, he embedded HMM ransiion is compleed by leing he new sae sequence, x (i), be equal o (x [s0] 0, x [s1] 1,..., x [sn 1] n 1 ) 3 Proof of Correcness To show ha a Markov chain wih hese ransiions will converge o π, we need o show ha i leaves π invarian, and ha he chain is ergodic. Ergodiciy need no always hold, and proving ha i does hold may require considering he pariculars of he model. However, i is easy o see ha he chain will be ergodic if all possible sae sequences have non-zero probabiliy densiy under π, he pool disribuions, ρ, have non-zero densiy everywhere, and he ransiions R are ergodic. This probably covers mos problems ha arise in pracice. To show ha he ransiions Q( ) leave π invarian, i suffices o show ha hey saisfy deailed balance wih respec o π. This will follow from he sronger condiion ha he probabiliy of moving from x o x (saring from a sae picked from π) wih given values for he J and given pools of candidae saes, C, is he same as he corresponding probabiliy of moving from x o x wih he same pools of candidae saes and wih values J defined by J = J h, where h is he index (from K + J + 1 o J of x in he candidae pool. The probabiliy of such a move from x o x is he produc of several facors. Firs, here is he probabiliy of saring from x under π, which is π(x). Then, for each ime, here is he probabiliy of picking J, which is 1/K, and of hen producing he saes in he candidae pool using he ransiions R and R, which is J j=1 R (x [j] x [j 1] 1 j= K+J +1 R (x [j] x [j+1] = J 1 j=0 R (x [j+1] x [j] 1 j= K+J +1 R (x [j+1] x [j] ρ (x [j] ρ (x [j+1] (5) = ρ (x [ K+J+1] ρ (x [0] J 1 j= K+J +1 R (x [j+1] x [j] (6) Finally, here is he probabiliy of picking x from among all he sequences wih saes from he pools, C, which is proporional o π(x )/ ρ (x ). The produc of all hese facors is π(x) 1 n 1 K n ρ (x [ K+J+1] J 1 R (x [j+1] x [j] = =0 ρ (x [0] 1 π(x)π(x ) K n n 1 =0 ρ(x )ρ(x ) n 1 =0 j= K+J +1 ρ (x [ K+J+1] J 1 j= K+J +1 π(x ) n 1 =0 ρ (x ) R (x [j+1] x [j] (7) We can now see ha he corresponding expression for a move from x o x is idenical, apar from a relabelling of candidae sae x [j] as x [j h].

5 4 A simple demonsraion The following simple example illusraes he operaion of he embedded HMM. The sae space X and he observaion space, Y, are boh R, and each observaion is simply he sae plus Gaussian noise of sandard deviaion σ i.e., P (y x = N(y x, σ 2 ). The sae ransiions are defined by P (x x 1 ) = N(x anh(ηx 1 ), τ 2 ), for some consan expansion facor η and ransiion noise sandard deviaion τ. Figure 1 shows a hidden sae sequence, x 0,..., x n 1, and observaion sequence, y 0,..., y n 1, generaed by his model using σ = 2.5, η = 2.5, and τ = 0.4, wih n = The sae sequence says in he viciniy of +1 or 1 for long periods, wih rare swiches beween hese regions. Because of he large observaion noise, here is considerable uncerainy regarding he sae sequence given he observaion sequence, wih he poserior disribuion assigning fairly high probabiliy o sequences ha conain shor-erm swiches beween he +1 and 1 regions ha are no presen in he acual sae sequence, or ha lack some of he shor-erm swiches ha are acually presen. We sampled from his disribuion over sae sequences using an embedded HMM in which he pool disribuions, ρ, were normal wih mean zero and sandard deviaion one, and he pool ransiions simply sampled independenly from his disribuion (ignoring he curren pool sae). Figure 2 shows ha afer only wo updaes using pools of en saes, embedded HMM sampling produces a sae sequence wih roughly he correc characerisics. Figure 3 demonsraes how a single embedded HMM updae can make a large change o he sae sequence. I shows a porion of he sae sequence afer 99 updaes, he pools of saes produced for he nex updae, and he sae sequence found by he embedded HMM using hese pools. A large change is made o he sae sequence in he region from ime 840 o 870, wih saes in his region swiching from he viciniy of 1 o he viciniy of +1. This example is explored in more deail in [4], where i is shown ha he embedded HMM is superior o simple Meropolis mehods ha updae one hidden sae a a ime. 5 Discreizaion wihou discreizaion error A simple way o handle a model wih a coninuous sae space is o discreize he space by laying down a regular grid, afer ransforming o make he space bounded if necessary. An HMM wih grid poins as saes can hen be buil ha approximaes he original model. Inference using his HMM is only approximae, however, due o he discreizaion error involved in replacing he coninuous space by a grid of poins. The embedded HMM can use a similar grid as a deerminisic mehod of creaing pools of saes, aligning he grid so ha he curren sae lies on a grid poin. This is a special case of he general procedure for creaing pools, in which ρ is uniform, R moves o he nex grid poin and R moves o he previous grid poin, wih boh wrapping around when he firs or las grid poin is reached. If he number of pool saes is se equal o he number of poins in a grid, every pool will consis of a complee grid aligned o include he curren sae. On heir own, such embedded HMM updaes will never change he alignmens of he grids. However, we can alernaely apply such an embedded HMM updae and some oher MCMC updae (eg, Meropolis) which is capable of making small changes o he sae. These small changes will change he alignmen of he new grids, since each grid is aligned o include he curren sae. The combined chain will be ergodic, and sample (asympoically) from he correc disribuion. This mehod uses a grid, bu neverheless has no discreizaion error. We have ried his mehod on he example described above, laying he grid over he ransformed sae anh(x, wih suiably ransformed ransiion densiies. Wih K = 10, he grid mehod samples more efficienly han when using N(0, 1) pool disribuions, as above.

6 Figure 1: A sae sequence (black dos) and observaion sequence (gray dos) of lengh 1000 produced by he model wih σ = 2.5, η = 2.5, and τ = Figure 2: The sae sequence (black dos) produced afer wo embedded HMM updaes, saring wih he saes se equal o he daa poins (gray dos), as in he figure above Figure 3: Closeup of an embedded HMM updae. The rue sae sequence is shown by black dos and he observaion sequence by gray dos. The curren sae sequence is shown by he dark line. The pools of en saes a each ime used for he updae are shown as small dos, and he new sae sequence picked by he embedded HMM by he ligh line.

7 6 Tracking human moion Figure 4: The four-second moion sequence used for he experimen, shown in hree snapshos wih sreamers showing earlier moion. The lef plo shows frames 1-59, he middle plo frames 59-91, and he righ plo frames There were 30 frames per second. The orhographic projecion in hese plos is he one seen by he model. (These plos were produced using Herzmann and Brand s mosey program.) We have applied he embedded HMM o he more challenging problem of racking 3D human moion from 2D observaions of markers aached o cerain body poins. We consruced his example using real moion-capure daa, consising of he 3D posiions a each ime frame of a se of idenified markers. We chose one subjec, and seleced six markers (on lef and righ fee, lef and righ hands, lower back, and neck). These markers were projeced o a 2D viewing plane, wih he viewing direcion being known o he model. Figure 4 shows he four-second sequence used for he experimen. 1 Our goal was o recover he 3D moion of he six markers, by using he embedded HMM o generae samples from he poserior disribuion over 3D posiions a each ime (he hidden saes of he model), given he 2D observaions. To do his, we need some model of human dynamics. As a crude approximaion, we used Langevin dynamics wih respec o a simple hand-designed energy funcion ha penalizes unrealisic body posiions. In Langevin dynamics, a gradien descen sep in he energy is followed by he addiion of Gaussian noise, wih variance relaed o he sep size. The equilibrium disribuion for his dynamics is he Bolzmann disribuion for he energy funcion. The energy funcion we used conains erms peraining o he pairwise disances beween he six markers and o he heighs of he markers above he plane of he floor, as well as a erm ha penalizes bending he orso far backwards while he legs are verical. We chose he sep size for he Langevin dynamics o roughly mach he characerisics of he acual daa. The embedded HMM was iniialized by seing he sae a all imes o a single frame of he subjec in a ypical sance, aken from a differen rial. As he pool disribuion a ime, we used he poserior disribuion when using he Bolzmann disribuion for he energy as he prior and he single observaion a ime. The pool ransiions used were Langevin updaes wih respec o his pool disribuion. For comparison, we also ried solving his problem wih he paricle smooher of [2], in which a paricle filer is applied o he daa in ime order, afer which a sae sequence is seleced a random in a backwards pass. We used a sraified resampling mehod o reduce variance. The iniial paricle se was creaed by drawing frames randomly from sequences oher han he sequence being esed, and ranslaing he markers in each frame so ha heir cenre of mass was a he same poin as he cenre of mass in he es sequence. Boh programs were implemened in MATLAB. The paricle smooher was run wih 5000 paricles, aking 7 hours of compue ime. The resuling sampled rajecories roughly fi he 2D observaions, bu were raher unrealisic for insance, he subjec s fee ofen floaed above he floor. We ran he embedded HMM using five pool saes for 300 ieraions, aking 1.7 hours of compue ime. The resuling sampled rajecories were more realisic 1 Daa from he graphics lab of Jessica Hodgins, a hp://mocap.cs.cmu.edu. We chose markers 167, 72, 62, 63, 31, 38, downsampled o 30 frames per second. The experimens repored here use frames of rial 20 for subjec 14. The elevaion of he view direcion was 45 degrees, and he azimuh was 45 degrees away from a fron view of he person in he firs frame.

8 han hose produced by he paricle smooher, and were quaniaively beer wih respec o likelihood and dynamical ransiion probabiliies. However, he disribuion of rajecories found did no overlap he rue rajecory. The embedded HMM updaes appeared o be sampling from he correc poserior disribuion, bu moving raher slowly among hose rajecories ha are plausible given he observaions. 7 Conclusions We have shown ha he embedded HMM can work very well for a non-linear model wih a low-dimensional sae. For he higher-dimensional moion racking example, he embedded HMM has some difficulies exploring he full poserior disribuion, due, we hink, o he difficuly of creaing pool disribuions wih a dense enough sampling of saes o allow linking of new saes a adjacen imes. However, he paricle smooher was even more severely affeced by he high dimensionaliy of his problem. The embedded HMM herefore appears o be a promising alernaive o paricle smoohers in such conexs. The idea behind he embedded HMM should also be applicable o more general reesrucured graphical models. A pool of values would be creaed for each variable in he ree (which would include he curren value for he variable). The fas sampling algorihm possible for such an embedded ree (a generalizaion of he sampling algorihm used for he embedded HMM) would hen be used o sample a new se of values for all variables, choosing from all combinaions of values from he pools. Finally, while much of he elaboraion in his paper is designed o creae a Markov chain whose equilibrium disribuion is exacly he correc poserior, π(x), he embedded HMM idea can be also used as a simple search echnique, o find a sae sequence, x, which maximizes π(x). For his applicaion, any mehod is accepable for proposing pool saes (hough some proposals will be more useful han ohers), and he selecion of a new sae sequence from he resuling embedded HMM is done using a Vierbi-syle dynamic programming algorihm ha selecs he rajecory hrough pool saes ha maximizes π(x). If he curren sae a each ime is always included in he pool, his Vierbi procedure will always eiher find a new x ha increases π(x), or reurn he curren x again. This embedded HMM opimizer has been successfully used o infer segmen boundaries in a segmenal model for voicing deecion and pich racking in speech signals [1], as well as in oher applicaions such as robo localizaion from sensor logs. Acknowledgmens. This research was suppored by grans from he Naural Sciences and Engineering Research Council of Canada, and by an Onario Premier s Research Excellence Award. Compuing resources were provided by a CFI gran o Geoffrey Hinon. References [1] Achan, K., Roweis, S. T., and Frey, B. J. (2004) A Segmenal HMM for Speech Waveforms, Technical Repor UTML-TR , Universiy of Torono, January [2] Douce, A., Godsill, S. J., and Wes, M. (2000) Mone Carlo filering and smoohing wih applicaion o ime-varying specral esimaion Proc. IEEE Inernaional Conference on Acousics, Speech and Signal Processing, 2000, volume II, pages [3] Neal, R. M. (1993) Probabilisic Inference Using Markov Chain Mone Carlo Mehods, Technical Repor CRG-TR-93-1, Dep. of Compuer Science, Universiy of Torono, 144 pages. Available from hp:// radford. [4] Neal, R. M. (2003) Markov chain sampling for non-linear sae space models using embedded hidden Markov models, Technical Repor No. 0304, Dep. of Saisics, Universiy of Torono, 9 pages. Available from hp:// radford. [5] Sco, S. L. (2002) Bayesian mehods for hidden Markov models: Recursive compuing in he 21s cenury, Journal of he American Saisical Associaion, vol. 97, pp