THERE has been a growing interest in using switching diffusion

Transcription

1 1706 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 5, MAY 2007 Two-Time-Scale Approximation for Wonham Filters Qing Zhang, Senior Member, IEEE, G. George Yin, Fellow, IEEE, Johb B. Moore, Life Fellow, IEEE Abstract This paper is concerned with approximation of Wonham filters. A focal point is that the underlying hidden Markov chain has a large state space. To reduce computational complexity, a two-time-scale approach is developed. Under time scale separation, the state space of the underlying Markov chain is divided into a number of groups such that the chain jumps rapidly within each group switches occasionally from one group to another. Such structure gives rise to a limit Wonham filter that preserves the main features of the filtering process, but has a much smaller dimension therefore is easier to compute. Using the limit filter enables us to develop efficient approximations useful filters for hidden Markov chains. The main advantage of our approach is the reduction of dimensionality. Index Terms Hidden Markov chain, two-time-scale Markov process, Wonham filter. I. INTRODUCTION THERE has been a growing interest in using switching diffusion systems for emerging applications in wireless communication, signal processing, financial engineering. Unlike the pure diffusion models used in the traditional setup, both continuous dynamics discrete events coexist in the regimeswitching models. The hybrid formulation makes the models more versatile, but the analysis becomes more challenging. A class of hybrid diffusions uses a continuous-time Markov chain to capture the discrete event features resulting in a set of diffusions modulated by the Markov chain. To carry out control optimization tasks for regime-switching diffusions under partial observations, it is desirable to extract characteristics or features of the systems based on the limited information available, which brings us to the framework of hybrid filtering. Optimal filtering of hybrid systems typically yields infinite-dimensional stochastic differential equations. Efforts have been made to find finite-dimensional approximations. Some of these approximation schemes can be simplified if the conditional probability of the Markov chain given observation over time is available. In this paper, we consider the model in which a function of the Markov chain is observed in additive white Manuscript received April 12, 2005; revised January 23, This work was supported in part by the National ICT Australia, which is funded by the Australian Government Backing Australia s Ability Initiative in part through the Australian Research Council, in part by the ARC under discovery Grants A DP , in part by the National Science Foundation. Q. Zhang is with the Department of Mathematics, Boyd GSRC, The University of Georgia, Athens, GA USA ( qingz@math.uga.edu). G. Yin is with the Department of Mathematics, Wayne State University, Detroit, MI USA ( gyin@math.wayne.edu). J. B. Moore is with the Department of Systems Engineering, Australian National University, Canberra, ACT 0200, Australia ( john.moore@anu. edu.au). Communicated by A. Høst-Madsen, Associate Editor for Detection Estimation. Digital Object Identifier /TIT noise. We focus on the conditional probability of the chain given the observation. The corresponding filter, developed in [26], is known as the Wonham filter is given by the solution of a system of stochastic differential equations. A. Wonham Filter In the literature, after the Kalman filter was developed in the 1960s, the first rigorous development of nonlinear filters for diffusion-type processes came into being; see Kushner [18]. Nonlinear filter problems soon attracted growing continued attention. In contrast to the Kalman filter, which is finite dimensional, it is well known that nonlinear filters are generally infinite dimensional. There are only a hful of finite-dimensional nonlinear filters known to exist to date. The first of such finite-dimensional filters was the Wonham filter of [26]. Owing to its importance, it has received much attention. In the new era, because of the use of hidden Markov models, filters involving jump processes have drawn resurgent attention. For example, to characterize stock price movements, one may use to represent the stock trends. Suppose that we can observe, the percentage change of the stock price represented by plus white noise, is an appropriate function. For simplicity, suppose. For example, represents an up trend with adown trend with ); see [34] for more details related literature. In manufacturing applications, one may take as the discrete dem for a product that is corrupted by Gaussian noise. In a recent paper [25], Wang, Zhang, Yin considered Kalman-type filters for the partially observed system is the continuous state variable, is a finite-state Markov chain (a discrete-event state), is the observation process; they developed a quadratic variation test near-optimal filters by examining the associated system of Riccati equations. In the current paper, we concentrate on Wonham filters, in which only noisy corrupted observations of a Markov chain are available. Thus, there are no Riccati equations we can utilize. The methods developed in [25] do not apply. To proceed, we first summarize results about the Wonham filter. Let be a continuous-time Markov chain having finite state space generator. A function of the Markov chain together with additive Gaussian noise is observed. Let denote the observation given by is a positive constant is a stard Brownian motion independent of. (1) /$ IEEE

2 ZHANG et al.: TWO-TIME-SCALE APPROXIMATION FOR WONHAM FILTERS 1707 Let be the conditional probability of given the observations up to time, i.e., for. Let. Then the Wonham filter is given by initial probability vector (2) Here henceforth, we use as an identity matrix of appropriate dimension use as a generic constant with the convention. B. Brief Review of Literature Owing to its importance, filtering problems have received much attention, various developments have been made. For example, Caines Chen [4] derived an optimal filter when it involves a rom variable but with no switching; see also Hijab [12]. Haussmann Zhang [11] used two statistical hypothesis tests, the quadratic variation test the likelihood ratio test, to estimate the value of the rom variable to choose among competing filters on successive time intervals. These results are generalized in Zhang [31] to incorporate unobservable Markov chains. Concerning nonlinear filtering of a hybrid system in discrete time, Blom Bar-Shalom [3] proposed a numerical algorithm to compute the conditional expectation of the state given observations up to time. The algorithm seems to perform well numerically. However, there is no mathematical analysis for optimality or near optimality of these filters; see Li [20] for further discussions. For related work on filtering, see Dey Moore [6] Moore Baras [22] for risk sensitive filtering; Wang, Zhang, Yin [25] Yin Dey [27] for reduction of complexity of filtering problems involving large-scale Markov chains; Zhang [33], [32] for the most probable estimates in discrete-time continuous-time models, respectively; Liu Zhang [21] for numerical experiments involving piecewise approximation of nonlinear systems; Yin, Zhang, Liu [30] for numerical methods of Wonham filters. For a survey of results on filtering, we refer the reader to the books by Anderson Moore [1] on classical linear filtering. For hidden Markov models related filtering problems, see Elliott, Aggoun, Moore [8]. For general nonlinear filtering, see Kallianpur [16] Liptser Shiryayev [15]; see also the books by Bensoussan [2] Kushner [19] for related topics on partially observed systems. The primary concern of this paper is on constructing Wonham filters for Markov chains with a large state space. Note that related results may be found in Tweedie [24] for quasi-stationary distributions of Markov processes for a general state space, Huisinga, Meyn, Schutte [13] for a spectral theory approach to approximation of a complex Markov process with a simpler one, Jerrum [14] for further discussion. When the state space of the Markov chain is large, the number of the filter equations will be large comprising a switching diffusion system with large dimension. We focus on developing good approximations of such large-dimensional filters. The main idea is to use time-scale separation hierarchy within the Markov chain to reduce the computation complexity. In applications of manufacturing (see [23, Sec. 5.9]) in system reliability (see [28, Sec. 3.2]), the state space of the Markov chain can be partitioned to a number of groups so that the Markov chain jumps rapidly within a group of states less frequently (or occasionally) among different groups. Under such a setup, due to the fast variation of the Markov chain, it is difficult to pinpoint the exact location of the Markov chain. Nevertheless, it is much easier to identify if the chain belongs to certain groups. This leads to a two-time-scale formulation involving states having weak strong interactions. C. Contributions of this Paper Our contributions in this paper include the following: 1) presentation of a two-time-scale filter formulation; 2) construction of a limit filter; 3) proof of filter convergence to the desired limit Wonham filter as the rate of fluctuations of the Markov chain goes to infinity in each group of irreducible states; 4) construction of an approximation scheme based on the limit filter that is easier to compute; 5) proof that the original filter can be approximated in a twostage procedure under different topologies, establishment of the asymptotic optimality of the approximate filter. Before proceeding further, we point out: First, the time scale separation in this paper is formulated by using a small parameter. The asymptotic results to follow require that approaches zero. However, in applications, can be a fixed constant. For example, given the magnitude of other parameters being of order, can take the value or. Mainly, the small parameter brings out the different scales of the jump rates in different states of the Markov chain. Second, in the formulation, the Markov chain has a particular structure. Since any finite-state Markov chain has at least one recurrent state, conversion to such a canonical form is always possible; see for example, [28, Sec. 3.6] references therein. D. Outline The rest of the paper is organized as follows. In the next section, we present notation needed for two-time-scale Markov chains summarize relevant results to be used in this work. In Section III, we consider limit filters two-time-scale approximations verification of these results. In Section IV, we provide a numerical example to illustrate the main results of this paper. In Section V, extension of results to Markov chains with

3 1708 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 5, MAY 2007 transient states are considered. The paper is concluded with a few remarks. II. SINGULARLY PERTURBED MARKOV CHAINS Suppose that is a continuous-time Markov chain whose generator is. We say that the Markov chain or the generator is weakly irreducible if the system of equations we obtain a process with considerably smaller state space. To be more specific, the aggregated process is defined by when (6) Note that is not necessarily Markovian. However, using certain probabilistic arguments assuming to be weakly irreducible, we have shown the following in [28, Sec. 7.5]. (a) converges weakly to, which is a continuoustime Markov chain generated by has a unique solution satisfying for. The solution (row-vector-valued function) is termed a quasi-stationary distribution. Note that the definitions were used in our work [17] [28]. They are different from the usual definitions of irreducibility stationary distribution in that we do not require all the components ; they are also different than that of [24]. A. Time Scale Separation in Markov Chains In this work, we focus on Markov chains that have large state spaces with complex structures. Suppose that the states of the underlying Markov chain are divisible to a number of weakly irreducible classes such that the Markov chain fluctuates rapidly among different states within a weakly irreducible class, but jumps less frequently from one weakly irreducible class to another. To highlight the different rates of variation, introduce a small parameter assume the generator of the Markov chain to be of the form Throughout the paper, we assume both to be generators. As a result, the Markov chain becomes an -dependent singularly perturbed Markov chain. An averaging approach requires aggregating the states in each weakly irreducible class into a single state, replacing the original complex system by its limit, an average with respect to the quasi-stationary distributions. In this the following three sections, we concentrate on the case that the underlying Markov chain has weakly irreducible classes with no transient states, which specifies the form of as Here, for each is the weakly irreducible generator corresponding to the states in, for. The state space is decomposed as Note that governs the rapidly changing part describes the slowly varying components. Lumping the states in into a single state, an aggregated process, containing states, is obtained, in which these states interact through the matrix resulting in transitions from to. Thus, by aggregation, (3) (4) (5) is the quasi-stationary distribution of is an -dimensional column vector with all components being equal to is a block-diagonal matrix with appropriate dimensions. (b) For any bounded deterministic, then is the indicator function of a set. (c) Let. Then for some. Note that for the process, the state space is given by. For complete treatment of two-time-scale Markov chains in continuous time, see the book by Yin Zhang [28]. In addition, we would like to point out that the aggregation process depends on the decomposition of in (4). Substantial reduction of dimensionality can be achieved when. B. Two-Time-Scale Wonham Filters Let be the observation given by (7) (8) (9) (10) is a positive constant is a stard Brownian motion. We assume that are independent. Let denote the conditional probability of given the observation up to time, i.e., for. Let such that Let

4 ZHANG et al.: TWO-TIME-SCALE APPROXIMATION FOR WONHAM FILTERS 1709 We next derive the equation for. First, recall (8) the convergence of. Intuitively, we have (11) Then the corresponding Wonham filter can be rewritten as with given initial condition (12) III. LIMIT FILTER AND TWO-TIME-SCALE APPROXIMATION Therefore, we expect the weak limit of to have the form A. Limit Filter Intuitively, similar to the probability distributions of two-time-scale Markov chains, as, the conditional probability in Wonham filter for should converge to a limit filter. In this section, we first derive formally the limit filter then provide a verification theorem that shows that the limit filter is indeed the limit of the original filter as in an appropriate sense. Write (12) in its integral form note the boundedness of both. It follows that for some finite for all. Therefore, we have The proof of this can be found in Wang, Zhang, Yin [25]. Recall that. In (12), multiplying from the right by sending, we obtain (13) with initial condition This implies that Moreover, note that are conditional probability measures that are uniformly bounded between.if as for some, then necessarily This implies. In view of the block-diagonal structure of, the vector must have the following form: for (14) We will show in what follows that, for each can be approximated by in a two stage procedure as. B. Two-Time-Scale Approximation Note that the noise driving the limit filter is the weak limit of. In order to use the filter in real-time applications, one needs to feed the filter by the actual observation in (13). Let denote such a filter with given by (15) later. Recall the definition of is to be determined in (7). It follows that,

5 1710 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 5, MAY 2007 Then we have the following theorem. Theorem 3.1: The following assertions hold. (a) is an approximation to for small. More precisely for some constant. (b) converges weakly to in, denotes the space of -valued continuous functions defined on. Remark 3.2: This theorem reveals that the two-stage approximation of leads to the limit with. Stage 1 approximation provides a practical way for computing using that is governed by a system of stochastic differential equations (SDEs) of much smaller dimension. Stage 2 approximation leads to a theoretical weak limit for completeness of the two-time-scale analysis. Proof of Part (a): Let Write Recall that. It follows that Let Note that (16) or in differential form (17) Right-multiplying both sides by the matrix yields Integrating both sides from to leads to In view of this relation, (11) (16) yield that Moreover, owing to (12) (15), equation: satisfies the following Note that the term is a row vector the integr is a matrix. Recall that. It is easy to check that

6 ZHANG et al.: TWO-TIME-SCALE APPROXIMATION FOR WONHAM FILTERS 1711 with defined in (14). Recall the uniform boundedness of. We can show with initial, Moreover, in view of (15) (17), we have Using these (9), we have (18) Step 1. We show that converges to weakly. Recall that in distribution. By the Skorohod representation theorem (see Ethier Kurtz [7]), there exists a probability space processes such that for some constant. Using Cauchy Schwarz inequality noting,wehave (19) Moreover, recall that. It follows that almost surely (a.s.) in,. Let be a probability space upon which is a stard Brownian motion. Then, on the product space a.s. in Step 2. We show that for each,as. Let. Then. Moreover, in view of (15) (21) by reordering terms, we have (20) Let. It follows from (18) (20) that for some positive constants. Finally, the Gronwall s inequality (see [10, p. 36]) implies that. Write i.e., This proves Part (a). Proof of Part (b): The proof for Part (b) is divided into two steps. First, we introduce an intermediate process defined by It is easy to see that (21)

7 1712 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 5, MAY 2007 for some constant. This implies that In view of the Gronwall s inequality, it suffices to show. Let Then (8) implies, as. For a diffusion with bounded, let Then, using integration by parts, we have Finally, noting both have continuous sample paths a.s., the convergence, in fact, takes place on the space. Remark 3.3: The conditional probability vector does not converge in a neighborhood (of size )of due to an initial layer with thickness near the origin. Note that need not be the same as approaches for away from the initial layer. These observations are summarized in the next corollary. Corollary 3.4: The following assertions hold. (a). (b) For any. (c) For each in distribution. Proof: Parts (a) (b) are immediate from Theorem 3.1. To see Part (c), note that for all, wehave It follows that Take write IV. A NUMERICAL EXAMPLE In this section, we consider a simple example involving a four-state Markov chain. Let de- to obtain. Step 3. Finally, it is easy to show that the process fined in (15) satisfies the following conditions: The corresponding state space is for each for all (22) In this case (23) denotes the conditional expectation given. Using the tightness criterion [19, p. 47], it is easy to see that is tight. To complete the proof, it suffices to show the weak convergence of finite-dimensional distributions. To this end, note that following Steps 1 2, we have, for any (the space of functions whose second derivatives are bounded) The corresponding quasi-stationary distributions (in this case, also stationary distributions) are given by as solutions to. Note that the stationary distributions depend only on the block matrices in. Let denote the conditional probability vector its approximation by.define the norm In this example, we take for any constants. This implies converges to.

8 ZHANG et al.: TWO-TIME-SCALE APPROXIMATION FOR WONHAM FILTERS 1713 Fig. 1. Sample paths of (t); p (t); ~p (t) 0 p (t) with " =0:05. the discretization step size. A sample path of (with ) the corresponding conditional probabilities are given in the first five rows in Fig. 1. In Fig. 1, the states are labeled as. The differences between are plotted in the last four rows. As can be seen in Fig. 1, stays in group from to, jumps to group at, goes back to at, then to, finally ls in from to. The approximation filter tracks the corresponding conditional probabilities pretty well on these time intervals. TABLE I DEMONSTRATION OF ERROR BOUNDS In addition, we vary run 1000 samples for each. The results are recorded in Table I. As can be seen in Table I, the differences between the exact conditional probabilities their approximations are fairly small. The result validates the effectiveness of our approach in this simple example.

9 1714 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 53, NO. 5, MAY 2007 V. INCLUSION OF TRANSIENT STATES In the previous sections, we have concentrated on the case that the underlying Markov chain consists of weakly irreducible classes only. This section takes up the issue that the underlying Markov chain contains weakly irreducible classes as well as transient states. In this case, the state space is partitioned as (24) is the collection of the transient states, the generator is still of the form (3), but Write Define (28)... (25) To distinguish the transient states with that of the states in weakly irreducible classes, we use as an index for the transient states. We assume that is asymptotically stable, i.e., all of its eigenvalues belong to the left half of the complex plane. To proceed, define We proved in [29, Theorem 4.3] that converges weakly to, a continuous-time Markov chain generated by given in (28). We also obtained similar mean-squares estimates for the occupation measures as in the previous case. In fact for for Moreover, let. Then... (26) Define as follows: is an zero matrix, Let be a rom variable uniformly distributed on that is independent of. For each, define an integer-valued rom variable by We proceed to define the aggregated process, in which the aggregation is only taken over each weakly irreducible class. Define Using the partition if if for (27) with initial condition Similarly, define as above with replaced by. We can prove the following results similarly to the proof of Theorem 3.1. Theorem 5.1: Let. Then, we have (a), for some ; (b) converges to weakly on. VI. CONCLUDING REMARKS In this paper, we have developed approximate Wonham filters under the framework of two-time-scale Markov chains. The

10 ZHANG et al.: TWO-TIME-SCALE APPROXIMATION FOR WONHAM FILTERS 1715 advantage of this approach is the reduction of dimensionality. Our approach leads to a limit filter that is close to the original Wonham filter but of much smaller dimension. Such a filtering scheme is desirable in state estimation involving Markov chains with a large number of states such as in production planning of stochastic manufacturing systems. Recently, there has been resurgent interest in constructing robust filters inspired by the work of Clark [5]. In Elliott, Malcolm, Moore [9], unnormalized densities are considered. It will be interesting to see if the two-time-scale approach can be used to treat such problems. Nevertheless, care must be taken. Specifically, in such a setup, the counterpart of satisfies an ordinary differential equation; it is not bounded. More thoughts work are required for this. ACKNOWLEDGMENT The authors are very grateful to the referees the editors for many valuable comments suggestions, which led to improvements of the paper. REFERENCES [1] B. D. O. Anderson J. B. Moore, Optimal Filtering. Englewood Cliffs, NJ: Prentice-Hall, [2] A. Bensoussan, Stochastic Control of Partially Observed Systems. Cambridge, U.K.: Cambridge Univ. Press, [3] H. A. P. Blom Y. Bar-Shalom, The interacting multiple model algorithm for systems with markovian switching coefficients, IEEE Trans. Automat. Control, vol. 33, no. 8, pp , Aug [4] P. E. Caines H. F. Chen, Optimal adaptive LQG control for systems with finite state process parameters, IEEE, Trans. Automat. Control, vol. AC-30, no. 2, pp , Feb [5] J. M. Clark, The design of robust approximation to the stochastic differential equation for nonlinear filtering, in Communications Systems Rom Process Theory, J. K. Skwirzynski, Ed. Alphen aan den Rijn, The Netherls: Sijthoff Nordhoof, 1978, pp [6] S. Dey J. B. Moore, Risk-sensitive filtering smoothing for hidden Markov models, Syst. Control Lett., vol. 25, no. 5, pp , [7] S. N. Ethier T. G. Kurtz, Markov Processes: Characterization Convergence. New York: Wiley, [8] R. J. Elliott, L. Aggoun, J. B. Moore, Hidden Markov Models: Estimation Control. New York, NY: Springer-Verlage, [9] R. J. Elliott, W. P. Malcolm, J. B. Moore, Robust control of a partially observed Markov chain, working paper. [10] J. Hale, Ordinary Differential Equations, 2nd ed. Malabar, FL: Krieger, [11] U. G. Haussmann Q. Zhang, Stochastic adaptive control with small observation noise, Stochastics Stochastics Rep., vol. 32, pp , [12] O. Hijab, The adaptive LQG problem-part I, IEEE, Trans. Automat. Control, vol. AC-28, no. 2, pp , Feb [13] W. Huisinga, S. P. Meyn, C. Schutte, Phase transitions metastability in markovian molecular systems, Ann. Appl. Probab., vol. 14, pp , [14] M. Jerrum, On the Approximation of One Markov Chain by Another, School of Informatics, Univ. Edinburgh. Edinburgh, U.K., Working paper, [15] R. S. Lipster A. N. Shiryayev, Statistics of Rom Processes. New York: Springer-Verlag, 1977, two volumes. [16] G. Kallianpur, Stochastic Filtering Theory. New York: Springer- Verlag, [17] R. Z. Khasminskii, G. Yin, Q. Zhang, Asymptotic expansions of singularly perturbed systems involving rapidly fluctuating Markov chains, SIAM J. Appl. Math., vol. 56, pp , [18] H. J. Kushner, On the differential equations satisfied by conditional probability densities of Markov processes with applications, SIAM J. Control, vol. 2, pp , [19] H. J. Kushner, Approximation Weak Convergence Methods for Rom Processes With Applications to Stochastic Systems Theory. Cambridge, MA: MIT Press, [20] X. R. Li,, C. T. Leondes, Ed., Hybrid Estimation Techniques, in Control Dynamic Systems. San Diego, CA: Academic, 1996, vol. 76. [21] R. H. Liu Q. Zhang, Nonlinear filtering: A hybrid approximation scheme, IEEE Trans. Aerosp. Electron. Syst., vol. 37, no. 2, pp , Apr [22] J. B. Moore J. S. Baras, Finite-dimensional optimal controllers for nonlinear plants, Syst. Control Lett., vol. 26, pp , [23] S. P. Sethi Q. Zhang, Hierarchical Decision Making in Stochastic Manufacturing Systems, ser. Systems & Control: Foundations & Applications. Boston, MA: Birkhäuser, [24] R. L. Tweedie, Quasi-stationary distributions for Markov chains on a general state space, J. Appl. Probab., vol. 11, pp , [25] J. W. Wang, Q. Zhang, G. Yin, Two-time-scale hybrid filters: Near optimality, SIAM J. Control Optim., vol. 45, pp , [26] W. M. Wonham, Some applications of stochastic differential equations to optimal nonlinear filtering, SIAM J. Control, vol. 2, pp , [27] G. Yin S. Dey, Weak convergence of hybrid filtering problems involving nearly completely decomposable hidden Markov chains, SIAM J. Control Optim., vol. 41, pp , [28] G. Yin Q. Zhang, Continuous-Time Markov Chains Applications: A Singular Perturbation Approach. New York: Springer-Verlag, [29] G. Yin, Q. Zhang, G. Badowski, Asymptotic properties of a singularly perturbed Markov chain with inclusion of transient states, Ann. Appl. Probab., vol. 10, pp , [30] G. Yin, Q. Zhang, Y. J. Liu, Discrete-time approximation of wonham filters, J. Control Theory Applic., vol. 2, pp. 1 10, [31] Q. Zhang, Nonlinear filtering control of a switching diffusion with small observation noise, SIAM J. Control Optim., vol. 36, pp , [32] Q. Zhang, Optimal filtering of discrete-time hybrid systems, J. Optim. Theory Applic., vol. 100, pp , [33] Q. Zhang, Hybrid filtering for linear systems with non-gaussian disturbances, IEEE Trans. Automat. Control, vol. 45, no. 1, pp , Jan [34] Q. Zhang G. Yin, Nearly optimal asset allocation in hybrid stock-investment models, J. Optim. Theory Applic., vol. 121, pp , 2004.