Estimation and Control across Analog Erasure Channels

Transcription

1 Estimation and Control across Analog Erasure Channels Vijay Gupta Department of Electrical Engineering University of Notre Dame 1 Introduction In this chapter, we will adopt the analog erasure model to describe the communication channel present inside a control loop. his model, also referred to as the packet erasure or packet loss model, can be described as follows [8]. he channel operates in discrete time steps. At every time step, the channel accepts as input a finite dimensional real vector rk. he value of the output of the channel yk is chosen according to an erasure process. At every time step, the erasure process assumes either the value or the value R. If the value at time k is, yk + 1 = rk and a successful transmission is said to have occurred. Otherwise, yk + 1 = φ and an erasure event, or a packet loss, is said to have occurred at time k. he symbol φ denotes that the receiver does not receive any data; however, the receiver is aware that an erasure event has occurred at that time. Note that we have assumed that the channel introduces a constant delay of one time step. he analog erasure model aims to capture the data loss effect due to a communication channel. Due to effects such as interference and fading in wireless channels, congestion in shared networks, or even overload and interrupts at a micro-controller, various parts of a control loop between the sensor and controller, or the controller and actuator, may exhibit information loss. By considering the idealization in which every successful transmission results in the communication of a real vector of a bounded dimension, such situations can be modeled using an analog erasure representation. While an analog erasure model has an infinite capacity in an information theoretic sense, it is often a useful representation for the cases when the communication protocols allow for large data packets to be transmitted at every time step. For instance, the minimum size of an ethernet data packet is 72 bytes. his is much more space for carrying information than usually required inside a control loop. If the data packets allow for transmission of control and sensing data to a high fidelity, the quantization effects are often ignored and an analog erasure model adopted. Various descriptions of the erasure process are possible. he following two models are the most popular: 1. Maximum Allowable ransmit Interval MAI based models: his model see, e.g., [3] is described using two integer values n 1 and n 2. For an n 1, n 2 model, in any interval of length n 1, at most n 2 erasure events can occur. Apart from this constraint, the erasure process is arbitrary. 2. Stochastic erasure event based models: In this model see, e.g., [12], the erasure process is a random process. he simplest case is when the erasure events are independent and identically distributed at different time steps. In such a case, the erasure process is described by the erasure probability p def = Probyk = φ, at any time step k. More complicated models when the erasure process can be described by, e.g., a Markov chain possibly on the lines of the Gilbert-Eliot channel model [4], can also be considered. In this chapter, we will concentrate on the stochastic erasure event based models. As with any networked control system, two questions can be asked. 1. What is the effect of introducing channels on the structure of the estimators and controllers? 2. What are the optimal encoders and decoders that transmit the maximum amount of control relevant information to achieve the fundamental limits of performance in such systems? hese questions are answered in the next two sections, respectively. Section 4 lists some extensions to the simple model considered here for pedagogical ease, and points out some open research directions. Some results on Markovian jump linear systems that are used in the chapter are presented in the Appendix. 1

2 Figure 1: Basic set-up of the estimation problem. 2 Effect on Estimation and Control Performance 2.1 Estimation We begin with the problem of estimating a linear time invariant process across an analog erasure channel. Consider the setup as shown in Figure 1. he process with state xk R n evolves as xk + 1 = Axk + wk, where wk is the process noise modeled as white, zero mean, Gaussian with covariance R w > 0. he initial condition x0 is assumed to be drawn from a Gaussian distribution with zero mean and covariance Π0. he state is observed by a sensor of the form yk = Cxk + vk, where vk R p is measurement noise modeled by white, zero mean, Gaussian with covariance R v > 0. We assume that the sources of randomness x0, {wj} and {vj} are mutually independent. he sensor transmits its measurements to an estimator across an analog erasure channel with erasure probability p at every time step. he pair A, C is assumed to be observable. he estimator receives those measurements that are transmitted successfully across the channel. It has access to the constant matrices A, C, R w, R v and Π0. he estimator aims at generating the minimum mean squared error MMSE estimate ˆxk of the process state xk based on the information it has access to at time k. If no erasure events were to occur, the optimal estimate is given by the Kalman filter, and the estimate error covariance evolves according to a discrete Riccati recursion. In particular, for the assumptions as made above, the error covariance is stable in the sense of being bounded as time k increases. We wish to extend this analysis to the case when erasure events occur with a probability p at every time step. Since the event of a packet drop is known to the estimator, the problem is equivalent to estimation of the following Markov jump linear system with the jumps occurring according to a Bernoulli process: xk + 1 = Axk + wk yk = C rk xk + vk, where rk is the Markov state such that rk = 1 with probability 1 p and rk = 2 otherwise. Moreover, C 1 = C and C 2 = 0. We can thus utilize the standard results from Markovian jump linear system theory 1. In particular, from Corollary 5.1 we obtain that the optimal estimator for such a system is provided by a time varying Kalman filter. Due to probabilistic erasure events, the estimate error covariance Πk evolves as a random variable. An upper bound of the expected estimate error covariance E[Πk] is provided by the quantity V k that evolves as V k + 1 = AV ka + R w 1 pav kc CV kc + R v CV ka, with the initial condition V 0 = Π0. A sufficient condition for stabilizability can also be obtained through Corollary 5.1. We can also express the condition as a Linear Matrix Inequality LMI [14]. he system is stabilizable if there exists a matrix X > 0 and a matrix K such that X 1 pxa + KC pxa 1 pa X + C K X 0 > 0. pa X 0 X For our problem, we can also derive a lower bound on E[Πk] as follows. At time step k, the error covariance Πk is lower bounded by R w if a packet is received by the estimator, and is equal to AΠk 1A + R w if a packet is not received. hus, E[Πk] is lower bounded by Sk which evolves as Sk + 1 = p ASkA + R w + 1 prw = paska + R w. his is a discrete algebraic Lyapunov recursion. Using the property for the recursion to converge, we obtain that a necessary and sufficient condition for stability of Sk and, thus, a necessary condition for stability of E[Πk] is given by pρa 2 < 1, where ρa is the spectral radius of A. 1 A brief review of such results is provided in the Appendix. 2

3 Figure 2: Basic set-up of the control problem with a single channel present between the sensor and the controller. 2.2 Control We now move on to the control problem considered, e.g., in [9, 14]. o begin with, consider the set-up shown in Figure 2 that has only one channel in the control loop, present between the sensor and the controller. Such a situation can arise, e.g., when the controller is co-located with the process and the sensor is remote, or the controller has access to large transmission power. he linear time invariant process now evolves as xk + 1 = Axk + Buk + wk, where the additional variable uk R m is the control input chosen to minimize the cost [ K J LQG = E x kqxk + u kruk ] + x K + 1P K + 1xK + 1, k=1 where the expectation at time k is taken with respect to the future values of the packet erasure events, the initial condition, and the measurement and process noises. Further, the matrices P K + 1, Q and R are all assumed to be positive definite. he pair A, B is assumed to be controllable. We can utilize the Markov state representation to solve the LQG problem as well. he system can again be written as a Markov jump linear system of the form xk + 1 = Axk + Buk + wk yk = C rk xk + vk, where rk is the Markov state such that rk = 1 with probability 1 p and rk = 2 otherwise. Moreover, C 1 = C and C 2 = 0. hus, we can utilize the separation principle from Section A.3 to obtain the optimal controller as the combination of the LQ optimal control, with the MMSE estimate of the state used in place of the state value. Note that the MMSE estimate can be calculated as in Section 2.1. Moreover, since neither the matrix A nor the matrix B depend on the Markov state, the LQ optimal control corresponds to the control input for the system xk + 1 = Axk + Buk, that minimizes the cost function [ K J = x kqxk + u kruk ] + x K + 1P K + 1xK + 1, k=1 when the controller has full state information, i.e., to calculate the input uk at time k, the state x0, x1,, xk is available. We can also consider the case of two channels being present. Consider the system setup shown in Figure 3, with the process and the sensor as described above. In addition to the sensor-controller channel, there is an additional channel between the controller and the actuator. Assume that the erasures on the controller-actuator channel occur in an i.i.d. fashion, with probability of erasure q at any time step. Moreover, the erasure events on this channel are independent of all other random variables in the system. In this case, it is also important to specify the action that the actuator takes when it does not receive a packet. he action depends on the amount of processing, memory and information about the process that is assumed to be available at the actuator. We consider the simplest choice, which is to apply zero control input if no packet was received. Other actions by the actuator can be treated in a similar fashion. For this case, the Markov jump linear system representation of the system is now given by xk + 1 = Axk + B rk uk + wk yk = C rk xk + vk, 3

4 Figure 3: Set-up of the control problem with two channels. where rk is the Markov state that can take values 1, 2, 3 and 4 with probabilities 1 p1 q, p1 q, 1 pq and pq respectively. he system matrices are given by B 1 = B 2 = B, B 3 = B 4 = 0, C 1 = C 3 = C and C 2 = C 4 = 0. he solution of the LQG problem requires one additional assumption. he separation result in heorem 6 assumes that the controller at time k has knowledge of the control inputs that were previously applied till time k 1. Indeed, if the past control inputs are not available at the controller, then the control will have a dual effect. For our problem, this implies that the controller must know whether or not the transmission over the controller actuator channel has been successful. o provide this information to the controller, we will assume a perfect acknowledgement from the actuator to the controller for any data packet received by the actuator. his is often called the CP-like case. he case when acknowledgements are not available is termed as the UDP-like case. For the UDP-like case, the separation principle does not hold and the form of the optimal controller is unknown, in general. Note that since the controller can detect any packet drops on the sensor-controller channel, and receives acknowledgements about transmissions over the controller-actuator channel, the controller has access to the Markov states r0,, rk 1 at time k, but not rk. In particular, at time step k, the controller does not know the value of B rk. While in general heorem 6 requires knowledge of the current Markov state at the controller, in this particular case the problem is still solvable. o see this, assume that the controller uses the value B rk = B to calculate the optimal control input. If indeed r k = 1 or 2, the actuator successfully receives this packet and the control input is optimal. If r k = 3 or 4, the optimal control input should be zero since for those states B rk = 0. However, at these time steps, the transmission by the controller is not successful and the actuator applies zero as the control input. hus, once again, the optimal control input is applied. hus, we see that the LQG problem can be solved for this case using heorem 6. We can also identify stability conditions and performance bounds by utilizing the Markovian jump linear systems theory. 3 Optimal Coding We now turn to the more general question of identifying fundamental limits on the performance of a system being controlled across an analog erasure channel, and the design of encoders and decoders to achieve such limits as discussed, e.g., in [7]. o proceed, we must define the class of encoders that we will consider. he information theoretic capacity of an analog erasure channel is infinite. hus, the only constraints we impose on the encoder are that the transmitted vector is some causal possibly time-varying function of the measurements available to the encoder until time k and that the dimension of the vector is finite. he encoder is collocated with the sensor, while the decoder is located at the estimator / controller. We will sometimes refer to the encoder as an encoding algorithm. We begin by considering the system setup in Figure 2 and the associated assumptions about the process, the sensor and the cost function are as in Section 2.2. However, instead of transmitting measurements yk at every time step k, the sensor can now calculate a vector sk = f k, {yj} k and transmit it. Note that we have not assumed that the encoder has access to any acknowledgements from the decoder about which transmissions have been successful. However, we will show that the presence of such acknowledgements does not improve the optimal performance achievable by a suitable encoder. Denote by Ik the information set that the decoder can utilize to calculate the control at time k. As an example, if no erasures were happening, Ik = {y0, y1,, yk 1}. More generally, given any packet erasure pattern, we can define a time stamp t s k at every time step k such that the erasures did not allow any information transmitted by the encoder after time t s k to reach the decoder. Without loss of generality, we can restrict attention to informationset feedback controllers. For a given information set I., denote the minimal value of the cost J LQG that can be achieved with the optimal controller design by JLQG I, and the smallest sigma algebra generated by the information set as I.. If two information sets I 1. and I 2. are such that I 1 k I 2 k, we have JLQG I2 JLQG I1. Consider an algorithm A 1 in which at every time step k, the sensor transmits all measurements y0, y1,, yk to the decoder. Note that this algorithm is not a valid encoding algorithm since the dimension of the transmitted 4

5 vector is not bounded, as k increases. However, with this algorithm, for any drop sequence, the decoder has access to an information set of the form I max k = {y0, y1,, yt s k}, where t s k k 1 is the time stamp defined above. his is the maximal information set that the decoder can have access to with any algorithm, in the sense that Ik I max k, for any other algorithm that yields the information set Ik. hus, one way to achieve the optimal value of the cost function is to utilize an algorithm that makes I max k available to the sensor at every time k along with a controller that optimally utilizes this set. Further, one such encoder algorithm is A 1. However, as discussed above, A 1 is not a valid encoding algorithm. Surprisingly, as shown below, we can achieve the same performance with an algorithm that transmits a vector with finite dimension. We begin with the following separation principle when the decoder has access to the maximal information set. Denote by ˆαk βk the minimum mean squared error MMSE estimate of the random variable αk based on the information βk. heorem 1 Separation Principle with Maximal Information Set. Consider the control problem as defined above, when the decoder has access to the maximal information set I max k at every time step. hen, the optimal control input is given by uk = û LQ k I max k, {uj} k 1, where u LQ k is the optimal LQ control law. he proof of this result is similar to the standard separation principle see, e.g., [10, Chapter 9] and is omitted here. For our setting, the importance of this result lies in the fact that it recognizes that û LQ k I max k, {uj} k 1 or, in turn, ˆx LQ k I max k, {uj} k 1 is a sufficient statistic to calculate the control input that achieves the minimum possible cost for any encoding algorithm. Utilizing the fact that the optimal MMSE estimate of xk is linear in the effects of the maximal information set and the previous control inputs, we can identify the quantity that the encoder should transmit that depends only on the measurements. We have the following result. heorem 2 Separation of the Effect of the Control Inputs. he quantity ˆx LQ k I max k, {uj} k 1 can be calculated as ˆx LQ k I max k, {uj} k 1 = xlq k I max k + ψk, where x LQ k I max k depends only on I max k but not on the control inputs and ψk depends only on the control inputs {uj} k 1. Further both x LQ k I max k and ψk can be calculated recursively. Proof. he proof follows readily from noting that ˆx LQ k I max k, {uj} k 1 can be obtained from the Kalman filter which is affine in both measurements and control inputs. We can identify where xj + 1 j evolves as x LQ k I max k = A k t sk 1 xt s k + 1 t s k k t sk 2 ψk = A k t sk 1 ψts k A i Buk i 1, i=0 M 1 j j = M 1 j j 1 + C Rv 1 C M 1 j j xj j = M 1 j j 1 xj j 1 + C Rv 1 yj Mj j 1 = AMj 1 j 1A + R w xj j 1 = A xj 1 j 1, with the initial conditions x0 1 = 0 and M0 1 = Π0, and ψj evolves as with the initial condition ψ0 = 0. ψj = Buj 1 + Γj 1 ψj 1 Γj = AM 1 j 1 j 1Mj 1 j 2, Now consider the following algorithm A 2. At every time step k, the encoder calculates and transmits the quantity xk k using the algorithm in the above proof. he decoder calculates the quantity ψk. If the transmission is successful, the decoder calculates ˆx LQ k + 1 I max k + 1, {uj} k = xlq k + 1 I max k ψk = A xk k + ψk. 5

6 If the transmission is unsuccessful, the decoder calculates ˆx LQ k + 1 I max k + 1, {uj} k = A k t s k x LQ k + 1 I max t s k ψk, where the quantity x LQ k + 1 I max t s k + 1 is stored in the memory from the last successful transmission note that only the last successful transmission needs to be stored. Using the heorems 1 and 2 clearly allows us to state the following result. heorem 3 Optimality of the Algorithm A 2. Algorithm A 2 is optimal in the sense that it allows the controller to calculate the control input uk that minimizes J LQG. Proof. At every time step, the algorithm A 2 makes ˆx LQ k + 1 I max k + 1, {uj} k available to the controller. hus, the controller can calculate the same control input as with the algorithm A 1 which together with an LQ controller yields the minimum value of J LQG. Note that the optimal algorithm is non-linear in particular, it is a switched linear system. his is not unexpected, in view of the non-classical information pattern in the problem. Remarks Boundedness of the ransmitted Quantity: It should be emphasized that the quantity xk k that the encoder transmits is not the estimate of xk or the state of some hypothetical open loop process based only on the measurements y0,, yk. In particular under the constraint on the erasure probability that we derive later, the state xk is stable and hence the measurements yk are bounded. hus, the quantity xk k is bounded. his can also be seen from the recursive filter used in the proof of heorem 2. If the closed loop system xk is unstable due to high erasure probabilities, xk k would, of course, not be bounded. However, the optimality result implies that the system cannot be stabilized by transmitting any other bounded quantity such as measurements. Optimality for any Erasure Pattern and the Washing Away Effect: he optimality of the algorithm required no assumption about the erasure statistics. he optimality result holds for an arbitrary erasure sequence, and at every time step not merely in an average sense. Moreover, any successful transmission washes away the effect of the previous erasures in the sense that it ensures that the control input is identical to the case as if all previous transmissions were successful. Presence of Delays: We assumed that the communication channel introduces a constant delay of one time step. However, the same algorithm continues to remain optimal even if the channel introduces larger or even timevarying delays, as long as there is the provision of a time stamp from the encoder regarding the time it transmits any vector. he decoder uses the packet it receives at any time step only if it was transmitted later than the quantity it has stored from the previous time steps. If this is not true due to packet re-ordering, the decoder continues to use the quantity stored from previous time steps. Further, if the delays are finite, the stability conditions derived below remain unchanged. Infinite delays are equivalent to packet erasures, and can be handled by using the same framework. Stability and Performance: Both the stability and performance of the system with this optimal coding algorithm in place can be analyzed by assuming specific models for the erasure process. For pedagogical ease, we adopt the i.i.d. erasure model, with an erasure occurring at any time step with probability p. Due to the separation principle, to obtain the stability conditions, we need to consider the conditions under which the LQ control cost for the system, and the covariance of the estimation error between the state of the process xk and the estimate at the controller ˆxk remain bounded, as time k increases. Under the controllability and observability assumptions the LQ cost remains bounded, if the control value does. Define the estimation error and its covariance as ek = xk ˆxk P k = E [ eke k ], where the expectation is taken with respect to the process and measurement noises, and the initial condition but not the erasure process. Due to the washing away effect of the algorithm, the error of the estimate at the decoder evolves as { ēk + 1 no erasure ek + 1 = Aek erasure event, where ēk is the error between xk and the estimate of xk given all control inputs {uj} k 1 {yj} k 1. hus, the error covariance evolves as { Mk + 1 with probability 1 p P k + 1 = AP ka + R w with probability p, and measurements 6

7 where Mk is the covariance of the error ēk. hus, we obtain E[P k + 1] = 1 pmk pr w + pae[p k]a, where the extra expectation for the error covariance is taken over the erasure process in the channel. Since the system is observable, Mk converges exponentially to a steady state value M. hus, the necessary and sufficient condition for the convergence of the above discrete algebraic Lyapunov recursion is pρa 2 < 1, where ρa is the spectral radius of A. Due to the optimality of the algorithm considered above, this condition is necessary for stability of the system with any causal encoding algorithm. In particular, for the strategy of simply transmitting the latest measurement from the sensor as considered in Section 2, this condition turns out to be necessary for stability though not sufficient for a general process model. For achieving stability with this condition, we require an encoding strategy, such as the recursive algorithm provided above. his analysis can be generalized to more general erasure models. For example, for a Gilbert-Eliot type channel model, the necessary and sufficient condition for stability is given by q 00 ρa 2 < 1, where q 00 is the conditional probability of an erasure event at time k + 1, provided an erasure occurs at time k. In addition, by calculating the terms E[P k] and the LQ control cost of the system with full state information, the performance J LQG can also be calculated through the separation principle proved above. he value of the cost function thus achieved provides a lower bound to the value of the cost function achievable using any other encoding or control algorithm, for the same probability of erasure. An alternative viewpoint is to consider the encoding algorithm above as a means for transmitting data with lesser frequency to achieve the same level of performance, than, e.g., transmitting measurements to the controller. Higher Order Moments: It can be seen that the treatment above can be extended to consider the stability of higher order moments of the estimation error, or the state value. In fact, the entire steady state probability distribution function of the estimation error can be calculated. 4 Extensions and Open Questions he above framework was explained for a very simple set-up of an LQG problem. It is natural to consider its generalization to other models by removing various assumptions. We consider some of these assumptions below. We also point out some of the open questions. Channel between Controller and Actuator: he encoding algorithm presented above continues to remain optimal when a channel is present between the controller and the actuator as considered in Figure 3, as long as there is a provision for acknowledgement from actuator to controller for any successful transmission, and the protocol that the actuator follows in case of an erasure is known at the controller. his is because these two assumptions are enough for the separation principle to hold. If no such acknowledgement is available, the control input begins to have a dual effect and the optimal algorithm is still unknown. Moreover, the problem of designing the optimal encoder for the controller-actuator channel can also be considered. his design will intimately depend on the information that is assumed to be known at the actuator e.g., the cost function, the system matrices and so on. Algorithms that optimize the cost function for such information sets are largely unknown. A simpler version of the problem would involve either analyzing the stability and performance gains for given encoding and decoding algorithms employed by the controller and the actuator respectively, or, considering algorithms that are stability optimal, in the sense of designing recursive algorithms that achieve the largest stability region for any possible causal encoding algorithm. Both these directions have seen research activity. For the first direction, algorithms typically involve transmitting some future control inputs at every time step, or the actuator using some linear combination of past control inputs if an erasure occurs. he second direction has identified the stability conditions that are necessary for any causal algorithm. Moreover, recursive designs that can achieve stability when these conditions are satisfied have also been identified. Surprisingly, the design is in the form of a universal actuator that does not require access to the model of the plant. Even if such knowledge were available, the stability conditions do not change. hus, the design is stability optimal. Presence of a Communication Network: So far we have concentrated on the case when the sensor and the controller are connected using a single communication channel. A typical scenario, particularly in a wireless context, 7

8 would instead involve a communication network with multiple such channels. If no encoding algorithm is implemented, and every node in the network including the sensor transmits simply the measurements, the network can be replaced by a giant erasure channel with the equivalent erasure probability being some measure of the reliability of the network. he analysis in Section 2 carries over to this case; however, the performance degrades rapidly as the network size increases. If encoding is permitted, such an equivalence breaks down. he optimal algorithm is an extension of the single channel case, and is provided in [6]. he stability and performance calculations are considerably more involved. However, the stability condition has an interesting interpretation in terms of the capacity for fluid networks. he necessary and sufficient condition for stability can be expressed as the inequality p max-cut ρa 2 < 1, where p max-cut is the max-cut probability calculated in a manner similar to the min-cut capacity of fluid networks. We construct cut-sets by dividing the nodes in the network into two sets with one set containing the sensor, and the other the controller. For each cut-set, we calculate the cut-set erasure probability by multiplying the erasure probabilities of all the channels from the set containing the sensor to the set containing the controller. he maximum such cut-set erasure probability over all possible cut-sets denotes the max-cut probability of the network. he improvement in the performance and stability region of the system by using the encoding algorithm increases drastically with the size and the complexity of the network. Multiple Sensors: Another direction in which the above framework can be extended is to consider multiple sensors observing the same process. As with the case with one sensor, one can identify the necessary stability conditions and a lower bound for the achievable cost function with any causal coding algorithm. hese stability conditions are also sufficient and recursive algorithms for achieving stability when these conditions are satisfied have been identified. hese conditions are a natural extension of the stability conditions for the single sensor case. As an example, for the case of two sensors described by sensing matrices C 1 and C 2 that transmit data to the controller across erasure channels for which erasure events are i.i.d. with probabilities p 1 and p 2 respectively, the stability conditions are given by the set p 2 ρa 1 2 < 1 p 1 ρa 2 2 < 1 p 1 p 2 ρa 2 < 1, where ρa i denotes the spectral radius of the unobservable part of the system matrix A, when the pair A, C i is represented in the observability canonical form. However, the problem of identifying distributed encoding algorithms to be followed at each sensor for achieving the lower bounds on the achieved cost function remains largely open. his problem is related to the track fusion problem that considers identifying algorithms for optimal fusion of information from multiple sensors that interact intermittently e.g., see [1]. hat transmitting estimates based on local data from each sensor is not optimal is long known. While algorithms that achieve a performance close to the lower bound of the cost function have been identified, a complete solution is not available. Inclusion of More Communication Effects: Our discussion has focussed on modeling the loss of data transmitted over the channel. In our discussion of the optimal encoding algorithms, we also briefly considered the possibility of data being delayed or received out of order. An important direction for future work is to consider other effects due to communication channels. Both from a theoretical perspective, and for many applications such as underwater systems, an important effect is to impose a limit on the number of bits that can be communicated for every successful transmission. Some recent work [11, 13] has considered the analog digital channel in which the channel supports n bits per time step and transmits them with a certain probability p at every time step. Stability conditions for such a channel have been identified and are a natural combination of the stability conditions for the analog erasure channel above and the ones for a noiseless digital channel, as considered elsewhere in the book. he performance of optimal encoding algorithms and the optimal performance that is achievable remain unknown. Another channel effect that has largely been ignored is the addition of channel noise to the data received successfully. More General Performance Criteria: Our treatment focussed on a particular performance measure - a quadratic cost, and the stability notions emanating from that measure. Other cost functions may be relevant in applications. hus the cost function may be related to target tracking, measures such as H 2 or H [15], or some combination of communication and control costs. he analysis and optimal encoding algorithms for such measures are expected to differ significantly. An an example, for target tracking, the properties of the reference signal that needs to be tracked can be expected to play a significant role. Similarly, for H related costs, the sufficient statistic, and hence the encoding algorithms to transmit it, may be vastly different than the LQG case. Finally, a distributed control problem with multiple processes, sensors and actuators is a natural direction to consider. More General Plant Dynamics: he final direction is to consider plant dynamics that are more general than the linear model that we have considered. Moving to models such as jump linear systems, hybrid systems, and general non-linear systems will provide new challenges and results. As an example, for non-linear plants concepts 8

9 such as spectral radius no longer hold. hus, the analysis techniques are likely to be different and measures such as Lyapunov exponents and the Lipschitz constant for the dynamics will likely become important. A Some Results on Markovian Jump Linear Systems We present a short overview of Markov jump linear systems. A more thorough and complete treatment is given in [2]. Consider a discrete time discrete state Markov process with state rk {1, 2,, m} at time k. Denote the transition probability Probrk + 1 = j rk = i by q ij, and the resultant transition probability matrix by Q. Also denote Probrk = j = π j k, with π j 0 as given. he evolution of a Markovian jump linear system MJLS, denoted by S 1 for future reference, can be described by the following equations xk + 1 = A rk xk + B rk uk + F rk wk 1 yk = C rk xk + G rk vk, where wk is zero mean white Gaussian noise with covariance R w, vk is zero mean white Gaussian noise with covariance R v and the notation X rk implies that the matrix X {X 1, X 2,, X m } with the matrix X i being chosen when rk = i. he initial state x0 is assumed to be a zero mean Gaussian random variable with variance Π0. For simplicity, we will consider F rk = G rk I for all values of rk in the sequel. We also assume that x0, {wk}, {vk} and {rk} are mutually independent. A.1 LQ Control he Linear Quadratic Regulator LQR problem for the system S 1 is posed by assuming that the noises wk and vk are not present. Moreover the matrix C rk I for all choices of the state rk. he problem aims at designing the control input uk to minimize the finite horizon cost function J LQR = K k=1 E {rj} K j=k+1 [ x kqxk + u kruk ] + x K + 1P K + 1xK + 1, where the expectation at time k is taken with respect to the future values of the Markov state realization, and P K + 1, Q and R are all assumed to be positive definite. he controller at time k has access to control inputs {uj} k 1, state values {xj}k and the Markov state values {rj}k. Moreover, the system is said to be def J stabilizable if the infinite horizon cost function J = lim LQR K K is finite. he solution to this problem can readily be obtained through dynamic programming arguments. he optimal control is given by the following result. heorem 4. Consider the LQR problem posed above for the system S 1. At time k, if rk = i, then the optimal control input is given by where for j = 1, 2,, m, P j k = m t=1 uk = R + B i P i k + 1B i B i P i k + 1A i xk, q tj Q + A t P t k + 1A t A t P t k + 1B t R + B t P t k + 1B t B t P t k + 1A t, and P j K + 1 = P K + 1, j = 1, 2,, m. Assume that the Markov states reach a stationary probability distribution. A sufficient condition for stabilizability of the system is that there exist m positive definite matrices X 1, X 2,, X m and m 2 matrices K 1,1, K 1,2,, K 1,m, K 2,1,, K m,m such that for all j = 1, 2,, m, X j > m q ij A i + K i,j Bi X i A i + K i,j Bi + Q + K ij RKij. i=1 9

10 Note that the sufficient condition can be cast in alternate forms as linear matrix inequalities, that can be efficiently solved. We omit such representations. A special case of Markov jump linear systems is when the discrete states are chosen independently from one time step to the next. Since this is the case we have concentrated on in this chapter, we summarize the results pertaining to this case below. Corollary 4.1. Consider system S 1 with the additional assumption that the Markov transition probability matrix is such that for all states i and j, q ij = q i in other words, the states are chosen independently and identically distributed from one time step to the next. Consider the LQR problem posed above for the system S 1. At time k, if rk = i, then the optimal control input is given by uk = R + B i P k + 1B i B i P k + 1A i xk, where P k = m t=1 q t Q + A t P k + 1A t A t P k + 1B t R + B t P k + 1B t B t P k + 1A t. Assume that the Markov states reach a stationary probability distribution. A sufficient condition for stabilizability of the system is that there exists a positive definite matrix X, and m matrices K 1, K 2,, K m such that m X > q i A i + K i Bi XA i + K i Bi + Q + K i RKi. A.2 MMSE Estimation i=1 he minimum mean squared error estimate problem for the system S 1 is posed by assuming that the control u rk is identically zero. he objective is to identify at every time step k, an estimate ˆxk + 1 of the state xk + 1 that minimizes the mean squared error covariance Πk + 1 = E {wj},{vj} [ xk + 1 ˆxk + 1xk + 1 ˆxk + 1 ], where the expectation is taken with respect to the process and measurement noises but not the Markov state realization. he estimator at time k has access to observations {yj} k and the Markov state values {rj}k. Moreover, the error covariance is said to be stable if the expected steady state error covariance lim k E {rj} k 1[Πk] is bounded, where the expectation is taken with respect to the Markov process. he estimator at time k has access to the measurements y0, y1,, yk, and Markov state values r0, r1,, rk. Since the estimator has access to the Markov state values till time k, the optimal estimate can be calculated through a time-varying Kalman filter. hus, if at time k, r k = i, the estimate evolves as ˆxk + 1 = A iˆxk + Kk yk C iˆxk, where Kk = A i ΠkC i Πk + 1 = A i ΠkA i Ci ΠkC i + R v + R w A i ΠkC i Ci ΠkC i + R v Ci ΠkA i. he error covariance Πk is available through the above calculations. However, calculating E {rj} k 1[Πk] seems to be intractable. Instead, we present an upper bound to this quantity 2 that will also help in obtaining sufficient conditions for the error covariance to be stable. he intuition behind obtaining the upper bound is simple. he optimal estimator presented above optimally utilizes the information about the Markov states till time k. Consider an alternate estimator that at every time step k, averages over the values of the Markov states r 0,, r k 1. Such an estimator is sub-optimal and the error covariance for this estimator forms an upper bound for E {rj} k 1[Πk]. A formal proof using Jensen s inequality is given in [5, heorem 5]. We present the statement below while omitting the proof. heorem 5. he term E {rj} k 1[Πk] obtained from the optimal estimator is upper bounded by Mk = m j=1 M jk where m M j k = q tj R w + A t M t k 1A t A t M t k 1Ct Rv + C t M t k 1Ct Ct M t k 1A t, t=1 2 We say that A is upperbounded by B if B A is positive semi-definite. 10

11 with M j 0 = Π0 j. Moreover, assume that the Markov states reach a stationary probability distribution. A sufficient condition for stabilizability of the system is that there exist m positive definite matrices X 1, X 2,, X m and m 2 matrices K 1,1, K 1,2,, K 1,m, K 2,1,, K m,m such that for all j = 1, 2,, m, X j > m q ij Ai + K i,j C i X i A i + K i,j C i + R w + K ij R v Kij. i=1 We can once again consider the special case of states being chosen in an independent and identically distributed fashion. Corollary 5.1. Consider the estimation problem posed above for the system S 1 with the additional assumption that the Markov transition probability matrix is such that for all states i and j, q ij = q i in other words, the states are chosen independently and identically distributed from one time step to the next. he term E {rj} k 1[Πk] obtained from the optimal estimator is upper bounded by Mk where m Mk = q t R w + A t Mk 1A t A t Mk 1Ct Rv + C t Mk 1Ct Ct Mk 1A t t=1 with M0 = Π0. Further, a sufficient condition for stabilizability of the system is that there exists a positive definite matrix X, and m matrices K 1, K 2,, K m such that, m X > q i Ai + K i C i XA i + K i C i + R w + K i R v Ki i=1. A.3 LQG Control he Linear Quadratic Gaussian LQG problem for the system S 1 aims at designing the control input uk to minimize the finite horizon cost function [ K J LQG = E x kqxk + u kruk ] + x K + 1P K + 1xK + 1, k=1 where the expectation at time k is taken with respect to the future values of the Markov state realization, and the measurement and process noises. Further, the matrices P K + 1, Q and R are all assumed to be positive definite. he controller at time k has access to control inputs {uj} k 1, measurements {yj}k and the Markov state values {rj} k. he system is said to be stabilizable if the infinite horizon cost function J J = lim LQG K K is finite. he solution to this problem is provided by a separation principle and using heorems 4 and 5. We have the following result. heorem 6. Consider the LQG problem for the system S 1. At time k, if rk = i, then the optimal control input is given by uk = R + B i P i k + 1B i B i P i k + 1A iˆxk, where for P i k is calculated as in heorem 4 and ˆxk is calculated using a time-varying Kalman filter. We can also obtain the conditions for stabilizability of the system by utilizing heorems 4 and 5. References [1] K. C. Chang, R. K. Saha and Y. Bar-Shalom, On optimal track-to-track fusion, IEEE rans. Aerospace and Electronic Systems, AES- 334: , [2] O. L. V. Costa, M. D. Fragoso and R. P. Marques, Discrete-time Markov Jump Linear Systems, Springer, [3] D. Dacic and D. Nesic, Quadratic stabilization of linear networked control systems via simultaneous controller and protocol synthesis, Automatica, vol. 43, No. 7, July, 2007, pp [4] E. 0. Elliott, Estimates of error rates for codes on burst-noise channels, Bell Systems echnical Journal, vol. 42, pp , Sept def 11

12 [5] V. Gupta,. Chung, B. Hassibi and R. M. Murray, On a Stochastic Sensor Selection Algorithm with Applications in Sensor Scheduling and Dynamic Sensor Coverage, Automatica, Vol. 42, No. 2, February 2006, Pages: [6] V. Gupta, A. F. Dana, J. Hespanha, R. M. Murray and B. Hassibi, Data ransmission over Networks for Estimation and Control, IEEE ransactions on Automatic Control, o appear Aug [7] V. Gupta, D. Spanos, B. Hassibi, and R. M. Murray, Optimal LQG control across a packet-dropping link, Systems and Controls Letters, vol. 56, no. 6, pp , June [8] J. Hespanha, P. Naghshtabrizi, Y. Xu, A Survey of Recent Results in Networked Control Systems Proc. of IEEE, Special Issue on echnology of Networked Control Systems, vol. 95, no. 1, pp , Jan [9] O.C. Imer, S. Yuksel and. Basar, Optimal Control of LI Systems over Communication Networks, Automatica, 429: , Sep [10]. Kailath, A.H. Sayed and B. Hassibi, Linear Estimation, Prentice-Hall, [11] N. C. Martins and M. A. Dahleh, Feedback Control in the Presence of Noisy Channels: Bode-Like Fundamental Limitations of Performance, IEEE ransactions on Automatic Control, 527: , Aug [12] A. Matveev and A. Savkin, he problem of state estimation via asynchronous communication channels with irregular transmission times, IEEE ransactions on Automatic Control, vol. 48, No. 4, pp , Apr [13] P. Minero, M. Franceschetti, S. Dey, G. Nair, Data rate theorem for stabilization over time-varying feedback channels, IEEE rans. on Automatic Control, in press. [14] L. Schenato, B. Sinopoli, M. Franceschetti, K. Poolla and S. S. Sastry, Foundations of Control and Estimation over Lossy Networks, Proc. of IEEE, 951: , January [15] P. Seiler and R. Sengupta, An H approach to networked control, IEEE ransactions on Automatatic Control, vol. 50, no. 3, pp , March