RECENT advances in technology have led to increased activity

Transcription

1 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 49, NO 9, SEPTEMBER Stochastic Linear Control Over a Communication Channel Sekhar Tatikonda, Member, IEEE, Anant Sahai, Member, IEEE, and Sanjoy Mitter, Life Fellow, IEEE Abstract We examine linear stochastic control systems when there is a communication channel connecting the sensor to the controller The problem consists of designing the channel encoder and decoder as well as the controller to satisfy some given control objectives In particular, we examine the role communication has on the classical linear quadratic Gaussian problem We give conditions under which the classical separation property between estimation and control holds and the certainty equivalent control law is optimal We then present the sequential rate distortion framework We present bounds on the achievable performance and show the inherent tradeoffs between control and communication costs In particular, we show that optimal quadratic cost decomposes into two terms: A full knowledge cost and a sequential rate distortion cost Index Terms Certainty equivalent control, communication constraints, networked control, separation, sequential rate distortion, stochastic linear systems I INTRODUCTION RECENT advances in technology have led to increased activity in understanding and designing networked control systems In this paper, we examine a stochastic control problem there is a communication channel connecting the sensor to the controller This problem arises when the plant and the controller are geographically separated and there is a band-limited and possibly noisy communication channel connecting them In addition, communication constraints can arise when there is no major geographical separation between the controller and the plant, but there is a shared communication medium that is being used along with other users in the same local area, or as a part of the larger system Although we do not explicitly examine the networking issues per se in this paper, we believe that a fundamental understanding of the role of communication constraints will be essential to a more complete theory of networked control The system we consider consists of a plant, an encoder, a channel, a decoder, and a controller The plant and the channel are given to us Our task is to design the encoder, decoder, Manuscript received June 4, 2003; revised December 19, 2003 Recommended by Guest Editors P Antsaklis and J Baillieul This work was supported by the Army Research Office under MURI Grant: Data Fusion in Large Arrays of Microsensors DAAD , and by the Department of Defense under MURI Grant: Complex Adaptive Networks for Cooperative Control Subaward S Tatikonda is with Yale University, New Haven, CT USA ( sekhartatikonda@yaleedu) A Sahai is with the University of California, Berkeley, CA USA S Mitter is with the Massachusetts Institute of Technology, Cambridge, MA USA Digital Object Identifier /TAC and controller to satisfy some given control objectives Since we have a distributed system the choice of information pattern, [26], can have a dramatic effect on what control performance is achievable We discuss the effect that the choice of information pattern at the encoder has on the communication requirements needed to achieve the control objective In particular, we examine the role communication has on the classical linear quadratic Gaussian (LQG) problem To this end we present the sequential rate distortion (SRD) framework We derive bounds on the achievable performance and show the inherent tradeoffs between control and communication costs In particular, we show that the optimal LQG cost decomposes into two terms: a full knowledge cost and a sequential rate distortion cost There are two classical notions of separation that we examine in this paper The first notion is the control-theoretic separation between state estimation and control We present conditions that insure the optimality of the certainty equivalent control law These build on the work of Bar-Shalom and Tse [3] The second notion is the information-theoretic separation between source encoder and channel encoder In particular, in the limit of long delays, it is known that one can, without loss of generality, design the source encoder and the channel encoder separately [11] This separation is known to hold quite broadly, [25], but, in general, fails for both short delays and for unstable processes In the limit of large delays, [18] showed that the estimation of unstable processes can be covered by a suitably modified separation theorem, but this information-theoretic result does not extend to the case of limited delay Since delay is an important issue in control applications we cannot apply the information-theoretic separation results to our problem To deal with this delay issue, we present the sequential rate distortion framework first introduced in [13] and further developed in [19], [20], and [23] Bansal and Basar examine the simultaneous design of the measurement and control strategies for a class of stochastic systems [2] Borkar and Mitter introduced the present problem of LQG control under communication constraints [6] There they looked at stable systems and noiseless digital channels They introduced the innovations coding scheme Unfortunately, their scheme does not work for unstable systems We generalize these results to unstable systems and noisy channels Moreover, we give conditions under which our coding scheme is optimal, the separation principle holds, and the certainty equivalent controller is optimal Nair and Evans [17] examined mean-square stabilizability over a noiseless channel In contrast, we examine the LQG performance over both noisy and noiseless channels Reference [7] examines the role of nonclassical information patterns for Markov decision problems We extend those results to the LQG problem with differing information pattern at the /04$ IEEE

2 1550 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 49, NO 9, SEPTEMBER 2004 encoder The design of optimal sequential quantization schemes for uncontrolled Markov processes was examined in [8] Recently, there has been a good deal of work studying deterministic systems under communication constraints: [10], [19], [21], [22], [15], [1], [27], and [28] In [21] and [22], we examined deterministic systems over both noiseless and noisy channels Our work here differs in that we treat stochastic systems, provide a general separation result, apply the SRD [23] framework to the LQG problem, and discuss conditions under which the plant is matched to the channel The work in this paper first appeared in preliminary form in [24] and represents a portion of the first author s PhD dissertation [19] In Section II, we formulate the problem In Section III, we examine the LQG problem under different information patterns and in different communication channels In Section IV, we introduce the sequential rate distortion framework and give closed form solutions to the sequential rate distortion function for Gauss Markov sources In Section V, we examine some particular scenarios II PROBLEM FORMULATION Here, we present the different components of our problem a) Plant: We consider the following discrete-time, stochastic, linear system: is a -valued state process, is a -valued control process, and is an independent and identically distributed (IID) sequence of Gaussian random variables with zero mean and covariance The initial position is Gaussian with zero mean and covariance Let and assume is a controllable pair The decoder output process is a -valued process The output represents the decoder s estimate of the state of the system at time ; see Fig 1 Upper case variables, like, represent random variables and lower case variables, like, represent particular realizations Throughout this paper, log refers to logarithm base two Let b) Channel: The channel input and output alphabet spaces are denoted and, respectively In this paper, we restrict ourselves to time-invariant memoryless channels which can be modeled as stochastic kernels: Specifically, for each realization of the conditional probability of given is denoted by At time the encoder produces a channel input symbol and the channel outputs the channel output symbol according to the probability The Shannon capacity is defined as is the mutual information between the channel input and output See the Appendix for a review of mutual information In this paper we study two particular channels Noiseless digital channel with rate The channel input and output alphabets are the same: The alphabet size is is called the rate of the channel The channel is noiseless and memoryless: ( is a Dirac measure) In this case (1) Fig 1 System Memoryless vector Gaussian channel: The channel input and output alphabets are: The channel is memoryless: is an IID sequence of Gaussian random variables with zero mean and covariance The input symbols satisfy the power constraint: This channel is often used as a simplified model of a wireless channel The supremizing input distribution can be shown to be a zero mean, vector-valued, Gaussian random variable Hence, See [9] for more details The capacity is given by denotes the determinant of c) Information Pattern: Our task is to design an encoder, decoder, and controller to achieve given control objectives Thus we need to specify the information pattern of each of these components [26] There are five types of signals: state, channel input, channel output, decoder output, and control The following time-ordering represents the causal ordering in which the events of the system occur: Encoder: We model the encoder at time as a stochastic kernel: Deterministic encoders are modeled as Dirac measures Note that, in general, there are five potential kinds of feedback to the encoder We will examine three different information patterns A) : This information pattern represents full knowledge at the encoder B) : In this information pattern, the channel output and decoder output are not available to the encoder C) : In this information pattern, only the current state is available to the encoder Information pattern B), along with time-ordering (2), implies that is independent of conditioned on Similarly information pattern C implies is independent of conditioned on Decoder: We model the decoder at time as a stochastic kernel: The information pattern here along with the time-ordering (2) implies that is independent of given As we will see later, the output of the decoder can be taken to be an estimate of the state of the plant (2)

3 TATIKONDA et al: STOCHASTIC LINEAR CONTROL OVER A COMMUNICATION CHANNEL 1551 Controller: We model the controller at time as a stochastic kernel: The information pattern here along with the time-ordering (2) implies that is independent of conditioned on Note that we are assuming the controller takes as input only the decoder output Thus, there is a separation structure between the decoder and the controller We will give conditions under which this separation structure can be assumed without loss of generality Clearly, more information available at the encoder will lead to better control performance Information pattern A) can be viewed as the best scenario and information pattern C) can be viewed as the worst scenario There are many information patterns in between these two extremes and information pattern B) represents one such information pattern As we will see later, the important feature of information pattern A) is that the encoder knows the decoder s state We will discuss scenarios this naturally occurs The important feature of information pattern B) is that the encoder has access to the previous control signals Here, we are envisioning situations the encoder is geographically collocated with the plant and, hence, can observe both the state of the plant as well as the control actions applied to the plant Finally, information pattern C) is useful for modeling situations the control signals are not observed at the encoder This information pattern can be used for scenarios we want to model encoders that need to be simple and memoryless d) Interconnection and Performance Objective: In order to consider a performance objective, we need to insure that there is a well-defined joint measure over the variables of interest We are given the plant and channel For each encoder, decoder, and controller satisfying the required information patterns we can interconnect the different stochastic kernels together to produce the following joint measure: e) Sequential Rate Distortion: One role of the feedback link from the encoder to the decoder is to convey information aimed at reducing the controller s uncertainty about the state of the system At time the evolution of the state is given by Since the decoder has access to the control signals the uncertainty in is determined by the primitive random variables, and The decoder s estimate of the state is given by The information transmitted across the channel should only be relevant for determining and Hence, it makes sense to examine the uncontrolled dynamics: We will examine the uncontrolled dynamics in Section IV we introduce the sequential rate distortion framework This turns out to be the appropriate framework for understanding what information, relevant to the control objective, should be transmitted over the communication channel III LQG PERFORMANCE OBJECTIVE Here, we consider conditions that insure the optimality of the certainty equivalent controller We then show that the LQG cost can be decomposed into a full knowledge cost and a partial knowledge cost Recall our goal is to minimize the long term average cost given by (3) Under full state observation, it is well known that the optimal steady state control law is a linear gain of the form (4) and satisfies the Riccati equation Note that this joint measure preserves the dynamics of the plant and the channel as well as maintains the information patterns of the encoder, decoder, and controller Let denote the set of all such joint measures: Our performance objective is the LQG cost The optimal cost is given by (5) (6) and S are positive definite Our goal is to minimize the LQG cost (3) over the set of all measures in that are consistent with the given plant, channel, and information pattern We are also interested in understanding how the properties of the channel, like rate and power, and the choice of information pattern at the encoder can effect the minimum LQG performance cost (3) Furthermore, these results continue to hold for the LQ problem the process disturbances in (1) are no longer Gaussian but are uncorrelated with zero mean and common covariance matrix That is the separation result continues to hold if only second-order statistics are specified These results are standard and can be found in [5] The addition of a communication channel converts the fully observed LQG control problem shown previously into a partially observed LQ control problem The structure of the observation is determined by both the communication channel and the choice of encoder and decoder

4 1552 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 49, NO 9, SEPTEMBER 2004 A Certainty Equivalence We now present conditions that insure the optimality of the certainty equivalent control law: Where is the decoder s estimate of the state of the plant To that end, we discuss the no dual effect property [3] Fix a memoryless channel, an information pattern for the encoder, and an encoder Let the state estimation error be We know that for any control sequence we have Define to be the uncontrolled state Similarly, let denote the channel output when all the controls are set to zero Specifically for each particular realization of the primitive random variables:, and (the latter is included if the channel is the Gaussian channel) represents the channel output under the control sequence and represents the channel output when all the controls are set to zero The control has no dual effect if for all and The condition of no dual effect essentially states that the error covariance is independent of the control signals applied The term dual comes from the control s dual role: Effecting state evolution and probing the system to reduce state uncertainty If the control has no dual effect, the latter probing property will not be in effect Proposition 31: The optimal control law for system (1) is a certainty equivalent control law if and only if the control has no dual effect Proof: Bar-Shalom and Tse [3] prove this result for the case when the output measurement model,, has the following functional form is a given sequence of functions and is an IID sequence of random variables that are independent of the process For our case, depending on the information pattern of the encoder and the memoryless channel, we have a more general output measurement model,, with the following functional form The proof for this more general output measurement model is a straightforward generalization of the proof given in [3] We now give a necessary condition insuring no dual effect Lemma 31: If the sigma-fields are nested,, and then there is no dual effect Proof: Note that The last two equalities holds by hypothesis Thus, is independent of the control sequence applied We now use Lemma 31 to give conditions for the control to have no dual effect for the noiseless digital channel and the memoryless Gaussian channel Noiseless digital channel: Because the channel is noiseless the encoder information patterns and are equivalent (they define the same sigma-field) Also, for information patterns and, both the encoder and decoder have access to the control signals hence they both can subtract out the effect of the control Furthermore, as shown in [8], the optimal encoder will have the form of a quantizer applied to the innovation:, is a quantizer We will show that and forms a Markov chain and, hence, Clearly, Note that because conditioned on the control is independent of and This implies because Hence, Now because is a function of and and is independent of By the induction hypothesis, assume that and for Note that because, conditioned on, the control is independent of and Now, by the induction hypothesis and the fact that is independent of,wehave Therefore, This implies because Hence, Now, because is a function of and and is independent of Hence, there is no dual effect Memoryless vector Gaussian channel: Here, we restrict ourselves to encoders that are deterministic and linear: the s are matrices of appropriate dimension Some of the s may be set to zero depending on the information pattern of the encoder (We show in the next section that we may choose linear encoders without loss of generality) Due to the Gaussian channel we have Since the plant dynamics, the encoder, and the channel are linear we can rewrite in terms of the primitive random variables and the controls: the s are matrices of appropriate dimension Similarly, the uncontrolled channel output can be written as with the same matrices Hence we can write for appropriate matrices The information in relevant to estimating is summarized by Thus, for linear encoders the control has no dual effect; see [5, Lemma 521] for more details We have shown that for any linear encoder the no dual effect condition holds We have also shown that no dual effect can hold for certain nonlinear encoders In particular, it holds for those nonlinear encoders that first remove the effect of the control before applying the nonlinearity This was done for the noiseless digital channel with information pattern B Reduction to Fully Observed LQ Model We now reduce our partially observed control problem into a fully observed LQ control problem As before let be the decoder s estimate of the state of

5 TATIKONDA et al: STOCHASTIC LINEAR CONTROL OVER A COMMUNICATION CHANNEL 1553 the system and let The running cost at time be the state estimation error can be written for every the last equation holds because Note that if the control has no dual effect the last term of the running cost,, will not depend on We now define a new fully observed process with the decoder s estimate of the state,, as the new state If there is no dual effect then the second term,, is independent of the control actions chosen This term represents the cost due to the encoder and channel We have shown previously that the process is uncorrelated Therefore, the dynamics (7) with cost given by the first term of (8) is an LQ control problem Hence, the optimal control law is the certainty equivalent control Note that If we further assume that for all, then we can directly apply the results described in (4) (6) to get with process disturbance: Our new system has the dynamics The term represents the new information being transmitted from the encoder to the decoder To see this, note that At time, the decoder s prediction error in estimating the state at time is After receiving the error reduces to The difference, represents the new information sent to the encoder To complete the formulation of the fully observed LQ problem we need to show the process disturbances,,in (7) are uncorrelated Lemma 32: The random variables are uncorrelated Proof: We need to show that for all We will prove The general case will then follow First, note that is uncorrelated with because the error in estimating is uncorrelated with the estimate Second, note that is uncorrelated with because the process disturbance at time is independent of everything that occurs before time Now, Where the second equality follows because is measurable with respect to We can now put all the pieces together Let be the covariance of the state estimation error As stated before, the cost is (7) (8) We see that the optimal cost decomposes into two terms The first term is the cost under full state observation and the second term is a cost that depends only on the steady state estimation error covariance Thus, we have reduced the problem of computing the optimal cost to that of minimizing for a given information pattern and channel We have presented conditions under which the certainty equivalent controller is optimal In the next section, we discuss the encoder and decoder design IV SEQUENTIAL RATE DISTORTION We first review traditional rate distortion theory and the concept of separation between the source encoder and the channel encoder We then describe the failings of the traditional theory These failings have to do with the long delays, noncausality, and the failure of separation to hold for traditional rate distortion theory in sequential settings To correct for these failings we review the notion of source-channel matching [19], [12], and introduce the SRD framework [13], [23] Given a source and a channel our job is to design an encoder and decoder that can transmit the source over the channel while maintaining a given end-to-end distortion criterion; see Fig 2 For our source, we consider the Gauss Markov system (1) with the controls set to zero We will focus on the weighted squared error distortion measure For, let the weight matrix is positive definite The channel capacity theorem [11, Th 564] states that any rate can be transmitted over the channel with probability of decoding error decreasing to zero exponentially with the number of channel uses Specifically, there exists a channel encoder that can transmit messages, equivalently bits, by using the channel times and incurring a small probability of decoding error (9)

6 1554 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 49, NO 9, SEPTEMBER 2004 Fig 2 Source-channel coding The source-channel separation theorem gives us conditions under which we can separate the encoder into two pieces: a source encoder that encodes the source into bits and a channel encoder that converts those bits into channel symbols Similarly the decoder is split into two pieces: a channel decoder and a source decoder; see Fig 2 The rate distortion function for horizon is The next theorem follows from [11, Th 963] Theorem 41: We are given a Gauss Markov process as described in (9),, and a channel with capacity If, then there does not exist a coding scheme that achieves distortion over channel uses If for sufficiently large then there exists a coding scheme that can achieve a distortion arbitrarily close to This theorem shows that there exists a coding scheme that achieves the rate distortion bound Specifically, we can construct a source encoder that quantizes our source into bits We can then design independently a channel encoder that will transmit these bits over a noisy channel with small probability of channel decoding error Hence, one can design the source encoder and the channel encoder separately so as to achieve the required end-to-end distortion [11] There is another coding scheme that can sometimes achieve the end-to-end distortion bound over a channel Note that the definition of the rate distortion function involves an infimization over stochastic kernels These stochastic kernels can be viewed as channels connecting the source to the reproduction If the true communication channel equals the rate distortion infimizing channel, then we say the channel is matched to the source [19], [12] Specifically, for all, wehave, and Note that the communication channel may need to have memory In the source-channel matching case, we do not need to separately design the source encoder and channel encoder In fact, we can send the source over the channel in an uncoded fashion Here, there is no explicit separation between the source and channel encoding decoder can compute the estimate of The time-ordering is (10) In our control situation, this delay is unacceptable The natural causal time-ordering we require is (11) In time-ordering (10), the random variable is the th event as in time-ordering (11) is the second event Similarly, in the source-channel matching scheme there is a delay Note that the rate distortion infimizing stochastic kernel in Theorem 41 can be viewed as a channel that factors: Under the time-ordering (10) this is a causal channel Under the time-ordering (11), this is a noncausal channel because the reconstruction depends on future s We now define the sequential rate distortion function This function involves an optimization of the mutual information over all causal, with respect to time-ordering (11) channels Definition 41: The sequential rate distortion function is with Note that in the definition we are expected to achieve a distortion at each time step We provide a necessary condition on the channel capacity to achieve a given end-to-end distortion causally To that end, we state the data-processing inequality [9] Lemma 41: Let be a Markov chain Then, Proposition 41: A necessary condition on a given memoryless channel with capacity to achieve the distortion causally as described in Definition 41 is Proof: Let be any joint measure which satisfies the distortion and channel requirements Then A Causality and Sequential Rate Distortion Unfortunately, under the source-channel coding scheme, there is a delay of at least time units in computing the reconstruction of Specifically, at time, we observe It then takes steps to observe the remainder of the source, At this point the channel input symbols,, are produced It takes another time steps until all the channel output symbols are received Then the the second inequality follows from the data-processing inequality Finding sufficient conditions on the channel to achieve the SRD end-to-end distortion is more difficult The source-channel separation principle does not generally hold in situations delay is an issue But there do exist channels with capacity over which an end-to-end distortion can

7 TATIKONDA et al: STOCHASTIC LINEAR CONTROL OVER A COMMUNICATION CHANNEL 1555 be achieved causally [23] This can occur when the communication channel is matched to the sequential rate distortion infimizing channel Definition 42: We say a channel is matched to the SRD infimizing channel if there exists an encoder and a decoder such that Here, is the joint measure determined by interconnecting the source, the encoder, the channel, and the decoder In words, a communication channel is matched to the SRD infimizing channel if we can find an encoder and decoder for this communication channel such that the source-to-reconstruction behavior behaves like the SRD infimizing channel B Computations Here, we compute the sequential rate distortion function and determine the structure of the infimizing channel law for the Gauss Markov source described in (9) We first review the solution for a single Gaussian source Then, we describe the sequential situation 1) Gaussian Source: Let our source be First, we present a simplifying lemma Lemma 42: Let Then, 1), and 2) is the identity matrix Let be the unitary matrix that diagonalizes Let Then, 3), and 4) Proof: (1) holds because mutual information is invariant under injective transformations (see Appendix) To see (2) note Both (3) and (4) hold because mutual information and squared error distortion with weight matrix are invariant under unitary transformations Thus, without loss of generality we can restrict our attention to the case and the source covariance is diagonal In practice, the encoder would preprocess the observation by applying and to it Similarly the decoder would postprocess its output by and Let and Then, [4], [9] if (12) is chosen so that This is the so called water-filling solution It is useful to view as the rate, or channel capacity, allocated to reconstructing the th component of The error covariance is given by For the formula (12) reduces to Specifically, the distortion accrued on each component is the same: The rate used for each component may be different though We now characterize the infimizing channel Following [11], we first define the backward channel,, before defining the forward channel, The backward channel is given by the error is zero mean with covariance The channel output is Gaussian with zero mean and covariance Thus, we can compute the joint Gaussian measure The forward channel is given by and is a zero mean Gaussian vector with covariance The optimal channel has the property that There are many ways to realize the channel One such way is to let be an invertible transformation such that Let and For this case, we say the channel with power constraint is matched to the source 2) SRD for Gauss-Markov Sources: We now compute the sequential rate distortion function for the Gauss Markov source described in (9) We first present some results on the structure of the optimal sequential rate distortion infimizing channel in Definition 41 In the previous section, we showed that the static Gaussian source is matched to a memoryless Gaussian channel Here, we show that the source (9) is matched to a Gaussian channel with memory Furthermore, this Gaussian channel can be realized over a memoryless channel with feedback [23] See the Appendix for the proof of the following Lemma 43: The optimal SRD infimizing channel for the Gauss Markov source (9) is a Gaussian channel of the form This lemma states that the channel output at time only depends on the current channel input and not on the previous channel inputs Specifically, it has the form (13), and is an independent sequence of Gaussian random variables This channel has memory due to its dependence on the past channel outputs This channel can be realized over a memoryless Gaussian channel with feedback To see this, note The channel outputs are available to the encoder via feedback The encoder first computes the scaled innovation and then transmits over the channel The decoder upon receiving can then compute The memoryless A-B channel has power constraint: is the error covariance in estimating from For this realization, the total mutual information separates into a sum of mutual information terms: To see this, first note that

8 1556 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 49, NO 9, SEPTEMBER 2004 The second equality follows because we can restrict ourselves to channels of the form (13) Now, each addend can be written as and is chosen so that The error covariance at time is then given by (16) The second equality comes from the fact differential entropy is invariant under translations The third equality follows by noting is independent of Since we may conclude that is independent of We are now ready to compute the SRD function and the SRD infimizing channel for the Gauss Markov source (9) We first address the scalar case Let be a sequence of scalar memoryless Gaussian channels that realize, under noiseless feedback, the sequential rate distortion infimizing channel Let be the mutual information for the th channel use As we have shown, one can reconstruct a scalar Gaussian source with variance over a matched Gaussian channel of capacity within a distortion For the scalar Gauss Markov source, we know that the encoder first computes the innovation: the error is It then scales this innovation and transmits it over the channel The distortion at time is Hence, the variance of the innovation is Then, the reconstruction has distortion and that The interpretation of (16) is as follows First the covariance of the innovation is diagonalized Each component is then transmitted at a rate of The decoder receives the channel output and then computes the state estimate This state estimate has error covariance given by Note that in (15), for low enough distortion, specifically, we get for each that diagonalizes Here, represents the th eigenvalue of Thus, (16) becomes Hence, the rate required on the th component is and The total rate required at time is To compute hence we require each Thus for and Now (14) We have reduced the sequential rate distortion problem for the Gauss Markov process to a standard rate distortion problem for a single Gaussian source For small enough to satisfy the previous condition, we have If the source is unstable, then by Proposition 41 and (14) a necessary condition to insure a bounded distortion at each time step is that the channel capacity be at least We now treat the -dimensional vector valued Gauss Markov source Let be the covariance of the distortion error At time the encoder first computes the innovation, which has covariance Let be the unitary matrix that diagonalizes That is By our discussion, for the single Gaussian source we see that for a distortion we need a rate: if (15) (17) Note that This result relating the channel rate to the eigenvalues of the open-loop system is related to the stability requirements presented in [21] In particular, is the minimum rate required to insure bounded state estimation error For the Gauss Markov source we have shown that if the communication channel is matched to the SRD infimizing channel then the SRD distortion can be achieved In many situations the communication channel is not matched to the SRD infimizing channel In the case when the channel is Gaussian but not matched to the source, Lee and Peterson [14] derive the optimal linear encoder and decoder For the unmatched case, though, the optimal encoder and decoder are generally

9 TATIKONDA et al: STOCHASTIC LINEAR CONTROL OVER A COMMUNICATION CHANNEL 1557 nonlinear and difficult to characterize The problem of joint source-channel coding with small latencies is currently an active area of research C Noiseless Digital Channel and Operational SRD We have computed the SRD function for the Gauss Markov source We have also shown that if the communication channel is a memoryless Gaussian channel with noiseless feedback matched to the source then the SRD function can be achieved Here we examine the situation the communication channel is a digital noiseless channel and hence not matched to the source In this case the encoder must quantize the information it wants to transmit (as opposed to scaling and rotating as it did in the case of the Gaussian channel with noiseless feedback) Definition 43: A sequential rate distortion quantizer is a sequence of measurable functions such that the range of each function is at most countable Specifically takes In the following, the superscript o represents operational Definition 44: The operational sequential rate distortion is It is easy to see that Specifically let be any sequential quantizer such that Then Computing can be difficult However, some structural properties are known In [8], it is shown that the optimal quantizer for the Gauss Markov source is itself Markov Unfortunately, designing optimal sequential quantizers is difficult In the limit of high rate (low distortion) more can be said The quantizer can be chosen to be a uniform quantizer (see Appendix) Also, the difference between the operational SRD function and the SRD function becomes negligible Thus, the SRD function is a useful approximate measure for the rate required to achieve a given distortion [23], [16] See the Appendix for a proof of the following Proposition 42: In summary, with the difference becoming negligible as The sequential rate distortion function can be considered a relaxation of the operational SRD function the relaxation comes from extending the class of deterministic quantizers to the class of random quantizers In the SRD framework the random quantizers are represented by the infimizing channel laws The randomization can be viewed as coming from the noise in the infimizing channel law Hence for the matched case the randomization comes from the true channel noise A Noiseless Digital Channel As stated before, for the noiseless digital channel, the information patterns and are equivalent For these two information patterns let be the operational SRD function for the uncontrolled dynamics with weight matrix Recall that the optimal LQG cost has the form: The cost can be achieved over a noiseless digital channel with rate As discussed earlier, hence, is a good approximation to the channel rate needed in the low distortion regime There is a tradeoff between the channel rate and the control performance Notice that if we are interested in engineering the total cost to within some percentage of the optimal value, then beyond a certain point it is no longer worth trying to lower by increasing the quality of communication since the full knowledge cost,, will dominate the total cost B Memoryless Gaussian Channel With Information Pattern A In this case, the encoder has access to the past channel outputs There are many ways to realize this in practice One possibility is that there is a direct noiseless feedback path for the memoryless Gaussian channel Another possibility is that encoder can infer the channel output from the observation of the previous control signals and knowledge of the controller policy and decoder policy The encoder also has access to the previous controls as well Let be the SRD function for the uncontrolled process with weight Then the LQG cost can be achieved over a memoryless Gaussian channel if that Gaussian channel is matched to the SRD infimizing channel law Following Lemma 42, we can redefine the dynamics Let, and Then We first examine the scalar case For Gaussian channels with capacity, we have shown that the steady state distortion is Thus, the optimal LQG cost equals Note that if then the LQG cost equals infinity For the vector case, we can get an explicit solution in the low distortion regime To achieve an LQG cost of, we need a memoryless Gaussian channel matched to the source with capacity Recall from before that for small enough we have V EXAMPLES In this section, we apply the results of Sections III and IV to four different scenarios We compute or bound the cost in each scenario In the case when the Gaussian channel is not matched to the source the expression is a lower

10 1558 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 49, NO 9, SEPTEMBER 2004 bound on the optimal LQG cost for a channel with capacity C Memoryless Gaussian Channel With Information Pattern B In this case, the encoder has access to the past controls We are assuming that the encoder is collocated with the plant and, hence, can observe both the state and the control actions applied to the plant Computing the SRD function for this information pattern is difficult because encoder does not know the decoder s state One potential suboptimal encoding scheme consists of first computing the encoder s innovation and then transmitting that across the Gaussian channel This innovations coding scheme was introduced in [6] At time the encoder computes the innovation using the encoder s information: Specifically, and for we have Note that the innovation computed by the encoder is different than the innovation computed by the decoder: This is because the encoder does not have access to the channel outputs This innovation coding scheme only works for systems with a stable matrix To see this, we examine the scalar case Let the Gaussian channel have capacity Then, at time zero, the estimation error at the decoder is For general times, the decoder upon receiving the channel output will compute the estimate of the state as follows: Hence, the error variance satisfies the recursion If is unstable then In the case when is stable we can compute the steady state error variance: Hence, the LQG cost is Note that even over a noiseless digital channel this innovations coding scheme will not be stable if is unstable This is because the innovations are quantized This quantization error leads to errors in the decoder s state estimate The new information sent to the decoder will never correct these previous errors Because is unstable, this error will grow We now examine the vector case At each time step we only transmit information about This is nothing more than the standard rate distortion problem discussed before Let be the decoder s reconstruction of and let be the error in reconstruction Let be a unitary matrix that diagonalizes Specifically, Let be a choice of rates for each component Then If all the then error covariance The rate required to achieve this distortion for the reconstruction of is given by The state estimation error covariance evolves as: For stable this Lyapunov equation has the steady state solution In the case when all the we get See [6] for more details on this innovations encoding scheme including a discussion on coding versus delay For the noiseless digital channel with information pattern A) and B) and for the memoryless Gaussian channel with information pattern A) we are able to treat unstable systems However, for the memoryless Gaussian channel with information B), we are not able to treat unstable systems This difference arises for information pattern B) because the encoder is using different information than the decoder s information to base its decisions This asymmetry in information can cause the filters employed by the encoder and the decoder to lose synchronization with each other In the stable case such errors in synchronization will decay away However, in the unstable case they can grow in an unbounded fashion The problem with encoder s employing information pattern B) and transmitting over a noisy channel is that the encoder is not made aware of the channel noise corrupting the signal received by the decoder The failure of innovations coding in the unstable case does not mean that it is impossible to design control systems faced with information pattern B) More complicated coding is necessary to insure synchrony between the encoder and decoder It is unclear what the fundamental penalty is in terms of rate for this situation For the cases treated so far, the channel input symbols,, transmitted over the channel have been uncorrelated This is consistent with the source coding idea that we should only transmit new information But due to channel noise the encoder may lose synchronization with the decoder Hence, the natural idea is to add error correction, or redundancy, to the channel coding In the next section, we discuss this in the context of information pattern C) D Memoryless Gaussian Channel With Information Pattern C Here, the encoder s information pattern at time consists only of The only linear encoder is one that scales and transmits it across the channel: for some gain Note that this is the standard LQG problem with linear observation equation except that the s need to be chosen so as to satisfy the channel power constraint In the previous settings, the channel input symbols have been uncorrelated They represented new information about the source We saw that for information pattern B) if the system is unstable and there is channel noise then we can lose synchronization due to the fact that the encoder no longer knows the decoder s state In such settings, it makes sense to add redundancy to the channel input symbols For the classical LQG problem the input symbols are correlated because the process is correlated Assume we have (1) with a linear observation equation of the form is an observable pair As is well known, the optimal control law is the certainty equivalent control law as described in (4) (6) Let be the steady state covariance of the process under this control law Then, in

11 TATIKONDA et al: STOCHASTIC LINEAR CONTROL OVER A COMMUNICATION CHANNEL 1559 steady state the capacity of the memoryless Gaussian channel should be at least We have a new interpretation for the classical LQG problem with linear observation model The linear observation model adds redundancy to the information about the innovation to combat channel noise This is required because the encoder is not able to compute the decoder s state estimate Part a) follows because is jointly Gaussian Part b) follows from noting that independence or conditional independence of some random variables implies that those same random variables are uncorrelated or conditionally uncorrelated has the same second-order properties as thus it inherits the same independence properties For c), note APPENDIX The mutual information between two random variables and is defined as else and is the Radon Nikodym derivative If is a discrete random variable then its entropy is defined as: and its conditional entropy is defined as: If is a random variable admitting a density,, then its differential entropy is defined as: and its conditional differential entropy is defined as: The following useful properties can be found in [9] a) if if is a discrete random variable admits a density for each This implies conditioning reduces entropy b) c) is a Markov chain if and only if d) Mutual information is invariant under injective transformations In particular, let be an injective function Then, e) The differential entropy of a -dimensional Gaussian random variable is is the determinant of f) The differential entropy is invariant under translations Lemma 43: The optimal SRD infimizing channel for the Gaussian source (9) is a Gaussian channel of the form Proof: We first show the optimal channel is Gaussian Let be a jointly Gaussian source admitting a density Let be a channel Call the resulting joint measure Let be a jointly Gaussian measure with the same second-order properties as Then a) is a Gaussian channel; b) has the same independence properties as ; c) The second equality follows because has the same secondorder properties as and because For the weighted squared error distortion measure we see that the distortion is the same under and Thus, for the Gaussian source the best channel that minimizes mutual information while maintaining a given squared error distortion is the Gaussian channel Now, we show the optimal Gaussian channel can be chosen of the form Let be any Gaussian channel Once given the Gauss Markov source (9) we can determine the joint measure Now define a new channel: As before once given the Gauss Markov source (9) we can determine the joint measure We will prove It is straightforward to verify that holds for all Assume the result holds for all We now prove the induction step For any,wehave Thus, the distortion under is the same as under Wenow show that the mutual information under is less than or equal to that under Proposition 42: For any the limit Proof: We sketch the proof for the scalar source given in (9): Following [16], let represent a scalar quantizer defined as follows: if Let denote

12 1560 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL 49, NO 9, SEPTEMBER 2004 the componentwise application of to each component of Note that the entropy is an upper bound to because the squared error distortion under the quantizer is at most Now, for with finite-differential entropy it can be shown that [16] By the previous results, we know that for (18) small enough Since is jointly Gaussian, we can write the differential entropy as Using (18) with, we get a) follows because Thus See [16] and [19] for details ACKNOWLEDGMENT The authors would like to thank V Borkar and N Elia for many helpful discussions [5] D Bertsekas, Dynamic Programming and Optimal Control Belmont, MA: Athena Scientific, 2000 [6] V Borkar and S Mitter, LQG control with communication constraints, in Communications, Computation, Control, and Signal Processing: A Tribute to Thomas Kailath Norwell, MA: Kluwer, 1997 [7] V Borkar, S Mitter, and S Tatikonda, Markov control problems under communication constraints, Commun Inform Systems (CIS), vol 1, no 1, pp 15 32, Jan 2001 [8], Optimal sequential vector quantization of Markov sources, SIAM J Control Optimiz, vol 40, no 1, pp , Jan 2001 [9] T Cover and J Thomas, Elements of Information Theory New York: Wiley, 1991 [10] N Elia and S Mitter, Stabilization of linear systems with limited information, IEEE Trans Automat Contr, vol 46, pp , Sept 2001 [11] R Gallager, Information Theory and Reliable Communication New York: Wiley, 1968 [12] M Gastpar, B Rimoldi, and M Vetterli, To code, or not to code: Lossy source-channel communication revisited, IEEE Trans Inform Theory, vol 49, pp , May 2003 [13] A Gorbunov and M Pinsker, Prognostic epsilon entropy of a Gaussian message and a Gaussian source, Trans Problemy Peredachi Informatsii, vol 10, no 2, pp 5 25, Apr June 1973 [14] K H Lee and D P Petersen, Optimal linear coding for vector channels, IEEE Trans Commun, vol 24, pp , Dec 1976 [15] D Liberzon and R Brockett, Quantized feedback stabilization of linear systems, IEEE Trans Automat Contr, vol 45, pp , July 2000 [16] T Linder and R Zamir, Causal coding of stationary sources and individual sequences with high resolution, IEEE Trans Inform Theory, 2004, submitted for publication [17] G Nair and R Evans, Mean square stabilisability of stochastic linear systems with data rate constraints, in Proc 41st IEEE Conf Decision and Control, Dec 2002, pp [18] A Sahai, Anytime information theory, PhD dissertation, Dept EECS, Mass Inst Technol, Cambridge, MA, Feb 2001 [19] S Tatikonda, Control under communication constraints, PhD dissertation, Dept EECS, Mass Inst Technol, Cambridge, MA, Aug 2000 [20], The sequential rate distortion function and joint source-channel coding with feedback, in Proc 41st Allerton Conf Communication, Control, and Computing, Oct 2003 [21] S Tatikonda and S Mitter, Control under communication constraints, IEEE Trans Automat Contr, vol 49, pp , July 2004 [22], Control over noisy channels, IEEE Trans Automat Contr, vol 49, pp , July 2004 [23], The sequential rate distortion function, IEEE Trans Inform Theory, 2004, submitted for publication [24] S Tatikonda, A Sahai, and S Mitter, Control of LQG systems under communication constraints, presented at the 37th IEEE Conf Decision and Control, Tampa, FL, Dec 1998 [25] S Vembu, S Verdu, and Y Steinberg, The source-channel separation theorem revisited, IEEE Trans Inform Theory, vol 41, pp 44 54, Jan 1995 [26] H Witsenhausen, Separation of estimation and control for discretetime systems, Proc IEEE, vol 59, pp , Nov 1971 [27] W Wong and R Brockett, Systems with finite communication bandwidth constraints I: State estimation problems, IEEE Trans Automat Contr, vol 42, pp , Sept 1997 [28], Systems with finite communication bandwidth constraints II: Stabilization with limited information feedback, IEEE Trans Automat Contr, vol 44, pp , May 1999 REFERENCES [1] J Baillieul, Feedback designs in information-based control, presented at the Workshop on Stochastic Theory and Control, Lawrence, KS, Oct 2001 [2] R Bansal and T Basar, Simultaneous design of measurement and control strategies for stochastic systems with feedback, Automatica, vol 25, no 5, pp , 1989 [3] Y Bar-Shalom and E Tse, Dual effect, certainty equivalence, and separation in stochastic control, IEEE Trans Automat Contr, vol AC-19, pp , Oct 1974 [4] T Berger, Rate Distortion Theory Upper Saddle River, NJ: Prentice- Hall, 1971 Sekhar Tatikonda (S 92 M 00) received the PhD degree in electrical engineering and computer science from the Massachusetts Institute of Technology, Cambridge, in 2000 From 2000 to 2002, he was a Postdoctoral Fellow in the Computer Science Department at the University of California, Berkeley He is currently an Assistant Professor of Electrical Engineering at Yale University, New Haven, CT His research interests include communication theory, information theory, stochastic control, distributed estimation and control, statistical machine learning, and inference

13 TATIKONDA et al: STOCHASTIC LINEAR CONTROL OVER A COMMUNICATION CHANNEL 1561 Anant Sahai (M 02) received the BS degree in electrical engineering and computer sciences from the University of California, Berkeley, in 1994, and the MS and PhD degrees in electrical engineering and computer science, from the Massachusetts Institute of Technology, Cambridge, in 1996 and 2001, respectively In 2001, he developed adaptive signal processing algorithms for software radio at Enuvis, San Francisco, CA He joined the Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, as an Assistant Professor in 2002 His current research interests are in control over noisy channels, information theory, and wireless communication Sanjoy Mitter (M 68 SM 77 F 79 LF 01) received the PhD degree from the Imperial College of Science and Technology, London, UK, in 1965 He joined the Massachusetts Institute of Technology (MIT), Cambridge, in 1969, he has been a Professor of Electrical Engineering since 1973 He was the Director of the MIT Laboratory for Information and Decision Systems from 1981 to 1999 He was also a Professor of Mathematics at the Scuola Normale, Pisa, Italy, from 1986 to 1996 He has held visiting positions at Imperial College, London, UK; University of Groningen, Groningen, The Netherlands; INRIA, Paris, France; Tata Institute of Fundamental Research, Tata, India; and ETH, Zürich, Switzerland He was the McKay Professor at the University of California, Berkeley, in March 2000, and held the Russell-Severance-Springer Chair in Fall 2003 His current research interests are communication and control in networked environments, the relationship of statistical and quantum physics to information theory, and control and autonomy and adaptiveness for integrative organization Dr Mitter is a Member of the National Academy of Engineering and the winner of the 2000 IEEE Control Systems Award