Diploma Thesis Networked Control Systems: A Stochastic Approach

UNIVERSITY OF PATRAS SCHOOL OF ENGINEERING DEPARTMENT OF ELECTRICAL & COMPUTER ENGINEERING Diploma Thesis Networked Control Systems: A Stochastic Approach Panteleimon M. Loupos Advisor Prof. George V. Moustakides Patras, June 2011

Óôïí ÐáôÝñá

Acknowledgements It is dicult not to overstate my deep sense of gratitude to my advisor Professor George V. Moustakides for his enormous support and constant guidance during my last two years of undergraduate studies. Above all, I would like to thank him for inspiring me love for Science. ÄÜóêáëå, åõ áñéóôþ ãéá üëá. I am grateful to my uncle Professor Nicos Christodoulakis and to Professor George Papavasilopoulos, for their unfailing love and unconditional encouragement to pursue my dreams. I am also greatly indebted to my teachers Tassos Bountis and George Bitsoris for helping me mould my personality and instilling me with values and ideals forgotten in our day and age. They are real pedagogues. Last but not least, I would like to express my appreciation to Professor Dimitris Toumpakaris for our fruitful discussions and rewarding interaction, and to my friend Panagiotis Niavis for his ingenious lessons on programming and his witty sense of humor.

i Contents List of Figures iii 1 Introduction 1 1.1 General about Networked Control Systems............... 1 1.2 Evolution of Control Theory....................... 4 1.3 Fundamental Issues in NCS....................... 5 1.4 This Dissertation............................. 8 2 Event Triggered Sampling 9 2.1 Lebesgue Sampling............................ 9 2.1.1 Lebesgue Integral......................... 9 2.1.2 Comparison of Riemann and Lebesgue Sampling........ 13 2.1.3 Level Crossing Sampling..................... 18 2.2 Optimal Stopping Times......................... 20 2.2.1 Backward Induction Method................... 20 2.2.2 Optimal Estimation with Limited Measurements over a Finite Time Horizon........................... 22 3 Conclusions and Future Research 39

ii Bibliography 41

iii List of Figures 1.1 Encoder blocks map measurements into streams of symbols that can be transmitted across the network. Encoders serve two purposes: they decide when to sample a continuous-time signal for transmission, and what to send through the network. Conversely, decoder blocks perform the task of mapping the streams of symbols received from the network into continuous actuation signals. (reproduced from [3])........ 2 1.2 Direct and Hierarchical Structure of an NCS. (reproduced from [12]). 3 1.3 Evolution of Control Theory. (reproduced from [2]).......... 5 1.4 Actuator and Sensor Delays in an NCS. (reproduced from [10]).... 6 2.1 Riemann versus Lebesgue Integral. (reproduced from Britannica).. 13 2.2 Integrator with Riemann (blue) and Lebesgue sampling (red)..... 16 2.3 Comparison of V L and V R in a rst order stochastic system. (reproduced from [1]).............................. 17 2.4 LSC scheme for d = 1. (reproduced from [5]).............. 19 2.5 Gaussian sampling thresholds for 2 = 1................. 29 2.6 Numerical Computations for 30, 50 and 100 samples.......... 32 2.7 V n and C n for N = 10 and a = 0:5.................... 36 2.8 Sampling thresholds of the AR(1) for dierent values of a....... 37

iv

1 Chapter 1 Introduction 1.1 General about Networked Control Systems Networked Control Systems (NCS) are spatially distributed systems comprising sensors, actuators and controllers whose operation is coordinated through a shared bandlimited digital communication network. However, one may see the notion of NCS from a more general perspective. For example, in the context of biology systems, the components might be identied as neurons, muscles, neural pathways and the cerebral cortex.thus, the universal feature of NCS is the spatial distribution of its components. The components may operate in an asynchronous manner, but they cooperate in order to achieve some overall objective. Figure 1.1 illustrates the general architecture of an NCS. Generally speaking, there are two major groups of systems in which networked control system conguration can be applied, namely complex control systems and remote control systems. A complex control system is a large-scale system containing several subsystems that collaborate together and share resources. In the past, it was challenging to install and maintain such systems, because direct electrical wiring was needed to connect their components. The advent of networked control system conguration has greatly reduced the

2 Figure 1.1: Encoder blocks map measurements into streams of symbols that can be transmitted across the network. Encoders serve two purposes: they decide when to sample a continuous-time signal for transmission, and what to send through the network. Conversely, decoder blocks perform the task of mapping the streams of symbols received from the network into continuous actuation signals. (reproduced from [3]) complexity of the connections, as it provides more exibility in installation, and eases maintenance and troubleshooting. On the other hand, the term remote control system refers to a system that is controlled by a controller located far away (also known as tele-operation control). Remote control systems have been used for two reasons, namely convenience and safety. A remote control system saves place-to-place traveling time of human operators and protects them from dangers in hazardous environments such as space or war zones. In the past, a remote control system typically required a specic connection link or medium, which was often limited to a point-to-point connection and had an expensive set-up cost. With the evolution of communication technologies, an emerging alternative to expand remote systems to comprise more connections is to utilize wireless data network resources by conguring the system as an NCS. Moreover, there are two general approaches to designing an NCS, as depicted in gure 1.2. Direct Structure: An NCS employing the direct structure approach is composed of a controller

3 Figure 1.2: Direct and Hierarchical Structure of an NCS. (reproduced from [12]) and a remote system containing a physical plant, and sensors and actuators attached to the plant. The control signal is encapsulated in a frame or a packet and is sent to the plant via the network. The plant then returns the system output to the controller by putting the sensor measurement into a frame or also a packet. In a practical system, multiple controllers can be implemented in a single hardware unit that manages multiple NCS loops in the direct structure. Hierarchical Structure: In this case, there are several subsystems forming a Hierarchical structure and a main controller. Each subsystem contains a sensor, an actuator, and its own controller. Periodically, the main controller computes and sends the reference signal to the system in a packet via a network. The remote system then processes the reference signal to perform local closed-loop control, and returns the sensor measurement to the main controller for networked closed-loop control. Note, however, that the aforementioned dierence between the two structures

4 has nothing to do with the network. That is, from the perspective of NCS, these two structures do not signicantly dier. 1.2 Evolution of Control Theory Control theory is an interdisciplinary branch of engineering and mathematics, that deals with the behavior of dynamical systems. Although modern control theory relies on mathematical models for its implementation, control systems of various types date back to antiquity; long before mathematical tools were available. The rst systematic approach in this eld was made by the physicist James Clerk Maxwell in 1868 in his publication entitled On Governors. The major breakthrough, however, was made by the work of Nyquist in 1932, as it provided general methods of design and analysis that could be applied to virtually any feedback system. During the World War II, the need of designing re-control systems, guidance systems and other military equipment gave a great impetus to the automatic control theory. In particular, the pioneering work of Nyquist, Bode, Nichols, and Evans laid a solid theoretical foundation for frequency domain methods. After 1950, with the advent of digital computers and microprocessor (in 1969) control design has been gradually shifting away from frequency-domain techniques, and digital control has become the de facto design method. After many years of research, one may be sure that the foundations of digital control is now rmly established. As mentioned before, control systems with spatially distributed components have been used in several applications, such as reneries, power plants, and airplanes. In these applications, the components of the systems were connected with hardwired connections. The high cost of wiring and the diculty in introducing new components, together with the availability of low-cost, low-power small embedded processors have raised the necessity of adopting a networked control system conguration; giving birth to a complete new direction of control theory, that of NCS. Thus, low-cost microprocessors can be installed at remote locations, and information can be transmitted

5 Figure 1.3: Evolution of Control Theory. (reproduced from [2]) reliably via shared digital networks and wireless connections. Although NCS have a great commercial impact in industrial implementations; mainly due to ad-hoc approaches, there is an increasing interest on applying NCS to a more general potential framework. Consequently, NCS have been nding application in a broad range of areas such as the automotive and aerospace industries, mobile sensor networks, remote surgery, automated highway systems, and unmanned aerial vehicles. 1.3 Fundamental Issues in NCS In this section, we briey discuss the most basic problems occurring in NCS (see [11] for more details). Network Induced Delay: Since NCS are composed by a large number of interconnected devices operating over a network, data transfer between the controller and the remote system will induce network delays. Network-induced delay can be constant, time varying, or random. Its delay characteristics depend on the medium access control (MAC) protocol of the control network, on the scheduling method used, and on other uncertain factors of the medium. It is responsible for degrading control system' s quality of performance (QoP), and can even cause destabilization of the system. More specic, network-induced delays are categorized based on the direction

6 of data transfer. Thus, we have the sensor-to-controller delay sc, and the controller-to-actuator delay ca as depicted in gure 1.4. Note that we can easily deal with the consequences of the delay sc by using time-stamps and state estimators. On the contrary, it is much more dicult to face the delay ca, since the controller has no information on when the computed control signal will arrive at the actuator. Figure 1.4: Actuator and Sensor Delays in an NCS. (reproduced from [10]) Finally, apart from the delays sc and ca, there are several other types of delays, most important of which is the network access delay. Network access delay is the time that a node has to wait, because of the competition, in order to get access to the network. Network-induced delay is one of the most important characteristics of NCS and thorough research has been conducted on this eld. Single-Packet versus Multiple-Packet Transmission: For a variety of reasons data in networks is transmitted in packets, which are sequences of bytes. There are two dierent situations in NCS, i.e. single or multiple packet transmission. In single-packet transmission sensor or actuator data are lumped together into one network packet and transmitted at the same time, whereas in multiple-packet transmission sensor or actuator data are trans-

7 mitted in separate network packets. Multiple-packet transmission is imposed by bandwidth and packet size constraints. Thus, large amounts of data must be separated into multiple packets to be transmitted. The other main reason is that sensors and actuators in an NCS are often distributed over a large physical area, and it is impossible to put the data into one network packet. The main disadvantage of this method is that the controller and actuator have to wait for the arrival of all the data packets, before they are able to calculate their actions. Data Packet Dropout: Data packet dropout often occurs while transmitting data among devices due to node failures or message collisions, and it is a potential source of instability and poor performance of an NCS. There are two dierent strategies for dealing with this problem, either to send the packet again or simply discard it. In communication networks, these two strategies are called Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) respectively. In NCS, UDP is used in most applications due to the real-time requirement and the robustness of control systems. That is for real-time feedback control data, it is better to discard the old untransmitted message and transmit a new packet instead, in order for the controller to receive fresh data for control calculation. In most cases, lower bounds of the packet transmission rate are computed, to determine the certain amount of data loss that the system can tolerate. Network Scheduling: The problem of network scheduling in NCS is to assign a transmission schedule to each transmission device (sensor, controller, actuator) based on a scheduling algorithm (a set of rules that determines the order in which messages are transmitted). The need of a scheduling algorithm is imposed by the limited bandwidth of the network, which creates a situation where all the subsystems can not access the network resource at the same time.

8 1.4 This Dissertation The aim of this diploma thesis is to discuss and present existing techniques that apply event-triggered sampling to linear system state estimation, and to evaluate how these methods aect the overall performance of the system. As already mentioned, sampling rate constraints arise in NCS due to the limited bandwidth available. This restriction on the number of samples clearly aects the mean square estimation distortion error, and the question is how we should choose the sampling instants in order to minimize it. That is, we want to { [ 1 min E N ]} N (x n ˆx n ) 2 : n=1 The thesis is organized in three chapters. The rst chapter constitutes a brief introduction to Networked Control Systems. The applications of NCS and some of the key design issues are introduced. This chapter also contains a brief history of Control Theory, which will hopefully elucidate how the necessity of NCS has emerged. In Chapter 2, which is the core of the thesis, event-triggered sampling strategies are presented. Level-triggered sampling (in particular Lebesgue sampling) and optimal sampling are discussed. Finally, the third chapter is dedicated to the conclusions of the thesis and to potential future research topics.

9 Chapter 2 Event Triggered Sampling Event-triggered sampling is a particular strategy of sampling in which the sampling instants are random variables. It exploits the idea that the system itself would decide when to sample according to the evolution of its dynamics. Thus, event-triggered sampling oers extreme versatility resulting in signicant performance gains in the state estimation problem. 2.1 Lebesgue Sampling In this section, we discuss the notion of a particular type of event-triggered sampling, that of Lebesgue sampling. We begin by briey introduce the Lebesgue integral, and we continue with the presentation of some basic results. 2.1.1 Lebesgue Integral Everybody is familiar with the notion of Riemann integral of a function f between limits a and b, which can be interpreted as the area under the graph of f. Although Riemann integration is well behaved for a large class of functions (Riemann

10 integrable functions), it does not interact well with taking limits of sequences of functions; making such limiting processes dicult to analyze. For this reason, French mathematician Henri Lebesgue proposed a dierent way of integration, the so called Lebesgue integral. Lebesgue integral is really an extension of the Riemann integral, in the sense that it allows for a larger class of functions to be integrable, and it does not succumb to the shortcomings of the latter. For example, the Dirichlet function, which is 0 where its argument is irrational and 1 otherwise, has a Lebesgue integral, but it does not have a Riemann integral. In what follows, the Lebesgue integral of an nonnegative function is dened, and nally the general Lebesgue integral is presented (for more details and analytic proofs see [9]). Let (E, X, ì) be a measure space where E is a set, X is a ó-algebra of subsets of E and ì is a (non-negative) measure on X of subsets of E. In Lebesgue's theory, integrals are dened for a class of functions called measurable functions. A function f is measurable if the pre-image of every closed interval is in X: f 1 ([a; b]) X for all a < b: We build up an integral f d = f (x) (dx) ; E E for measurable real-valued functions f dened on E. Indicator function: Suppose X is a set with typical element x, and let S be a subset of X. The indicator function of the subset S is a function 1 S : X [0; 1];

11 dened as 1 if x S 1 S (x) = 0 if x = S To assign a value to the integral of the indicator function 1 S consistent with the given measure ì we set: of a measurable set S 1 S d = (S): Simple function: A nite linear combination of indicator functions a k 1 Sk ; k where the coecients a k are real numbers and the sets S k are measurable, is called a measurable simple function. We extend the integral by linearity to non-negative measurable simple functions. When the coecients a k are non-negative, we set ( a k 1 Sk ) d = k a k 1 Sk d = k a k (S k ): k Even if a simple function can be written in many ways as a linear combination of indicator functions, the integral will always be the same. If B is a measurable subset of E and s a measurable simple function one denes s d = B 1 B s d = k a k (S k B):

12 Non-negative functions: Let f be a non-negative measurable function on E which we allow to attain the value. We dene E { } f d = sup s d : 0 s f; s simple : E We need to show this integral coincides with the preceding one, dened on the set of simple functions. When E is a segment [a, b], there is also the question of whether this corresponds in any way to a Riemann notion of integration. It is possible to prove that the answer to both questions is yes. We have dened the integral of f for any non-negative extended real-valued measurable function on E. Now, we are going to give the general Lebesgue integral. The General Lebesgue Integral: Let f + and f be the positive and the negative part of a function f respectively. That is: f + = max {f; 0} and f = max { f; 0} Note that f = f + + f and f = f + + f If f d < ; then f is called Lebesgue integrable. In this case, both integrals satisfy

13 f + d < ; f d < ; and it makes sense to dene f d = f + d f d: Intuitive interpretation: The main dierence between the Lebesgue and Riemann integrals is that the Lebesgue method takes into account the values of the function, subdividing its range instead of just subdividing the interval on which the function is dened, as demonstrated in the following gure. Getting the intuition is the key in understanding the notion of Lebesgue Sampling, and this is the reason why all the above were presented. Figure 2.1: Riemann versus Lebesgue Integral. (reproduced from Britannica) 2.1.2 Comparison of Riemann and Lebesgue Sampling The traditional approach in designing control systems is to sample the signals periodically in time. Analogously to Riemann integration, we call this scheme Riemann sampling. There are several alternatives to Riemann sampling. The most common one is to sample the signal when it passes certain limits, namely Lebesgue Sampling. Because of its simplicity, Lebesgue sampling was very popular in early feedback systems,

14 and much work on these systems was done in the period 1960-1980. However, the control analysis and design of such systems becomes very dicult and complicated. Moreover, no general theory has been developed, contrary to the well-established theory of time-driven sampled systems. Hence, the interest in this direction had faded away until the appearance of hybrid systems. Lately, with the new trend of NCS, Lebesgue sampling has regained popularity. K. J. Aström and B. M. Bernhardsson in their work [1] investigate the benets of Lebesgue sampling in the simple cases of an integrator and a rst order stochastic system. In what follows, we present the case of the integrator, and briey discuss the case of the rst order stochastic system. 2.1.2.1 Integrator dynamics The dynamics of an integrator are described by the equation: dx = udt + dv; where the disturbance v(t) is a Wiener process with unit incremental variance and u is the control signal. We want to control the system, in the sense that we wish to keep the state close to the origin. We compare traditional periodic sampling with Lebesque sampling where control actions are taken only when the output is outside an interval, i.e. d < x < d. Notice that in order for the comparison to be fair, we use impulse control for both schemes. Periodic sampling In the case of periodic sampling with period h, the sampled system is described by x(t + h) = x(t) + u(t) + w(t): The mean variance over one sampling period, when we use impulse control is a Wiener

15 process which is periodically reset to zero. It is V R = E [ x 2] = 1 [ h ] h E w 2 (t)dt = 1 h Lebesgue sampling 0 h 0 tdt = h 2 : In this case, impulse control actions are taken when x(t k ) = d, resulting in x(t + k ) = 0. Using this control law the closed loop system becomes a Markovian diusion process investigated in Feller (1954a). Let denote the stopping time, i.e the rst time when the process reaches the threshold d starting from the origin. Then the fact that t x 2 t martingale permits us to compute the mean stopping time. Thus h L E [] = E [ x 2 ] = d 2 : Hence, the average Lebesgue sampling period is h L = d 2 : between two impulses is a The stationary probability distribution of x is given by the stationary solution of the Kolmogorov forward equation, and is found to be f(x) = (d x ) d 2 ; which is symmetric and triangular since d x d: The steady state variance is V L = d d x 2 f(x)dx = d2 6 = h L 6 : In order to be able to compare the results obtained, we assume that average sampling rates are the same in both schemes, i.e. h L = h. Thus, we obtain that V R V L = h=2 h=6 = 3: The above formula means that we must sample 3 times faster with Riemann sampling to get the same mean error variance. gures, where we have selected d = 0:1 and w = d. The results are illustrated in the following

16 Figure 2.2: Integrator with Riemann (blue) and Lebesgue sampling (red). In the above simulations, the decrease in output variance using Lebesgue sampling is clearly demonstrated. Moreover, in the particular realization there are 83 and 69 control actions with Riemann and Lebesgue sampling respectively. 2.1.2.2 A First Order System In the case of a rst order stochastic system dx = axdt + udt + dv; it turns out that the value of a clearly aects the improvement of Lebesgue sampling.

17 Moreover, as shown in gure 2.3, the performance gain of Lebesgue sampling is larger for unstable systems and large sampling periods. This is explained by the fact that for unstable systems the variance V R increases much faster as the sampling period gets larger. Note that in this simulation, V R is the minimum variance obtained by solving a Riccati equation. Figure 2.3: Comparison of V L and V R in a rst order stochastic system. (reproduced from [1]) 2.1.2.3 Remarks At this point, it is important to highlight some important dierences between Riemann and Lebesgue sampling. In the case of Riemann sampling, we know exactly all the sampling instants. On the contrary, in Lebesgue sampling if we are given a time instant t, we cannot draw any conclusion on its distance from an actual sampling instant (including whether it is a sampling instant). In Riemann sampling, we sample independently from the evolution of the state of the system. Thus, we have to minimize the variance for every time instant

18 1 h and then take their expectation, i.e. h 0 E[x2 (t)]dt. This means that during the rst time instants after the sampling point the performance is good, because control has been applied, but can degrade signicantly as the distance from the sampling point increases because no boundary is placed on the evolution of the state. However, in Lebesgue sampling we limit the evolution of the state into one region. Since we do not know whether a particular time instant is a sampling instant, every time instant is treated in the same way, resulting in the same distribution function for each t. 2.1.3 Level Crossing Sampling E. Kofman and J. H. Braslavsky in their work [5] extend the idea of Lebesgue sampling scheme proposed in Aström et al [1]. In particular, they introduced the use of level crossing sampling (LCS) scheme based on hysteretic quantization for feedback stabilization. That is, they divide the range space of the signal into quantization levels regularly spaced by d, and they allow the sampled signal to hold on the triggered value until a new sample is generated. Thus, LCS may be viewed as a Lebesgue sampling scheme in which the quantizer includes hysteresis. Let us now present the above idea in more detail. LCS Scheme Given a continuous function y(t) : R R and a quantization interval d, we dene the level-crossing sampled sequence {y s ( n )} n=0 by the piecewise constant function y s (t) : R R, y s (t) = { y(0 )=h h if 0 t < 1 y( n ) if n t < n+1 where denotes the integer part, and the sampling instants { n } n=0 are dened by

19 n = inf{t > n 1 : y(t) y( n 1 ) > d}: A visualization of LCS scheme is shown below. In the gure, the output LCS signal together with the input signal that generates it are plotted for a quantization interval d = 1. Figure 2.4: LSC scheme for d = 1. (reproduced from [5]) The main advantage of LCS is that it allows one bit coding representation. This is because successive samples always dier in ±d, so we can adopt the following coding strategy { 1 if ys ( n ) > y s ( n 1 ) 0 if y s ( n ) < y s ( n 1 ) If no samples are produced, no transmission takes place. Notice, however, that we still have information when no samples are produced; namely, the knowledge that the output y(t) remains within its quantization band.

20 2.2 Optimal Stopping Times The aim of the present section is to present basic results of the general theory of optimal stopping in the discrete time case. We begin the presentation by introducing the backward induction method and its mathematical formalism. We then use the method to solve an NCS problem. 2.2.1 Backward Induction Method We rst consider the martingale approach, which is then followed by the Markovian approach. 2.2.1.1 Martingale Approach We dene on a ltered probability space (Ω; F; (F n ) n 1 ; P ) the sequence of random variables G = (G n ) n 1. G n represents the gain obtained if we stop the observation of G at time n. Moreover, G is adapted to the ltration (F n ) n 1 meaning that each G n is F n -measurable. This means that we base all our decisions, in regard to optimal stop at time n, on the information available up to time n (no anticipation is allowed). Having said all that, we are now ready to give the denition of a stopping time. Denition: A random variable : Ω {1; 2; :::; } is called a Markov time if { n} F n for all n 1. A Markov time is called a stopping time if < P -a.s. The general optimal stopping problem seeks to solve: V = sup E [G ] ; where for the E[G ] to be well dened for all, we impose the condition: [ ] E sup n k N G k < :

21 The above problem involves two tasks, namely to compute the value function V as explicitly as possible and to nd the optimal stopping time at which the supremum is attained. Now, in the case of nite time horizon (N < ), we solve this problem using the method of backward induction. That is, we construct a sequence of random variables (V N n ) 1 n N and let the time go backward and proceed recursively as follows. Let's say that we are asleep and we wake up at time n = N. Then, then only option we have is to stop and our gain V N N equals G N. If now we wake up at n = N 1, we have two options, i.e. to stop or to continue. If we stop our gain V N N 1 will be equal to G N 1, and if we continue optimally our gain V N N 1 will be equal to E[V N N F N 1]. Notice that we must take the expected value of V N N because of the fact that our decision must be based on the information contained in F N 1 only. It follows that if G N 1 E[V N N F N 1] then we have to stop, and if G N 1 < E[V N N F N 1] then we have to continue. For n = N 2; :::; 1 the considerations are continued analogously. To sum up, we have the sequence of random variables (V N n ) 1 n N dened recursively as follows: V N n = G N for n = N and V N n = max { G n ; E [ V N n+1 F n ]} for n = N 1; :::; 1: 2.2.1.2 Markovian Approach Now, we consider a time-homogeneous Markov chain X = (X n ) n 1 dened on a ltered probability space (Ω; F; (F n ) n 1 ; P x ), which takes values in a measurable space (E; B) where for simplicity we assume that E = R d for some d 1 and B = B(R d ) is the Borel -algebra on R d. Remember that a stochastic sequence X = (X n ; F n ) n 1 is called a time-homogeneous Markov chain (in a wide sense) if the random variables X n are F n =E -measurable

22 and the following Markov property (in a wide sense) holds: P (X n+1 B F n )(!) = P (X n+1 B X n )(!) P-a.s. for all n 1 and B B. The term time-homogeneous refers to the fact that P (x; B) does not depend on n. Note also that F n = F X n generated by the rst n observations. = (X 1 ; X 2 ; :::; X n ) is the -algebra It is assumed that the chain X starts at x under P x for x E, and that the mapping x P x (F ) is measurable for each F F. It follows that the mapping x E x [Z] is measurable for each random variable Z. Given now a measurable function G : E R satisfying the condition [ ] E x sup G(X n ) < : 1 n N for all x E, the nite horizon optimal stopping problem becomes: V N (x) = sup E x [G(X )] : 1 N To solve the problem, we set G n = G(X n ). This way, the solution of the problem reduces to the solution given in the martingale approach, where instead of P and E we have P x and E x for x E, exploiting here the Markovian structure of the problem. 2.2.2 Optimal Estimation with Limited Measurements over a Finite Time Horizon In this section, we consider a sequential estimation problem with limited measurements over a nite horizon (N) in the discrete time case. As stated in the introduction, state estimation under communication rate constraints is a fundamental issue for NCS. Specically, there are three types of communication rate constraints (as discussed in [7]): 1) Average rate limit: This is a `soft constraint', which sets an upper limit on the average number of transmissions; 2) Minimum waiting time between transmissions: Here, there is a mandatory minimum waiting time between two

23 successive transmissions from the same node; 3) Finite transmission budget : This is a `hard constraint', which allows a limited number of transmissions from the same node over a given time window. The simplest version of this type of constraint is to set the constraint's window to be the problem's entire time horizon. In this thesis, we adopt as our communication constraint the simple version of the nite transmission budget. That is, we allow the sensor to transmit exactly M (where M < N) samples of an underlying stochastic process to a supervisor over a noiseless and error-free channel. On the other side of the channel, the supervisor sequentially estimates the state of the process based on the causal sequence of samples it receives. The sensor and the supervisor have the common objective to minimize the average mean-square distortion error. Relationship to previous works In their work [4], Imer and Basar consider the same problem of nite horizon adaptive sampling in a discrete-time setting. Using the optimal form of the estimator under adaptive sampling (see the partial proof in proposition 1 of [4]), they derive dynamic programming equations to be satised by the optimal sampling policy. Moreover, since there is no known method to carry out this optimization problem over innite measurable sets on the real line, they solve it for the specic case of the sets being in the form of symmetric intervals. Rabi, Moustakides and Baras in [7] provide the solution of the problem in a continuoustime setting. They nd the optimal sampling times to be the rst hitting times of time-varying, double-sided and symmetric envelopes. When the signal follows a Brownian motion, they characterize the optimal sampling envelopes analytically. Moreover, in the case of the Ornstein-Uhlenbeck process, they provide a numerical procedure for computing these envelopes.

24 General Problem Statement Given a stochastic sequence {x n }, we want to: { [ 1 min E N ]} N (x n ˆx n ) 2 n=1 subject to the constraint of M channel uses, where M < N. Notice that this is a joint problem of nding the optimal sampling policy and the optimal estimator. We begin with the presentation of the optimal estimation policy and the optimal sampling policy for the case where the sequence {x n } is i.i.d. We derive and simulate the analytical expression for the sampling thresholds in the i.i.d. Gaussian case. Moreover, we propose a numerical approach for the computation of the sampling thresholds. We then consider the case where the sequence {x n } follows an Autoregressive model of order 1. 2.2.2.1 Optimal Estimator for the I.I.D. Case First case: Deterministic Sampling In this case, is known, which permits us to interchange the order of expectation and summation. Thus, we can write: [ N ] 1 E (x n ˆx n ) 2 = E [ (x n ˆx n ) 2] N + E [ (x n ˆx n ) 2] Let's examine the term: n=1 n=1 n=+1 I = E [ (x n ˆx n ) 2] = E [ xn] 2 2E [xnˆx n ] + E [ˆx ] [ 2 n = E x 2 n] 2ˆxn E [x n ] + ˆx 2 n To nd the optimal estimator, we solve: @I @ˆx n = 0

25 0 2E [x n ] + 2ˆx n = 0 ˆx n = E [x n ] = : Hence, the optimal estimator in the deterministic case is the mean value of the stochastic process. Second case: Optimal Sampling The equation of the optimal estimator in this case diers, because the sampling instant is now a random variable. Thus, we cannot put the expectation into the sum as we did before. Hence, we adopt the following approach: [ N ] [ N ] [ N ] E (x n ˆx n ) 2 = E (x n ˆx n ) 2 1{ > n} + E (x n ˆx n ) 2 1{ < n} n=1 n=1 n=1 The use of the indicator function allows the limits of the summation to be deterministic, which in turn permits us to get the expectation into the sum. Letting the estimator to be a deterministic function of x 0, i.e. ˆx n = '( ; x 0 ), we can write for the rst term: I a = E [ (x n ˆx n ) 2 1{ > n} ] = E [ x 2 n1{ > n} ] 2ˆx n E [x n 1{ > n}]+ˆx 2 ne [1{ > n}] Solving again to nd the optimal estimator, we have: @I a @ˆx n = 0 0 2E [x n 1{ > n}] + 2ˆx n E [1{ > n}] = 0 ˆx = E [x n1{ > n}] E [1{ > n}] = E [x n1{ > n}] P ( > n) = E [x n > n] : The knowledge that > n makes the computation of the optimal estimator extremely dicult. What changes now is that at the time instants we do not sample, we still

26 have information, i.e. that our stochastic process is inside a region and has not crossed the sampling thresholds. Instead of solving E[x n > n], we rst considered the case of E[x n x 0 ]. Since the stochastic sequence is i.i.d., this ends up to be the mean value, i.e. E[x n x 0 ] =. Afterwards, we tried to solve E[x n > n] for a specic case, that of x n n, where for n we used the thresholds found in the case of E[x n x 0 ] =. It is: E [x n > n] = E [ ] x n x 1 1 ; x 2 2 ; :::; x n n = P (x < x n x + dx; x 1 1 ; x 2 2 ; :::; x n n ) P ( x 1 1 ; x 2 2 ; :::; x n n ) = P (x dx; x n n )P (x dx; x n 1 n 1 ) P (x dx; x 1 1 ) P ( x n n )P ( x n 1 n 1 ) P ( x 1 1 ) = P (x dx; x n n ) ; P ( x n n ) where in the third equality we used the fact that the samples are independent. Note that in the case of a symmetric pdf with mean value 0, the above expectation is equal to the mean value, as well. Actually, this is the case in our simulations, since we treat a zero mean i.i.d. Gaussian sequence. 2.2.2.2 Optimal Sampling Policy for the I.I.D. Case On a decision horizon of length N, we seek to nd the increasing and causal sequence { 1 ; 2 ; :::; M }, which minimizes the average mean-square distortion error. Since we have a nite horizon problem, we will use the method of backward induction discussed in section 2.2.1. We start by examine the optimal choice of a single sampling instance 1, i.e. M = 1, where for simplicity we drop the subscript 1. This is due to the fact that knowing how to choose i+1 optimally, we can obtain an optimal choice for i by solving the same problem over the horizon of length i+1 1 this time.

27 Optimal Sampling for a Single Sample To nd the optimal sampling policy for a single sample we want to: { [ ]} 1 N min E (x n ˆx n ) 2 N n=1 Setting the estimator to be the mean value of the stochastic process, i.e. ˆx = = 0, we can write the above as (we drop the 1 since it is a constant): N { [ N ]} N min E x 2 n1{ > n} + x 2 n1{ < n} n=1 n=1 { max x 2 } : The above equation means that we should not transmit the most likely outcomes, and that the sample should be generated only when it contains suciently new information. We have the following backward recurrence: V 1 N(x N ) = x 2 N for n = N V 1 N 1(x N 1 ) = max { x 2 N 1; E [ V 1 N(x N ) ]} = max { x 2 N 1; 2} for n = N 1 V 1 N 2(x N 2 ) = max { x 2 N 2; E [ V 1 N 1(x N 1 ) ]} for n = N 2 resulting in the equation: V 1 n (x n ) = max { x 2 n; E [ V 1 n+1(x n+1 ) ]} for n = N 1; N 2; :::; 1: V 1 n (x n ) denotes the minimum distortion incurred by sampling at discrete time instants no less than n, where the superscript refers to the number of samples allowed. Notice that for every time n there exists a threshold, i.e. C n = E[V 1 n+1(x n+1 )], such that if x 2 n C n we sample, otherwise we go to the next time instant.

28 Multiple Samples Having found the solution for one sampling instant, we can now generalize for multiple samples. Let's rst take a look at the case of two samples. The minimal distortion of the two samples would be: V 2 N 1(x N 1 ) = x 2 N 1 + E [ V 1 N(x N ) ] = x 2 N 1 + 2 for n = N 1 V 2 n (x n ) = max { x 2 n + E [ V 1 n+1(x n+1 ) ] ; E [ V 2 n+1(x n+1 ) ]} for n = N 2; :::; 1: This means that if we decide to sample at time instant n, in the remaining time for the end of the horizon the best we can do is E [ V 1 n+1(x n+1 ) ]. Thus, sampling occurs if the quantity x 2 n + E [ V 1 n+1(x n+1 ) ] is greater than the expected minimum distortion of the two samples. Otherwise, we continue to the next time instant. Generalizing the above, we get: V M+1 n (x n ) = max { x 2 n + E [ Vn+1(x M n+1 ) ] ; E [ Vn+1 M+1 (x n+1 ) ]} for n = N M; :::; 1: 2.2.2.3 I.I.D. Gaussian We now derive and compute the integral equations of the sampling thresholds in the case of a zero mean i.i.d. Gaussian sequence with variance 2. It is: C N 1 = E[V 1 N(x N )] = E [ x 2 N C N 2 = E [ V 1 N 1(x N 1 ) ] = ] = 2 V 1 N 1(x N 1 )f(x N 2 )dx = max { x 2 N 1; 2} f(x N 2 )dx = x 2 N 1f(x N 2 )dx + 2 f(x N 2 )dx x 2 N 1 >2 x 2 N 1 <2 = 2 x 2 N 1f(x N 2 )dx + 2 f(x N 2 )dx x 2 N 1 > ( ) ) ( ) 1 2 1 = (erfc 2 + + 2 erf 2 e 2

29 and C n = E [ Vn+1(x 1 n+1 ) ] = V 1 n+1(x n+1 )f(x n )dx = max { } x 2 n; C n+1 f(xn )dx = x 2 nf(x n )dx + C n+1 f(x n )dx x 2 n >C n+1 C n+1 > x 2 n C n+1 = 2 x 2 nf(x n )dx + C n+1 f(x n )dx x n>c n+1 C n+1 ) = 2 (erfc ( Cn+1 2 2 + 2Cn+1 2 ) C n+1 e 2 2 + C n+1 erf ( Cn+1 2 2 ) where in the above integral calculations we used the transformation u = x= 2 2, dx = 2du and f(x) is the well known Gaussian pdf. Figure 2.5 depicts the Gaussian sampling thresholds derived from the analytic solution for N = 10 and N = 100, respectively. We see that the optimum policy is a timevarying threshold, which is decreasing in time. Figure 2.5: Gaussian sampling thresholds for 2 = 1.

30 2.2.2.4 Numerical Computation of the Sampling Thresholds Although we were able to compute the analytic solution of the sampling thresholds in the Gaussian case, this is not feasible in general. There are several distributions, for which the analytic computation of the integral equations is very dicult, even impossible. For this reason, we present here two dierent schemes for the numerical evaluation of the integral equations. Probability Density Function Approximation We need to compute the integral equation: C n = V 1 n+1(x n+1 )f(x n )dx: We can replace the pdf f(x) by a vector f = [f(x 0 ); f(x 1 ); :::; f(x L )]; where a = x 0 < x 1 < ::: < x L = b constitutes a sampling of the interval [a; b]: A similar sampling is applied to the function V 1 n+1 resulting in the vector V 1 n+1 = [V 1 n+1(x 0 ); V 1 n+1(x 1 ); :::; V 1 n+1(x L )] t : Then, we can evaluate the integral by the following approximation: L j=0 where A = dxf and dx = b a L. V 1 n+1(x j )f(x j )dx = AV 1 n+1; Cumulative Distribution Function Approximation We can rewrite the integral equation as: C n = V 1 n+1(x n+1 )df (x n );

31 where F (x) is the cdf, i.e. f(x) is the derivative of F (x), and let a = x 0 < x 1 < ::: < x L = b be again the sampling of the interval [a; b]: Then, we have the approximation: 1 2 L j=1 (F (x j ) F (x j 1 )) ( V 1 n+1(x j ) V 1 n+1(x j 1 ) ) = 1 2 = 1 2 L j=1 j=1 (F (x j ) F (x j 1 )) V 1 n+1(x j ) + 1 2 L j=1 j=0 (F (x j ) F (x j 1 )) V 1 n+1(x j 1 ) L (F (x j ) F (x j 1 )) Vn+1(x 1 j ) + 1 L 1 (F (x j+1 ) F (x j )) V 1 2 n+1(x j ) = 1 2 (F (x 1) F (x 0 )) Vn+1(x 1 0 ) + 1 L 1 (F (x j+1 ) F (x j 1 )) V 1 2 n+1(x j ) j=1 + 1 2 (F (x L) F (x L 1 )) V 1 n+1(x L ) = 1 2 [F (x 1) F (x 0 ); F (x 2 ) F (x 0 ); :::; F (x L ) F (x L 2 ); F (x L ) F (x L 1 )] V 1 n+1: In Fig. 2.6, we compare the sampling thresholds received from the numerical approaches with these received from the analytic solution. We chose the sampling interval to be [a; b] = [ 3; 3], because of the fact that 99.7% of the values drawn from a normal distribution are within three standard deviations from the mean (known as 3-sigma rule). For a small number of samples, the cdf approximation gives an overestimate of the sampling thresholds, whereas for 50 samples and above it gives better results than the pdf approximation. In any case, the numerical results are very close to the analytic ones.

32 Figure 2.6: Numerical Computations for 30, 50 and 100 samples.

33 2.2.2.5 Autoregressive Model of Order 1 We assume now that the stochastic sequence {x n } follows an Autoregressive model of order 1 (AR(1)). That is: x n = ax n 1 + w n ; where w n is an i.i.d. Gaussian process with zero mean and variance w. 2 Moreover, we assume that the initial value x 0 is known, and for simplicity we set x 0 = 0. As previously, we rst solve the joint problem of nding the optimal estimator and the optimal sampling policy for a single sampling instance, and we then generalize it for multiple samples. Note that we will use as an estimator the optimal estimator for the case of deterministic sampling, and we will determine the optimal sampling policy for this estimator. Optimal Estimator for the AR(1) Model As we have already found in the i.i.d. case, the optimal estimator in the deterministic sampling is: ˆx n = E[x n ]: For the AR(1) model, we can write the above as: E [x n ] = E [ax n 1 + w n ] = ae [x n 1 ] + E [w n ] = ae [x n 1 ] ; since the mean value of the noise is zero. As can be seen, this is a recursive equation, which results in: ˆx n = a n x 0 = 0 for n = 1; 2; :::; 1; and ˆx n = a n x for n = + 1; + 2; :::; N:

34 Optimal Sampling Policy for the AR(1) To determine the optimal sampling policy for a single sample, as before, we want to nd: min { [ 1 E N ]} N (x n ˆx n ) 2 : In the AR(1) case, this can be written as: { [ 1 ]} N ( e = min E x 2 n + xn x a n ) 2 n=1 n= { [ N N ( = min E x 2 n + 2xn x a n + x 2 a 2(n ))]} n= n=1 Let's examine the term: [ N ] [ N ] N E x n x a n = E ( x n x a n )1{ = k} = E = E = n= n=1 k=1 [ N n= k=1 n=k [ N N n=1 k=1 n=1 k=1 ] N x n x k a n k 1{ = k} ] n x n x k a n k 1{ = k} n E [ x n x k a n k 1{ = k} ] : The random variable 1{ = k}] depends on x 1 ; x 2 ; :::; x k. Thus, we can write: E [ x n x k a n k 1{ = k} ] = E [ E [ x n x k a n k 1{ = k} ]] x 1 ; :::; x k = E [ E [x n x 1 ; :::; x k ] x k a n k 1{ = k} ] = E [ E [x n x k ] x k a n k 1{ = k} ] = E [ (a n k x k )x k a n k 1{ = k} ] = E [ x 2 ka 2(n k)] ; where in the third equality, we used the Markov property of the AR(1) model.

35 Hence, it is: [ N ] [ N ] E x n x a n = E x 2 a 2(n ) : n= n= Substituting the above in e, we get: { [ N ]} N e = min E x 2 n x 2 a 2(n ) n=1 n= { [ N ] [ ]} N = min E x 2 n E x 2 a 2(n ) ; n=1 n= and since E[ N n=1 x2 n] is a constant, we end up with: { [ ]} e = max E x 2 1 a 2(N +1) : 1 a 2 Thus, we have the following backward recurrence: VN(x 1 N ) = x 2 1 a 2(N N+1) N = x 2 1 a 2 N for n = N { V 1 n (x n ) = max x 2 1 a 2(N n+1) n ; E [ V 1 a n+1(x 1 2 n+1 ) ] } x n for n = N 1; N 2; :::; 1: We observe that C n = E [ V 1 n+1(x n+1 ) x n ] is not anymore a constant value, but a deterministic function of x n. Moreover, note that for a = 0, we get the i.i.d. case. In Fig. 2.7, we simulate the functions V n and C n for a time horizon N = 10. We have used the cdf approximation presented before for the sampling interval [ 4; 4], and we have selected a = 0:5 and x = w = 1. An interesting observation is that E [ V 1 n+1(x n+1 ) x n ] are convex functions. Finally, in Fig. 2.8, we plot the sampling thresholds for dierent values of a. We observe that the sampling thresholds grow as the value of a increases.

36 Figure 2.7: V n and C n for N = 10 and a = 0:5.

37 Figure 2.8: Sampling thresholds of the AR(1) for dierent values of a. Now, we are able to nd the optimal sampling policy for multiple samples. minimal distortion of the two samples would be: The For n = N 1: VN 1(x 2 N 1 ) = x 2 1 a 2(N 1 (N 1)+1) N 1 + E [ V 1 a N(x 1 2 N ) ] x N 1 = x 2 N 1 + E [ ] x 2 N x N 1 = x 2 N 1 + a 2 x 2 N 1 + 2 w For n = N 2; :::; 1: { V 2 n (x n ) = max x 2 1 a 2(N 1 n+1) n + E [ V 1 a n+1(x 1 2 n+1 ) ] [ x n ; E V 2 n+1(x n+1 ) ] } x n The only thing that changes is the term that multiplies x 2 n. Specically, it is decreased by 1 due to the fact that if we not transmit any sample until the time instant N 1, then we have to sample at N 1.

38 Generalizing the above, we get: { V M+1 n (x n ) = max x 2 1 a 2(N M n+1) n + E [ V 1 a n+1(x M 2 n+1 ) ] [ x n ; E V M+1 n+1 (x n+1 ) ] } x n for n = N M; :::; 1:

39 Chapter 3 Conclusions and Future Research We have discussed the Lebesgue sampling strategy and its benecial performance and applicability in NCS. Moreover, we have furnished methods to obtain good sampling policies for the nite horizon ltering problem. We have derived the analytic solution of the sampling thresholds when the stochastic sequence to be kept track of is an i.i.d. Gaussian process. We also provided numerical methods to determine the best sampling policies and their performance for other possible distributions. Finally, we treated the case of an Autoregressive Model of order 1. We conclude that the optimal sampling strategy is the best strategy for minimizing the mean-square estimation distortion error. One possible extension of the problem would be the case where the sensor has access only to noisy observations of the signal instead of perfect observations. That is: x n = ax n 1 + w n y n = bx n + v n ; and we want to estimate x n from the noisy observation y n. Another set of unanswered questions involves the performance of the aforementioned

40 sampling policies when we have multiple sensors. Then, we have: X n = AX n 1 + W n Y n = BX n + V n : In this case, we have two possible sampling policies: 1) We sample all components of Y n at the same time (the whole vector Y n ), and 2) To sample each component of Y n independently from the others. In other words, we will employ for each component a separate sampling mechanism. Finally, another possible extension is the case where the samples are not reliably transmitted but may be lost in transmission.

41 Bibliography [1] Karl Johan Aström and Bo M. Bernhardsson. Comparison of Riemman and Lebesgue sampling for rst order stochastic systems. Proceedings of the 41st IEEE conference on Decision and Control, 2:2011{2016, 2002. [2] Panos Antsaklis and John Baillieul. Special issue on technology of networked control systems. Proceedings of The IEEE, 95:9{28, 2007. [3] P. Hespanha, Payam Naghshtabrizi, and Yonggang Xu. A survey of recent results in networked control systems. Proceedings of The IEEE, 95:138{162, 2007. [4] Orhan C. Imer and Tamer Basar. Optimal estimation with limited measurements. Int. J. Systems, Control and Communications, 2:5{29, January 2010. [5] Ernesto Kofman and Julio H. Braslavsky. Level crossing sampling in feedback stabilization under data-rate constraints. In Conference on Decision and Control, pages 4423{4428, 2006. [6] Goran Peskir and Albert Shiryaev. Optimal Stopping and Free-Boundary Problems. Birkhäuser, 1988. [7] Maben Rabi, George V. Moustakides, and John S. Baras. Adaptive sampling for linear state estimation. Accepted for publication in the SIAM Journal on Control and Optimization. [8] Maben Rabi, George V. Moustakides, and John S. Baras. Multiple sampling for

42 estimation on a nite horizon. In Proceedings of the 45th IEEE Conference on Decision and Control, CDC, San Diego, USA, Dec. 2006. [9] H. L. Royden. Real Analysis. Macmillan Publishing Company, 1988. [10] J. C. So. Delay modeling and controller design for networked control systems. Master's thesis, University of Toronto, 2003. [11] Wei Zhang, Michael S. Branicky, and Stephen M. Phillips. Stability of networked control systems stability of networked control systems. IEEE Control Systems Magazine, pages 84{99, 2001. [12] Yun-Bo Zhao. Packet-Based Control for Networked Control Systems. PhD thesis, University of Glamorgan, 2008.