Decentralized Control of Stochastic Systems

Transcription

1 Decentralized Control of Stochastic Systems Sanjay Lall Stanford University CDC-ECC Workshop, December 11, 2005

2 2 S. Lall, Stanford Decentralized Control G 1 G 2 G 3 G 4 G 5 y 1 u 1 y 2 u 2 y 3 u 3 y 4 u 4 y 5 u 5 K 1 K 2 K 3 K 4 K 5 Control design problem is to find K which is block diagonal, whose diagonal blocks are 5 separate controllers. u 1 u 2 u 3 u 4 = u 5 K y 1 0 K y K y K 3 0 y K 3 y 5

3 3 S. Lall, Stanford Example: Communicating Controllers Often we do not want perfectly decentralized control; we may achieve better performance by allowing controllers to communicate, both to receive measurements from other aircraft, and to exchange data with other controllers. G 1 G 2 G 3 D D D D K 1 K 2 K 3 Here, we need to design 3 separate controllers.

4 4 S. Lall, Stanford Research directions Spacetime models Infinite and finite spatial extent Synthesis procedures via SDP Decentralization structure via information passing Structured synthesis Decentralized structure specified in advance Quadratic invariance conditions on solvability See talk by Mike Rotkowitz Dynamic programming Stochastic finite state systems Relaxation approach

5 5 S. Lall, Stanford Finite-State Stochastic Systems In this talk Look at dynamics specified by Markov Decision Processes. Not linear, but finite state New semidefinite programming relaxations Randy Cogill, Mike Rotkowitz, Ben Van Roy Many applications: Decision problems: e.g., task allocation problems for vehicles Detection problems: e.g., physical sensing, communication

6 6 S. Lall, Stanford Example: Medium-Access Control Two transmitters, each with a queue that can hold up to 3 packets p a k = probability that k 1 packets arrive at queue a p 1 = [ ] p 2 = [ ] At each time step, each transmitter sees how many packets are in its queue, and sends some of them; then new packets arrive Packets are lost when queues overflow, or when there is a collision, i.e., both transmit at the same time

7 7 S. Lall, Stanford Example: Medium-Access Control We would like a control policy for each queue, i.e., a function mapping number of packets in the queue number of packets sent One possible policy; transmit all packets in the queue. Causes large packet loss due to collisions. The other extreme; wait until the queue is full Causes large packet loss due to overflow. We d like to find the policy that minimizes the expected number of packets lost per period.

8 8 S. Lall, Stanford Centralized Control Each transmitter can see how many packets are in the other queue In this case, we look for a single policy, mapping pair of queue occupancies pair of transmission lengths Decentralized Control Each transmitter can only see the number of packets in its own queue In this case, we look for two policies, each mapping queue occupancy transmission length

9 9 S. Lall, Stanford Medium Access: Centralized Solution The optimal policy is Transmitter 2 state Transmitter 1 state (0,0) (0,1) (0,2) (0,3) 1 (1,0) (1,0) (0,2) (0,3) 2 (2,0) (2,0) (2,0) (0,3) 3 (3,0) (3,0) (3,0) (3,0) Easy to compute via dynamic programming Each transmitter needs to know the state of the other We would like a decentralized control law

10 10 S. Lall, Stanford Markov Decision Processes The above medium-access control problem is an example of a Markov Decision Process (MDP) n states, and m actions, hence m n possible centralized policies However, the centralized problem is solvable by linear programming Decentralized control NP-hard, even with just two policies, even non-dynamic case Would like efficient algorithms Hence focus on suboptimal solutions

11 11 S. Lall, Stanford Static Problems Called classification or hypothesis testing e.g., a radar system sends out n pulses, and measures y reflections The cost depends on the number of false positives/negatives prob(y X 1 ) prob(y X 2 ) p(y X 1 ) = prob. of receiving y reflections given no aircraft present p(y X 2 ) = prob. of receiving y reflections given an aircraft present

12 12 S. Lall, Stanford Static Centralized Formulation x is the state, with prior pdf Prob(x) = p(x) y = h(x, w) is the measurement w is sensor noise Choosing control action u results in cost C(u, x) Represent sensing by transition matrix A(y, x) = Prob(y x) Problem: find γ : Y U to minimize the expected cost: p(x)a(y, x)c ( γ(y), x ) x,y

13 13 S. Lall, Stanford Static Decentralized Formulation x is the state, with prior pdf Prob(x) = p(x) y i = h i (x, w i ) is the i th measurement w i is noise at i th sensor Choosing control actions u 1,..., u m results in cost C(u 1,..., u m, x) y = (y 1,..., y m ) given by transitions A(y, x) = Prob(y x) Problem: find γ 1,..., γ m to minimize the expected cost: p(x)a(y, x)c ( γ 1 (y 1 ),..., γ m (y m ), x ) x,y

14 14 S. Lall, Stanford Change of variables Represent the policy by a matrix { 1 if γ(y) = u K yu = 0 otherwise This matrix K satisfies K1 = 1, that is K is a stochastic matrix. K is Boolean Then the expected cost is p x A yx C ( γ(y), x ) = p x A yx C ux K yu x,y,u x,y

15 15 S. Lall, Stanford Optimization Let W yu = x C uxa yx p x, the cost of decision u when y is measured. minimize W yu K yu y,u subject to K1 = 1 K yu {0, 1} for all y, u nm variables K ij, with both linear and Boolean constraints Just m easy problems; pick γ(y) = arg min u W yu

16 16 S. Lall, Stanford Linear Programming Relaxation Another solution: relax the Boolean constraints: For estimation: interpret K yu as belief in the estimate u For control: we have a mixed policy K yu = Prob(u y) Relaxation gives the linear program: minimize W yu K yu y,u subject to K1 = 1 K 0 It is easy to show that this gives the optimal cost and rounding gives the optimal controller.

17 17 S. Lall, Stanford Decentralization Constraints Decentralization imposes constraints on K(y 1,..., y m, u 1,..., u m ). Measurement constraint: controller i can only measure measurement i Prob(u i y) = Prob(u i y i ) Independence constraint: for mixed policies, controllers i and j use independent sources of randomness to generate u i and u j Prob(u y) = m Prob(u i y i ) i=1

18 18 S. Lall, Stanford Decentralization Constraints We call a controller K decentralized if K(y 1,..., y m, u 1,..., u m ) = K 1 (y 1, u 1 )... K m (y m, u m ) for some K 1,..., K m If K is a deterministic controller y = γ(u), this is equivalent to γ(u 1, u 2 ) = ( γ 1 (u 1 ), γ 2 (u 2 ) )

19 19 S. Lall, Stanford The Decentralized Static Control Problem minimize y,u W y1 y 2 u 1 u 2 K y1 y 2 u 1 u 2 subject to K y1 y 2 u 1 u 2 = K 1 y 1 u 1 K 2 y 2 u 2 K i 0 K i 1 = 1 K i ab {0, 1} for all a, b This problem is NP-hard In addition to linear constraints, we have bilinear constraints

20 20 S. Lall, Stanford Boolean Constraints One can show the following facts Removing the Boolean constraints does not change the optimal value i.e., we can simplify the constraint set Then there exists an optimal solution which is Boolean i.e., using a mixed policy does not achieve better performance than a deterministic policy

21 21 S. Lall, Stanford Relaxing the Constraints Now we would like to solve minimize y,u W y1 y 2 u 1 u 2 K y1 y 2 u 1 u 2 subject to K y1 y 2 u 1 u 2 = K 1 y 1 u 1 K 2 y 2 u 2 K i 0 K i 1 = 1 Still NP-hard, because it would solve the original problem. Removing the bilinear constraint gives the centralized problem. This is a trivial lower bound on the achievable cost

22 22 S. Lall, Stanford Valid Constraints For any decentralized controller, we know K y1 y 2 u 1 u 2 = Ky 2 2 u 2 u 1 The interpretation is that, given measurements y 1 and y 2, averaging over the decisions of controller 1 does not affect the distribution of the decisions of controller 2 Easy to see, since K 1 is a stochastic matrix, and K y1 y 2 u 1 u 2 = K 1 y 1 u 1 K 2 y 2 u 2

23 23 S. Lall, Stanford Valid Constraints Adding these new constraints leaves the optimization unchanged. But it changes the relaxation. Dropping the bilinear constraint gives a tighter lower bound than the centralized problem. minimize y,u W y1 y 2 u 1 u 2 K y1 y 2 u 1 u 2 subject to K y1 y 2 u 1 u 2 = K 1 y 1 u 1 K 2 y 2 u 2 u 1 K y1 y 2 u 1 u 2 = K 2 y 2 u 2 u 2 K y1 y 2 u 1 u 2 = K 1 y 1 u 1 new valid constraints K i 0, K i 1 = 1

24 24 S. Lall, Stanford Synthesis Procedure 1. Solve the linear program minimize subject to W y1 y 2 u 1 u 2 K y1 y 2 u 1 u 2 y,u u 1 K y1 y 2 u 1 u 2 = K 2 y 2 u 2 K y1 y 2 u 1 u 2 = Ky 1 1 u 1 u 2 K i 0, K i 1 = 1 The optimal value gives a lower bound on the achievable cost 2. If K y1 y 2 u 1 u 2 = Ky 1 1 u 1 Ky 2 2 u 2 then the solution is optimal. If not, use K 1 and K 2, as approximate solutions.

25 25 S. Lall, Stanford Lifting Lifting is a general approach for constructing primal relaxations; the idea is Introduce new variables Y which are polynomial in x This embeds the problem in a higher dimensional space Write valid constraints in the new variables The feasible set of the original problem is the projection of the lifted feasible set

26 26 S. Lall, Stanford Example: Minimizing a Polynomial We d like to find the minimum of f = 6 k=0 a kx k Pick new variables Y = g(x) where 1 x x 2 x 3 g(x) = x x 2 x 3 x 4 x 2 x 3 x 4 x 5 and C = x 3 x 4 x 5 x 6 Then an equivalent problem is a a 1 a 2 a a a 5 2 a 6 minimize trace CY subject to Y 0 Y 11 = 1 Y 24 = Y 33 Y 22 = Y 13 Y 14 = Y 23 Y = g(x) Dropping the constraint Y = g(x) gives an SDP relaxation of the problem

27 27 S. Lall, Stanford The Dual SDP Relaxation The SDP relaxation has a dual, which is also an SDP. Example Suppose f = x 6 + 4x 2 + 1, then the SDP dual relaxation is maximize subject to t 1 t λ 2 λ 3 0 2λ 2 λ 3 λ λ 2 λ 3 2λ 1 0 λ 3 λ this is exactly the condition that f t be sum of squares

28 28 S. Lall, Stanford Lifting for General Polynomial Programs When minimizing a polynomial, lifting gives an SDP relaxation of whose dual is an SOS condition When solving a general polynomial program with multiple constraints, there is a similar lifting This gives an SDP, whose feasible set is a relaxation of the feasible set of the original problem The corresponding dual SDP is a Positivstellensatz refutation Solving the dual certifies a lower bound on the original problem

29 29 S. Lall, Stanford Example: The Cut Polytope The feasible set of the MAXCUT problem is C = { X S n X = vv T, v { 1, 1} n } A simple relaxation gives the outer approximation to its convex hull. Here n = 11; the set has affine dimension 55; a projection is shown below

30 30 S. Lall, Stanford Example Suppose the sample space is Ω = {f 1, f 2, f 3, f 4 } {g 1, g 2, g 3, g 4 } The unnormalized probabilities of (f, g) Ω are given by g 1 g 2 g 3 g 4 f f f f Player 1 measures f, i.e., Y 1 is the set of horizontal strips and would like to estimate g, i.e, X 1 is the set of vertical strips Player 2 measures g and would like to estimate f

31 31 S. Lall, Stanford Example Objective: maximize the expected number of correct estimates g 1 g 2 g 3 g 4 f f f f Optimal decision rules are K 1 = Y 1 f 1 f 2 f 3 f Xest 1 g 2 g 4 g 1 g K 2 = Y 2 g 1 g 2 g 3 g 4 X 2 est f 3 f 1 f 2 f 2 The optimal is These are simply the maximum a-posteriori probability classifiers

32 32 S. Lall, Stanford Example Objective: maximize the probability that both estimates are correct Optimal decision rules are g 1 g 2 g 3 g 4 f f f f Y 1 f 1 f 2 f 3 f 4 X 1 est g 2 g 4 g 1 g 3 Y 2 g 1 g 2 g 3 g 4 X 2 est f 3 f 1 f 4 f 2 The relaxation of the lifted problem is tight The optimal probability that both estimates are correct is MAP estimates are not optimal; they achieve 0.5

33 33 S. Lall, Stanford Example Objective: maximize the probability that at least one estimate is correct g 1 g 2 g 3 g 4 f f f f The relaxation of the lifted problem is not tight; it gives upper bound of The following decision rules (constructed by projection) achieve Y 1 f 1 f 2 f 3 f 4 X 1 est g 2 g 4 g 1 g 1 Y 2 g 1 g 2 g 3 g 4 X 2 est f 1 f 3 f 1 f 4 MAP estimates achieve

34 34 S. Lall, Stanford Heuristic 1: MAP f 1 1 f g 1 g 2 g 3 g f f The optimal probability that both estimates are correct is 4 7, achieved by Player 1 sees: f 1 f 2 f 3 f 4 Player 1 guesses: g 1 g 2 g 3 g 4 Player 2 sees: g 1 g 2 g 3 g 4 Player 2 guesses: f 1 f 2 f 3 f 4 If both just guess most likely outcome, achieved probability is 1 4.

35 35 S. Lall, Stanford Heuristic 2: Person by Person Optimality 1. Choose arbitrary initial policies γ 1 and γ Hold γ 2 fixed. Choose best γ 1 given γ Hold new γ 1 fixed. Choose best γ 2 given this γ Continue until no more improvement. u 1 = 0 u 1 = 1 u 2 = u 2 = PBPO value is 1000 times the optimal! Almost as bad as the worst policy.

36 36 S. Lall, Stanford Lift and Project Performance Guarantee We would like to maximize reward; all rewards are nonnegative. Use the above LP relaxation together with a projection procedure The computed policies always achieve a a reward of at least optimal reward min{ U 1, U 2 } This gives an upper bound, a feasible policy, and a performance guarantee

37 37 S. Lall, Stanford Markov Decision Processes We will now consider a Markov Decision Process where X i (t) is the event that the system is in state i at time t A j (t) is the event that action j is taken at time t We assume for simplicity that for every stationary policy the chain is irreducible and aperiodic Transition probabilities: A ijk = Prob(X i (t + 1) X j (t) A k (t)) Mixed policy: Cost function: K jk = Prob(X j (t) A k (t)) W jk = cost of action k in state j

38 38 S. Lall, Stanford Markov Decision Processes The centralized solution minimize subject to W jk K jk j,k K ir = r j,k K 0 F jk = 1 j,k A ijk K jk

39 39 S. Lall, Stanford Decentralized Markov Decision Processes Two sets of states X p = {X p 1,..., Xp n} Two transition matrices A p ijk = Prob(Xp i (t + 1) Xp j (t) Ap k (t)) Two controllers K p jk = Prob(Xp j (t) Ap k (t)) Cost function W j1 j 2 k 1 k 2 = cost of actions k 1, k 2 in states j 1, j 2

40 40 S. Lall, Stanford Decentralized Markov Decision Processes minimize W j1 j 1 k 1 k 2 K j1 j 2 k 1 k 2 j 1,j 1,k 1,k 2 subject to K j1 j 2 k 1 k 2 = Kj 1 1 k 1 Kj 2 2 k 2 K p ir = A p ijk Kp jk (1) r j,k K p 0 (2) K p jk = 1 (3) j,k Each of constraints (1) (3) can be multiplied by K 3 p to construct a valid constraint in lifted variables K The resulting linear program gives a lower bound on the optimal cost

41 41 S. Lall, Stanford Exact Solution If the solution K to the lifted linear program has the form K j1 j 2 k 1 k 2 = K 1 j 1 k 1 K 2 j 2 k 2 then the controller is an optimal decentralized controller. This corresponds to the usual rank conditions in e.g., MAXCUT. Projection If not, we need to project the solution K defines a pdf on X 1 X 2 U 1 U 2 We project by constructing the marginal pdf on X p U p

42 42 S. Lall, Stanford Example: Medium-Access Control Two transmitters, each with a queue that can hold up to 3 packets p a k = probability that k 1 packets arrive at queue a p 1 = [ ] p 2 = [ ] At each time step, each transmitter sees how many packets are in its queue, and sends some of them; then new packets arrive Packets are lost when queues overflow, or when there is a collision, i.e., both transmit at the same time

43 43 S. Lall, Stanford Example: Medium Access This is a Decentralized Markov Decision Process, where Each MDP has 4 states; the no. of packets in the queue Each MDP has 4 actions; transmit 0, 1, 2, 3 packets State transitions are determined by arrival probabilities and actions Cost is total number of packets lost; Each queue loses all packets sent if there is a collision Each queue loses packets due to overflows

44 44 S. Lall, Stanford Example: Medium Access Optimal policies for each player are queue occupancy number sent queue occupancy number sent Expected number of packets lost per period is The policy always transmit loses per period queue 2 successful collisions overflows

45 45 S. Lall, Stanford Summary Decentralized control problems are inherently computationally hard We have developed Efficient algorithms to construct suboptimal solutions Upper bounds on achievable performance Guaranteed performance ratio Much more topology independent controllers algorithms that exploit structure peer-to-peer techniques other applications: robotics, sensor networks